AI-in-a-box puts large languages models, translation into embedded
A US startup is integrating large language models and translation AI into embedded designs.
Useful Sensors has developed an AI development kit that can create captions in real time and translate 15 different languages on an embedded board.
The company is starting crowd funding campaign on Crowdsupply for an embedded AI-in-a-box dev kit running the Llama 2 transformer LLM to provide a voice interface and translation without having to have a cloud connection.
“We want this to be Android for everyday objects,” said Pete Warden, CEO and founder.
The $279 kit runs on a RockChip 3588S ARM processor with a neural processor unit (NPU) with 8Gbytes of memory.
- OKdo selected by Google sensor spin-off as SBC partner
- Small dev board enables low-power ML in IoT devices
“We are putting together a board with an integrated display, speaker, microphone and USB so that you can also use the box as an interface to a computer,” said Warden. “and “We have an open source library that accelerates that accelerates running transformers. Speech requires more horse power so its needs a bigger board, especially LLMs that are memory bound. We need 8Gbytes on this board we can run on 4G but it runs more slowly.”
Warden was CTO of JetPac developing deep learning AI on phones in 2014 before it was acquired by Google and was a founding member of TensorFlow.
The company has already developed a person sensor with a microcontroller and camera that keeps on the data on the device. It has a single pin output that goes high when a person goes by as a $10 drop in replacement for a PIR sensor.
“This is what I’ve been working towards with Useful Sensors, to walk in a room, look at a light and say on, and to interact with objects like we interact with people using language and using presence without having to do stuff online, that works when you plug it in and has the privacy,” said Warden.
“I would like anything that has a switch or a button eventually to have a voice interface. I would like to get the cost of this module down to 50c but starting off with things that have a plug with a voice interface,” he said.
The memory is the key problem for embedded designs. The real time speech recognition needs 40Mbytes of RAM, while the open source translation networks need 100Mbytes. “There have been some recent LLMs in the 1Gbyte range and the question is then how to fine tune them,” said Warden.
He sees this AI being used in all kinds of embedded applications.
“One of the things I’d love to do is have a hardware store such as B&Q with a box on each pillar that you can ask a question and it will direct you to the right aisle,” he said. “We are working with large consumer manufacturers and I’ve love to give closed captions and translations running on a TV.”
He is planning to license the software but also provide boards.
“We think there’s going to be interest in this kind of interface layer so we want to provide the easiest to use solutions for this, so that people can buy a single board that does these functions out of the box and they can integrate it into their systems.”
“We are happy to do software licensing but I do think there will be hardware specialisation around these capabilities to co-design the hardware with partners.”
github.com/usefulsensors/useful-transformers
If you enjoyed this article, you will like the following ones: don't miss them by subscribing to :
