TinyML in Action: Jon Norby on emlearn, Zephyr, and Edge AI
With the Elektor online conference, “Zephyr – The Open RTOS for Future Devices,” right around the corner in November, we spoke with speaker Jon Nordby, co-founder of Soundsensing and creator of the open-source TinyML library emlearn. In this conversation, Jon shares his insights on deploying machine learning to microcontrollers, how Zephyr RTOS enables smarter edge devices, and much more.
Elektor: What were the biggest technical challenges you faced when designing emlearn to work across such a wide range of devices with only a C99 compiler available?
John Nordby: emlearn strives to keep things as simple as possible. For example, we utilize portable C code instead of doing a lot of device or architecture specific optimizations. C99 is thankfully very widely supported by now. We support some standard neural network architectures, but not arbitrary neural network/compute graphs. This limits the amount of technical challenges that we need to tackle, and keep the project high quality and maintainable for a small team.
Elektor: How do you balance model accuracy with the memory and power constraints of embedded devices when converting models from scikit-learn or Keras?
John Nordby: emlearn aims to have optimized inference for the most popular models. In general, we do this by supporting fixed-point arithmetic and by precomputing as much as possible during the training process, to simplify the calculations needed during inference. We also have some specific optimizations for particular models, that are not widely implemented elsewhere. For example, the inference for decision-tree ensembles (Random Forest etc.) supports soft-voting with quantized leaf values (class probabilities), which enables a small model without loosing accuracy the way hard-voting does. We are working on a technical report that describes this in more detail.
There are other very important aspects to finding a good tradeoff between model predictive performance vs computational costs, such as model hyperparameters, feature selection, and feature engineering/preprocessing. This is outside the scope of emlearn as a tool, but we try to include some useful pointers on these topics in our documentation.
Elektor: What types of sensor data (e.g., audio, vibration, radar) have been most challenging to model effectively with TinyML, and how did emlearn address those challenges?
John Nordby: emlearn is rather agnostic to the nature of the input data, as the user has to provide feature extraction code. So it is easiest to apply for data domains and tasks where one can find references for appropriate feature extraction. Audio and accelerometer data is very common, for example, where-as radar is a bit less so (in my experience). Searching Google Scholar for papers, or using your favorite LLM is a good approach here. We also have in the documentation links to projects using emlearn, and they usually
describe how they adapted to the data they used.
Elektor: What unique advantages does Zephyr RTOS bring to TinyML deployments that you don’t typically see in other RTOS platforms?
John Nordby: Zephyr provides a portable Hardware Abstraction Layer (HAL), and has built-in support for a lot of connectivity options, with many drivers included. This is outside the scope for many other RTOS platforms, so it is relatively unique to Zephyr. Using the Zephyr HAL makes it easy to access sensor data, which is essential in TinyML applications. And
many TinyML applications typically send outputs from models over BLE, WiFi, LoRa, etc. I also appreciate the good built-in support for filesystems, since it is very useful for collecting data and when validating the data processing pipeline.
Elektor: Could you walk us through a practical example of how Zephyr’s sensor API integrates with emlearn in a real-world application?
John Nordby: The simplest use case is when there is a one-to-one correspondence between sensor data to be read, and execution of the machine learning model. For example, if you are building a device that sorts objects by color (say M&Ms or Skittles). Then you can make a few calls to the Zephyr sensor “Fetch and Get” sensor API to collect red, green, blue, and place those values (the features) into a 3-element array. You then pass this feature values into the skittels_detect_predict() function created by emlearn.convert, and you will get as output the classification. Based on the class value, you could for example drive a servo to sort out all the purple ones.
Another common family of tasks is using time-series sensor data, such as accelerometer or audio. In that scenario, one usually wants to read fixed-length chunks of data instead of individual measurements, from a few milliseconds at a time to some seconds depending on the task. With the sensor API in Zephyr, there are two ways of achieving this: Either via the new Read & Decode sensor API to get such chunks of data directly, or by using a dedicated higher-priority thread that calls Fetch and Get repeatedly to build up a chunk, then passed to the model inference thread (via a queue).
I am working on some code examples to illustrate these in practice for the upcoming talk. Note that audio and video have dedicated chunk-based APIs in Zephyr that are preferred over the generic sensor API.
Elektor: From your perspective, which industries or domains stand to benefit the most from TinyML adoption in the next five years?
John Nordby: I believe it is relevant in most domains that need to automatically ingest and analyze large amounts of sensor data. In this sense, there is a lot of overlap with applications or industries that benefited a lot from related technologies, like MEMS sensors (Micro-Electro-Mechanical Systems) and wireless sensor networks. But the key characteristics that makes TinyML particularly relevant is when the input data rate (from sensor) is relatively high, but the output data must be quite low. For example, when using audio/video/radar/IMU sensors, but running on battery power, and transmitting over low-power protocol such as LoRa. Or there are other concerns that limit the ability to transmit the raw sensor data, such as privacy.
For more information on Elektor’s online conference “Zephyr – The Open RTOS for Tomorrow’s Devices” on 5 November, see the conference website.
Editor’s note: eeNews Europe is an Elektor International Media publication.
If you enjoyed this article, you will like the following ones: don't miss them by subscribing to :
