AI creates 3D images from a simple ToF sensor
European researchers backed by Amazon have developed a technique to use a simple time of flight (ToF) to generate moving 3D images without using spatial daya.
Rather than capturing where photons land on an image sensor array, a ToF sensor measures the time a photon arrives. These are already well established, low cost sensors used in mobile phones for example. But they only provide 1D data to measure a distance. 3D ToF sensors use an array of pixels to combine special and temporal data such as the latest array from Teledyne e2V: HIGH RESOLUTION SENSOR FOR 3D VISION
In a paper published in the journal Optica, the researchers at the University of Glasgow, TU Delft and the Polytechnic University of Milan and backed by e-commerce giant Amazon, used a ToF SPAD (single photon avalanche detector) sensor receiving data from a pulsed laser that illuminated a scene. This created a graph of the time of all the received photons bouncing off objects in the scene.
The trick is to train a neural network with the graph and conventional photographs of the same scene. After several thousand training sets, the AI network had learned enough about how the temporal data corresponded with the photos that it was capable of creating highly accurate images from the temporal data alone. This will shift the processing from the digital signal analysis into on-chip AI accelerators. This could also drive on-chip AI learning where the neural network is refined.
In the proof-of-principle experiments, the team managed to construct moving images at about 10 frames per second from the temporal data, although the hardware and algorithm used has the potential to produce thousands of images per second.
The advantage is that the technique can be used with any type of photon, including radar. This opens up low cost sensors in driverless cars and delivery drones. Amazon, which is developing delivery drones, was one of the funders of the research.
Next: AI with ToF
“Creating images with a single pixel alone is impossible if we only consider spatial information, as a single-point detector has none. However, such a detector can still provide valuable information about time. What we’ve managed to do is find a new way to turn one-dimensional data – a simple measurement of time – into a moving image which represents the three dimensions of space in any given scene,” said Dr Alex Turpin, Fellow in Data Science at the University of Glasgow’s School of Computing Science.
“The most important way that differs from conventional image-making is that our approach is capable of decoupling light altogether from the process. Although much of the paper discusses how we’ve used pulsed laser light to collect the temporal data from our scenes, it also demonstrates how we’ve managed to use radar waves for the same purpose.”
“We’re confident that the method can be adapted to any system which is capable of probing a scene with short pulses and precisely measuring the return ‘echo’. This is really just the start of a whole new way of visualising the world using time instead of light.”
Currently, the neural net’s ability to create images is limited to what it has been trained to pick out from the temporal data of scenes created by the researchers. However, with further training and even by using more advanced algorithms, it could visualise a much varied range of scenes, widening its potential applications in real-world situations say the researchers.
“The single-point detectors which collect the temporal data are small, light and inexpensive, which means they could be easily added to existing systems like the cameras in autonomous vehicles to increase the accuracy and speed of their pathfinding,” said Turpin.
“Alternatively, they could augment existing sensors in mobile devices like the Google Pixel 4, which already has a simple gesture-recognition system based on radar technology. Future generations of our technology might even be used to monitor the rise and fall of a patient’s chest in hospital to alert staff to changes in their breathing, or to keep track of their movements to ensure their safety in a data-compliant way,” he said.
“We’re very excited about the potential of the system we’ve developed, and we’re looking forward to continuing to explore its potential. Our next step is to work on a self-contained, portable system-in-a-box and we’re keen to start examining our options for furthering our research with input from commercial partners.”
‘Spatial images from temporal data’, is published in Optica. The research was funded by the Royal Academy of Engineering, the Alexander von Humboldt Stiftung, the Engineering and Physical Sciences Research Council (ESPRC) and Amazon.
Related ToF articles
- 3D SENSING EMITTER MODULE
- ESPROS: ENHANCED FACE ID SECURITY USING 3D CAMERA
- MEMS-BASED SONAR-ON-A-CHIP DELIVERS EXTENDED RANGE
- WIDE SPECTRUM GESI SENSOR SAFELY DELIVERS 3D SCANS
Other articles on eeNews Europe
- Sensor combines megapixel visible and SWIR images
- German law aims to be first for driverless cars
- Marvell’s 5nm ASIC business shakeup
- Single pair ethernet moves into aircraft designs
If you enjoyed this article, you will like the following ones: don't miss them by subscribing to :
