Activity detection in the vehicle interior takes privacy into account
In automated driving, the vehicle decides what it has to do – it steers, brakes and accelerates. However, until the time comes when vehicles can do without a driver altogether, partially automated vehicles will support the driver and give him or her increasingly more freedom. Naturally, partially automated vehicles require handovers between the car and the driver, for example at a construction site on the motorway or during the transition into city traffic after a motorway journey. The vehicle needs to keep a constant eye not only on the surroundings, but also on the driver to determine how quickly he might take control of the vehicle if necessary. Existing driver observation systems are mainly limited to detecting drowsiness; they do not primarily evaluate the images from the cameras.
Researchers at Fraunhofer IOSB are pursuing this approach in a current project. “With our technology, we not only recognise the face, but also the current poses of the driver and passengers,” explains Michael Voit, group leader at Fraunhofer IOSB. From this, the researchers want to reliably determine what the driver and passengers are currently doing.
The core of the development lies in algorithms and machine learning methods. The algorithms analyse the camera data in real time and find out what the driver’s attention is focused on. The technology thus goes beyond pure image recognition and interprets activities in context. The researchers first learned the system by annotating numerous camera shots by hand: Where are the hands, feet, shoulders of the people, where are objects such as smartphones, books and co. recognisable? They then evaluated the algorithms with new images and corrected or verified their results.
The system abstracts images of the driver and passengers into a digital skeleton – an abstract, reduced representation of the person’s body pose. From their movement and a complementary object recognition, it infers the activity. The algorithm thus recognises whether someone is sleeping or looking at the road, how distracted a person is and how long it takes until full attention can be directed back to the traffic. Both classic video cameras and infrared cameras that can see in the dark are supported as image sensors, as well as 3D cameras that measure the distance of objects to the camera. The system also gives interior designers freedom in the placement of the cameras.
The researchers are working on questions relating to activity detection in vehicle interiors in several collaborative projects with car manufacturers such as Audi and Volkswagen, but also with suppliers such as Bosch and Continental. “The technology is ready for pilot production,” explains Voit. The research institute is already in contact with companies that want to use its technology, he adds.”
What goes on inside the vehicle is very private and subject to strict data protection laws. That’s why the researchers have taken the issues of privacy and security into account from the very beginning. “The camera data is evaluated in real time, is not stored and never leaves the vehicle at any time. Personalised models are not needed for this either – and thus no personal data is collected,” says Dr Pascal Birnstill, Senior Scientist at Fraunhofer IOSB. The technology therefore respects privacy from the outset and thus complies with the strict regulations and the high level of data protection awareness in the EU.
An EU regulation shows just how important activity recognition is: Driver Monitoring is to become mandatory for automated cars in the future. With the technology from Fraunhofer IOSB, vehicle manufacturers can not only comply with this directive, but also make numerous visions of autonomous driving a reality. One example: speech recognition quickly reaches its limits when it comes to communication between humans and cars. For example, the command “park there” is not meaningful in itself. However, through body pose and activity recognition, the system knows which parking space the user is pointing to at that moment. The system can also help with safety aspects of driverless vehicles: While drivers currently still make sure that all passengers observe the safety rules and buckle up, for example, this will have to be done by the driverless vehicle in the future – for example in autonomously driving taxis or buses. Here, too, reliable interior monitoring is indispensable.
https://www.iosb.fraunhofer.de/en.html
Related articles:
ST, Eyeris team up for in-cabin monitoring
Camera-based DMS fights distracted driving
German law aims to be first for driverless cars
Sensor/MCU combo enables ultra-short range radar applications
$73m deal creates European automotive AI juggernaut
If you enjoyed this article, you will like the following ones: don't miss them by subscribing to :
