Communication between drivers and other traffic participants plays an important role in road safety. This applies to an even greater extent to autonomous vehicles. An ideal task for AI, according to Fraunhofer researchers.
In order to be able to interact with other road users – especially pedestrians – a vehicle in autonomous mode must recognise which pedestrians could become relevant and then detect and interpret their behaviour. The Fraunhofer Institute of Optronics, System Technologies and Image Exploitation (IOSB) in Karlsruhe has now presented a prototype of a system that is supposed to do just that using artificial intelligence.
What a human driver perceives intuitively and without thinking, for example on the basis of location, line of sight and gestures, an autonomous vehicle first has to learn in order to be able to act safely and independently in a residential area or in front of a school. Artificial intelligence (AI) methods offer the potential to analyse video images in this way – but they must first learn to draw the right conclusions based on large amounts of training data. The computer scientist and research associate at Fraunhofer IOSB is working on this as part of the research project “Intelligent Human-Technology Communication in Mixed Traffic”, or “Initiative” for short.
- Research project makes life safer for pedestrians and cyclists
- Hella mulls light for communication in road traffic
“We have implemented a research prototype that assesses whether a pedestrian wants to cross the road, analyses his or her gestures and thus creates the basis for interaction,” explains Manuel Martin, research associate at IOSB. The system consists of a stereo camera that can see spatially and thus detect the exact position of passers-by, and an AI algorithm that detects the positions of the limbs and draws conclusions from them. At a project presentation at the halfway point of Initiative, Martin successfully demonstrated this system. Now it is a matter of further training the AI and refining the system as a whole so that it can recognise pedestrians’ intentions as accurately as possible in all conceivable situations, the researcher explains.
Capturing the behaviour of occupants and other road users from a single source
Fraunhofer IOSB is thus bridging the gap between the observation of the vehicle interior and the exterior, as the head of the “Perceptual User Interfaces” research group, Dr Michael Voit, points out: “We are now bringing together what were previously separate worlds: The intelligent recording of the behaviour of drivers and, if applicable, passengers by our Advanced Occupant Monitoring System – and the recording of other road users and their intentions as part of the Initiative research project.” This means that it is now also possible to record interactions between drivers and passers-by.
In the Initiative project, the detection of pedestrian intentions is only one piece of the puzzle – the big goal is to enable AI-supported adaptive communication between different road users in order to integrate automated vehicles into mixed traffic scenarios. To this end, comprehensive communication interfaces are ultimately to be developed for the vehicle’s interaction with other road users as well as with its own occupants. For example, the car will be able to tell a pedestrian who wants to cross the road that it is about to stop or pass by means of an unambiguous light indicator.
Launched in April 2021, the research project is funded by the German Federal Ministry of Economics with a total of 4 million euros. Other participants in the project include the electronics supplier Hella, the Würzburg Institute for Traffic Sciences (WIVW), and the Institute for Control Systems at the Karlsruhe Institute of Technology (KIT).