Extended Reality in the car: BMW records first successes
In a perhaps not-too-distant future, when cars steer themselves through traffic, drivers will be able to turn their attention to other activities – including gaming with virtual reality technologies. For passengers, especially those in the back seat, this could already be an option today. And it doesn’t have to be just about gaming – vehicle and traffic-related information could also be presented in this way better than today. But before such visions become reality, a number of problems have to be solved.
In any case, BMW sees great potential in the use of augmented reality (AR) glasses and virtual reality (VR) devices in cars. BMW refers to the common ground of AR (in which digital information is displayed in the otherwise normal optical field of view through special electronic glasses) and VR (in which users are presented with completely synthetic environments via a corresponding display strapped in front of their heads) as Extended Reality (XR). The vehicle manufacturer from Bavaria is conducting research in this area together with Meta Reality Labs from California.
The two companies have now presented the first results of their joint research activities. Displaying XR content in a moving car proved to be far from trivial, as the system must be able to distinguish the user’s head movements from the pitch, yaw and roll movements of the vehicle on the road: Even if the car turns, goes over a bump, or brakes suddenly, the XR display has to deliver an interference-free image. VR headsets use cameras and motion sensors to determine their exact position. So when the user moves their head or looks around, the virtual content remains stable. However, in fast-moving vehicles, the headset’s sensors are disturbed by conflicting signals from the device’s movement in the vehicle and the vehicle’s movement in the world: the headset’s motion sensors state that the car is moving, but the cameras believe it is stationary when they view the car’s interior.
- BMW brings CinemaScope-like HUD to series production
- “Holokraktor” demos potential of Augmented Reality in the car
This discrepancy, known as motion in a non-inertial frame of reference, means that a VR headset cannot stably display virtual content when riding in a fast-moving vehicle. To solve this challenge, researchers from BMW and Meta collaborated to integrate information from vehicle sensors into the tracking system of a VR headset.
In this effort, BMW and Meta Reality recorded their first successes: the two companies linked Meta Reality’s tracking system (implemented in the Meta Quest VR headset) with the car’s sensor data. In the process, the engineers were able to realize stable stable gaming, entertainment, productivity and meditation experiences. They were able to do this because the tracking system could simultaneously calculate the position of the device relative to the vehicle and the position of the moving vehicle relative to the world. This accurately anchored virtual objects, a prerequisite for compelling VR and MR experiences while driving.
The two partners have now succeeded in the latter. By combining vehicle-generated motion data with the XR headset’s sensing capabilities, vehicle motion can essentially be subtracted and the headset’s movement in a vehicle can be tracked more accurately. This, in turn, enables the display of content that is synchronized with the movement of the vehicle in real time – both inside and outside the vehicle at high speeds and regardless of whether the vehicle is moving on a flat, straight road or on a bumpy road with sharp curves.
Together, BMW and Meta Reality developed a proof-of-concept prototype that processes motion in six degrees of freedom while calculating the headset’s position relative to the vehicle. In addition, the system must incorporate the position of the vehicle relative to the environment. The latter is made possible by integrating the vehicle signals into the VR system.
Next step: Industry standard
The long-term goal of the partnership is to explore new extended reality (XR) experiences for passengers and drivers in future vehicles. The partners also want to help advance industry standards for integrating XR consumer devices into the digital vehicle ecosystem.
“It’s still hard to predict when and how this technology will become available to end customers,” explained Claus Dorrer, head of BMW’s Technology Office in Mountain View. But Dorrer can envision several use cases for XR devices in vehicles – such as alerting drivers to hazards, displaying important information about the vehicle’s condition, or helping to find the vehicle in a crowded parking lot. “The potential of future AR glasses and VR devices – for both passengers and drivers – is promising. The research partnership with Meta allows us to find out what immersive XR experiences in the vehicle might look like in the future and to drive the seamless integration of such experiences,” Dorrer said.
The next stage goal now is to create an industry standard for connecting XR devices to vehicles. To that end, BMW says it is open to involving other partners.
Related articles:
BMW brings CinemaScope-like HUD to series production
Ericsson to demo 5G remote driving tech that UK seeks to ban
Seeing Machines, ADI team for smarter driver monitoring systems
“Holokraktor” demos potential of Augmented Reality in the car
Startup shows immersive holographic head-up display technology
Here teams for AR 3D map displays in cars
Volkswagen introduces AR head-up displays in the compact class
