From Supercar to Supercomputer?
Cars that can “see” have moved successfully from film fantasy into real life in many of today’s high-end and mid-range road cars. Car manufacturers are keen to continue exploring the technology, and safety legislators may soon require any or all vehicles approved for sale to be fitted with certain active safety systems that can bet realised by implementing automotive image recognition systems. This pattern is already established in e.g. the EU, where safety systems such as ABS that were once considered high-end accessories are now mandatory in all vehicles.
Engineers developing vision-based safety systems such as pedestrian detection and collision avoidance now need to consider whether the design decisions they take today will allow them to deliver solutions suitable for mass-market applications in the near future. Some chipmakers targeting this market are already beginning to paint exciting images of tomorrow’s cars as mobile supercomputers, leveraging technologies such as intensive multi-core architectures as currently used in high-performance PC-graphics cards and gaming platforms. However, important factors such as cost and power consumption in particular may demand a more focused approach.
In addition to supporting various types of hazard recognition, visual recognition is expected to complement collision avoidance technologies such as radar and ultrasonic, sometimes even by replacing them. Additional potential safety applications include systems to eliminate blind spots or to replace conventional mirrors for better rearward visibility. Ultimately, vision-based systems will play a major role in autonomous cars that are capable of navigating safely to a destination and parking without human intervention to respond to road conditions, observe traffic signs and avoid numerous types of hazards.
The industry’s vision for vision suggests that in-car systems will become increasingly complex, with more and more cameras and screens around and inside the vehicle. On-board image-processing capability will need to scale accordingly, while continuing to deliver reliable real-time performance, with acceptablely low latency.
Image processing: Hardware or Software
Historically, image recognition has been largely reliant on software algorithms running on one or more digital signal processors (DSPs). As more and more video channels need to be handled simultaneously, the demand on processing resources increases. In a desktop computing system, this capability can be satisfied by increasing the operating frequency of the graphics processor. In an automotive application, however, the power dissipated by a high-performance CPU/DSP array running at high-megahertz frequencies would present unacceptable thermal challenges.
For space reasons, a significant proportion of the system electronics must be mounted inside ECUs with narrow space constraints. These are typically mounted behind the windscreen and often exposed to direct sunlight. It is therefore important to ensure real-time system performance without resorting to power-hungry architecture that dissipate large amounts of heat.
An image recognition processor architecture featuring dedicated hardware blocks for complex or frequently-used processing functions such as transforms, filters, histograms and pyramid matching can provide a platform for more energy-efficient processing generating less heat while also benefiting from faster algorithm execution resulting in consistent real-time performance. This type of platform can also scale linearly to support extra ADAS functions or additional channels for multiple cameras.
To answer demands to handle an increasing number of camera feeds for vision-based driver-assistance systems with image-recognition capabilities, Toshiba has developed an image-recognition processor family specifically for automotive applications. The architecture comprises high-performance Media-Processing Engines (MPEs), dedicated camera inputs and an array of hardware acceleration blocks. The current fourth-generation TMPV7608XBG processor family integrates up to eight MPE units with up to eight camera inputs and multiple hardware accelerators optimised for functions such as pixel calculation and filtering, enhanced affine transforms for distortion reduction and image sizing, and functions for histogram manipulation and matching. These hardware accelerators incorporate the latest and most advanced techniques, to deliver even faster performance than processors from previous generations (figure 1).
Figure 1. Media-Processing Engines and dedicated hardware accelerators ensure high-performing yet power-efficient processing. For full resolution click here.
Automotive-Specific Algorithms
In addition, Toshiba has implemented proprietary processing algorithms such as its CoHOG (Co-occurrence Histograms of Oriented Gradients), enhanced CoHOG and Structure from Motion (SfM) functions in dedicated hardware. With these enhancements, this fourth-generation processor delivers ten times greater performance than the previous generation of processors featuring four MPE units.
Toshiba created its CoHOG technology to enhance detection of human beings in various situations such as walking, cycling, sitting, or people in wheelchairs. This is achieved by comparing processed detected data with known characteristics of human body shapes and movement. CoHOG looks for pairs of gradients within certain upper and lower limits to detect features such as shoulders, arms, legs or hips. Knowing also that the features will have a certain orientation relative to each other, within physiological limits such as maximum and minimum limb lengths, range of movement and overall height helps ensure correct identification. The technique can also be extended to allow detection of other objects such as animals.
More recently, Toshiba has also developed the enhanced CoHOG accelerator, which provides extremely high pedestrian-recognition accuracy during night-time by analysing colour-based gradients of images from multiple Full-HD cameras (figure 2).
The SfM accelerator enables accurate identification of unknown obstacles that are not included in the image library, such as guardrails, kerbs or small objects on the road surface (figure 3). Conventional pattern recognition has only limited ability to identify unknown objects, whereas the SfM accelerator allows the use of 3D reconstruction techniques by analysing the image feed from a monocular camera at high speed.
Figure 3. The SfM accelerator enables the system to deal more effectively with unusual situations.
The extensive integration of hardware-based accelerators seen in this family of processors is effective in lowering overall processor power consumption.
Project development is supported via a Software Development Kit (SDK) with drivers and sample application programs, as well as a dedicated Integrated Development Environment (IDE), media processor debugger and simulator. A set of APIs for the software drivers, as well as hardware accelerators and an image-processing library are also included. Toshiba also has a number of ready-to-use algorithms such as pedestrian detection, vehicle detection and line detection, and others are at beta stage and preparing for release. Engineers can also implement their own algorithms if preferred.
Special Automotive Requirements
If automotive systems designers are to use the latest image-recognition technology in future vision-based systems, other important issues besides performance and power must be considered. Unlike the situation in consumer markets, automotive product lifecycles are relatively long and devices such as processors must be supported for periods of many years.
Moreover, systems such as collision avoidance and pedestrian detection are considered safety-critical and therefore must meet the appropriate Automotive Safety Integrity Level (ASIL). Toshiba has developed its image-recognition processor families including the TMPV7608XBG using its own ISO26262-certified process flow, and has designed the devices to satisfy the requirements for ASIL-C/D (Automotive Safety Integrity Level) systems.
Conclusion
The automotive sector is not the first to make use of extensive hardware acceleration to perform complex signal-processing algorithms at high speed and low power. The case for hardware acceleration is already proved in applications such as the software-defined radios of 3G and 4G cellular base stations, as well as equipment such as high-speed internet infrastructure, data-centre computing, and military signalling and radar systems.
Advanced vision-based driver assistance can take advantage of image-recognition algorithms and power-efficient hardware developed specifically for automotive application. This approach enables today’s driver aids to deliver real-time performance within tight power, thermal and cost constraints, and allows scalability to support increasingly complex ADAS functions.
About the author:
Klaus Neuenhüskes is Manager Automotive Solution Marketing at Toshiba Electronics Europe.
If you enjoyed this article, you will like the following ones: don't miss them by subscribing to :
