Reducing AI energy consumption with on-chip training
Researchers in the Netherlands have developed a technique for reducing the power consumption with on-chip training in AI chips.
Researchers at Eindhoven University of Technology have developed a device capable of on-chip training that eliminates the need to transfer trained models to the chip, opening up more energy efficient AI chips in the future
The development uses a neuromorphic architecture but has been adapted for mainstream AI frameworks rather than the spiking networks.
Training neuromorphic networks can be tedious, time-consuming, and energy-inefficient given that the model is often first trained on a computer and then transferred to the chip.
“In a neuromorphic chip there are memristors (which is short for memory resistors). These are circuit devices that can ‘remember’ how much electrical charge has flowed through them in the past,’ says Yoeri Van de Burgt, associate professor at the Department of Mechanical Engineering at TU/e. who worked with Evaline van Doremaele (above) on the project. “And this is exactly what is required for a device modeled on how brain neurons store information and talk to each other.”
World’s largest neuromorphic supercomputer aims at 10bn neurons
With training done on a computer, the weights from the network are mapped to the chip hardware. The alternative is to do the training in-situ or in the hardware, but current devices need to be programmed one by one and then error-checked. This is required because most memristors are stochastic and it’s impossible to update the device without checking it.
“These approaches are costly in terms of time, energy, and computing resources. To really exploit the energy-efficiency of neuromorphic chips, the training needs to be done directly on the neuromorphic chips,” says Van de Burgt.
For the researchers, the main challenge was to integrate the key components needed for on-chip training on a single neuromorphic chip.
“A major task to solve was the inclusion of the electrochemical random-access memory (EC-RAM) components for example,” says Van de Burgt. “These are the components that mimic the electrical charge storing and firing attributed to neurons in the brain.”
The researchers fabricated a two-layer neural network based on EC-RAM components made from organic materials and tested the hardware with an evolution of the widely used training algorithm backpropagation with gradient descent.
The hardware implementation of the backpropagation algorithm progressively updates each layer using in situ stochastic gradient descent, avoiding the storage requirements. The design includes in situ error calculation and a progressive backpropagation method in a multilayer hardware-implemented neural network. This has identical learning characteristics and classification performance compared to conventional backpropagation in software and the researchers showed it can be scaled to large and deep neural networks, enabling highly efficient training of AI systems.
“We have shown that this works for a small two-layer network,” says van de Burgt. “Next, we’d like to involve industry and other big research labs so that we can build much larger networks of hardware devices and test them with real-life data problems.”
This next step would allow the researchers to demonstrate that these systems are very efficient in training, as well as running useful neural networks and AI systems. “We’d like to apply this technology in several practical cases,” says Van de Burgt. “My dream is for such technologies to become the norm in AI applications in the future.”
If you enjoyed this article, you will like the following ones: don't miss them by subscribing to :
