d-Matrix delays chiplet processor to better address generative AI

News | June 29, 2023

By Peter Clarke

Memory & Data Storage Datacenter Artificial Intelligence

d-Matrix Inc. (Santa Clara, Calif.), a well-connected startup formed to address AI inference, is optimizing its chiplet-based processor, to better address generative-AI.

In April 2022 multiple reports said d-Matrix expected to launch the company’s first product, code-named Corsair, in 2H23. Now Sheth has told eeNews Europe that Corsair is being “augmented” to specifically support transformer, generative AI models.

Corsair was already configured to address large-model inference using digital in-memory computation (DIMC) and a broad variety of datatypes including block floating point (BFP).

“The comment lasts year were made about six months before the generative AI frenzy kicked in. We decided to augment the platform to specifically support generative AI models, transformer models generally. This product redesign to address generative AI delayed our development to accommodate these changes,” said Sheth.

Sheth said he now expects 6nm silicon back from TSMC in 1H24 and sampling in 1H24 with production volumes in 2H24.

Well-connected

The company is well-connected and appears to have the runway to sustain the delay.

The company was founded in 2019 by Sheth and Sudeep Bohja who serves as CTO. Both have backgrounds in developing power-efficient compute and interconnect solutions for datacenters. Both had worked at Broadcom before joining Inphi Corp. which was acquired by Marvell Technology for about US$10 billion.

Back in April 2022, d-Matrix raised $44 million in a Series A venture capital round. Investors included Playground Global, Microsoft through their M12 venture fund and memory maker SK Hynix Inc. They joined existing investors Nautilus Venture Partners, Marvell Technology Group Ltd. and Entrada Ventures LLC.

In November 2022 Microsoft announced the Microsoft Project Bonsai reinforcement learning would be supported on the d-Matrix DIMC technology.

Sheth said that when the company was founded much of the focus was on training of AI models in the datacentre, a field in which Nvidia’s GPU-based silicon has been highly successful.

“The models were a lot simpler then. A lot of the inference was done on server CPUs back then. And we thought, in five years AI will be all about inference. We decided not to look at high-performance computing, analytics, graphics and just focus on inference for large models,” Sheth said.

Corsair

The company has already designed a couple of processors as demonstrator circuits. These are called Nighthawk and Jayhawk but the first commercial product is called Corsair.

The chiplet-style of development provides for scalability and when it appears it will come in two, four and eight processor die versions. The chiplet itself is designed in a logic process but is essentially a sea of highly-connected SRAMs, Sheth said. These are used to perform efficiently the billions of multiply-accumulate operations required in AI algorithms. “The chiplet is about 60 percent SRAM and then 30 to 40 percent is interconnect engines needed.” The SRAM is used for both processing and memory but is placed close to DRAM.

Such a memory-rich architecture allow more of a transformer model and neural network weights to be retained on a single PCIe card within the server.

Sheth showed that in a variety of benchmarks Corsair is showing 10x to 20x performance improvement over Nvidia’s A100. “We are already well-equipped to address generative AI but it is also important to maximise differentiation from the incumbent. After Corsair comes out we will have quick cadence of products.”