d-Matrix launches Corsair for AI inference without GPUs, HBM

New Products | November 19, 2024

By Peter Clarke

d-Matrix Inc. (Santa Clara, Calif.), a Microsoft-backed startup, has launched Corsair, its first AI processor, designed to speed through inferencing tasks.

Corsair offers performance of 60,000 tokens/second at 1 ms/token for Llama3 8B in a single server and 30,000 tokens/second at 2 ms/token for Llama3 70B in a single rack, d-Matrix said. As a result Corsair provides performance, energy efficiency, and cost savings as compared to GPUs and other alternatives, the company asserted.

Corsair is based on the Nighthawk and Jayhawk II tiles implemented in 6nm manufacturing process technology. Nighthawk includes four neural cores and a RISC-V CPU.

The company was founded in 2019 and was expected to launch its first product in 2H23. However, with the rapidly rising significance of generative AI the company decided to re-spin the architecture with augmentations to support transformer and generative AI models.

The chip was already configured to address large-model inference using digital in-memory computation (DIMC) and a broad variety of datatypes including block floating point (BFP).

“We saw transformers and generative AI coming, and founded d-Matrix to address inference challenges around the largest computing opportunity of our time,” said Sid Sheth, cofounder and CEO of d-Matrix. “The first-of-its-kind Corsair compute platform brings blazing fast token generation for high interactivity applications with multiple users, making Gen AI commercially viable.”

The emergence of reasoning software agents – agentic AI – and interactive video generation is the next step up in AI capability and in power consumption, triggering a need for improved processing architectures, d-Matrix asserted.

Corsair makes use of chiplet packaging and tight integration of memory and computation and d-Matrix provides the Aviator software stack to support AI developers.

Corsair comes in an industry standard PCIe Gen5 full height full-length card form factor, with pairs of cards connected via DMX Bridge cards. Each Corsair card is powered by multiple DIMC compute cores with 2400 TFLOPs of 8-bit peak compute, 2Gbytes of integrated performance memory, and up to 256Gbytes of off-chip capacity memory. The DIMC architecture delivers ultra-high memory bandwidth of 150Tbytes/s.

d-Matrix is providing samples of Corsair to early-access customers and will be broadly available in 2Q25.

Vik Malyala, senior vice president for technology and AI at Supermicro: said “Our high-performance end-to-end liquid- and air- cooled systems incorporating Corsair are ideal for next-level AI compute.”