End-to-end design and verification for PCIe 6.0
PCI Express (PCIe) 6.0 is gaining traction in AI, HPC, and data centres, operating at 64GT/s, twice as fast as the previous generation. Network servers, SSDs, switches, and AI accelerators are all early adopters of PCIe 6.0, and network interface cards (NIC) and CPU host chips are on the horizon.
PCIe 6.0 is a transformative serial bus interface technology, a kind of sea change in interconnect based on several technological shifts in this version of the specification, say Gary Ruggles, Senior Product Manager and Madhumita Sanyal, Sr. Staff Technical Product Manager at Synopsys.
PAM-4 pulse amplitude signaling at four voltage levels produces three eyes, a shift from traditional non-return to zero (NRZ) signaling. Precoding and forward error correction (FEC) will reduce errors for analog and digital, respectively. This delivers 64GT/s bandwidth with low latency.
Flow control unit (FLIT) packet delivery is a new architecture for packet delivery (required due to the FEC) not only supports the increased bandwidth but also enables your system to support it.
The L0p low-power state allows some lanes to go into a sleep mode as bandwidth requirements decrease in the system. This gives you the ability to optimize your power consumption while never shutting down the link.
The specification uses data object exchange (DOE) as the PCIe security building block at lower bandwidth levels with cryptographic data and keys. Component Measurement Authentication (CMA) provides a firmware cryptographic signature. Integrity and Data Encryption (IDE) gives your system packet-level security to prevent physical attacks. By coupling IDE with your controllers, it’s efficient security at high-bandwidth 64GT/s speeds.
There are a couple important factors to consider in adopting PCIe 6.0.
The expense incurred in adopting a new PCIe specification: Added costs can come from moving to a smaller foundry process, new IP, and what the implication of adoption means to the whole system, including losses associated with cables and backplanes and adding new cards.
The timing of PCIe 6.0 maturity means the PCIe 6.0 ecosystem hasn’t solidified and compliance. This, gated on PCI-SIG consortium workshops, is roughly a couple years out.
Moving to a new generation of a standard has cost considerations. Whether these costs are born now or later depends on the need for the advantages PCIe 6.0 has to offer weighed against the market pressures.
In terms of PCIe 6.0 maturity, the current adoption trajectory is similar to previous generations of the spec. Because PCIe is ubiquitous, the ecosystem to support the 6.0 generation is growing. Adopting PCIe 6.0 now gives the advantages while helping to future proof a design as as the 6.0 ecosystem comes online.
Backwards compatibility
Despite the significant technology advancements, PCIe has carefully written the specification for backward compatibility with previous versions of the spec. This means it is not necessary to rely on a robust PCIe 6.0 ecosystem for a system to work. Whether it’s an endpoint or a complex system, PCIe 6.0 designs can plug into technology that uses any version of the PCIe standard, including PCIe 1.0 @ 2.5GT/s. If a device cannot support the new PCIe 6.0 64GT/s data rate, the link will be negotiated at the highest data rate supported by both link partners.
However, once a link negotiates 64GT/s FLIT mode, it must remain in FLIT mode: if an anomalous event impacts signal integrity—for instance, if a giant motor turns on causing a glitch in the power supply, or there is extra noise, or somebody moves a cable—the system may need to fall back to the previous PCIe generation data rate (such as 32GT/s or even 16GT/s).
Even when a glitch such as this causes a fall back, the system will remain in FLIT mode. Supporting a previous generation while remaining in FLIT mode is a new state that never existed before. While this is not necessarily likely, it is a possible complication. To hedge for this possibility, support for th eFLIT mode should be implemented for all PCIe data rates.
PCIe 6.0 is already part of a whole PCIe ecosystem that dominates the industry, providing ubiquitous connectivity. Afterall, you cannot change the entire data centre and all the devices in it at once when you move from Gen. 5 to 6. PCIe enables a fall back option, so the whole ecosystem doesn’t need to upgrade simultaneously.
Network Switch Design with PCIe 6.0
Navigating network switch design development illustrates the complexity of an implementation of PCIe 6.0 in practical application. To get the advantages of 64GT/s and PAM-4 signal integrity in a design with 256 lanes, for instance, will not only need the full switching solution, but will also need to integrate with technology beyond the system.
The higher speed means designer need to think through the co-design of the package, the board, and all the integrations, as well as how it is all assembled. This entails activities such as optimizing the bump map, reduction of escaping signals, and optimizing the beachfront of the die, on all die edges.
Designing a switch with PCIe 6.0 requires an understanding of a vast array of different perspectives so the system will work holistically. Implementing PCIe 6.0 in a switch requires different configurations with multiple links and multiple controllers.
- First protocol analyzer/exerciser for PCIe 6.0
- Security IP and controller for TDISP in PCI Express 6.0
PCIe 6.0 Verification
The changes introduced with PCIe 6.0 affect all layers, creating increased verification complexity.
At the physical layer, 64GT/s speed support is achieved using PAM4 encoding. PCIe 6.0 introduces 256B FLIT, which demands certain packing rules for protocol packets into FLITs, increasing design complexity. It also brings in FEC complexity along with the existing CRC mechanism.
For backward compatibility reasons, 256B FLITs are supported at 2.5/5/8/16/32 GT/s speeds. This demands verification of FLIT mode at all supported speeds.
At the data link layer, addition of new DLLP types–namely optimized updatefc and link management, for exchanging link information, and change in sequence number/replay rules–demands in-depth verification of sequence numbering, FLIT replay command handshakes, and selective/full replay mechanisms to provide guaranteed FLIT transfer across to the link partners.
Along with the introduction of FLIT, new TLP framing rules are also defined, which requires extensive verification.
PCIe 6.0 also introduces a new power state L0p, which enables power reduction without impacting the traffic flow. Link management DLLPs are used to establish L0p handshake between link partners. This adds to the design complexity necessitating in-depth verification.
The disruptive nature of PCIe Gen6 specification will create new verification challenges not only for backward compatibility, bandwidth, and performance of the interface but also for dependent NVMe, SSD, and other PCIe-based storage technologies. Synopsys Verification IP (VIP) and test suite are designed to handle this verification complexity. Synopsys VIP is used to verify silicon-proven Synopsys IP.
In addition, running system-level payload on SoCs requires a faster hardware-based pre-silicon solution. Synopsys transactors based on Synopsys IP enable fast verification hardware solutions including Synopsys ZeBu emulation systems and Synopsys HAPS prototyping systems for validation use cases.
Synopsys PCIe 6.0 Controller and PHY IP in an end-to-end host-to-device system, using Teledyne LeCroy’s Interposer and Analyzer showing the impact of payload size on throughput at PCI-SIG DevCon 2023.
If you enjoyed this article, you will like the following ones: don't miss them by subscribing to :
