Meet the power challenges of high-speed interconnects in the cloud
With the addition of cloud services (for example, Apple iCloud) to the already massively connected Internet, data centers are seeing an unprecedented increase in sheer compute and storage requirements. This growth directly impacts energy consumption. As it grows, engineers are seeking solutions to keep the power under control. In this article we examine specifically the interconnect power budgets as massively connected systems move beyond 10 gigabit per second (Gbps) interconnects and solutions to lower power consumption in these high-speed channels.
Today it is without question that Internet traffic is increasing at a rapid pace. The latest Visual Networking Index (VNI) forecast from Cisco (June 2011) clearly shows this trend (see Figure 1) with the interesting part of the forecast in the growth of the mobile space. With the introduction of “cloud” computing and storage, a new paradigm has been introduced driving an even larger consumption of bandwidth. As mobile users move from simple texts to high definition photos and videos, product’s that replicate these to cloud storage, transcode the videos for publication, and replicate the media to the user’s myriad of devices (not to mention publishing to social networks) will place even larger demands on these services. This performance pressure ultimately will require improvements in both processing and communications.
Figure 1. Internet bandwidth trends 2010 through 2015
However, these increases come at a cost – not only in a monetary sense, but also in power consumption. The designers of these next generation servers and networks are already struggling with the power being consumed – both from a cost of ownership (CoO) perspective and a practical thermal design point of view. How will systems be architected to both improve performance and reduce power? This is a never-ending battle driven by the explosive growth of the information age.
Where to look first
As in any system design, the next generation should improve on the performance of the last. In the architecture of cloud computing, services are often moved as loading changes. No longer is a “server” really a discrete piece of hardware. In most cases, the actual hardware hosting a service may be anywhere within a service provider’s infrastructure, which introduces a sort of “uncertainty” of where it is at any moment in time. This type of performance throttling is referred to as “virtualization” or the encapsulation of a service within a software framework that allows it to move freely between hardware hosts. This allows service providers the ability to vary the resources on demand and improve the power consumption of the infrastructure.
As services are throttled, there is a great deal of “machine-to-machine” (M2M) activity. In most data centers, most of the traffic is between machines and not connected to the outside world. The simple addition of virtualization has driven the need to migrate from one gigabit per second interconnects (standard on many mid-decade servers) to 10 Gbps. Today, the demand is driving the move to 25 Gbps interconnects. Many of these connections are less than 5 meters with the majority less than one meter in length. They exist this way due to the architecture of the server farms. A single rack will have blade centers stacked and connected to a switch at the top (or bottom) of the rack. Many racks in a row are then aggregated through concentrators where the information is routed to other rows of servers or network attached storage.
With one gigabit connections, small gauge wires could easily carry the bits without considerable loss of signal integrity. This was important for several reasons – one important consideration is the reduction of airflow out of the servers due to wires blocking the outflows. Another is the bend radius which dictates how many wires you can route in the rack (see Figure 2).
Figure 2. Cable wiring within a rack
With the move to 10G Ethernet, signal integrity became more of an issue and passive cables started using larger gauge wire to compensate. The airflow / bend radius issues began to show up and installers / designers started looking to fiber interconnects as a way to fix the problem. This move to fiber introduced several issues such as increased cost and power consumption. A typical single 10G Ethernet SFP+ module dissipates about a watt of power. With tens of thousands of ports, the amount of power required just for the fiber interconnects increased significantly, along with issues introduced by the increased power dissipation – a rise in rack temperature.
Cabling interconnect issues
If passive cable used for high-speed interconnects suffers from bulk and bend radius issues, then fiber solutions suffer from increased power consumption and higher cost. It would seem that there must be a middle ground to solve this issue. The answer lies in a technology called “active copper cable” – a clever idea that embeds active components into the shell of the connectors to compensate for the high frequency loss introduced by smaller gauge wire. This solution allows smaller wires which have a “fiber like” bend radius and bulk while dissipating much lower power. This typically is less than 65 mW per channel at 10 Gbps for devices such as the DS100BR111, which is commonly used in SFP+ active cable applications.
The technology of improving the signal integrity of cables is limited to lengths less than 15 meters in most cases for 10 Gbps Ethernet. However, as mentioned earlier, most interconnects are less than three meters allowing for easy replacement of passive or fiber cables with active copper versions. This is actually very common today for 10 Gbps interconnects. However, the future is approaching rapidly and even 10 Gbps interconnects will not be fast enough.
In the world of fiber interconnects, there are basically two realms:1) interconnects for short (less than a kilometer); and 2) long (much greater than a kilometer) communications. The longer fiber interconnects form the back-bone of our modern Internet infrastructure and has commonly used 100 Gbps WDM fiber optic technology. To lower the cost of this technology, companies that include Google, Brocade Communications, JDSU and others ratified in March of 2011 a 10 x 10 Gbps multi source agreement (MSA) for a physical medium dependant (PMD) sub-layer that would provide a common architecture for a C-Form factor (CFP) module.
The CFP connector is fine for the low count / long distance interconnects requiring 100 Gbps communications. However, SFP and quad small form-factor pluggable (QSFP) connectors provide much higher density, which is required on local switches and routers. The QSFP form factor is used today for 40 Gbps Ethernet by combining four channels of 10 Gbps data. The next evolution will be to move from 10 Gbps to 25 Gbps channels. This will provide the equivalent of 100 Gbps of data traffic over small QSFP connectors as well as provide a backward compatibility mode for 40 Gbps Ethernet systems that do not support the 100 Gbps standard. Ultimately, this form factor could be used for fiber modules since the 10-to-4 lane conversion used in the CFP modules will no longer be required.
This type of technology has been already demonstrated by several vendors an,d once again, provides infrastructure designers with a road-map to higher speed interconnects. But the interconnection at the back of a switch or server is not the only area where this problem will appear. The electrical interconnects within the servers and network attached storage devices are seeing the same problems.
Power challenges of high-speed interconnects
Distance is not your friend
As the waveform of a digital bit transverses a transmission line and connectors, physics takes over and attempts to completely destroy the original signal by variable attenuation by frequency, reflections from impedance mismatches as well as cross-talk from adjacent channels. The data itself is also to blame since symbols transmitted earlier interfere with the current bits being transmitted. This is referred to as inter-symbol interference, or ISI. By the time the signal covers the distance from an ASIC to the back of the router or switch, the bits are no longer discernable. The same effects that killed the passive cables ability to transfer error free bits apply here as well.
On earlier designs, the switch ASICs used multiple paths of slower data (typically 3.125 Gbps) to connect to a physical layer device (PHY) to create the 10-Gbps NRZ connection at the SFP connector. The PHY would sit very close to the physical connector so the loss of signal integrity was greatly minimized. However, as ASIC technology moved to smaller geometries, faster interconnects became native which absorbed the 10 Gbps interface. At first, this change would seem to improve the overall power consumption of the electrical interconnect by removing the PHY. However, the loss of signal integrity at the edges of the PCB required either more expensive, low-loss board materials or the re-introduction of an active solution.
The same devices used to combat signal loss in cables are also now being applied to the interconnections within these high-performance routers, switches and servers. With the introduction of low-power buffer-repeaters and re-timers, standard FR-4 PCB material can be used (controlling cost) with very low power dissipation. In practice, these devices are used in a similar fashion to the 10 Gbps NRZ Ethernet PHYs to recover and re-time the data to meet the specifications of the connector.
Making the grade
In servers, standards abound including PCI express (PCIe). As data moves faster, the ability for core processors to transfer the information to and from the core has driven the standards such as PCIe to much higher speeds. The latest of which is generation 3.0, which touts an 8 Gbps interconnect. As mentioned previously, in many cases the physical distance within equipment doesn’t change due to processor hardware, connector counts and spacing. Servers are no exception and suffer from signal integrity issues and the power dissipated as well. Previous designs using PCIe generation 1 or 2 where able to meet the operational specifications by careful layout and connector selection. However, as servers are moving to generation 3.0, the board material and connectors are affecting the signal integrity to a point where the standard is no longer being met.
Standards such as PCIe and others introduce another issue that makes fixing this problem somewhat difficult, while keeping power consumption low. This problem is out-of-band (OoB) signaling that occurs in the early training process of the channel. Since a standard PCIe board doesn’t know anything about the channel when it is plugged in, it must communicate with the root complex and make adjustments to the channel to help the signal integrity. This communication is done OoB and, if it fails (is blocked for any reason), the channel will fail to initialize.
Some vendors of PCIe integrated circuit (IC) repeaters use a method that replicates the root complex. This method splits the channel into two segments, effectively shorting the distance and greatly improving the signal integrity (less connectors / shorter distance). The problem with this approach is power. To replicate the root complex requires understanding the traffic moving over the channel and replicating it correctly at both sides. It also introduces excessive latency due to the serialization and de-serialization processes.
Other vendors have worked around this problem by simply using an analog method to condition both the in-band and OoB signals removing any processing of the information. Devices such as the DS80PCI402 use this technique requiring only 65 mW per channel. The insertion of this device into the PCIe channel effectively shortens the channel between the end node and root complex, which does not interfere with the OoB process and greatly improves the signal integrity of the 8 Gbps data while using a small amount of energy.
Other areas for improvement
Our information infrastructure is growing to meet the demand of both increasing user numbers and technologies such as cloud computing. The interconnect power budgets are just one piece of the total power consumption of these systems. Vendors are looking at ways to produce cores that consume much less power along with the interconnections between them. Due to the simplicity and extremely low-power consumption of the ARM core, interest is rising in using this engine in cloud servers. Additionally, specialized processors are also finding their way into the infrastructure to provide services such as real-time transcoding of video and images, speech recognition, and more. These specialized services would usually require floating point function performed in a general purpose processor. These specialized processors provide energy efficient ways to perform the same function.
Conclusion
As cloud computing and storage continues to grow in both scale and capacity, the interconnections between nodes will continue to increase in capacity. The challenges for designers will be to keep power at a minimum while continuing to increase the throughput of the network. Not only will these solutions be challenged by the ever-increasing requirements for higher bandwidths, but they will be stretched to the limits to minimize the power they consume.
References
For more information on signal conditioning, visit: Signal Conditioning
About the Author
Richard Zarr is a technologist at Texas Instruments focused on high-speed signal and data path technology. He has more than 30 years of practical engineering experience and has published numerous papers and articles worldwide. Richard is a member of the IEEE, holds a BSEE from the University of South Florida, as well as several patents in LED lighting and cryptography. Richard can be reached at ti_richardzarr@list.ti.com.
If you enjoyed this article, you will like the following ones: don't miss them by subscribing to :
