contact button

Data Center Interconnect for All

portrait of Tim Doiron

October 5, 2023
By Tim Doiron
Vice President, Solutions Marketing

Meeting the economic and capacity challenges of the evolving data center landscape

Earlier this year, I highlighted the increasing modularity and distributed nature of new data center builds. Evidence of these new smaller, modular, less power-hungry data centers is easy to find as I bike around the Chicago suburbs near my home or travel throughout Southeast Asia as I did this summer. These smaller, more modular data centers are easier and faster to build by utilizing smaller parcels of land and distributing their impact on local power grids. They may also deliver increased diversity and reduced latency when located closer to the consumers of their data.

But data centers do not exist in isolation. They require local, regional, and sometimes submarine connectivity – high-speed optical connectivity – to bind them to other data centers and the people and applications that utilize their compute and storage resources.

As an industry, how can we evolve and adapt data center interconnect (DCI) solutions to support the growing capacity demands of expanding hyperscale data centers while also supporting the diversity of smaller, modular, and distributed ones?

The answer is threefold: more compact modular optical platform choices, innovations in embedded and pluggable coherent optical engines, and increased transmission spectrum per fiber pair. In short, we need to right-size initial DCI capacity and investment to match the data center’s day-one transmission demands while also enabling cost- and energy-efficient expansion over time.

A Chassis to Fit Every Environment

While the Data center interconnect category started with small, server-like transponders with coherent optical engines, the industry has quickly evolved to more flexible compact modular platforms with sleds that can be mixed and matched to support practically any desired functionality, as seen in Figure 1. Chassis are available in a wide range of depths to support deployment in diverse environments. These platforms can mix optical line system and transponder functionality and can be extended with multi-chassis connectivity that enables single network element manageability with easy capacity expansion. This pay-as-you-grow model enables operators to expand only when needed, matching cost and power consumption to capacity.

Compact modular DCI platform with mix-and-match sledsFigure 1: Compact modular DCI platform with mix-and-match sleds

Data Center Interconnect: Moving at the Speed of Light

With increased vertical integration, leading coherent optical engines are evolving in two directions simultaneously: 1) smaller, lower-power pluggables that can reach 1,000 km or more and 2) embedded, sled-based optical engines with sophisticated transmission and reception technology that maximize capacity-reach and spectral efficiency.

With increasing capabilities in small 400G QSFP-DD pluggables, IP over DWDM (IPoDWDM) is starting to be realized with deployments directly into routers and switches. Basic 400ZR pluggables support Data center interconnect applications up to 120 km with fixed settings and only support Ethernet traffic. More advanced 400G ZR+ pluggables offer increased programmability, support for OTN and Ethernet traffic, and better optical performance to support metro/regional and some long-distance connectivity. 400G XR pluggables support both high-performance point-to-point applications like ZR+ and point-to-multipoint deployments, where a single 400G pluggable optic can communicate simultaneously with multiple 100G pluggable optics in 25 Gb/s increments.

Today’s embedded engines like Infinera’s ICE6 deliver 800 Gb/s per wavelength at distances approaching 1,000 km, and those same 800G engines can deliver 600 Gb/s up to 3,000 km and 400 Gb/s almost everywhere in the world, including submarine spans that can measure 10,000 km. But 1.2 Tb/s-per-wavelength engines and beyond are emerging from development labs and will be ideal for long-distance DCI connectivity solutions.

Embedded engines are also ideal where fiber is scarce and high spectral efficiency is required. As an example, in situations where a data center operator is leasing fiber, embedded optical engines can reduce operational costs by maximizing the amount of data transmission over a single fiber pair to avoid leasing a second fiber pair or trenching new fiber. The power consumption of today’s 800G coherent optical engines is also drastically improved, utilizing 89% less power per bit than similar engines 10 years prior.

Putting More Lanes and Automobiles on the Highway

In most of the world, DWDM networks have only used the C-band of fiber spectrum, but webscale operators with hyperscale data centers were early adopters of C+L-band transmission on the same fiber. Just like adding lanes to a highway, expanding the usable spectrum on a fiber delivers more capacity. But in recent years, operators have focused on getting capacity gains by using more spectrally efficient coherent optical engines with advanced modulation schemes. As these efficiency gains diminish with each successive generation, advancements in optical line system components like amplifiers and wavelength-selective switches (WSS) can cost-effectively increase the transmission spectrum from 4.8 THz to 6.1 THz in both the Super C- and Super L-bands, trading a small incremental line system infrastructure cost to realize a 27% gain in incremental spectrum and transmission capacity per fiber pair. Although still early, look for Super C and Super L deployments in future Data center interconnect networks.

Super-C and Super-L spectrum expansionFigure 2: Super C and Super L spectrum expansion

Data center construction shows no signs of abating, and while still critical and growing, massive hyperscale data centers are giving way to an increasing number of smaller, modular, and diverse data centers. This change will only accelerate compact modular deployments with a variety of sleds and chassis to fit different types of environments, a collection of embedded and pluggable optical engines, and increasing transmission spectrum per fiber. When combined, this solution architecture enables us to have a low entry price, reduced power consumption, and a small footprint, while also enabling flexible scalability to meet future capacity demands.