Reducing Cost and Latency in Ring-based Multi-Layer Networks (AKA Lessons Learned on the Circle Line)
In the London rail system, there is a line called the Circle Line that ferries riders past many famous tube stops in central London – Victoria Station, Paddington, etc. On my last trip to London, I got on the train going the wrong direction. So instead of only needing to transit eight or so stops, I ended up going the other way around the ring, going through 15 or so stops, resulting in a considerable delay, not to mention the stress and anxiety that went along with knowing I might miss my flight home.
In transit systems, it’s a well-known fact that in general you want to avoid as many transit stops as possible – in air travel that means direct flights are preferred over multi-hop flights with layovers, and for the Circle Line, it sure would be great if they had rails that could cut across the ring to create shortcuts. Imagine the savings in total travel time that would come from eliminating transit point delays (where passengers get on and off) as well as potentially reducing actual time in motion. From a passenger’s perspective, an ideal transit system would always provide a single hop to one’s final destination. This would reduce the size of the railcars needed, because each one could be sized for the number of people going from station A to station Z, but the cost of realizing such a vision on the Circle Line is clearly impractical and cost-prohibitive never mind the fact that it would be extremely challenging to engineer. So, instead, we live with the current model of a large-capacity train, sized to accommodate passengers getting on and off at various stations, enduring the “cost” of going through each station en route.
In many ways, multi-layer ring-based Internet protocol (IP)/optical networks have similar challenges. If we consider a physical fiber-based ring that interconnects multiple cities, and the variable/bursty traffic demands that might go between any two cities, it makes sense to deploy IP routers at each location. These routers not only terminate services or traffic at these sites, but also act as intermediate transit nodes for traffic that is just passing through. This works well for up to several nodes, but starts to become inefficient as the percentage of transit traffic at a router site becomes too large, proportionally to the add/drop traffic. At some point, creating a partial mesh topology becomes highly desirable for diversity as well as traffic optimization reasons. But that’s a topic for another day. For this discussion, let’s consider a fixed physical fiber ring, because many fiber rings with 10 or fewer sites exist today, and an assumption of traffic following a general “anywhere-to-anywhere” pattern with a mix of small and large flows.
There are multiple options for building such a fiber ring-based network. The simplest way is to deploy routers and use static point-to-point wavelengths (via wavelength-division multiplexed or WDM optics) between each pair of neighboring nodes, very much like the Circle Line model. While simple to engineer, this method incurs a high proportion of transit traffic at each router location, and is typically the most expensive to scale. From a multi-layer networking perspective, it is a somewhat rudimentary approach that does not provide network operators with ways to optimize the optical transport layer. As such, let’s look at three multi-layer options that leverage a flexible optical transport layer:
- Option 1: routers and wavelength-granular switching using WDM/reconfigurable add-drop multiplexers (ROADMs)
- Option 2: routers and optical transport network (OTN) switching (WDM/OTN)
- Option 3: routers and packet-aware OTN switching (WDM/P-OTN)
Option 1: Wavelength Granular Switching
When routers and wavelength-switched WDM systems are employed over a fiber ring, we immediately gain some benefit of multi-layer optimization over the static Circle Line model. Although the fiber ring is physically circular, it is possible to directly interconnect routers across the ring using dynamic ROADMs or static fixed optical add-drop multiplexers (FOADMs) to avoid intermediate hops. With wavelength-granular switching, router port speeds need to be exactly matched with optical wavelength technology, and a partial mesh of wavelengths over the physical ring can be created. Many network operators generally wish to optimize capacity on the fiber, however, which means deployment of advanced higher-capacity optical technologies are directly coupled with the deployment of higher-speed router ports. This means that the two technologies cannot evolve (or be upgraded) independently. Hence, a tradeoff (or compromise) must be made when trying to increase the “meshiness” across a fiber ring or when upgrading router interface speed. With growing adoption of 100 gigabit per second (100G) optical wavelength technology, a substantial amount of aggregate traffic between two routers on the ring is often a necessary condition before the deployment of a direct express link between those sites can be considered, because doing so requires deployment of new router interfaces at each end as well as activation of a new wavelength. In addition, the impact of changing the IP topology must be analyzed because the creation of express links may reduce the amount of transit traffic at other locations while also influencing the routing of other traffic. In some scenarios, traffic engineering may be an option to help load-balance the traffic, but that could result in higher operational complexity.
When compared to using static point-to-point wavelengths between adjacent routers around the ring (the Circle Line model), however, this approach can help to scale the network, as the percentage of transit traffic through routers at each site is reduced. But at some point incrementally adding too many express links to reduce transit traffic can result in lower interface utilization, thereby stranding capital expenditure (CapEx) in the network. As such, a careful balance needs to be struck.
Option 2: OTN (Sub-wavelength) Granular Switching
A second option is to employ routers along with optical transport systems that provide converged WDM transmission along with integrated sub-wavelength switching. Topologically similar to option 1, this approach enables operators to create digital express links beneath the IP layer and create a partial mesh. There are two key differences between the two options, however.
First, the newly created express link has sub-wavelength granularity, and thus the corresponding router interface can be less than the wavelength rate). If the actual aggregate traffic between site A and site B is much less than 100G, then the network operator has the added flexibility to deploy lower cost Nx10G or 40G router interfaces, while still maintaining the benefits of 100G wavelength strategy on the fiber plant. This reduces incremental expenditures through lower-cost router ports, and also improves utilization of deployed optical wavelengths. And it means upgrades to the wavelength technology can be made independently of the router port upgrades. It may even make better utilization of the existing router system’s hardware resources, such as the backplane.
Second, this approach introduces the option of using sub-50 millisecond (ms) digital protection schemes within the transport layer as an alternative to or in conjunction with IP layer protection schemes. While this solution provides added flexibility in preserving and leveraging existing router assets, by not requiring a mass upgrade of every router port to 100G, the transport layer is still essentially interconnecting router ports with “opaque” pipes, without any awareness of the packet flows into and out of those transport pipes.
Option 3: Packet/OTN Switching (AKA Follow the Traffic, NOT the Fiber)
A new option that many operators are starting to consider is deploying packet intelligence into the WDM/OTN infrastructure, which in many cases is already providing “transparent wavelength” connectivity services. By deploying packet-aware ports into transport systems, a new level of flexibility is created that enables network engineers to leverage Ethernet virtual circuits and fine-grained OTN circuits (sized in increments of 1.25G) as a way to interconnect routers rather than entire opaque, coarse-grained 10/40/100G wavelength services. In a fiber ring, when a mesh pattern between routers is desired, instead of deploying a dedicated fixed-rate router port to connect with each of the other routers that are part of the mesh, the operator can use the packet-aware transport system to split any single high-speed router port into multiple variable-sized single-hop virtual IP links, and switch them to multiple router destinations. Similarly, at the destination site, multiple streams of packet-aware transport tunnels, each supporting an IP link, can be aggregated and handed off to the router with a single high-speed interface.
With this approach, a single 100G router port can be partitioned into multiple differently-sized virtual router interfaces (say, 40G and 60G), mapped efficiently into a corresponding amount of OTN bandwidth (a 40G circuit and a 60G circuit), and switched at the OTN layer to other 100G router interfaces that are also virtualized in a similar fashion. With this technology, network operators can not only efficiently bypass transit routers via the underlying switched WDM/OTN layer, but they can also achieve more flexible IP link topologies by virtue of the transport layer being fully packet-aware, and by mapping specific packet flows directly into flexibly-sized Layer 1 express tunnels.
This approach also has an important implication on protection strategy. With packet-aware Layer 1 tunnels now supporting these flexible IP links, sub-50 millisecond protection schemes natively supported in the transport systems, such as Infinera’s Fast Shared Mesh Protection (FastSMP) can be fully leveraged. This provides a cost-effective alternative to more conventional IP protection schemes, which often require significant over-provisioning of the IP links, and which in turn leads to stranded optical bandwidth that is often only utilized in failure situations. With this approach, transport-layer protection can still provide sub-50ms protection performance, and restore IP link connectivity across the ring without triggering re-convergence, while also providing the added benefit of reducing the over-dimensioning of the IP links, and thereby increasing IP link utilization.
So, while packet blades on WDM/OTN systems can be considered as an interesting way to enable high-speed Carrier-Ethernet-over-WDM or Metro Ethernet Forum (MEF) compliant services directly from the transport layer (without going through higher-layer routers), the same technology can also be used to improve IP link engineering design and topologies. As the traffic patterns evolve or change over time, the packet-aware optical transport layer can correspondingly adapt to create the best suited transport topology supporting the needs of the IP layer. Compared to the traditional static “Circle Line” model, this approach enables network operators to realize a much higher performing network by creating a flexible and adaptive mesh topology at the IP layer that facilitates router bypass, as well as a highly efficient network by creating configurable “express” links across the fiber ring, sized to actual traffic needs. With the simple addition of packet-awareness to WDM/OTN transport, this new level of flexibility and cost efficiency can be realized without any fiber upgrades.
Fortunately, despite the lack of any express options on the Circle Line, I did make my flight that day. For more information, click here to contact us.
- White Paper: The Evolving Economics of 100G Transport Networks
- White Paper: The New Packet Optical Core for a Software-driven World
- White Paper: An Introduction to Fast Shared Mesh Protection
- Brochure: Infinera Packet Switching Module