Early in 2014, the Computer History Museum added a Juniper M40 router to its collection. This router, initially released in 1998, was the first internet router to use custom-designed silicon to accelerate the movement of internet traffic in the largest nation-wide internet backbone networks. The M40 launched a race that continues to this day: keeping ahead of the growth of internet traffic with highly-specialized silicon technology.
All of the major Internet Service Providers, companies like AT&T, Verizon, Deutsche Telekom and many others, have national backbone networks. These are made up of routers placed at strategic points around the country, connected together by thousands of miles of fiber optic cable.
AT&T IP Backbone Network, 2Q 2000
At each point in the network where several fiber optic links come together, the network designers place a router, which acts as a traffic exchange. The resulting web of connections can get traffic from any place to any other place by hopping from one router to the next.
Internet traffic, such as a web-page download, is sent as a series of packets, each of which contains an Internet Protocol (IP) address to show where it’s going, and some data (usually no more than 1500 bytes). Needless to say, when only a few hundred words will fit in each packet, it takes many packets to transmit even a simple web page, let alone a movie or phone call. Multiply that by millions of people using the Internet, and there are a lot of packets to be sent!
As a traffic exchange point, the key function of the router is to look at the IP address in each packet and forward the packet to another router that might be closer to the ultimate destination (e.g., your laptop!), a process called forwarding. Looking up an IP address in a table isn’t so hard to do, but the list of possible destinations (called ‘routes’) in the Internet is large (currently over half a million), and it has to be done very quickly – a single Gigabit Ethernet interface can deliver 1.4 million small packets every second, each potentially with a different destination.
Basic router operation
And while looking up the address to find the next place to send the packet (called the Next Hop) is the essential function of a router, the forwarding process actually has many other steps, to manage congestion (too much traffic trying to go to the same place), multicast (sending the same packet to many destinations, such as broadcast TV) and a myriad of security functions. Then there’s the whole challenge of how the router even knows where the next hop should be – that’s done by Routing Protocols, which has its own long and storied history!
Juniper M40 Router
Routers have been fundamental building blocks since the invention of the Internet, but early routers were simply computers with network interfaces running specialized software. All functions – looking up packet addresses, shuffling packets from incoming to outgoing interfaces, as well as computing routing tables – were done by carefully-written software running in the router.
But the amount of traffic passing through routers grew rapidly in the early 1990s, to the extent that ordinary computers just couldn’t keep up. Recognizing that even routers made of many general-purpose computers weren’t going to be fast enough, the founders of Juniper Networks took advantage of the growing capability of customized silicon chips called Application Specific Integrated Circuits (ASICs) to make special-purpose hardware that could perform the key elements of the router’s function much faster than ordinary microprocessors. This formed the basis of the Juniper M40 router, the first commercially-successful product to use specialized ASIC technology to forward packets from the router’s incoming to outgoing interfaces without being slowed by software intervention.
The result was a huge increase in router capacity, from a few Gbit/s to tens of Gbit/s for a single router. These are modest numbers by today’s standards, but it was a game-changer in 1998, enabling the growth of the Internet that continues today.
Internal structure of a router
The Juniper M40 router, and most high-performance Internet routers since, are built from several key functions.
The most obvious router component would be its External Interfaces. The M40 router is modular, which means that small interface modules called PICs can be plugged in, providing connections to many different kinds of copper or optical fiber transmission media, such as Ethernet, FDDI, or a telco favorite, SONET/SDH.
Inside the router, there’s a Forwarding Plane, which is where the high-speed custom silicon devices that move each packet are found. The forwarding plane’s job is to receive each packet one at a time from the interface modules, look up the packet’s destination address in a route table, and then send it to the proper outbound interface. The M40 router’s forwarding plane actually does a number of additional functions to manage network overload, or to implement security policies, but the key idea is that the forwarding plane does all the work to get packets from any input to any output, and is carefully optimized to do it quickly. With a maximum capacity of over 40 Gbit/s, the M40 forwarding plane is capable of forwarding 40 million packets each second.
Modern high-capacity routers still use highly optimized ASIC forwarding planes, but capacity for a single router can easily run to several Terabits/sec (i.e., several thousand Gb/s).
The third element in the M40 is the Control Plane. The forwarding plane is all about lightning-fast decisions on where to send each individual packet. But it’s the control plane that informs the forwarding plane of the rules to use for these lookups.
While the forwarding plane is built from highly-specialized custom circuitry, the control plane comprises very complex software running on a high-end microprocessor. In the M40 router, an x86 processor running the FreeBSD operating system is used to control the router. The control plane has several tasks:
Managing the chassis, that is, initializing and configuring all the interfaces and the forwarding plane when the router starts; Computing the routing tables. This complex task involves communicating with the other routers in a network to learn the best way to get to any given Internet destination. There’s a veritable alphabet soup of protocols used by the control plane, such as BGP, ISIS, RIP, OSPF and many others, but at the core, they all are different ways to figure out the best way from point A to point B; Communicating with the network operator. Internet Service Providers usually have one or more operations centers where network operators can monitor and control routers and other gear. Another set of communication protocols are used to transmit problem reports and status to the network operators. Many of the principles developed in the M40 are still in use in today’s largest Internet routers, including:
