The Ethernet Switching Landscape Part 02 – Finding meaning in latency measurements.


This is one of a multi-part series on the Ethernet switching landscape I’m writing in preparation for a 2-hour presentation at Interop Las Vegas 2014 (hour 1 & hour 2). Part 1 of this written series appeared on NetworkComputing.com. Search for the rest of this series.

One often quoted statistic in Ethernet switch specifications is latency. Latency figures are usually cited in microseconds or nanoseconds. With ever decreasing latency numbers as new switch platforms emerge, what is a networker to make of latency? Is latency a key measurement to focus on when evaluating Ethernet switches? The answer to that question comes in two parts. First, there needs to be a good understanding of what latency is. Second, there needs to be a business use-case where latency matters. For many consumers, once you understand what Ethernet switch latency is, you might decide it’s not a big factor in your buying decision.

What is latency?

Generally speaking, latency measures the time it takes for a frame to enter and then exit a switch. In other words, latency is a measure of how long it takes for the switch to do its job – that of “switching” a frame in one port and out another. Latency is sometimes described as “port-to-port” latency. Think of it as the amount of time a frame spends inside the switch. That’s a rough description (and not completely accurate), so let’s look a little closer.

According to Gary Lee of Fulcrum Microsystemslatency can be measured in a number of different ways.

“There are several ways to measure latency through a switch: first-bit-in to last-bit-out (FILO), last-bit-in to first-bit-out (LIFO), first-bit-in to first-bit-out (FIFO) and last-bit-in to last-bit-out (LILO). In each case, latency is measured at the switch ingress and egress ports.”

I only raise this point about the different measuring techniques only because as you Google around looking for information on switch latency, you run into this data. Various ways to measure switch latency might seem a concern. How do you know that the vendors you are comparing are using the same method when citing their latency specifications? Reality is that practically all modern switch architectures are cut-through (at least when it comes to low-latency forwarding), meaning that the switch is forwarding the frame before the entirety of the frame has been received. For low-latency operation, cut-through switching is expected. The alternative to cut-through is store-and-forward. In this mode, a frame must be received in its entirety before being forwarded, which notably increased the time it takes for a frame to be switched, as well as making it variable. The point is this. Assuming cut-through forwarding, the only accurate way to measure latency is either LILO or FIFO.  To quote Gary again,

“These methods [FIFO/LILO] are effectively the same and are the only way to properly measure the latency through a cut-through switch.”

Implicitly, then, I believe it’s safe to assume that when looking at latency numbers for cut-through switching operation, numbers from different vendors are directly comparable. If you can find a latency measurement reported by a vendor, it should indicate a FIFO/LILO measurement of a switch in cut-through mode.

How important is latency?

Let’s take a look at latency measurements as reported by their vendors for a few Ethernet switches. Note that this is not an “apples-to-apples” comparison. These switches have different ASICs, may have different host-facing physical ports, and likely first came to market in different years. I have a point, though – bear with me.

These switches all have widely varying latency characteristics, which are market differentiators for them. But I believe that latency is only a differentiator to certain market segments. Unless you are building a network where nanoseconds count – such as high frequency trading – then the port-to-port latency of one Ethernet switch over another just isn’t going to make any appreciable difference in the performance of the applications running across your network.

Another consideration is that the Ethernet PHY (the physical Ethernet medium you use, as in the transceiver) will also impact latency. For example, 10GBase-T (10GbE over twisted-pair copper) PHYs introduces noticeable higher latency than SFP+ based PHYs. Paul Kish in the Belden Right Signals blog opines,

“With simplified electronics, SFP+ also offers better latency—typically about 0.3 microseconds per link. 10GBASE-T latency is about 2.6 microseconds per link due to more complex encoding schemes within the equipment.”

But again, the question comes back to one of business needs and application. 10GBASE-T has a use-case in top-of-rack (ToR) or end-of-row (EoR), considering 10GBASE-T server LAN-on-motherboard modules (LOMs) are coming to market. Should the higher latency of 10GBASE-T introduced by the PHY put you off? Only if microseconds matter.


Latency, low latency, ultra low latency and the rest are relevant terms that indicate how long it takes a switch to move traffic through itself. However, beyond the physical medium and switch architecture, many other things could impact application latency. At the end of the day, it might be fun to argue about microseconds and nanoseconds in the context of your switchs, but what difference will it make if you have bottlenecks in other places? Consider these elements of your IT engine that are far more likely to negatively impact your application performance than Ethernet switching latency.

  • Underperforming storage reads & writes.
  • A CPU or memory bound host.
  • A congested network path.
  • A slowly responding DNS service.
  • Traversal of a long-distance WAN link.
  • An overloaded proxy.
  • Traversal of a DPI device.

These sorts of problems can introduce significant fractions of a second (or even whole seconds!) of latency into an application transaction. For most shops, these sorts of issues are where the performance battles need to be fought. Buying the lowest latency switch you can find just isn’t going to appreciably help, unless you’re one of the very few with that sort of application requirement. And you already know who you are.

About the author

Ethan Banks

Most people know me because I write & podcast about IT on the Packet Pushers network. I also co-authored "Computer Networks Problems & Solutions" with Russ White.

Find out more on my about page.


  • 0.6us is a good cut through latency but for the corner cases such as High Frequency Trading where having the lowest latency is the difference between making or losing money and being second fastest to execute trades equals a loss, is it fast enough?
    It is a fair point that in such niche environments as HFT, network transit delay accounts for no more than 20% of trade execution latency (at the very most). There are a multitude of factors in capital markets trading which can add to latency and it is common for these elements to make switching latencies seem largely irrelevant.
    There are a few switches now on the market achieving in the region of 0.2us (200 nanoseconds FIFO)
    I won’t name these models. It is safe to say that many of the lowest latency switches on the market have deficiencies in other areas. Post sales support being a typical failing, others are from startups which make for a risky investment in infrastructure.
    The issue I think for equipment manufacturers is that to achieve market beating low latency typically requires them to make a heavy investment in silicon and R&D with little opportunity to recoup that investment from such a niche market. This then results in those manufacturers using commodity silicon as a more accessible route into that market. The thing about commodity silicon is that several manufacturers can select exactly the same ASIC and therefore end up with a virtually identical latency profile. A switch using commodity silicon is unlikely to be the fastest on the market.
    In spite of all of this, for the majority of use cases that you will see, commodity silicon provides more than adequate latency for high performance applications and if the latent elements have not been driven out of the rest of the trading infrastructure stack, the decision process of selecting a .6us over a typically more expensive .2us switch is moot and somewhat irrelevant.

  • It’s also worth confirming with the vendor the load or throughput at which latency measurements are made.
    There was a time when not all switching functions were carried out in hardware & consequently xPU load had an effect on latency (& throughput).
    Yes, this is not so prevalent nowadays but strictly speaking in order to qualify performance measurement, the total test traffic load should be quoted also.

    Good work!

    • For network engineers, I personally think end-to-end path latency is a more important metric that the latency of an individual switch. Measuring individual switch latency is hard, requiring highly precise testing equipment from folks like Spirent.

      Measuring end-to-end path latency is a different beast, and can be partially conquered with something a simple as ICMP messages (ping, some flavors of traceroute), or finding RTT using Wireshark to measure interpacket gaps between source and destination. That sort of a capture can be a little deceiving, as there’s the time taken for a remote host to respond to inbound traffic, and that chunk of time isn’t technically part of the end-to-end path latency. But, it’s still going to be a good indicator of latency between the two endpoints assuming neither endpoint is CPU bound. More interesting measurements can be found via Cisco IP SLA operations, as that’s a network device to network device test. Open source perfSONAR would also be useful for infrastructure testing. A major goal of perfSONAR is to figure out where the slow bits are in a given network path.


Most people know me because I write & podcast about IT on the Packet Pushers network. I also co-authored "Computer Networks Problems & Solutions" with Russ White.

Find out more on my about page.

Subscribe via Email

Receive complete, ad-free posts in your inbox as I publish them here.