Ethan Banks Not writing about IT.

FPGAs: A Way Forward For Networking Silicon?

F

I’ve become somewhat fascinated with the silicon that moves packets around our networks. My knowledge is admittedly shallow on the topic. That said, I believe I can make the following generalizations about the 3 major kinds of networking silicon to be found on the market today.

1. X86. General purpose CPUs are not especially well suited to moving packets around, which isn’t to say there’s some adequate-for-their-purpose performance coming out of hypervisor vSwitches. But we’re talking tens of gigabits. Not hundreds. That said, x86 is the most flexible in terms of programming. Therefore, vSwitches in hypervisors might end up with the ability to do new encapsulations or execute new forwarding paradigms before that same function can be done in dedicated network silicon. Thus, we’ll see the proposed Geneve overlay running in a vSwitch before we see it in a hardware switch, if we ever do. By the same token, we also see the latest OpenFlow specifications making their way into Open vSwitch (OVS) before we see hardware vendors implementing them.

2. ASICs. Custom designed silicon built expressly for the purpose of forwarding specific network frames and packets across the wire at line rate offer the greatest performance, but the least amount of flexibility. ASICs are hard for manufacturers to make; they take roughly 1.5-3+ years to design & fabricate, and incur a significant up-front cost. Choosing what features to put into an ASIC requires the foresight of Nostradamus coupled with a gambling man’s nerves of steel. Listen to the Packet Pushers Podcast #185 for more discussion on this interesting topic. The key to understand is that an ASIC, once built, is stuck with the features built into it. This is why it’s not possible to, for example, run a code upgrade to gain TRILL functionality on an ASIC that was never designed to do TRILL encaps. Or why VTEP is not a pervasive feature in data center class switches yet. If no one saw the VXLAN storm coming or wasn’t confident that VXLAN would be a winner, then the ASICs to support VTEP weren’t being spun up.

3. FPGAs. Field programmable gate arrays promise the performance of an ASIC with the programmability of X86. That said, I understand FPGAs to be even harder for manufacturers to create than ASICs and somewhat error-prone to actually program and deliver new functionality on. So yes, FPGAs seem to hit a sweetspot, but might not be so wonderful in real-life networks. Even so, some manufacturers are making FPGAs and putting them into their networking products.

Huawei is one such manufacturer of FPGA-based network processors. Huawei has created the Ethernet Network Processor, which is available in their S12700 switch, and will eventually be found in other switches in the Huawei lineup.

Huawei S12700 series agile switches are core switches designed for next-generation campus networks. Using a fully programmable switching architecture, the S12700 series allows fast, flexible function customization and supports a smooth evolution to software-defined networking (SDN). The S12700 series uses Huawei Ethernet Network Processor (ENP) and provides native wireless access controller (AC) to help build a wired and wireless converged network.

huawei-s12700The big idea behind the ENP is that whatever comes down the line in the form of, say, OpenFlow enhancements or new encapsulation types, the ENP can be programmed to handle it. An FPGA is, theoretically, future proof. That’s a very compelling idea in a networked world that keeps changing. Chip programmability takes the Nostradamus element out of the chip design equation, although certainly places a major burden back on the shoulders of vendors’ development teams.

Are FPGAs a way forward for networking silicon, then? I believe they are an option, but the key is in execution. FPGAs are not new; I remember tracking Xilinx back in college many years ago. If FPGAs were easy, all manufacturers would be making them, and we networking practitioners would be consuming them instead of ASICs. Yet, ASICs dominate. I assume that’s because doing FPGAs right is hard, despite the strong desire for programmability in an unpredictable network world.

Anyone well-versed on ASICs vs. FPGAs care to comment? Feel free below.

11 comments

  • One reason why you would prefer an ASIC over an FPGA is that given the same task a specialized ASIC will almost always outperform an FPGA (aussuming equal die size and manufacturing process). Of course the same way an FPGA will outperform an x86 CPU.

    For management, and routing protocol updates a general purpose CPU is a good fit.
    For the forwarding/encapsulation/crypto party I think FPGAs make some sense because you can add new encapsulations and maintain high performance. I think it is probably efficient to use a combination of ASIC+FPGA.

    • Just to expand on this a little bit: a modern, high-end FPGA may be able to fit something like 16 to 24 10GbE ports, and the associated switching logic, and a modest amount of buffer memory (a couple of MB, maybe). Compare that to modern switching silicon from the likes of Broadcom (Trident2 has >100 10GbE links). That FPGA is probably going to cost more than the dedicated switch chip, and burn at least as much power.

      You’ll want to use FPGAs for high-value, low(ish) volume, low-to-medium density applications like firewalls, load balancers, or to implement your own custom packet processing logic, like in the Arista 7124SX that Roy mentioned.

      But if you wanted to implement a “vanilla” switch, which “only” performs “normal” packet forwarding with high port density and low power per port, you could never do it competitively in an FPGA.

      • High end FPGAs are capable of much more than that. You can get your hands on FPGAs that have 96 10G serdes on them today. You can build quite a powerful packet processors out of that.

        The biggest challenge with FPGAs is getting and maintaining the right talent to make them do what you want them to do. For a decent packet processor built out of FPGAs you’ll need a 10-12 person FPGA team (including DV). Its a very specialized skill that does not come cheap. Dare I say it’s a dying breed? This makes some of the specialized hybrids mentioned below (I would add Tilera to that list) interesting. Comparing any of them to the densest Tridents is futile, they serve a different purpose….

  • You don’t mentioned about programmable network processors like EZchip NP, Marvell Xelerated, Broadcom XLP families, etc. Network processors are some how a mix of CPU and ASIC structure which offer quite nice flexibility, are programmable with C or assembler and have great perfomance. Cavium Octeon network processor is almost like standard multicore CPU where you can try adapt and deploy software written for x86. So spectrum of choices is continuous and you can choose what you like. Many network manufactures uses NP instead of typical ASIC. I think Huawei Ethernet Network Processor is the same category – are you sure that it contains FPGA inside? The problem with FPGA is that they are very energy consuming. FGPA are also more easly accessible for research and universities comparing to programmable network processors.

  • I am genuinely tired of pointing to F5’s products but they are a good example of a company that have been using a mix of NPs and ASICs or NPs and FPGAs.

    Equally important though is quality of code. This would apply to say F5’s Virtual Edition FastL4 processing that’s still pretty quick on x86 and the work Brocade did with Vyatta to massively increase it’s throughput on x86. Making maximum use of NIC offloading functions also helps.

    Also worth pointing out that many overlay technologies really don’t perform too well right now as many ASICs and NPs don’t understand/support the encapsulation formats.

  • The reliability of ASICs and FPGAs is not the same. It could be important to critical applications like Lossless Ethernet and SAN.

    The configuration bitstream of FPGAs is mostly stored in RAM cells. That means that the algorithms, the core logic is defined by RAM contents. In case of single bit error there is a danger of algorithm modification, a severe malfunction of processing logic. There’s no ECC correction like one we find in server DRAM chips. To protect algorithms from bit errors one need to use TMR (Triple Modular Redundancy), safe finite state machines, etc. This increases the gap between ASICs and FPGAs resource usage by 3-4 times.

    Vendors like Xilinx, Altera have mitigation logic in FPGA chips that performs periodic scrubbing of configuration RAM contents and recover from SEU (Single Event Upset) errors using ECC-like codes. This does not protect from short time malfunction, only reduces the risks.

    FPGAs are very well suited for prototyping and pre-ASIC phase of development.

By Ethan Banks
Ethan Banks Not writing about IT.

You probably know Ethan Banks because he writes & podcasts about IT. This site is his, but covers other stuff.

Get the details on his about page.