I attended the Open Networking User Group meeting in NYC May 5 & 6, 2014, attached to Tech Field Day. ONUG was not a super-technical event overall, but it is an excellent chance for folks interested in SDN and open source networking tools to get together and discuss practical matters related to adoption of cutting edge networking technology.
One of the sessions I attended was “Understanding Whitebox Switching” with Big Switch Networks’ CTO, Rob Sherwood. If you’re a Packet Pushers listener, you’ve heard us talk about or with Big Switch several times. Big Switch is now in the space of whitebox switching, providing a network operating system called Switch Light that runs on a small but growing number of hardware switches or as a vSwitch in certain hypervisor & OVS environments.
Rob’s ONUG session wasn’t pitching Big Switch, not that he didn’t bring his own company up. Naturally, he did, but shilling for BSN wasn’t the point at all. The point of the talk was to orient folks on what whitebox switching is, and why they might be interested in such an approach. I gleaned a number of points from it, most of which I’d heard before, but I thought I’d share them here. I felt Rob’s presentation wrapped many points about whitebox switching up into a nice, tidy package. Many of my points are right from my notes on Rob’s presentation, so if you have heard this presentation of Rob’s before, what I write might sound familiar at times.
Whitebox switching is the idea that a networking consumer can purchase a generic Ethernet switch without a network operating system, and install an operating system of their choice. Alternatively, a whitebox switch could be bought from a vendor that will supply both the switch and the network operating system. But the key is that the long-term choice of network operating system remains with the consumer.
If we break this down into more detail, there’s really four major components that make up a network switch.
- Silicon. This is the switching chip that pushing packets and frames through the network. The silicon will have some set of functions that it is able to perform very quickly, such as forward an IPv4 packet according to a forwarding table, filter traffic via an access list, prioritize traffic according to a policy, encapsulate and decapsulate traffic using tunneling protocols, and so on.
- Box. This is the hardware chassis itself with a power supply, circuit board, cooling fans, Ethernet ports, etc. Someone’s got to make the box. As an aside, Rob mentioned visiting a manufacturing facility overseas where he saw identical switches branded with two different vendors insignia and shipped out the door. He was making the point that not all hardware is different on the inside just because there’s a label making people comfortable on the outside.
- Network OS and drivers. There has to be software sitting on the box that make it possible to program the silicon in the box to do something. That’s the job of the network operating system and low-level drivers.
- Applications. The applications are the daemons running on the network operating system providing networking services, such as routing protocols.
Whitebox switching is the idea that the silicon & box can be bought as one thing, and the network OS and applications as other things. This idea is counter to the traditional norm of network switch acquisition. Today, most customers purchase switching as a vertically integrated stack. The same vendor supplies the network hardware as well as the network operating system & applications. The network operating system is not able to be changed by the networking consumer.
The answer to why a consumer might be interested in whitebox switching isn’t an obvious as you might think. My knee-jerk reaction was a lower cost of acquisition. Assuming that vertically-integrated switching is expensive, whitebox switching should be less expensive. But of course, that’s really not the whole story to an IT purchase. Capex is just the beginning. Opex is just as significant a concern as capex, critically so in some organizations I’ve worked for. But even beyond that, there’s this matter of choice. Rob ranked these three whitebox switching motivators as follows.
- Avoid vertical lock-in. In other words, choice and flexibility is perhaps the greatest gain an organization makes moving to whitebox. This begins to make sense if you consider the server market. X86 architectures are popular in part because you can do whatever you like with them. Run ESXi. Or a different hypervisor. Run Linux. Or Windows. The point is that you can do what you like, change your mind in six months and do something else, and the manufacturer of the hardware won’t deny you that choice. It’s about compute power, not what you do with the cycles. That’s not so with networking, but the whitebox movement is pointing that it could well be this way: choice in networking like there’s choice in servers.
- Lower opex by building a networking environment that fits your needs. The idea here is that a customer can spend less in training, staffing requirements, day-to-day operations and the like if they can choose how to administrate their networking gear. In other words, why be locked into a Cisco-like CLI? Or into Junos? Or into SNMP? Instead, find a means of configuring, monitoring, and maintaining network hardware that works with processes your IT is already familiar with and use that instead.
- Lower capex via “horizontal competition” as Rob put it. This is simple economics. If a box is a box is a box, then why not buy from the lowest bidder? That will work for some, but not everyone. I’m sure there’s many reading this that can raise at least one obvious objection to this line of thinking. We’ll get to that a bit later in this piece.
Whitebox is a growing movement, but it is still admittedly small. Most major vendors still sell their products in the way that networking customers are used to: as a vertically integrated stack. While I don’t think I’ll be providing a definitive list, here are several groups of manufacturers and brands that are interesting.
- QuantaQCT. Quanta QCT is a division of Quanta Computer, Inc.; they are manufacturers of “the box.” IW Networks gear is rebranded Quanta hardware.
- Accton. Accton is another maker of “the box.” Edge-Core Networks gear is rebranded Accton.
- Delta Networks. DNI is yet another maker of “the box.” Agema Systems gear is rebranded DNI.
Network Operating Systems
- Big Switch Networks. BSN sells the Switch Light OS.
- Pica8. Pica8 sells PicOS.
- Cumulus Networks. Cumulus Networks sells Cumulus Linux, a full-blown Linux OS that presents Ethernet interfaces as NICs to the operator.
Notably, Big Switch, Pica8, and Cumulus are all very happy to sell you a vertically-integrated solution featuring a whitebox switch and their network operating system. And this speaks to the “lowering capex” caveat I alluded to above. Whitebox isn’t exclusively about a short-term reduction in spend, although that will be a factor for many. Many whitebox consumers are going to be happy to pay a little more for a supported solution from supplied by a single vendor.
Dell Networks is catering to this kind of consumer. Dell recently announced a partnership with Cumulus, where Cumulus Linux will be delivered by Dell on a Dell-branded switch, and Dell will fully support the solution. Dell has also partnered with Big Switch in the same vein. Does that mean we’re back to the vertically integrated, vendor lock-in problem? Not really. While a vertically integrated solution is nice from a support perspective, the consumer still has choice.
The mechanics of how whitebox switching is actually done gets interesting, and there’s a few disjointed points I want to mention here in the form of a Q&A.
Q: Could Cisco kit containing merchant silicon (the same ASICs many whitebox switches run) in the box run these whitebox network operating systems?
A: Not without doing things that would void warranties and possibly violate laws, but that doesn’t mean it isn’t theoretically possible. Think of it like jailbreaking your iPhone. Practically speaking, you perhaps *could* do this, but the average networker isn’t going to want to.
Q: So if a whitebox switch with no network operating system shows up on the dock, how does the NOS get loaded?
A: There is a project that facilitates this called Open Network Install Environment, or ONIE. Rob described ONIE as sort of like a hybrid of a PC BIOS and GRUB or LILO. ONIE provides the OS-less box enough smarts to boot up and get the NOS installed. The intent of ONIE is to provide network administrators a way to predictably install or uninstall a NOS from network hardware, so ONIE support is potentially going to be a standard feature to expect when looking at a whitebox switch.
Q: How does one acquire whitebox hardware? Direct, via a partner, or through a reseller?
A: It depends. There’s no one right answer here, as each answer has its own pros and cons related to pricing and support. Practically speaking, acquiring whitebox switches is going to be a similar experience to acquiring any other sort of IT hardware. Think it through, consider your processes and staff, and make the choice appropriate for your organization. You might be interested to know that whiteboxswitch.com is a legitimate thing, although the pricing should be considered “list” and not the best you’re likely to be able to get.
Q: How will whitebox switches be licensed?
A: That’s the million dollar question, and one that the market will help craft. Rob suggested in his presentation that two models are likely to work, both the traditional “perpetual plus annual support” license, as well as the more modern trend of “subscription-based” licensing. Time will tell here.
Q: How do I know my choice of box and NOS will work together?
A: NOS vendors are creating hardware compatibility lists that describe what hardware works with their operating systems. For example, the Cumulus HCL is published here.
Would you? Should you?
I see no reason the industry should stay married to the vertically integrated vendor lock-in networking model. Whitebox switching puts some power back into the hands of consumers. Frankly, most vertically integrated switching & routing platforms are loaded with features the vast majority of consumers will never use. Ironically, many of the features that consumers want to use come at a premium price. While that’s a heck of way to sell hardware/software combos at amazing profit margins, the approach is hard on an IT department’s budget. Thus, network refreshes have 5-7 year cycles, out of financial necessity. It’s too hard to get the depreciation time down much lower. That’s a big problem, in that a 5-7 year network refresh cycle of a monolithic network stack stifles innovation.
Whitebox switching puts new ground rules in place, lowering the cost of acquisition and ownership of a powerful packet forwarding engine, while at the same time allowing consumers to pick and choose the features and functionality they are willing to pay for. As the whitebox switching and NOS market begin to broaden, I believe what the market will see is a broad array of applications that consumers can choose to add to their newly flexible networks. I don’t think whitebox switching is merely about choosing a hardware manufacturer or NOS provider. The path whitebox switching ultimately takes us down is choice of applications running on those switches. That’s a whole different market. That’s a whole different way to think about networking. And really, it’s a different way to think about IT. Whitebox switching is sitting at the table where other IT disciplines like virtualization and devops have already pulled up a chair. What they’re talking about is the way IT moves forward to meet organizational needs. That’s a conversation worth having.
We’re seeing big vendors like Dell recognize the value here, and we’re even seeing Cisco offer a limited amount of choice on the Nexus 9K platform. So I think for the right customer, yes…you would. And probably should. See what’s going on with whitebox and explore. There’s lots of interesting doors the model opens up. Let’s start working through them.