Building a 10G Network – Part 1


One of my current projects is to build out a 10G network to support server consolidation we’re doing. How did this come about? Well, for a long time, we were corporately going the route of isolating specific services to specific servers, growing these little pizza box 1U servers everywhere. All of a sudden, the data center managers are having trouble providing rack space, power, and cooling to the zillion or so little servers. So, the new push is to free up RUs, power, and heat by migrating all these little servers into blade centers, in some cases blade centers running VMWare.

The one major problem with this strategy is the network. The pipes we need to plumb into the blade centers have to be huge: we need big bandwidth going into these things. So, either we plumb a ton of 1G copper lines into the blade centers, or else we consolidate the network pipes from a ton of 1G coppers into a much smaller number of 10G pipes. 10G is clearly the way to go, but in my corner of the world, that has its challenges:

  1. I don’t have 10G ports anywhere. Over the last 2 years, I’ve built a sizable data center on 1G copper, mostly using 6500s in a combined core/distribution layer and 3750s at the access layer. While the new 3750Es support 10G uplinks, the legacy 3750s do not. The 10G need happened all of a sudden, as so many things do in a fast-moving company. It’s not like we had a chance to get it built before it was actually needed.
  2. The 10G standards do not include any sort of UTP copper. Blech. I have a data center full of Cat6a, which we were hoping would eventually become a 10G copper standard, but it ain’t happened yet, assuming it ever will. From what I hear, 10G over UTP is a problem.
  3. 10G is expensive right now…really expensive. And if you want high port density, the options are few. My beloved 6500 platform isn’t a great answer for high 10G port density. I should restate that – you can get port density, but you’ll oversubscribe the 6500 badly. To eliminate or at least reduce the problem of 10G oversubscription at high port density, you have to get into the Nexus 5K and 7K boxes. Those boxes are so new, they ship with lab techs, never mind that I don’t have the budget for them. So I’m waiting until 2009 to do a serious Nexus eval.

Nonetheless, I still need to get 10G out to the incoming blade centers, so here’s what I’ve done. I’ve added 8 port 10G blades to my core/dist 6500s. To get a physical infrastructure going, I’ve got contractors on-site building a new fiber MDF. While we’re still hoping that we’ll be able to use our Cat6a to carry 10G at some point, for now the answer is multi-mode fiber. (CX4 is okay, but doesn’t go the distances we need.)

Now, to deal with port density…I’m ignoring that particular problem right now, at least as far as the oversubscription issue is concerned. What I ended up doing was buying 4900Ms to act as the 10G access layer. I’ve got the 4900Ms coming back into my 6500s using 2x10G etherchannels (or will just as soon as I’m done building it). So the 4900Ms pick up the blade centers, and the 6500s pick up the 4900Ms. Yes, this design is oversubscribed – this is not Cisco’s new “data center ethernet” proposed standard, which includes the concept of lossless ethernet.

I’ll blog about the 4900Ms later on. I’ve run into a couple of interesting “new to me” features getting them configured.

About the author

Ethan Banks

Most people know me because I write & podcast about IT on the Packet Pushers network. I also co-authored "Computer Networks Problems & Solutions" with Russ White.

Find out more on my about page.


  • I was chatting with the consultant we use to build the AutoCADs and produce the build specs for our centers a few weeks ago and he was telling us that a lot of people are now opting to go entirely fiber on their uplinks. Not only is the price of copper going up, but with a single pair of multi-mode fiber you can run 1G all the way up to 40G eventually, so “future-proofing” becomes more than some dumb marketing buzzword.

    Our consultant also mentioned that as copper standardizes on higher qualities, we’re going to have more and more trouble finding low-end copper runs for stuff like ILO that really needs about 1Mbps, and we’ll end up running Cat6 everywhere or something nasty.

    I’d be interested in hearing more on the 4900Ms; we’ve been eyeing them for some time, but haven’t had a reason to buy one yet.

  • Ethan,

    6500-E and new SUPs/10GE cards will have support for 80Gbps backplane in the future, so oversubscription will hopefully get better.

    Nexus seems a very good option, but i’m also waiting until it gets a detailed evaluation from other companies.


Most people know me because I write & podcast about IT on the Packet Pushers network. I also co-authored "Computer Networks Problems & Solutions" with Russ White.

Find out more on my about page.

Subscribe via Email

Receive complete, ad-free posts in your inbox as I publish them here.