From the blog.

Managing Digital Racket
The more I tune out, the less I miss it. But that has presented me with some complex choices for a nuanced approach to curb
Complexity – My Friend, My Enemy
Over my years of network engineering, I've learned that the fewer features you can implement while still achieving a business goal, the better. Why? Fewer
feeling-lucky

Why Nexus Validation Testing Is Worth Every Penny Cisco Spent

1,390 Words. Plan about 9 minute(s) to read this.

Cisco announced Nexus Validation Testing (NVT), which promises to greatly reduce the number of bugs that make it into production released NX-OS code. To achieve this feat, the NVT process creates customer-like switch topologies, deploys a customer-like group of features, then tests. This as opposed to testing individual features in a vacuum. Bugs that are found in the NVT process are enough to halt the release of NX-OS to customers. We cover NVT in more detail with Cisco’s N7K evangelist Ron Fuller on Packet Pushers Weekly Show 152, if you want more details on it.

The point came up on Twitter that Cisco spent real money to bring NVT to life. I don’t know how much of an investment that was, but let’s make up a number of $10M for sake of discussion. (Yes, I truly made that up. I have no idea what Cisco spent other than “a lot.”) My opinion is that NVT was worth every penny. Every single one.

Cisco has demonstrated a commitment to fixing a problem. What problem? Buggy code. Buggy code on a data center switching platform is a serious business risk. I’ll loosely categorize bugs into two big categories: bugs that crash hardware and bugs that break functionality. Either way, those aren’t problems you want to have to face in a data center. Working, reliable code is more important than feature-laden, bugtastic code. With NVT, Cisco is telling customers they get that. And they are also saying that they are committed to making that problem go away. Bravo. Am I more likely to consider Nexus as a data center switch, and/or stick with the Nexus gear I’ve already got? You bet.

Cisco wants engineers to trust them again. Let me tell you something about network engineers. We just want our stuff to work. It was hard enough to plan the installation, research the design, commit to a project, test the prototype, bring the kit online, and cut traffic over it. With that behind us, we’ve moved onto the next project(s). We don’t need the stupid thing crashing or some key feature doing random crap the moment our attention is elsewhere. NO TIME FOR THAT. GOT TO GO. OTHER THINGS TO DO. So believe me, if NVT accomplishes what it purports to – releasing field-tested NX-OS to customers – I’ll feel really good about implementing that code. I’ll have some trust in Cisco, Nexus gear and NX-OS. Network gear I trust is network gear I love. And network gear I love shines favorably upon the vendor that supplied it to me. Am I likely to keep buying Nexus and/or another product from that vendor? You bet.

Cisco wants customers to feel safe adopting new features. Over the years, engineers have learned that there’s a ton of risk in turning on the new stuff. Dot zero releases of software translates roughly to, “Do you feel lucky, punk?” In discussing with project teams the pros and cons of a new feature, I’d always qualify my assessment with “it’s dot zero” or “it’s dot one+.” That implicitly defines a risk characteristic of the software. We know software isn’t tested that well by the vendor…so…we’ll wait for the early adopters to lose their jobs, THEN we’ll give it a shot. With NVT, it’s more likely that I’ll be comfortable bringing that new dot zero feature into production. That means I can light it up sooner rather than later. And if I can bring it up sooner, that’s another chance Cisco has to hold my attention. I’m less likely to look at other vendors who’ve brought interesting features to market sooner. In other words, that’ll help keep me in the fold. Should that be worth it to Cisco? Every sales rep with a quarterly quota to fill says, “You bet!”

feeling-lucky

Cisco knows they need to preserve their existing customer base, and reduce the number of defections. If you look at the switching marketplace right now, there’s a large number of dogs nipping at Cisco’s heels. Some of them are trying to shift the Ethernet switching paradigm into something else. Some are trying to sell a superior product. Some are competing on price. I believe that a big reason many of these competing approaches have traction is because customers have become open to new (read “non-Cisco”) ideas. Why stick with Cisco if the darn data center switch they sold me isn’t the rock-solid foundation my business can count on? NVT says that Cisco grasps this, and is doing everything in their power to put a product in a rack that just runs. Why would I look somewhere else if the product I’m running is just working? Well, I probably wouldn’t. I’d be more likely to stay in the Cisco camp. That’s sales today, and sales tomorrow. NVT worth the investment? A thousand beancounters just said, “You betcha.”

There’s more to sales than keeping the customers you’ve already got. You also want to gain the customers you don’t. I see NVT as a differentiator for Cisco, and therefore an opportunity for Cisco to win more business. How so? As a Cisco salesperson, I can walk in to a potential deal, sell the product on its technical merits, and then bring it home with, “Oh, and by the way…this stuff works out of the gate. Do you know about our Nexus Validation Testing program? NVT builds a network like you would, loads up all the features we see customers loading up, then tests the systems at punishing levels. When the code is bug-free, we release the code. Not before. You buy this switch, it’ll just run.” Are there other vendors with a program like NVT? Maybe…but not that I’ve heard about. Squelching FUD is a massive hammer for a sales team to drive a deal home with, especially when selling into verticals where uptime is everything. When asked whether or not NVT matters, financial service providers are likely to yell, “You bet!”

With NVT, Cisco is demonstrating a long-term commitment to customer relationships. NVT isn’t just a “we’re going to try it this year and see how it goes” approach. There’s already talk of extending the testing model into other Cisco product lines. Plus, there’s that money that’s already been spent, again demonstrating commitment. Cisco’s in this code quality improvement thing for the long-haul. NVT could be the cornerstone of Cisco remaking its image into that of the company that makes reliable code. Ponder that for a moment. Imagine going back to the place in your mind where you can say, “Well yeah, I’m sure your stuff is good Mr. Non-Cisco Vendor. But I’ve been running Cisco kit for a long time. It just works.” Or, imagine moving the bar from NVT to “Cisco Validated Infrastructure,” where the building blocks of a Cisco data center are tested individually with the NVT methodology, and then tested holistically. The whole data center environment. *That* would be a selling point. “Yeah, but best-of-breed, and what about SDN + APIs that abstract hardware, etc.” you might say. And for some customers, mixing and matching their hardware and software will forever be their thing. But for other customers, getting a turnkey data center from a single vendor that’s tested interdependent device functionality in a customer-imitating way? “Oh my, yes please,” they’ll say. Sure, I just made up a “Cisco Validated Infrastructure” in a flight-of-fancy, but it’s not much of a leap to make since Cisco Validated Designs have been around for a long time.

Yes, but…

So, all this shiny, happy love I’m beaming towards Tasman Drive today does come with one big, fat caveat. And that’s this. If NVT doesn’t deliver the implied stability, it’s the Safe Harbor Apocalypse Redux. I learned over time to hate Safe Harbor. I got burned with a memory leak on core 6500s running Safe Harbor recommended IOS, and I’ve never really felt all that safe with the program since. My understanding is that Safe Harbor is dead now. Okay. If NVT does what it purports to do, I like to think that all of the blessings I outlined here will fall on Cisco. But if NVT fails, sending buggy code out the door anyway? I’m afraid those market share percentages are going to slip once again.

NX-OS 6.2 is going to be an interesting release.