This post is a simple challenge to all of us network engineers. Let’s document our networks better. By “better”, I mean more clearly, currently, and completely. And I will also make the case that new projects should not end with documentation. They should start with it.
I have frequently been in the position of inheriting networks from engineers that have moved on. My new manager and/or peers will share with me the documentation on hand, and I’ll use it as a starting point for my network discovery. All too often, the documentation I’m given is vague: icons, lines, and clouds connected to that generic Ethernet bus icon in Visio. Management IP address of the icon in question? Good luck. Hopefully, I’m smart enough to figure it out.
My personal takeaway from inheriting some frustratingly ambiguous diagrams over the years is to remember that I’m not building a diagram just for myself. I’m also building it for someone else. There’s lots of room to argue about how much is too much information to crowd onto a diagram – fair enough. But at least have the diagram be clear enough that someone who’s never seen the network before would be able to make a good start at managing it.
Another disease plaguing network documentation is antiquated information. We build these splendid network diagrams and procedures that go out of date quickly if not maintained. I’m as guilty as anyone here, as it can be difficult to remember, especially in a complex library of documentation, just what needs to be updated. This is a problem with no easy answer if you’re having to update manually. That said, I see some hope in automated software. I saw a demo of the NetBrain product a couple of weeks ago (disclaimer – NetBrain is a Packet Pushers sponsor), and I see in it promise of automating network diagrams. NetBrain achieves diagram automation by showing the network topology in real time based on its discovery and on the filtered views the user sets. The idea is compelling, and I felt that their interface worked well. I’m planning to do a full demo once I get an IPAM evaluation behind me.
In my mind, complete network documentation is a package providing a network operator with enough detail to properly understand the network that they are working on. This is not only a set of diagrams, but is also written documentation that explains why the network has been configured the way that it has. When I feel I’ve completely documented a network, I’ll have contributed to the following knowledge domains.
- Visual reference. These are the aforementioned diagrams. L3 diagrams are especially key for networks of any size. L2 diagrams can certainly be useful, but are difficult to generate as the network grows in complexity. Jaakko Rautanen, community blogger at Packet Pushers, wrote a very popular article entitled How to Draw Clear L3 Logical Network Diagrams worth a look.
- Standardized network configurations. I am a big believer in every network device configuration matching where possible. Using AAA? Use it everywhere. Configure SSH with particular tweaks? Tweak the same everywhere. Exec timeout set to a non-default value? That value should be the same everywhere. Etc. Okay. Once you’ve worked out the network device standards for your particular environment, then codify them. Use a wiki. Use SharePoint. But use something that takes a standard code block for a given device, explains the logic behind it, then explains the commands that make it work.
- Wiki-style posts explaining network functions. I believe that how things work is a critical element to network documentation. How does authentication work for VPN clients? How does route redistribution (export/import) work at the WAN edge? How does the application delivery controller manipulate the HTTP header to make that particular web application work? How does the dual BGP feed work at the Internet edge? How is MLAG set up between the aggregation layer switches?
- Contact lists, contract numbers, and IDs. You don’t want to have to search around for the support number to call when something is broken. Have that list prepared ahead of time. That’s part of network documentation. My usual trick for WAN circuits is to embed the support number & circuit ID in the description field of the circuit. If the circuit went down, I’d have to go no further than my NMS and/or alert that my NMS sent me to know who to call and what information to give them to get a ticket open. Along those lines, you really want your serial numbers & support contract numbers at hand when a network device has gone down. You can’t pull the serial number via an SNMP query if the device has crashed and can’t be accessed remotely.
- Scripted access to every device. I consider building a script that allows me to access every device I’m responsible for a part of documentation. Maybe you’d like to lump that in with automation, but the way I see it, if I’ve scripted access to the gear I have to manage, that’s documentation I don’t have to refer to. My favorite tool for this is my terminal emulator of choice, Emtec’s ZOC. I’ve used ZOC for many years now. There are native versions for Mac and Windows. I don’t claim it’s the best out there, because to be fair, I haven’t looked for a long time. ZOC is an old friend, does everything I want, works reliably, is maintained well by Emtec, and is portable if you need it to be. Alas, ZOC’s not free, but it’s one of the few applications I use every day. So the expense was worth it. With ZOC, I’ll build multiple scripts to access devices. One will access in-band management. Another will access out-of-band management if available, usually via a console server like the kit from Opengear (another Packet Pushers sponsor). And a third will use an alternate path if available, such as the external interface of an Internet-facing firewall. The reason I go to all this trouble is so that I don’t have to think about it when I’m troubleshooting a problem. I work far away from all of the gear I support, so I need to have multiple ways to access gear, without having to dig through documentation when under the gun to resolve a problem.
A final thought on complete network documentation is having access to that information when the network is completely down, including catastrophic power failures affecting your workstation. This might mean a hardcopy in a binder. This might mean a sync of a server share to a laptop with a battery, if your data policy permits it and/or this is acceptable considering the potential for data theft. This might mean a thumb drive updated regularly but otherwise kept under lock and key. But they critical part is to have access to the documentation you need when the lights are completely out. If you’re working in an especially sensitive & secure environment, this might actually be a difficult feat to pull off, as some environments strictly limit what hardcopies can be kept where. But that one day when you need that data, you’ll be glad you have it.
Starting With Documentation
Over the years, I’ve changed my project workflow to include documentation up front. I now start with documentation as opposed to finish with it when doing a network design. A layer 3 network diagram is generated, complete with IP addresses…all of which have been reserved in the IP management database. It may or may not be possible to have the wiki contributions sorted, all depending on the nature of the project. And I must add a confession. I’m not as good about all of this “starting with documentation” as I should be. When under the gun to get a project done, documentation is often the first ballast to get thrown overboard. But I do believe it’s the right way to go about a project. When working with other team members on the project, being able to point them at pre-existing documentation is enormously helpful. Plus, it helps when configuring gear to remember what’s being done in the overall context of the project. I find that with even slightly complex network deployments, working from scraps of paper, whiteboard sketches, or (horrors) my own memory means I’m thinking through the same things multiple times and second guessing myself – an error-prone waste of time. On the other hand, when I’ve taken the time to officially document what’s going on, I can comfortably proceed with rolling the design into the production environment.
New Year’s Thought
Network documentation matters. If it doesn’t matter to you, it matters to someone else. And even if you’re a Packet Militia of one, someone else will come after you. So document – early and often. And for you young engineers whose brains never leak a thing (and therefore, you don’t think you need to document), your time is coming. Now in my 40’s – hardly old – my brain is a bit leaky. The IPs don’t come to mind quite as quickly as they once did. That clever solution to a common problem doesn’t pop back to synaptic life even though I know the answer is in there somewhere. Documenting what I do when I do it has been a gift I’ve given myself more than once.