From time to time, the Packet Pushers inbox receives a message related to career development. This is such a set of questions along with my answers, modified here to improve clarity.
Q: I’d love you get your opinions on the relativity of Wireshark technical skills in today’s enterprise environments.
A: When you need a packet analysis tool, you need it. But you likely won’t use packet analysis everyday. Wireshark’s relevance to you depends a lot on how often you find yourself resorting to packet analysis to resolve a problem.
In my experience, the root causes of most issues are almost always higher-level problems — not packet-level problems. Therefore, packet analysis is my tool of last resort.
When I have resorted to packet analysis, Wireshark has helped me resolve issues by…
- Proving certain packets were or were not present, and on what network segments they were appearing.
- Decoding packets to expose odd behavior — for example, how MS Lync performs SIP signaling.
Those are just two examples from my experience. And for what it’s worth, any decent packet capture tool can be helpful for these sorts of tasks.
- At a data center site I supported for almost 4 years, we had a large WildPackets (now Savvius) install. I used it most often to troubleshoot complex issues with customers directly connected to our network over a WAN.
- Tcpdump at a Linux CLI has often been sufficient by itself. Even so, I’d often import a tcpdump capture into Wireshark for further analysis if the hex dump on the screen wasn’t helping or if the data was moving too fast.
To summarize, if the buck stops with you when it comes to troubleshooting strange and bizarre application behavior, you’ll want to be able to use a packet capture & analysis tool effectively. Wireshark is ubiquitous; most network engineers use it. Wireshark has an active user and development community. Plus, there is a commercial variant through Riverbed if you care to go that route. Therefore, I view Wireshark as a safe packet analysis tool to spend time learning intimately.
Q: I am at a crossroads of having my employer pay for Wireshark training, but they require a one year commitment of employment in exchange for the training.
A: Personally, I don’t care what the training is that is offered — I‘m not willing to indenture myself to get the training. If my employer gives me the training for “free” as long as I stay a year, my attitude is to be willing to pay for the training if I choose to jump ship. And if I’m not willing, I won’t make the commitment to the employer.
Employers that lock employees into contracts for what are, overall, nominal expenses to them don’t have the employee’s best interests at heart. You’re just a piece of meat they are trying to keep captive.
Now, hiring and retaining staff is hard — good staff even harder. I get that. I have been a manager several times in my career and am a small business owner with employees this very moment. However, a training class that ultimately benefits the employer should be viewed as a cost of doing business. By indenturing you for a year, they’ve twisted the class into leverage to keep you around. That arrangement is mostly downside for you, and mostly upside for them.
The way I see it, such servitude is a trap unless you’re willing to buy yourself out.
Q: What is your experience in the enterprise environment with Wireshark troubleshooting? Is it even relevant anymore with all of the enterprise applications and taps deployed in today’s large network environments?
I have not taken a Wireshark class. But if my past experience from almost 20 years ago taking an Etherpeek class is still relevant, you will learn far more in a packet-level troubleshooting class than simply how to use Wireshark. You’ll get down into the bowels of protocols, learn to read headers & flags, learn how to predict things like TCP acknowledgement & sequence numbers, etc. That’s fundamental knowledge that gets right down into the nerdy depths of protocol specifics that you really can’t get in too many other courses. If that sort of thing appeals to you, the class will be a great experience.
Of course, I presume you’ll also learn Wireshark as a tool and how to make the most of it. There’s lots of power there, many advanced features, a filtering language, the ability to write plugins, and so on.
Stepping back from Wireshark specifically, the question is really about packet-level network analysis. Is that relevant in a modern data center environment, considering other analysis tools that might be in play? My opinion is that packet analysis is now and always will be relevant, no matter how large an environment gets, and no matter what other network analysis tools might be deployed.
In my mind, packet analysis will forever be useful because devices in a network can change a packet in a variety of ways.
- Packets can be filtered (dropped). If you don’t know where a packet is being lost in a long chain of devices, a packet analyzer can help. Filtering is common with IPS appliances, firewalls, and routers.
- Packet payloads can be changed. If that payload is being changed in some way, understanding what exactly is being changed can help a troubleshooting effort. Payload munging is common with load balancers.
- Packets can be simply forwarded…but more slowly than expected. Packet analyzers can help identify the source of a delay by making it easy to quantify inter-packet gaps. Capture before a given device, and the flow seems normal. Capture after that same device, and the conversation gets ugly.
- Packets get encapsulated. Understanding which device encapsulated a packet in what sort of wrapper and where that encapsulated packet is headed can sometimes shed light on a complex problem, say where a packet is getting shoved into a GRE or VXLAN tunnel you weren’t expecting.
- Packets can be translated. The packet analyzer can help you determine which NAT deity to sacrifice a chicken to.
- Packets can be tagged. MPLS and 802.1q for starters.
- Packet headers can be manipulated. You won’t believe some of the things that show up in the TCP options field. Or the DSCP values being set (or not set) in the ToS byte. Etc.
And so on. Other sorts of analysis tools tend to function at a higher level, and aren’t good at correlating complex data flows and detecting problems. Admittedly, packet analysis tools provide a close-up, intimate view of data, and it can be hard to see the forest for the trees. But with training and experience, humans are often the best data correlators available. Therefore, my feeling remains that packet analysis might be the “last resort” tool in your toolbox, but as I said early in this post…when you need it, you need it.
By the way, don’t let me limit Wireshark to being a troubleshooting tool. Troubleshooting might be the primary purpose of Wireshark for most engineers, but it’s also a learning tool.
- Execute a random traffic capture with Wireshark, and challenge yourself to explain every flow you observe. It’s a bit like looking into the Matrix, and provides useful insights. (Hint: if you’ve never done this before, you’ll quickly learn what unknown unicast flooding is as well as just how much IPv6 you’ve deployed, knowingly or not.)
- Wireshark is also useful for reconstructing network conversations, which, by the way, may not be ethical, in accordance with corporate policy, or even legal depending on many criteria, but can be done.
- Wireshark is also an interesting discovery tool. What hosts are really out there on the wire, and what is a baseline of normal conversation for them? I admit to wiling away hours down the rabbit hole simply working with various conversations in packet traces, trying to comprehend what I was looking at.
Q: Have you covered Wireshark on Packet Pushers at all?
Packet Pushers has produced some content about Wireshark over the years. Click through the link below. A show we recorded with Gerald Combs will turn up, as well as some community blogger articles. None of the Wireshark content you’ll find is terribly recent, but it’s still content that you might find helpful.