You may be a network junkie or a Cisco pro who has spent tons of time configuring a wide range of network hardware to create complex interconnected networks. And you may have spent an equally extensive time troubleshooting a myriad of problems in your organisation’s network infrastructure. But there is a strong possibility that your network troubleshooting is incomplete.
Network engineers are tuned to address issues related to configuration congestion, and outages. Talk about a performance issue (the grey between the black and white) with them and they start to raise eyebrows. All that matters to them is the state of their network: hardware configurations, routing tables, node status, etc. Anything beyond these fundamentals does not seem to be in their checklist. And that is why they go bonkers whenever an IT user complains of slowness and blames the network.
For the network administrators, the network state is stable and everything seems to be fine. So they point towards the system administrators for a possible issue in the database, application or the server hardware. On the other hand, the system administrators claim everything to be fine on their end too, and use the fact that other IT users are able to use the services as a proof that the issue must be in the network. The usual blame game thus starts.
Here is where both of them are wrong: they need to come out of their functional silos. Data communication is not composed of one layer only. The OSI communication stack is comprised of 7 layers. And this is why it is important to think of the data communications happening over your network in the perspective of all 7 layers. This means you need to have visibility beyond your network layer (Layer-3), into the Transport layer (Layer-4), and up into the Application layer (Layer-7) as well.
Network engineers need to step up their mindset from their current network-usage level to network-performance level. They need to think beyond the fact of what traffic is flowing on their network to how that traffic is flowing and what is being experienced by that traffic.
Traditional NMS platforms are fine when it comes to monitoring the network state, but they fall short when you need to evaluate the network performance with respect to applications. Rather than relying on just the SNMP polled node statuses, network engineers should develop ‘packet analysis’ skills which are highly valuable at time of hard-to-diagnose performance issues.
Packet analysis requires the ability to inspect the raw protocol data inside the packets that are part of the data traffic flowing over your network. You would need two things to do this.
A Packet Analyzer
First, you would need a packet analyser (also known as packet sniffer), which is a tool (software or hardware) that can log, parse, and analyse traffic passing over a network. As data flows over the network, the packet analyser receives the captured data packets and decodes the packets’ raw data revealing the values of various fields in the packet (e.g. TCP header, session details, etc.).
You can then analyse these values according to the appropriate RFC specifications to deduce whether the packet underwent any abnormal behaviour during its transportation between the network points.
One of the most popular, and free, packet analysis tools available is WireShark, an open-source packet analyzer used for network troubleshooting and analysis. You may want to consider investing in a deeper monitoring solution, e.g. an NAPM platform.
An NAPM (Network & Application Performance Monitoring) platform goes beyond the analysis of raw packet data and performs advanced analysis based on algorithms and shows performance trends in graphs. It also stores the analysed data for a longer period of time, e.g. up to 1 year. However, NAPM platforms are expensive and are more suited for large enterprises.
A Packet Capture Tool
Second, you would need a packet capture mechanism for intercepting and capturing packets from your live traffic. There are two ways to do that: using port mirroring (SPAN) on your switch, or using a network TAP. I do not recommend going with the first approach – and there is a long list of reasons.
We cover all these reasons in detail in the article Comparing Network Monitoring Tools (TAP vs. SPAN). For now, in the context of this article, note that using SPAN limits your mobility to roam around various network locations during troubleshooting because you cannot be sure whether there is a vacant port available (of required capacity) on the specific switch in question.
Plus, you would never be able to see each and every packet from your mirrored traffic because SPAN ports cannot copy all the traffic streams from all the switch ports without dropping some of the packets (up to 50% in worst cases). This fails the original purpose of this activity – that is, to be able to capture & analyse all the packets for troubleshooting.This is why having a network TAP beats the SPAN on any given day, especially on your troubled day.
If you are one of those network pros who have spent most of their time under the hood of a router or a switch, chances are that you may not have had an encounter with a network TAP yet. And it is common. For example, I was talking to a friend, a seasoned network engineer working in Belgium, the other day, discussing various packet analyser tools, when we arrived to this topic. While he knew what a network TAP is, he was not fully aware of the various types and their usage according to various scenarios. (In case you are not completely aware of what TAPs are out there, you can check out Exploring the Different Types of Network TAPs on precisely this topic.)
A network TAP guarantees full capture of 100% of packets and feeds them to the packet analyser tool. Amongst the various types of TAPs available, portable TAPs are fast gaining popularity due to the flexibility to carry them on the field and deploy them instantly at any location.
They easily hook up to your laptop, and with a tool like Wireshark installed, your laptop turns into a mobile troubleshooting kit ready to dig into any network problem, especially those nagging performance issues. Using portable network TAPs is also suitable for those IT Managers who are not yet ready to deploy full-scale TAPs permanently in their network and just need an on-demand kit to be used as/when/where needed.
Talking about portable TAPs, most manufacturers have their own variety. However, not all of them are as good as they may sound. And it is important to have a tool that does not compromise performance with portability. That is why we recently published an article about the evolution of portable packet capture solutions.