I write on my personal time. Feel free to buy me a coffee or buy a copy of TCP/IP Illustrated Volume 1 to learn more about the protocols that run the internet. Check out Wireshark Network Analysis for an awesome hands-on guide to wireshark.

Network teams often use ICMP as a mechanism to determine the latency (propagation delay etc) and reachability between two endpoints using the trusty Ping utility. Ping appeared in late 1983 created Mike Muuss while working US Ballistics Research Laboratory. Additionally, what was interesting about 1983 is that it was the year the that the US military converged on IP (and TCP) mandating for any system connected to the ARPANET making Ping one of the oldest IP applications still in use today.

The naming for PING (Packet InterNet Groper) is a backronym for the sonar process used by submarines and other water-craft (as well as in nature). Which makes sense when you are trying to measure latency between nodes.

Ping uses ICMP (Internet Control Management Protocol echo(8)/echo-reply(7)) to communicate between nodes and is even mandated in the historic RFC1122 Requirements for Internet Hosts — Communication Layers(released in 1989) for internet connected hosts. This RFC is well worth a read to understand what was happening with IP and TCP in the aftermath of the congestion collapse events of the mid 1980s.

The problem with using Ping and ICMP as a measure of latency is that it is frequently blocked or placed in scavenger queues which distorts the latency detected (adding to it or making it appear jittery) anddoes not reflect the actual latency experienced by applications. The lack ICMP prioritisation makes sense, we want the actual user traffic coming through and processed at endpoints with a higher priority than our monitoring traffic. Secondly, Ping is usually run in intervals (eg. every 5 minutes) which means the that we wont be able to spot events between polling intervals.

This may have been good enough when we used IP networks for non-realtime applications (email and web browsing etc) where changes in latency and drops are not as important, but in the 2000s we started using IP SLA to inject synthetic traffic between to devices that support IP SLA and report on metrics like jitter and latency for the class of service or QoS markings desired. This was a good step further as now we understand how real traffic would perform while the IP SLA runs. This is (usually) run in intervals which means that still have gaps in our visibility. The good reason for using IP SLA (and other synthetics) is that traffic is being generated even when there is none being generated by users. A lot of vendors take this approach with their observability stacks, but it still leaves a gap between intervals and doesn’t necessarily reflect a users experience.

We can also monitor latency passively using captured TCP packets between nodes. NPM platforms like Alluvio AppResponse do this at a large scale, but we can also do this using Wireshark or TCPDump for ad-hoc reporting. The best bit is that we can now see the latency between any nodes that we can collect traffic between which has two big bennefits:

1. We have every connection.
2. It is all passive.

Using Wireshark we will look at how this is possible. I’ve talked about how TCP operates below the application layer and that an application has only a limited ability to influence socket behaviour. The OS kernel handles traffic acknowledgement, which has a very high priority in the Operating System scheduler. We can essentially ignore the TCP stack delay as negligible (unless it is behaving erratically which is a sign that the endpoint is over-subscribed).

The two TCP behaviours we will use to understand the latency of the connection are the 3-way handshake, and the TCP time to ACK two full segments.

Method 1 – The 3-way handshake

The 3-way handshake (also called connection setup time) is the process used to establish a reliable connection between two nodes. It involves the TCP client sending a specially marked segment called a SYN, The server responding with another specially marked segment called a SYN-ACK, followed by the client sending an ACK (with or without data). The delta between the SYN and the ACK collected anywhere between the nodes will give us the latency between the two nodes.

In this case we have a latency of 105ms between the SYN and the ACK. I’ve set the propagation delay of the backbone of this network to 100ms, which after we add small overhead on socket creation the server, we are very much right on the latency. I deliberately chose a capture that was was not on the client, or the server to show that this can be applied anywhere on the path.

We can also see this value of 105ms in each subsequent packet stored in the tcp.analysis.initial_rtt variable.

Method 2 — Time to ACK

We know that from RFC1122 we should see an ACK (for at least) every 2 full sized segments without delay, or after a packed marked with PSH is set. This behaviour is not impacted by the applications ability to process the data, and is solely the responsibility of the TCP stack in play. This method is best used close to the client (otherwise same additional math is required).

We can even graph this in Wireshark using the Round Trip Time option in the TCP Stream Graphs menu. You will also note some spikes in acknowledgements at and over 200ms, this is a topic willbe discussed in another article.

I like to add it as a coloumn in the packet list as below when troubleshooting TCP performance.

Using TCP to monitor latency has significant advantages over synthetics

If you made it this far, thanks for reading. Feel free to buy me a coffee or buy a copy of TCP/IP Illustrated Volume 1 to learn more about the protocols that run the internet. Check out Wireshark Network Analysis for an awesome hands-on guide to wireshark.

The post Performance Diagnostics Part 3 — Latency beyond Ping first appeared on Observe Ability.

To celebrate I’m going to offer my top 5 Wireshark tips for 2023!

Tip 1 — Use Profiles

Wireshark is an extremely flexible application allowing you to create profiles for any use case quickly changing any preference including which columns are displayed and colouring rules. For example I have a profile that is focused on TCP sequence analysis where I display TCP specific columns such as bytes in flight and window size as well as frame delta time.

You can create clones of the default profile by right clicking on the Profile text button in the bottom right hand corner of the screen and selecting Manage Profiles.

In the Configuration Profiles window you can select the copy button to the right of the minus (-) symbol.

You can switch profiles while looking at a capture by left clicking on the Profile text in the bottom right hand corner.

This allows you to crate as many customised profiles as you need to analyse your captures without needing to constantly change your settings back and forth.

Tip 2 — Use IO Graphs

IO Graphs provide the ability to graphs and statistics on virtually any metric and display filter you can imagine. Far more flexible than stream graphs, I can not only chose the type of graph, but have multiple types in the same window when comparing bytes in flight to TCP errors.

I can filter packets using display filters, change the colour, style. Where it gets interesting is that I can change the Y axis from the number of packets to bytes, bits, or perform statistics on a specific metric in the Y column.

The Y column can be anything that can be reached using a display filter from the number of bytes in flight, to the number SMB locks.

Don’t forget to change the intervals and smooth moving average fields as needed.

Tip 3  — TCP Analysis

tcp.analysis.flags highlights anything that may be of interest to do with TCP that might be of interest, from duplicate acts, retransmissions, suspected retransmissions, errors, window changes, out-of-order, and a bunch of other great pieces of information. This gives me a visual representation of the health of TCP at a glance. I could also use Expert information as shown in tip 4.

Simply type tcp.analysis.flags in the display filter bar at the top of the screen and you will be presented with packets of interest. It’s also worth pointing out the packets vs packets displayed text at the bottom of the window to get an idea of the percentage of packets that are of interest.

Tip 3  — Expert Information

Expert information contains the collective knowledge of protocol experts around the world. Found in the Analyze menu (as well as in the packet disectors), I can find out information about how different protocols are behaving and isolate problems quickly.

Expert information is part of the packet disectors for each protocol and by default are grouped by the summary (eg. Connection reset (RST)). In this case I can see many (1395) out of order segments which may indicate multi-pathing of TCP.

Expanding the summary gives me the list of packets that match the summary, and selecting a packet allows me to navigate automatically to the packet in question in the main Wireshark window.

Tip 4  — Colouring Rules

Colouring rules enable you to create custom rules for anything that can be referenced by a display filter. This is especially useful when you identify a problem and want to be able to recognise it again quickly. An example of this might be DNS requests without responses, or in a security context a packet that is an indicator of compromise (IOS).

Tip 5  — Stream Graphs

TCP Stream Graphs allow you to visualise typical questions in time-series. I use this feature in nearly every trace I open to help me understand bottlenecks in communications. This is a Stevens Graph which shows sequence numbers over time. Note the staircase pattern, which I will go into detail in another session.

TCPTrace Graph shows a little bit more detail that the Stevens graph including SACKs (Selective Acknowledgements). The green line at the bottom represents acknowledgements, while the green line above shows the available receive window.

The throughput graph allows us to see the throughput of a connection over different moving averages. This is very useful when looking microbursts and overall efficiency. The goodput option allows us to look at the throughput perceived by the application (minus headers and retransmissions) which is especially useful for understanding protocol efficiency.

Higher throughput does not equal a better user experience.

Round trip time allows us to visualise the round trip time (or time to ack) of a connection. This is useful when looking at applications perceived latency or the latency calculation that the OS uses for measuring SRTT (hopefully with a filter for delayed ack and retransmissions ).

The window scaling option allows us to look at the receive window (not the congestion window) vs the bytes in flight. Bytes in flight reaching the receive window will result in a window full event meaning that throughput is limited by the receiver as opposed to connectivity. Bytes in flight can of course exceed the bandwidth available, as it is calculated as unacknowledged data which could include dropped (policed or shaped) traffic.

For more information on window full vs zero windows, check out my video Demystifying TCP windows.

Bonus Tips

There are far too many to narrow down to 5, but these are my favourites. Apply as column has to be my number 6 must have! I want to hear from you! Comment below with your must haves.

For a great read on how to set up Wireshark for Troubleshooting, check out Wireshark 101: Essential Skills for Network Analysis – Second Edition: Wireshark Solution Series

The post Top 5 Wireshark Tips for Network Analysis first appeared on Observe Ability.

Who’s using my bandwidth is a good question, because it implies that someone (or something) is consuming more bandwidth than they’re supposed to. If we look at the major protocols that dominate the internet being a combination of TCP, and UDP (in the form of DNS and QUIC), surely there is some fairness built into them?

The answer to the question has a short answer, and a much longer one.

Short Answer

If we look at TCP/IP and QUIC specifically they do care about fairness and have congestion control built into them to back off if they detect the presence of congestion. The challenge can be that EVERY TCP connection on every device manages its own congestion control and may mean some connections may never get to equilibrium causing some endpoints getting more than their fair share.

UDP itself has no congestion control and can blast traffic at any rate, which can flood a connection or even DoS a host if it fails throttle/coalesce CPU interrupts (e.g ethtool InterruptThrottleRate and coalescing).

Quality of Service on un-contended connections can also help with fairness. Wendell Odem and Michael J. Cavanaugh do a fantastic job of explaining this in Cisco QOS Exam Certification Guide (IP Telephony Self-Study) (Official Cert Guide) 2nd Edition. The problem with traditional QoS is that it doesn’t work on connections that have variable or contended bandwidth, which is most consumer internet connections. The reason for this is that packets need to queue in software, before they can be prioritised.

The Longer Answer

TCP has undergone radical changes over the last 40 years since RFC793 was released in September of 1981. the original RFC didn’t care about congestion, the result of which was the congestion collapse events of the mid 1980s resulting in several important changes which I outline in this video.

The major changes I’ll outline here include:

• Congestion window
• TCP Slow Start
• Exponential backoffs

Congestion windows were introduced to back off in the event that congestion is detected using retransmission time outs. Slow Start is used at the beginning of each new connection (and after an idle period) which doubles the congestion window every round trip from an initial low value (eg. 10 x Maximum Segment Size) until congestion is detected. Exponential backoff timers were introduced to for a couple of reasons, one is to reduce the likelihood of global synchronisation, and also to give a problem time to resolve while reducing unnecessary traffic.

Most Internet TCP connections never get out of Slow Start before during their lifetime which means that they never get a chance to get to their fair share of bandwidth because they never discover the limits, and don’t cause congestion to trigger other connections to slow down.

To add to this challenge I’ve also looked at 17 TCP Congestion Control Algorithms (including the most popular CUBIC, BBR, and Reno), and most struggle to achieve equilibrium with bulk transfer traffic like iPerf.

For the congestion window to increase in either slow start or during congestion avoidance, ACKs need to return to the sender to tell the sender that data was received. This natural flow control means that TCP is self limiting (unlike UDP), AQM concepts like CoDel (Controlled Delay) can allow routers and TCP senders to effectively slow down a sender by injecting small amounts of delay to slow down TCP. this is the same idea as a receiver being overwhelmed and not acknowledging a packet quickly, resulting in the sender slowing down. I’m currently using this successfully at home with the CAKE (Common Applications Kept Enhanced) implementation on my Internet router to improve user (my family and I) experience, and reduce the impacts of bufferbloat. The beauty of CoDel is that we can target expected latency rather than just bandwidth (which in my case is variable).

If you got this far, thanks for reading!

1. My favourite FreeBSD OS book The Complete FreeBSD ︎

The post Who’s Using My Bandwidth? first appeared on Observe Ability.