Congestion - Observe Ability https://leighfinch.net Observability Fri, 15 Sep 2023 05:42:38 +0000 en-AU hourly 1 https://wordpress.org/?v=6.5.3 223568926 Top 5 Wireshark Tips for Network Analysis https://leighfinch.net/2023/09/15/top-5-wireshark-tips-for-network-analysis/ https://leighfinch.net/2023/09/15/top-5-wireshark-tips-for-network-analysis/#respond Fri, 15 Sep 2023 05:40:46 +0000 https://leighfinch.net/?p=80 I’ve been using Wireshark since it was named Ethereal back in the very early 2000s, and I still use it daily for research. Wireshark recently turned 25 with creator Gerald Combs announcing it on the Wireshark blog and celebrating it at Sharkfest ‘23 Asia and US. To celebrate I’m going to offer my top 5 Wireshark tips for 2023! Tip 1 — Use Profiles […]

The post Top 5 Wireshark Tips for Network Analysis first appeared on Observe Ability.

]]>
I’ve been using Wireshark since it was named Ethereal back in the very early 2000s, and I still use it daily for research. Wireshark recently turned 25 with creator Gerald Combs announcing it on the Wireshark blog and celebrating it at Sharkfest ‘23 Asia and US.

To celebrate I’m going to offer my top 5 Wireshark tips for 2023!

Tip 1 — Use Profiles

Wireshark is an extremely flexible application allowing you to create profiles for any use case quickly changing any preference including which columns are displayed and colouring rules. For example I have a profile that is focused on TCP sequence analysis where I display TCP specific columns such as bytes in flight and window size as well as frame delta time.

You can create clones of the default profile by right clicking on the Profile text button in the bottom right hand corner of the screen and selecting Manage Profiles.

In the Configuration Profiles window you can select the copy button to the right of the minus (-) symbol.

You can switch profiles while looking at a capture by left clicking on the Profile text in the bottom right hand corner.

This allows you to crate as many customised profiles as you need to analyse your captures without needing to constantly change your settings back and forth.

Tip 2 — Use IO Graphs

IO Graphs provide the ability to graphs and statistics on virtually any metric and display filter you can imagine. Far more flexible than stream graphs, I can not only chose the type of graph, but have multiple types in the same window when comparing bytes in flight to TCP errors.

I can filter packets using display filters, change the colour, style. Where it gets interesting is that I can change the Y axis from the number of packets to bytes, bits, or perform statistics on a specific metric in the Y column.

The Y column can be anything that can be reached using a display filter from the number of bytes in flight, to the number SMB locks.

Don’t forget to change the intervals and smooth moving average fields as needed.

Tip 3  — TCP Analysis

tcp.analysis.flags highlights anything that may be of interest to do with TCP that might be of interest, from duplicate acts, retransmissions, suspected retransmissions, errors, window changes, out-of-order, and a bunch of other great pieces of information. This gives me a visual representation of the health of TCP at a glance. I could also use Expert information as shown in tip 4.

Simply type tcp.analysis.flags in the display filter bar at the top of the screen and you will be presented with packets of interest. It’s also worth pointing out the packets vs packets displayed text at the bottom of the window to get an idea of the percentage of packets that are of interest.

tcp.analysis.flags

Tip 3  — Expert Information

Expert information contains the collective knowledge of protocol experts around the world. Found in the Analyze menu (as well as in the packet disectors), I can find out information about how different protocols are behaving and isolate problems quickly.

Expert information is part of the packet disectors for each protocol and by default are grouped by the summary (eg. Connection reset (RST)). In this case I can see many (1395) out of order segments which may indicate multi-pathing of TCP.

Expanding the summary gives me the list of packets that match the summary, and selecting a packet allows me to navigate automatically to the packet in question in the main Wireshark window.

Wireshark Expert information expanded

Tip 4  — Colouring Rules

Colouring rules enable you to create custom rules for anything that can be referenced by a display filter. This is especially useful when you identify a problem and want to be able to recognise it again quickly. An example of this might be DNS requests without responses, or in a security context a packet that is an indicator of compromise (IOS).

Tip 5  — Stream Graphs

TCP Stream Graphs allow you to visualise typical questions in time-series. I use this feature in nearly every trace I open to help me understand bottlenecks in communications. This is a Stevens Graph which shows sequence numbers over time. Note the staircase pattern, which I will go into detail in another session.

TCPTrace Graph shows a little bit more detail that the Stevens graph including SACKs (Selective Acknowledgements). The green line at the bottom represents acknowledgements, while the green line above shows the available receive window.

The throughput graph allows us to see the throughput of a connection over different moving averages. This is very useful when looking microbursts and overall efficiency. The goodput option allows us to look at the throughput perceived by the application (minus headers and retransmissions) which is especially useful for understanding protocol efficiency.

Higher throughput does not equal a better user experience.

Round trip time allows us to visualise the round trip time (or time to ack) of a connection. This is useful when looking at applications perceived latency or the latency calculation that the OS uses for measuring SRTT (hopefully with a filter for delayed ack and retransmissions 🙂 ).

The window scaling option allows us to look at the receive window (not the congestion window) vs the bytes in flight. Bytes in flight reaching the receive window will result in a window full event meaning that throughput is limited by the receiver as opposed to connectivity. Bytes in flight can of course exceed the bandwidth available, as it is calculated as unacknowledged data which could include dropped (policed or shaped) traffic.

For more information on window full vs zero windows, check out my video Demystifying TCP windows.

Bonus Tips

There are far too many to narrow down to 5, but these are my favourites. Apply as column has to be my number 6 must have! I want to hear from you! Comment below with your must haves.

For a great read on how to set up Wireshark for Troubleshooting, check out Wireshark 101: Essential Skills for Network Analysis – Second Edition: Wireshark Solution Series

The post Top 5 Wireshark Tips for Network Analysis first appeared on Observe Ability.

]]>
https://leighfinch.net/2023/09/15/top-5-wireshark-tips-for-network-analysis/feed/ 0 80
Who’s Using My Bandwidth? https://leighfinch.net/2023/09/13/whos-using-my-bandwidth/ https://leighfinch.net/2023/09/13/whos-using-my-bandwidth/#comments Wed, 13 Sep 2023 04:59:11 +0000 https://leighfinch.net/?p=66 One of the questions I hate is “who’s using my bandwidth?!?” and not at all because I was the child consuming and all of the available dial-up (28.8Kbps) bandwidth downloading the latest FreeBSD or Linux distribution image. In fact this was the age of magazines with CDs that contained Mandrake, RedHat, or if I was […]

The post Who’s Using My Bandwidth? first appeared on Observe Ability.

]]>
One of the questions I hate is “who’s using my bandwidth?!?” and not at all because I was the child consuming and all of the available dial-up (28.8Kbps) bandwidth downloading the latest FreeBSD or Linux distribution image. In fact this was the age of magazines with CDs that contained Mandrake, RedHat, or if I was lucky Slackware. Debian which is my current go-to wasn’t on my radar for many years later. I even recall ordering a copy of the latest FreeBSD by mail to run on one of my 386 boxes I’d collected1. I digress…

Who’s using my bandwidth is a good question, because it implies that someone (or something) is consuming more bandwidth than they’re supposed to. If we look at the major protocols that dominate the internet being a combination of TCP, and UDP (in the form of DNS and QUIC), surely there is some fairness built into them?

The answer to the question has a short answer, and a much longer one.

Short Answer

If we look at TCP/IP and QUIC specifically they do care about fairness and have congestion control built into them to back off if they detect the presence of congestion. The challenge can be that EVERY TCP connection on every device manages its own congestion control and may mean some connections may never get to equilibrium causing some endpoints getting more than their fair share.

UDP itself has no congestion control and can blast traffic at any rate, which can flood a connection or even DoS a host if it fails throttle/coalesce CPU interrupts (e.g ethtool InterruptThrottleRate and coalescing).

Quality of Service on un-contended connections can also help with fairness. Wendell Odem and Michael J. Cavanaugh do a fantastic job of explaining this in Cisco QOS Exam Certification Guide (IP Telephony Self-Study) (Official Cert Guide) 2nd Edition. The problem with traditional QoS is that it doesn’t work on connections that have variable or contended bandwidth, which is most consumer internet connections. The reason for this is that packets need to queue in software, before they can be prioritised.

The Longer Answer

TCP has undergone radical changes over the last 40 years since RFC793 was released in September of 1981. the original RFC didn’t care about congestion, the result of which was the congestion collapse events of the mid 1980s resulting in several important changes which I outline in this video.

The major changes I’ll outline here include:

  • Congestion window
  • TCP Slow Start
  • Exponential backoffs

Congestion windows were introduced to back off in the event that congestion is detected using retransmission time outs. Slow Start is used at the beginning of each new connection (and after an idle period) which doubles the congestion window every round trip from an initial low value (eg. 10 x Maximum Segment Size) until congestion is detected. Exponential backoff timers were introduced to for a couple of reasons, one is to reduce the likelihood of global synchronisation, and also to give a problem time to resolve while reducing unnecessary traffic.

Most Internet TCP connections never get out of Slow Start before during their lifetime which means that they never get a chance to get to their fair share of bandwidth because they never discover the limits, and don’t cause congestion to trigger other connections to slow down.

To add to this challenge I’ve also looked at 17 TCP Congestion Control Algorithms (including the most popular CUBIC, BBR, and Reno), and most struggle to achieve equilibrium with bulk transfer traffic like iPerf.

For the congestion window to increase in either slow start or during congestion avoidance, ACKs need to return to the sender to tell the sender that data was received. This natural flow control means that TCP is self limiting (unlike UDP), AQM concepts like CoDel (Controlled Delay) can allow routers and TCP senders to effectively slow down a sender by injecting small amounts of delay to slow down TCP. this is the same idea as a receiver being overwhelmed and not acknowledging a packet quickly, resulting in the sender slowing down. I’m currently using this successfully at home with the CAKE (Common Applications Kept Enhanced) implementation on my Internet router to improve user (my family and I) experience, and reduce the impacts of bufferbloat. The beauty of CoDel is that we can target expected latency rather than just bandwidth (which in my case is variable).

If you got this far, thanks for reading!

  1. My favourite FreeBSD OS book The Complete FreeBSD ↩

The post Who’s Using My Bandwidth? first appeared on Observe Ability.

]]>
https://leighfinch.net/2023/09/13/whos-using-my-bandwidth/feed/ 1 66