How to detect the SACK Panic vulnerability with Wireshark

The security team at Pentest-Tools.com has recently performed an in-depth analysis of the SACK Panic vulnerability (which was first disclosed in June 2019) to find out its exploitability against Linux machines.

Throughout this research, we’ve identified a new method to detect vulnerable servers using Wireshark, the popular network traffic analyzer.   

This article offers details about the SACK Panic vulnerability, a new detection method, and a set of mitigation measures recommended to protect against SACK Panic.

While the impact seems to be limited to denial of service attacks, this issue needs to be addressed to make users understand how these flaws can disrupt their network infrastructure and why patching is essential.

1. Key details about the SACK Panic vulnerability

Three security flaws have been identified and linked to the manipulations of TCP Selective Acknowledgement packets combined with a low Maximum Segment Size (MSS) value.

You can track the three vulnerabilities discovered by the following CVEs:

CVE-2019-11477 – This is the vulnerability that actually causes the kernel panic.

CVE-2019-11478 and CVE-2019-11479 are less impactful, generating only intense resource usage and a potential denial of service attack.

In order to better understand SACK Panic, we need to go through these definitions below:

TCP Selective Acknowledgment (SACK) is a technique used by TCP to help alleviate congestion that can arise due to the retransmission of dropped packets (segments).

The maximum segment size (MSS) is a parameter set in the TCP header of a packet that specifies the total amount of data contained in a reconstructed TCP segment.

 Generic Segmentation Offload (GSO) is a pure software offload that deals with cases in which device drivers cannot perform the offloads because of lacking support for the hardware.

In a traditional way, the data would be split up into small individual packets and go through the network stack one by one. GSO uses the network stack with one big packet that includes all the data.  If you want to dive deeper, you can read more info here.

2. How SACK Panic works

The SACK Panic option stands for Selective Acknowledgment. This is a smarter TCP mechanism created to eliminate redundant data. It was introduced back in 1996 by a Request for Comments (RFC) document from 2018.

TCP is a connection-oriented protocol that is fragmented into segments of the maximum segment size (MSS) during its transmission data. This is negotiated at the beginning of the connection in SYN packets, where the lowest MSS value is considered.

During the demo section, we will talk about the two systems as “sender” and “receiver”.

It can happen that, for various reasons, during transmission, a packet cannot be received by the other side. In this case, before SACK was introduced, the “sender” had to resend a whole window of packets for the receiver to get just the one that hadn’t made it.

For example, let’s say a “sender” has 10 packets and the “receiver” gets all the rest, except for packets 4 and 7. Then, in the next ACK system, the “receiver” can attach a SACK option stating that it received packets from 1-3; 5 and 6; 8-10. Thus, the “sender” will only need to resend 2 packets instead of 10.

To control all the transmission and retransmission queues, the Linux kernel uses a structure named sk_buff.

The struct sk_buff holds the data and can include different layers of headers.

This structure has an associated control buffer named struct tcp_skb_cb that keeps track of different aspects of the TCP packet.

In this control, the structure is also included, as well as the number of segments or the number of fragments (tcpgsosegs), which is displayed like this:

struct tcp_skb_cb {
  __u32 seq; /* Starting sequence number /
  __u32 end_seq; / SEQ + FIN + SYN + datalen /
  __u32 tcp_tw_isn;
  struct {
    U16 tcp_gso_segs;
    U16 tcp_gso_size;
  };
}

This tcpgsosegs field is an unsigned double-byte integer and limits it to 65536 (64K).

As defined here:

#define MAX_SKB_FRAGS (65536/PAGE_SIZE + 1)

the maximum number of fragments on Linux (PAGE_SIZE = 4096) in a sk_buff is exactly 17 fragments.

Each fragment of data can have up to 32KB, resulting in 17 * 32K of bytes of data maximum held in a sk_buff.

From here, the low MSS comes into play. If we set the MSS value to 48 (the minimum in a TCP Connection on Linux) and with TCP options amassing 40 bytes of it, we can create segments as small as 8 bytes of data each.

Considering the maximum data of 17 * 32KB in a sk_buff and dividing it by the mere 8 bytes, we get 69632 segments.

Looking at the limit of segments in tcpgsosegs reaching 64K, we can easily observe a potential overflow.

The mechanism behind SACK allows us to merge the packets during the retransmission of TCP and also combines multiple SKB queues resulting in a potential 17 packet SKB.

static bool tcp_shifted_skb (struct sock *sk, …, unsigned int pcount, ...) {
   tcp_skb_pcount_add(prev, pcount);
   BUG_ON(tcp_skb_pcount(skb) < pcount); <= Things go south
   tcp_skb_pcount_add(skb, -pcount);
}

In this source code, the BUGON() call can be triggered if the TCP fragment reaches 17 and the tcpgso_segs integer overflows.

3. How to detect SACK Panic with Wireshark

In order to detect if a target server is vulnerable, we will use Wireshark. We are going to refer to the two parts as the Client, your endpoint, and the target.

The client’s IP in our test is 192.168.222.1 and the target’s IP is 192.168.222.3

Before we get started, let’s do a quick setup on the client-side:

1. Set your MSS options

Add “advmss” options to the route to your target.

$ ip route change your_ip_route advmss 48

2. Make sure SACK is ON:  

$ cat /proc/sys/net/ipv4/tcp_sack

If it returns 1, then it is on.

3. Turn Wireshark and start listening.

$ wireshark OR Turn on Wireshark via GUI.

Select the interface and put a filter for the IP like this:

ip.addr == client_ip OR target_ip

4. Initiate a TCP connection from the client to the server using the following commands:

You can use nc -lvp port; nc ip_target port. Alternatively, you can try an HTTP GET on the target’s website.

On the target:
$ nc -lvp 1234

On the client:
$ nc 192.168.222.3 1234

Note: Don’t forget to change your IPs.

Going further to the actual detection:

In the image above, you can see how the options for a client should look. What’s important is that the MSS options value is 48 and the SACK-permitted option is ON.

This is how a vulnerable target responds. It shows that the SACK option is ON. In this case, the MSS is usually set by default, as stated in the above chapter, to the lowest MSS value used.

Some websites don’t respect the MSS negotiation, so you need to check if traffic is limited to 48 bytes further down the line. As you can see, the length of TCP data (‘Len = x’) is limited to 36 because of 12 bytes of TCP options.

To recap, if SACK is allowed on both sides and the MSS negotiation limits the data to 48 bytes minus the options, then the system is vulnerable.

To be sure, you can always check the Linux version by using:

$ nmap -O 192.168.222.3

This will fingerprint the Linux version and will also double-check the vulnerability.

4. How to mitigate the TCP SACK Panic vulnerability

To prevent your systems from being vulnerable to these attacks and improve their security, it is essential to apply one of the following fixes:

Fix #0: Patch your system.

Make sure you apply the latest patches available here or simply run the following command:

$ sudo apt-get dist-upgrade

Fix #1: Block connections with low MSS using filters.

These filters may break legitimate connections that rely on this setting, so practice caution when applying them.

 $ iptables -t mangle -A PREROUTING -p tcp -m conntrack --ctstate NEW -m tcpmss ! --mss 536:65535 -j DROP

Fix #2: Disable SACK processing

To do this, you have to be a superuser, because regular admin users don’t have permission to change this value.

$ echo 1 > /proc/sys/net/ipv4/tcp_sack

Please keep in mind that one of the security measures mentioned above should be enough to mitigate the TCP SACK Panic vulnerability. It is not necessary to apply all of them.

Interested in learning more?

Here are a couple of relevant articles and sources about SACK Panic that we selected for you:

• The TCP SACK panic

• TCP SACK PANIC – Kernel vulnerabilities – CVE-2019-11477, CVE-2019-11478 & CVE-2019-11479

• Technical details on these vulnerabilities in Netflix security bulletins