This backdoor almost infected Linux everywhere: The XZ Utils close call

It all started when Andres Freund, a Microsoft principal software engineer, became curious about why the SSH remote security code in the Debian Linux beta was running slowly. Freund did some digging and discovered the problem: A chief programmer and maintainer of the xz data compression library, Jia Tan, had put a backdoor in the code. Its purpose? To enable attackers to take over Linux systems.

Also: Linux might be your best bet for heightening your desktop computer security

Recently, it has become all too common for malicious hackers to insert bad code into software. Some open-source code repositories, such as the popular JavaScript package manager, Node Package Manager (npm), and the equally popular Python software repository Python Package Index (PyPI), have become infamous for hosting crypto mining and hacking malware.

There are also open-source malware programs, such as SapphireStealer, that seek to steal user IDs, passwords, and other secrets. While there has certainly been a lot of bad code written in Linux and its closely related utilities, no one has ever successfully hidden malware within it –until now.

Before you get too excited, note this: The corrupt xz code did not appear in any production Linux distros. If you were working with Fedora, Debian, openSUSE, Ubuntu, or other bleeding-edge beta distributions, you had something to worry about. Otherwise, you should be clear.

But, make no mistake: Linux dodged a bullet. Had this reached the Linux systems we all use every day — whether or not you’re ever aware of it — we’d be in a world of hurt. 

Ironically, while people are using the xz mess as an excuse to whip open source, the truth is that the attack failed because of open source. As Mark Atwood, Amazon’s open source program office principal engineer, noted, “The attack failed because it was open source. The way this attack works for non-open source is the attacker spends two years getting an agent hired by a contract software development vendor, they sneak it in, [and] nobody finds out.”

Also: Thinking about switching to Linux? 10 things you need to know

How can he say that? Because it’s the truth. For example, we still don’t know exactly how Microsoft allowed a Chinese hacking group to break into Microsoft Online Exchange last year. Thanks to Freund, we know a great deal about how the xz hack was accomplished. As Dimitri Stiliadis, Endor Labs CTO and co-founder, pointed out, “We were lucky that the attack happened against open-source software that anyone can look at and understand. If the same attack was against a closed source component, how would we even know?”

Amen. 

What we don’t know yet is who was behind the attack — or why. There’s much speculation that it was another Chinese hacking group; but at the end of the day, we’re left with educated guesses. 

For example, instead of international politics being behind the malware, it could have been an especially elaborate attempt to plant crypto miners into high-powered Linux systems. With current Bitcoin values hovering around $65,000 a coin, greed is a plausible motive.  

We do know that whoever was behind the name Jia Tan took a lot of time and trouble planting the malware. Tan began his dark work in 2021. He or she, with the aid of some sock puppets, gradually took control of the xz project. Tan and his colleagues then started pushing for the new backdoor-infected program to be fast-tracked into Linux distros.

It’s at this point that Freund’s digging into the code uncovered the plot. Today, Lasse Collin, the original XZ maintainer, has taken back control of the project and is cleaning the code.  

Also: The best Linux distros for beginners: Expert tested

There’s also been speculation that Tan and company had already placed malware in earlier xz versions. There doesn’t appear to be anything to this. 

Others are worried that xz was just the tip of the iceberg and that there are many other open-source malware programs hiding in Linux. But, as Eric S. Raymond, open-source co-founder, observed, “It sounds prudent and cautious to suppose that for any discovered exploit, there must be a large number of undiscovered ones. But we don’t actually know that, and even if it were true, it wouldn’t lead to actionable advice.”

So, what can we do about it? Lots!

Before this trap-door-equipped malware was discovered, the Open Source Security Foundation (OpenSSF) had proposed that we adopt policies for secure and responsible open-source software use. 

In the aftermath, Dan Lorenc, co-founder and CEO of open-source software supply chain company Chainguard, proposed that we reflect on the gaps this attack has surfaced and build up more in-depth defense across the entire open-source supply chain:  “Persistent threats aren’t going away, and we can’t magically stop them, but we can continue to raise the bar and make them harder.”

Also: 5 tips for securing SSH on your Linux server or desktop

Lorenc’s right. As he also stated, “We got incredibly lucky.”  

Open source, by its very nature, is potentially more secure than proprietary methods. But, it’s only more secure if we take a long, hard look at the code we use and make sure it really is safe. The idea that the code is safe just because it’s open is magical thinking at its worst. Wishing won’t make open-source or Linux secure; only hard work will do that.