Supply Chain Security: Has the Next SolarWinds Happened Yet?

More than two years after the now infamous SolarWinds hack, the incident is at the forefront of conversations about cybersecurity defenses, threat detection and incident response. As proof of this, the SolarWinds incident was the subject of five separate presentations and panels at the recent RSA conference in San Francisco. Dozens of other conversations addressed supply chain security issues that—directly or indirectly—created the SolarWinds incident.

Because the SolarWinds attack raises important – if not existential – questions for both software development and DevOps teams and the companies that run the software. First, if sophisticated cybercriminals can inject malicious backdoors directly into legitimate developers’ application code, what organization can afford to blindly trust software? Second: If it is now the responsibility of software teams To spot sophisticated, methodical trade-offs in their development pipelines, what is the best and most reliable means of doing so? Finally, if it is also the job of software consumers to monitor and detect compromised binaries from otherwise reputable vendors (like SolarWinds), how should they do so in the face of ongoing human and technological constraints?

The answers to these questions are being discussed in DevOps, software development, and information security communities. What is undisputed is that the SolarWinds incident was no coincidence. In the months since that attack, attackers – both criminal and nation-state-backed – have exploited vulnerabilities in the software supply chain to target development organizations to gain access to sensitive IT environments. While we haven’t heard of another incident of the magnitude or importance of SolarWinds, it’s perfectly reasonable to assume that the next compromise of SolarWinds caliber may already be unfolding, but has yet to be discovered and/or publicly acknowledged.

Vulnerabilities in the software supply chain

SunBurst, the malicious implant in SolarWinds software, remains a topic of interest and talking point for information security professionals as well as DevOps and development organizations precisely because it has exposed glaring vulnerabilities in the security of development organizations and software supply chains. As noted in mine Analysis of the hack in December 2020 and FireEye’s analysis Because of the same compromise, SolarWinds hackers gained access to the company’s development environment months before the actual attack. During that time, they closely studied the underlying source code and mimicked the SolarWinds style and nomenclature by adding malicious code and integrating it into the SolarWinds Orion application via a compromise in the company’s build system.

Though carefully planned and executed, the SolarWinds attack was not foolproof. The attackers left clues such as adding native Windows features, using cryptography, compression, and more. The attackers obfuscated these using compression and Base64 encoding to trick simple threat hunting rules that were looking for their plaintext variants and other suspicious strings. Still, the obfuscation repeated through the implanted code was enough to send a warning signal to anyone who bothered to look closely. This left the attack open to discoveries.

The problem is this: very few software publishers have the staff, expertise, or tools to comb through vast codebases to spot a handful of sketchy obfuscated strings, or analyze large binaries for deviant behavior before they are deployed. As proof, ReversingLabs sponsored a poll by more than 300 IT experts in software-driven companies. It found that 87% were aware that software tampering like the one seen in the SolarWinds attack could lead to security breaches in organizations. About 40% of respondents indicated that the exposure to the CI/CD toolchain poses a risk to their organization. Still, just under a third of those surveyed (37%) said they had the means to detect such software manipulation attacks.

Threats to software development organizations are increasing

Attackers are aware of this vulnerability in the software development pipeline, which is why threats to development organizations are metastasizing. In the last year alone, we’ve seen a number of attacks targeting development infrastructure and software supply chains. In one April 2021, where attackers were successful pushing malicious, unverified code directly into the main branch of PHP, ultimately leading to a compromised, formal version of PHP being pushed to all PHP websites. There was one in the same month Violation at the company CodeCov, which involved unauthorized access to and modification of the company’s bash uploader script. An investigation into this incident eventually revealed that modifying a single line of code in this script by an unauthorized third party allowed the attackers to (potentially) export information stored in users’ continuous integration (CI) environments.

More recently, ReversingLabs observed proof-of-concept and “white-hat” activity testing so-called “dependency confusion” attacks, in which attackers set up malicious packages in public repositories such as GitHub, referencing the names of private , support non-public repositories, and sport with high version numbers. Misconfigurations in package management systems can result in these malicious components being imported instead of their legitimate, privately managed namesakes. As ReversingLabs found out, a number of malicious modules discovered on the npm platform were used to target German technology, manufacturing and entertainment companies. It later turned out to be the work of a white hat firm conducting red team exercises on behalf of those firms. Aside from the Red Team exercises, RversingLabs research also discovered other malicious packages in nmand similar targets Ruby gems and Python Package Index (PyPi).

And that’s not even counting issues like the Travis CI bug that (potentially) exposed Millions of credentials of developer accounts on GitHub, Amazon Web Services, and Docker Hub. Such vulnerabilities can fuel supply chain attacks and allow malicious actors to inject malicious code into otherwise legitimate open-source or proprietary code repositories. After reading such headlines and without a chance to verify code intent, how can developers be confident that their software will be updated? And how can they do this while at the same time being forced to update their dependencies frequently to reduce or eliminate critical software vulnerabilities?

Development teams need new tools

For software development companies struggling to meet delivery deadlines and customer expectations, refocusing on threats and attacks in the amorphous software supply chain is too big a step. Ditto for organizations that struggle to staff security teams and simply keep track of targeted phishing attacks or threats to publicly-facing IT assets. All too often, these companies only come to believe in software supply chain attacks after they have happened.

What software development organizations need is a way to secure the CI/CD pipelines and release packages and ensure their software builds are free from code manipulation and other supply chain risks introduced by build systems. This should include the ability to detect compromises by monitoring for malicious software changes and enforcing security controls and error conditions on software build processes to ensure you can release your code with confidence.

On the customer side, organizations need a way to detect malicious behavior and unauthorized software changes in compiled binaries before they are released into production, while also monitoring for risks such as secrets being discovered or exploitable software vulnerabilities that might be lurking in compiled code.

Comments are closed.