Last updated at Fri, 26 Aug 2022 20:51:21 GMT

If you’re reading this, that means you survived the year 2021, so congratulations! For everyone in the software industry, and especially those in cybersecurity, the past 12 months probably felt like 12 rounds in the ring. Remember the Solarwinds attack and the resulting scramble to mitigate supply chain vulnerabilities? That all started just over a year ago, and things haven’t been quiet since.

Unprecedented security events have happened before, but last year, they became mainstream news. Even the White House and other US federal agencies are demanding more secure solutions from software vendors and better cybersecurity hygiene from any company that stores or processes sensitive data.

We can all feel the attention now, and because of this new level of awareness across the software industry, security vendors like Rapid7 are asked to respond to these widespread security events with speed and consistency to calm nerves and help prevent the spread of misinformation. That’s why I wanted to talk about our response to CVE-2021-44228 (a.k.a. Log4Shell), specifically with the Rapid7 InsightAppSec (IAS) platform.

What did we do?

When Log4Shell was disclosed, every major security vendor sprang into action to add coverage for the vulnerability to their products. At the time, many dynamic application security testing (DAST) platforms like IAS were struggling to add coverage due to the complexity of the attack, and we were no exception. But because of the severity of this vulnerability and the time-sensitivity of the situation, we knew we had to get our customers the appropriate DAST coverage as soon as possible, so we quickly created a new attack module that could provide that coverage.

How did we do it?

Before we could deliver a viable solution, we needed to address a substantial technical challenge – any Log4Shell exploit we developed needed to be an out-of-band (OOB) attack, meaning the information in our malicious request and the resulting response from the application wouldn’t be enough to prove the existence of the vulnerability. To better illustrate that definition, here is a traditional SQL Injection (SQLi) attack that IAS would perform on an application:

If you’re familiar with the HTTP protocol, URL formatting, URL encoding, and Structured Query Language (SQL), you can probably figure out what’s happening in this example HTTP request and response. If not, don’t worry — all you need to know is that IAS has essentially logged in as the ‘admin’ user in this application without knowing the password, using a malicious password value that exploits a SQLi on the server. Most attacks consist of one or more request/response pairs just like this example, with all communication happening over the same logical channel.

Of course, not all applications are willing to tell us so easily that we’ve successfully found and exploited a vulnerability. In these cases, we have to infer the success of our attack based on factors other than the content of the response, such as the time taken by the application to respond or whether we get a response at all.

But when it comes to performing a Log4Shell attack, we’re completely blind to the success of the attack if we only look at the application’s in-band response. This is because the Log4Shell vulnerability can only be exploited when malicious data is passed to a Log4j logging function. Since logging operations are not business-critical and shouldn’t be interrupting the business-critical functions that call them to record data, any errors thrown are suppressed by default. Not only that, but Log4j logging functions don’t return any data to the calling functions either, so any output from Log4j will not be included in the HTTP response sent to the client. That’s why we had to be a little more creative with our implementation of the Log4j attack module.

How does it work?

A picture is worth a thousand words, so let’s start with a diagram of our new attack:

As you can see, it’s not as simple as the in-band attack shown previously. This time, IAS isn't trying to log into the application, even though it’s attacking the login API. In fact, it doesn’t matter whether the login request was successful or really how the application responds to the request. What matters is that we can detect when the application passes the malicious username value to Log4j, which will process any lookups embedded in that value.

Since the username contains a valid JNDI Lookup string (starting with  ‘${jndi:’, then followed by a valid URL and a closing ‘}’), Log4j will try to retrieve some data from the server addressed by the URL in the lookup. This requires a DNS request to translate that URL into an IP address, and no matter which DNS server the application reaches out to first for the DNS lookup, that request will eventually reach a custom DNS server hosted in our Appspidered environment. Upon receiving the DNS request, our Appspidered server will parse out the unique token that was embedded in the URL as a subdomain (in this case, ‘12345’) and store it temporarily.

Meanwhile, after IAS has received an HTTP response from the application, it will periodically reach out to Appspidered to see if it has received any DNS requests for the Appspidered subdomain containing the unique token it provided to the application in the attack. If Appspidered confirms the DNS request was made, then we know the application must have parsed the attack payload and made the request, meaning it is vulnerable to Log4Shell.

And that’s it! Of course, the devil is in the details, so let’s try and answer any questions that you may have at this point.

How do we know which application endpoint was exploited?

In order to differentiate each attempted Log4Shell attack, IAS will generate a unique token for each request made to the application. The payload in each request will always have the format ‘${jndi:ldap://unique_token.oob.appspidered.rapid7.${lower:COM}}’, and each token will be a 160-bit random value represented by 40 hexadecimal characters, so the chances of two attacks using the same token are practically zero.

What’s with the ${lower:COM} part of the URL?

Log4j supports many different lookup types, and this is a lower lookup that simply rewrites ‘COM’ in its lowercase version ‘com’. This is a trick we use to disguise the URL and make sure no other components sitting between IAS and the vulnerable Log4j library notice the full URL and make their own DNS request to resolve it, as this would cause us to report a false positive Log4Shell vulnerability in the application.

Why do we detect DNS requests instead of the LDAP requests themselves?

By detecting DNS lookups instead of the preceding requests, we don’t need to worry about the requests protocol or whether it was blocked. Also, outbound DNS traffic is less likely to be blocked than other protocols in customer environments, and because of the recursive nature of DNS, the Appspidered domain won’t need to be whitelisted for vulnerable applications to provide confirmation of the exploit. Most other DAST solutions are using the same DNS-based technique for OOB attacks, presumably for the same reasons.

Does a DNS request really prove the Log4Shell attack would have worked?

It at least proves that the application’s Log4j library was intending on performing a JNDI message lookup and is therefore vulnerable, which is not supported in fixed and appropriately patched versions. It’s possible that later stages of the Log4Shell attack could be detected and/or prevented by security tools sitting between IAS and Log4j, so we can’t guarantee the vulnerability is exploitable just from the DNS request. However, it proves that there’s a Log4j dependency in the application that is both vulnerable and reachable from an attacker’s standpoint, so it should be remediated regardless.

So, is this better than a vulnerability management (VM) solution at finding Log4Shell vulnerabilities in my environment?

It depends on what you mean by “better.” Since VM solutions essentially scan your environment for vulnerable versions of the Log4 library, they will likely find more of those libraries than IAS would, especially if those libraries can only be attacked in very specific or complex scenarios. However, when it comes to scanner results, more isn’t always better, especially if you already have an overwhelming number of vulnerabilities in your environment and evaluating them for remediation priority is an expensive process.

Since IAS will only report instances of the Log4Shell vulnerability that are discoverable and exploitable from an attacker’s point of view, it is more likely to miss some instances compared to a VM solution, but you can be confident that vulnerabilities that do get reported are highly exploitable and should be high on the priority list. In a perfect world, you would use both a DAST solution like IAS alongside a VM solution to get the best of both worlds, leading to a breadth of visibility of your attack surface and a depth of understanding of your risk.

What’s next?

When Log4Shell hit, our priority was to deliver a working attack module as soon as possible, and despite the success, we feel there’s always room for improvement. Here are some of the enhancements we are considering:

  • Including this module in the default attack policy for regular use
  • Continue to update the Out of Band Attack Template to increase finding confidence and decrease scan time
  • Support for on-prem environments

Innovation happens fast at Rapid7, so stay tuned for updates on the Log4Shell attack module! And for more information on the Log4Shell vulnerability itself and why it was worth the attention it was given, I highly recommend you check out our Everyperson’s Guide to Log4Shell for a high-level understanding and our AttackerKB for a more technical analysis.

Additional reading

Get critical insights about defending against Log4Shell

Check out our resource center