Widespread Exploitation of Critical Remote Code Execution in Apache Log4j

Last updated at Fri, 01 Dec 2023 19:25:27 GMT

Need clarity on detecting and mitigating the Log4j vulnerability? Visit our Log4Shell Resource Center.

Rapid7 is continuously monitoring our environment for Log4Shell vulnerability instances and exploit attempts. At this time, we have not detected any successful exploit attempts in our systems or solutions. For further information and updates about our internal response to Log4Shell, please see our post here.

Information and exploitation of this vulnerability are evolving quickly. We will update this blog with further information as it becomes available.

Overview

On December 6, 2021, Apache released version 2.15.0 of their Log4j framework, which included a fix for CVE-2021-44228, a critical (CVSSv3 10) remote code execution (RCE) vulnerability affecting Apache Log4j 2.14.1 and earlier versions. The vulnerability resides in the way specially crafted log messages were handled by the Log4j processor. Untrusted strings (e.g. those coming from input text fields, such as web application search boxes) containing content like ${jndi:ldap://example.com/a} would trigger a remote class load, message lookup, and execution of the associated content if message lookup substitution was enabled. On December 13, 2021, Apache released Log4j 2.16.0, which no longer enables lookups within message text by default

Successful exploitation of CVE-2021-44228 can allow a remote, unauthenticated attacker to take full control of a vulnerable target system. CVE-2021-44228 is being broadly and opportunistically exploited in the wild as of December 10, 2021. Multiple sources have noted both scanning and exploit attempts against this vulnerability. Rapid7 researchers have developed and tested a proof-of-concept exploit that works against the latest Struts2 Showcase (2.5.27) running on Tomcat. We expect attacks to continue and increase: Defenders should invoke emergency mitigation processes as quickly as possible. CISA has also published an alert advising immediate mitigation of CVE-2021-44228.

A huge swath of products, frameworks, and cloud services implement Log4j, which is a popular Java logging library. Organizations should be prepared for a continual stream of downstream advisories from third-party software producers who include Log4j among their dependencies.

Affected versions

According to Apache’s advisory, all Apache Log4j (version 2.x) versions up to 2.14.1 are vulnerable if message lookup substitution was enabled.

Mitigation and detection guidance

Security teams and network administrators should update to Log4j 2.17.0 immediately, invoking emergency patching and/or incident response procedures to identify affected systems, products, and components and remediate this vulnerability with the highest level of urgency. They should also monitor web application logs for evidence of attempts to execute methods from remote codebases (i.e. looking for jndi:ldap strings) and local system events on web application servers executing curl and other, known remote resource collection command line programs. Furthermore, we recommend paying close attention to security advisories mentioning Log4j and prioritizing updates for those solutions.

According to Apache’s advisory for CVE-2021-44228, the behavior that allows for exploitation of the flaw has been disabled by default starting in version 2.15.0. Apache later updated their advisory to note that the fix for CVE-2021-44228 was incomplete in certain non-default configurations. CVE-2021-45046 has been issued to track the incomplete fix, and both vulnerabilities have been mitigated in Log4j 2.16.0. An additional Denial of Service (DoS) vulnerability, CVE-2021-45105, was later fixed in version 2.17.0 of Log4j.

• In releases >=2.10, this behavior can be mitigated by setting either the system property log4j2.formatMsgNoLookups or the environment variable LOG4J_FORMAT_MSG_NO_LOOKUPS to true.
• For releases >=2.7 and <=2.14.1, all PatternLayout patterns can be modified to specify the message converter as %m{nolookups} instead of just %m.
• For releases from 2.0-beta9 to 2.10.0, the mitigation is to remove the JndiLookup class from the classpath: zip -q -d log4j-core-*.jar org/apache/logging/log4j/core/lookup/JndiLookup.class.

Java 8u121 (see https://www.oracle.com/java/technologies/javase/8u121-relnotes.html) protects against RCE by defaulting com.sun.jndi.rmi.object.trustURLCodebase and com.sun.jndi.cosnaming.object.trustURLCodebase to false.

According to a translated technical blog post, JDK versions greater than 6u211, 7u201, 8u191, and 11.0.1 are not affected by the LDAP attack vector. com.sun.jndi.ldap.object.trustURLCodebase is set to false, meaning JNDI cannot load a remote codebase using LDAP. In Log4j releases >=2.10, this behavior can be mitigated by setting system property log4j2.formatMsgNoLookups to true or by removing the JndiLookup class from the classpath (e.g. zip -q -d log4j-core-*.jar org/apache/logging/log4j/core/lookup/JndiLookup.class). Java 8u121 protects against RCE by defaulting com.sun.jndi.rmi.object.trustURLCodebase and com.sun.jndi.cosnaming.object.trustURLCodebase to false. Rapid7 researchers are working to validate that upgrading to higher JDK/JRE versions does fully mitigate attacks.

Rapid7 Labs, Managed Detection and Response (MDR), and tCell teams recommend filtering inbound requests that contain the string “${jndi:” in any inbound request and monitoring all application and web server logs for similar strings.

EmergentThreat Labs has made Suricata and Snort IDS coverage for known exploit paths of CVE-2021-44228.

Researchers are maintaining a public list of known affected vendor products and third-party advisories releated to the Log4j vunlerability.

Rapid7 customers

InsightVM and Nexpose

[Update: December 17, 2021 4:50pm ET]

Authenticated and Remote Checks
InsightVM version 6.6.121 supports authenticated scanning for Log4Shell on Linux and Windows systems. After installing the product updates, restart your console and engine.

• Unix and Windows: apache-log4j-core-cve-2021-44228 authenticated vulnerability check performs a complete file system search for vulnerable versions of the Log4j on Linux and Windows systems. Windows file system search must be enabled in the scan template for the authenticated check to run in Windows environments.
• Windows, Linux, Mac: apache-log4j-core-cve-2021-44228-remote unauthenticated vulnerability check attempts to trigger a connection back to the scan engine to determine vulnerability.

Version 6.6.121 also includes the ability to disable remote checks.

Agent checks
Insight Agent version 3.1.2.36 was released on December 12, 2021 and includes collection support for Log4j JAR files on Mac and Linux systems so that vulnerability assessments of the authenticated check for CVE-2021-44228 will work for updated Agent-enabled systems.

Insight Agent collection on Windows for Log4j began rolling out in version 3.1.2.38 as of December 17, 2021. It will take several days for this roll-out to complete.

• If you are using the Insight Agent to assess your assets for vulnerabilities and you are not yet on version 3.1.2.38, you can uncheck the “Skip checks performed by the Agent” option in the scan template to ensure that authenticated checks run on Windows systems.
• If you rely on the Insight Agent for vulnerability management, consider setting the Throttle level to High (which is the default) to ensure updates are applied as quickly as possible.

Containers
InsightVM customers utilizing container security can assess containers that have been built with a vulnerable version of the library.

InsightIDR and Managed Detection and Response

Rapid7 InsightIDR has several detections that will identify common follow-on activity used by attackers. Additionally, our teams are reviewing our detection rule library to ensure we have detections based on any observed attacker behavior related to this vulnerability seen by our Incident Response (IR), MDR, and Threat Intelligence and Detection Engineering (TIDE) teams. Through continuous collaboration and threat landscape monitoring, we ensure product coverage for the latest techniques being used by malicious actors.

• [Update: December 10, 2021 3:30pm ET]
  • Our Threat Detection & Response team has deployed detection rules to help identify attacker behavior related to this vulnerability:
    • Attacker Technique - Curl or Wget To Public IP Address With Non Standard Port
    • Suspicious Process - Curl or WGet Pipes Output to Shell
  • We also identified an existing detection rule that that was providing coverage prior to identification of the vulnerability:
    • Suspicious Process - Curl to External IP Address
• [Update: December 10, 2021 7:00pm ET]
  • An additional rule has been deployed:
    • Attacker Technique - Curl Or WGet To External IP Reporting Server IP In URL

Velociraptor

A Velociraptor artifact has been added that can be used to hunt against an environment for exploitation attempts against Log4j RCE vulnerability. A second Velociraptor artifact was also added that hunts recursively for vulnerable Log4j libraries.

tCell

Along with the guidance below, our tCell team has a new, longer blog post on these detections and how to use them to safeguard your applications.

*New* Pattern update for monitoring rule

For tCell customers, we have updated our AppFirewall patterns to detect log4shell. This means customers can view monitoring events in the App Firewall feature of tCell should log4shell attacks occur.

*New* Default pattern to configure a block rule

In order to protect your application against any exploit of Log4j, we’ve added a default pattern (tc-cdmi-4) for customers to block against. Below is the video on how to set up this custom block rule (don’t forget to deploy!), or reach out to the tCell team if you need help with this. As research continues and new patterns are identified, they will automatically be applied to tc-cdmi-4 to improve coverage.

Identify vulnerable packages and enable OS Commands

tCell will alert you if any vulnerable packages (such as CVE 2021-44228) are loaded by the application.

tCell Customers can also enable blocking for OS commands. This will prevent a wide range of exploits leveraging things like curl, wget, etc. Please note, for those customers with apps that have executables, ensure you’ve included it in the policy as allowed, and then enable blocking.

InsightCloudSec

The InsightCloudSec and InsightVM integration will identify cloud instances which are vulnerable to CVE-2021-44228 in InsightCloudSec. Customers can use the context and enrichment of ICS to identify instances which are exposed to the public or attached to critical resources.

Applying two Insight filters  — Instance Vulnerable To Log4Shell and Instance On Public Subnet Vulnerable To Log4Shell — will enable identification of publicly exposed vulnerable assets and applications.

IntSights

IntSights researchers have provided a perspective on what's happening in criminal forums with regard to Log4Shell and will continue to track the attacker's-eye view of this new attack vector.

Attacks and campaigns

Rapid7 Labs is now maintaing a regularly updated list of unique Log4Shell exploit strings as seen by Rapid7's Project Heisenberg.

Bitdefender has details of attacker campaigns using the Log4Shell exploit for Log4j. Their technical advisory noted that the Muhstik Botnet, and XMRIG miner have incorporated Log4Shell into their toolsets, and they have also seen the Khonsari ransomware family adapted to use Log4Shell code.

External resources

The following resources are not maintained by Rapid7 but may be of use to teams triaging Log4j/Log4Shell exposure.

CISA now maintains a list of affected products/services that is updated as new information becomes available.

CISA also has posted a dedicated resource page for Log4j info aimed mostly at Federal agencies, but consolidates and contains information that will be used to protectors in any organization.

ShadowServer is a non-profit organization that offers free Log4Shell exposure reports to organizations.

NCSC NL maintains a regularly updated list of Log4j/Log4Shell triage and information resources.

Updates

[December 10, 2021, 5:45pm ET]
Rapid7 has posted a technical analysis of CVE-2021-44228 on AttackerKB.

[December 11, 2021, 11:15am ET]
Added additional resources for reference and minor clarifications.

[December 11, 2021, 4:30pm ET]
VMware has published an advisory listing 30 different VMware products vulnerable to CVE-2021-44228, including vCenter Server, Horizon, Spring Cloud, Workspace ONE Access, vRealize Operations Manager, and Identity Manager. Their response matrix lists available workarounds and patches, though most are pending as of December 11.

Rapid7 has observed indications from the research community that they have already begun investigating RCE exploitability for products that sit in critical places in corporate networks, including network infrastructure solutions like vCenter Server. VMware customers should monitor this list closely and apply patches and workarounds on an emergency basis as they are released.

[December 11, 2021, 10:00pm ET]
Apache Struts 2 Vulnerable to CVE-2021-44228
The Apache Struts 2 framework contains static files (Javascript, CSS, etc) that are required for various UI components. Finding and “serving” these components is handled by the Struts 2 class DefaultStaticContentLoader. The DefaultStaticContentLoader is vulnerable to Log4j CVE-2021-44228;
given the default static content, basically all Struts implementations should be trivially vulnerable. Rapid7's vulnerability research team has technical analysis, a simple proof-of-concept, and an example log artifact available in AttackerKB.

[December 12, 2021, 2:20pm ET]
InsightVM and Nexpose customers can now assess their exposure to CVE-2021-44228 with an authenticated vulnerability check. Note that this check requires that customers update their product version and restart their console and engine. See the Rapid7 customers section for details.

[December 13, 2021, 10:30am ET]
Updated mitigations section to include new guidance from Apache Log4J team and information on how to use InsightCloudSec + InsightVM to help identify vulnerable instances.

[December 13, 2021, 2:40pm ET]
Rapid7 researchers have confirmed and demonstrated that essentially all vCenter Server instances are trivially exploitable by a remote, unauthenticated attacker. Technical analysis, proof-of-concept code, and indicators of compromise for this vector are available in AttackerKB.

[December 13, 2021, 4:00pm ET]
tCell customers can now view events for log4shell attacks in the App Firewall feature. Additionally, customers can set a block rule leveraging the default tc-cdmi-4 pattern.

[December 13, 2021, 6:00pm ET]
We received some reports of the remote check for InsightVM not being installed correctly when customers were taking in content updates. Product version 6.6.119 was released on December 13, 2021 at 6pm ET to ensure the remote check for CVE-2021-44228 is available and functional. Customers will need to update and restart their Scan Engines/Consoles.

[December 13, 2021, 8:15pm ET]
Information on Rapid7's response to Log4Shell and the vulnerability's impact to Rapid7 solutions and systems is now available here.

[December 14, 2021, 08:30 ET]
Apache has released Log4j 2.16. This disables the Java Naming and Directory Interface (JNDI) by default and requires log4j2.enableJndi to be set to true to allow JNDI. It also completely removes support for Message Lookups, a process that was started with the prior update. It mitigates the weaknesses identified in the newly released CVE-22021-45046.

Rapid7 has posted resources to assist InsightVM and Nexpose customers in scanning for this vulnerability. This post, “Using InsightVM to Find Apache Log4j CVE-2021-44228” goes into detail on how the scans work and includes a SQL query for reporting. For product help, we have added documentation on step-by-step information to scan and report on this vulnerability.

[December 14, 2021, 2:30 ET]
Along with Log4Shell, we also have CVE-2021-4104 — reported on December 9, 2021 — a flaw in the Java logging library Apache Log4j in version 1.x. JMSAppender that is vulnerable to deserialization of untrusted data. This allows a remote attacker to execute code on the server if the deployed application is configured to use JMSAppender and to the attacker's JMS Broker. Note this flaw only affects applications which are specifically configured to use JMSAppender, which is not the default, or when the attacker has write-access to the Log4j configuration for adding JMSAppender to the attacker's JMS Broker.

[December 14, 2021, 3:30 ET]
CISA has posted a dedicated resource page for Log4j info aimed mostly at Federal agencies, but consolidates and contains information that will be used to protectors in any organization.

Rapid7 Labs is now maintaing a regularly updated list of unique Log4Shell exploit strings as seen by Rapid7's Project Heisenberg.

[December 14, 2021, 4:30 ET]
Added a section (above) on what our IntSights team is seeing in criminal forums on the Log4Shell exploit vector.

[December 15, 2021, 09:10 ET]
Added a new section to track active attacks and campaigns. See above for details on a new ransomware family incorporating Log4Shell into their repertoire.

[December 15, 2021, 10:00 ET]
Along with the guidance below, our tCell team has a new, longer blog post on these detections and how to use them to safeguard your applications.

[December 15, 2021 6:30 PM ET]
Apache has fixed an additional vulnerability, CVE-2021-45046, in Log4j version 2.16.0 to address an incomplete fix for CVE-2021-44228 in certain non-default configurations. Apache's security bulletin now advises users that they must upgrade to 2.16.0 to fully mitigate CVE-2021-44228. InsightVM and Nexpose customers can assess their exposure to CVE-2021-45046 with an authenticated (Linux) check.

Apache also appears to have updated their advisory with information on a separate version stream of Log4j vulnerable to CVE-2021-44228. They have issued a fix for the vulnerability in version 2.12.2 as well as 2.16.0. We are investigating the feasibility of InsightVM and Nexpose coverage for this additional version stream.

Version 6.6.120 of the Scan Engine and Console is now available to InsightVM and Nexpose customers and includes improvements to the authenticated Linux check for CVE-2021-44228.

[December 17, 2021 09:30 ET]
An "external resources" section has been added that includes non-Rapid7 resources on Log4j/Log4Shell that may be of use to customers and the community.

CVE-2021-45046 has been escalated from a CVSS score of 3.7 to 9.0 on the Apache Foundation website. The vulnerability was designated when it became clear that the fix for CVE-2021-44228 was “incomplete in certain non-default configurations'' and has now been upgraded in severity due to reports that it not only allows for DoS attacks, but also information leaks and in some specific cases, RCE (currently being reported for macOS). No in-the-wild-exploitation of this RCE is currently being publicly reported. The fix for this is the Log4j 2.16 update released on December 13. If you have not upgraded to this version, we strongly recommend you do so, though we note that if you are on v2.15 (the original fix released by Apache), you will be covered in most scenarios. Update to 2.16 when you can, but don’t panic that you have no coverage.
CVE-2021-45046 is an issue in situations when a logging configuration uses a non-default Pattern Layout with a Context Lookup. In this case, attackers with control over Thread Context Map (MDC) input data can craft malicious input data using a JNDI Lookup pattern.

[December 17, 12:15 PM ET]
Starting in version 6.6.121 released December 17, 2021, we have updated product functionality to allow InsightVM and Nexpose customers to scan for the Apache Log4j (Log4Shell) vulnerability on Windows devices with the authenticated check for CVE-2021-44228. This update now gives customers the option to enable Windows File System Search to allow scan engines to search all local file systems for specific files on Windows assets. Customers should ensure they are running version 6.6.121 of their Scan Engines and Consoles and enable Windows File System Search in the scan template. The update to 6.6.121 requires a restart.

Note: Searching entire file systems across Windows assets is an intensive process that may increase scan time and resource utilization. To allow this, you can enable Windows file system searching in the scan template in order to use the authenticated check for Log4j on Windows systems.

In some cases, customers who have enabled the “Skip checks performed by the Agent” option in the scan template may see that the Scan Engine has skipped authenticated vulnerability checks. If you have the Insight Agent running in your environment, you can uncheck “Skip checks performed by the Agent” option in the scan template to ensure that authenticated checks run on Windows systems.

[December 17, 4:50 PM ET]
Insight Agent collection on Windows for Log4j has begun rolling out in version 3.1.2.38 as of December 17, 2021. It will take several days for this roll-out to complete. If you are using the Insight Agent to assess your assets for vulnerabilities and you are not yet on version 3.1.2.38, you can uncheck the “Skip checks performed by the Agent” option in the scan template to ensure that authenticated checks run on Windows systems.

Read more about scanning for Log4Shell here.

[December 17, 2021, 6 PM ET]
Reports are coming in of ransomware group, Conti, leveraging CVE-2021-44228 (Log4Shell) to mount attacks. According to a report from AdvIntel, the group is testing exploitation by targeting vulnerable Log4j2 instances in VMware vCenter “for lateral movement directly from the compromised network resulting in vCenter access affecting US and European victim networks from the pre-existent Cobalt Strike sessions.” Expect more widespread ransom-based exploitation to follow in coming weeks.

[December 20, 2021 8:50 AM ET]
Added an entry in "External Resources" to CISA's maintained list of affected products/services.

[December 20, 2021 1:30 PM ET]
InsightVM and Nexpose customers can assess their exposure to CVE-2021-45105 as of December 20, 2021 with an authenticated vulnerability check. CVE-2021-45105 is a Denial of Service (DoS) vulnerability that was fixed in Log4j version 2.17.0. Our check for this vulnerability is supported in on-premise and agent scans (including for Windows).
Content update: ContentOnly-content-1.1.2361-202112201646
JarID: 3961186789

Please note that Apache's guidance as of December 17, 2021 is to update to version 2.17.0 of Log4j. While this is good guidance, given the severity of the original CVE-2021-44228, organizations should prioritize ensuring all Log4j versions have been updated to at least 2.16.0.

[December 22, 2021]
Rapid7 has released a new Out of Band Injection Attack template to test for Log4Shell in InsightAppSec. Learn more about the details here.

[December 23, 2021]
Apache has released Log4j 2.12.3 for Java 7 users and 2.3.1 for Java 6 users to mitigate Log4Shell-related vulnerabilities. Notably, both Java 6 and Java 7 are end-of-life (EOL) and unsupported; we strongly recommend upgrading to Java 8 or later. If you cannot update to a supported version of Java, you should ensure you are running Log4j 2.12.3 or 2.3.1.

[December 28, 2021]
Apache has released Log4j versions 2.17.1 (Java 8), 2.12.4 (Java 7), and 2.3.2 (Java 6) to mitigate a new vulnerability. CVE-2021-44832 is of moderate severity (CVSSv3 6.6) and exists only in a non-default configuration that requires the attacker to have control over Log4j configuration. This is an extremely unlikely scenario. Applications do not, as a rule, allow remote attackers to modify their logging configuration files. While keeping up-to-date on Log4j versions is a good strategy in general, organizations should not let undue hype on CVE-2021-44832 derail their progress on mitigating the real risk by ensuring CVE-2021-44228 is fully remediated.

[January 3, 2022]
InsightVM and Nexpose customers can assess their exposure to Log4j CVE-2021-44832 with an authenticated vulnerability check as of December 31, 2021. Please note that as we emphasized above, organizations should not let this new CVE, which is significantly overhyped, derail progress on mitigating CVE-2021-44228.

[February 4, 2022]

• The Log4j class-file removal mitigation detection is now working for Linux/UNIX-based environments.
• An issue with occassionally failing Windows-based remote checks has been fixed. Support for this new functionality requires an update to product version 6.6.125 which was released on February 2, 2022.
• Coverage has been added for CVE-2021-4506 (the Log4j weakness identified within VMware Horizon Connection Server).