Last updated at Wed, 29 Nov 2023 22:36:58 GMT

Back in June, I decided to spend some time looking at the VxWorks operating system. Specifically, I kept finding references to VxWorks-based devices running firmware images with the debug service (WDB Agent) enabled, but I could not find a description of the protocol or any estimates as to how prevalent this service was. After a couple days of digging around and a couple more days of scanning, I became aware of just how extensive this issue is.

For folks who aren't aware of what VxWorks is -- VxWorks was the most popular embedded operating system in 2005, it is a platform developed by Wind River Systems, which has since been acquired by Intel. Vendors who wish to build products using the VxWorks operating system will license it out by component, integrate their own application code, and then build images which can be installed on their products. VxWorks has been used to power everything from the Apple Airport Extreme access points to the Mars rovers and the C-130 Hercules aircraft. VxWorks itself is essentially a monolithic kernel with applications implemented as kernel tasks. This means that all tasks generally run with the highest privileges and there is little memory protection between these tasks (at least with version 5.x).

The WDB agent is a system-level debugger for the VxWorks operating system that runs on UDP port 17185. This service is modeled on the SunRPC protocol in terms of wire format and allows anyone with access to this port to read memory, write memory, call functions, and manage tasks. Since the protocol is UDP and there is no authentication, handshake, or session ID, requests to the WDB agent can also be spoofed by an attacker.

To determine how widespread this issue was, I wrote a scanner module for the Metasploit Framework and conducted a network survey that encompassed over 3.1 billion IP addresses. Of this set, over 250,000 systems were found with the WDB agent exposed. After digging around in the DShield database, it became obvious that an unknown party had already spent most of 2006 scanning for this service. I contacted the Carnegie Mellon CERT and provided them with the list of affected devices that were gleaned from the survey, with the goal of notifying as many vendors as possible in a reasonable amount of time. CERT assigned VU#362332 to this issue and plans to publish an advisory today (August 2nd, 2010). The Rapid7 Nexpose engineering team has created a check for this vulnerability and made this available through the standard product update mechanism.

While CERT began the coordination process for the WDB Agent issue, I ran into another vulnerability related to this VxWorks platform. VxWorks has a configuration setting entitled "INCLUDE_SECURITY"; when this setting is enabled, the definitions for LOGIN_USER_NAME and LOGIN_PASSWORD can be used to specify a set of credentials for accessing the device over FTP or Telnet. This credential set is baked into the firmware image, and while this backdoor account can be removed by application code calling loginUserDelete(), it is quite common for these credentials to be left in place for production builds. One of the Metasploit modules I wrote for the WDB Agent performs a complete physical memory dump of the target device. I noticed hardcoded credentials in the memory dumps obtained from a wide range of devices. In most situations, a memory dump would be enough to provide remote access to any exposed FTP or Telnet services, but VxWorks had one more trick that I had not accounted for.

Instead of storing the backdoor account password in clear-text, a home-grown hashing mechanism is used to obfuscate the password. Presumably, this was done so that anyone with access to an unencrypted firmware image could not login with the backdoor account just by reading the clear-text password. From an engineering perspective, the hashed password is obtained by passing the clear-text version to a proprietary utility called "vxencrypt". This utility, although undocumented, has had its hashing algorithm indexed by Google and is trivial to reverse engineer. The hashing process is basically an additive sum of all of the bytes making up the password, with some XOR thrown in for good measure, and a conversion routine to transform the final sum into a printable string. Even though VxWorks enforces a minimum password length of 8 characters (max 40), there are only around 210,000 possible hash outputs for any valid password.

To make matters worse, the number of passwords that are actually reasonable to type (not high or low ascii) fit within about 8,000 permutations. Keep in mind that there is no account lockout mechanism and that the FTP daemon allows four concurrent sessions and never drops the connection, regardless of the number of bad attempts. This allows almost any password to be brute forced, over FTP, in as little as 30 minutes. To only caveats are that the username (LOGIN_USER_NAME) is known and that the vendor in question did not replace the default authentication mechanism with their own implementation. To test this theory, I precalculated a password list that collides every possible hash output, then sorted this list so that typical passwords would be tested first. A single connection brute force using the Metasploit ftp_login module gained access to a local target device in about two hours. Once again, I enlisted the help of CERT, who assigned VU#840249 to this issue, coordinated the vendor notification process, and plans to publish an advisory today (August 2nd, 2010).

For more information, see my Fun with VxWorks presentation (PDF) from Security B-Sides Las Vegas and the Defcon 18 Skytalks. Wind River Systems, the maker of VxWorks, has notified their customer base about both issues, but has not indicated that they plan to disable the WDB Agent entirely or fix their hashing implementation. In the latter case, Wind River Systems has provided customers with sample code for replacing their existing hashing algorithm with SHA-1.

I would like to thank Dillon Beresford, Shawn Merdinger, David Maynor, R3L1K, and FX for their help identifying affected devices, reverse engineering firmware dumps in IDA, and generally lending a hand with the research. The two bugs mentioned in this post are just the tip of the iceberg and there is a lot more work left to do before the VxWorks platform is as tested as it needs to be.

I would also like to thank the security response team at Rockwell Automation, who took both issues seriously, did a deep assessment of their entire product line, and shared their findings. Not a single shipping Rockwell Automation product is affected by the vulnerabilities mentioned in this post.

Finally, I would like to thank the fine folks at CERT, who agreed to take on a 100-vendor coordination task in the 60 days leading up to the summer conferences. You guys kick ass and did an amazing job at both notifying the affected vendors and standing your ground on the disclosure schedule.

The Metasploit Framework SVN tree has been updated with a set of modules for detecting vulnerable systems and performing a remote memory dump. The device-specific WDB exploits and the master password list for the hashing vulnerability will be made available in early September. The example below demonstrates using the Metasploit Framework to identify an affected device and take a snapshot of the target's physical memory. More than likely, any device you find will NOT be in the survey results above -- the survey was limited to internet-exposed addresses and nearly every enterprise network I have tested has yielded additional affected products. You can either contact CERT (cert[at]cert.org) or try to contact the vendor directly and refer them to the CERT VU IDs above.

$ msfconsole
msf > use auxiliary/scanner/vxworks/wdbrpc_bootline
msf exploit(wdbrpc_bootline) > set RHOSTS 192.168.0.0/24   
msf exploit(wdbrpc_bootline) > run 

[*] 192.168.0.34: 5.4 Netro AirstarSAS 2 host:/zdev/vx_gz
[*] 192.168.0.34: BOOT> ffs(0,0)host:/zdev/vx_gz e=192.168.0.34:fffffff0...
[*] Auxiliary module execution completed

msf exploit(wdbrpc_bootline) > use auxiliary/admin/vxworks/wdbrpc_memory_dump
msf exploit(wdbrpc_memory_dump) > set RHOST 192.168.0.34
msf exploit(wdbrpc_memory_dump) > set LPATH /tmp/target.mem   
msf exploit(wdbrpc_memory_dump) > run 

[*] Attempting to dump system memory...
[*] 192.168.0.34 Connected to 5.4 - Netro AirstarSAS 2 (host:/zdev/vx_gz)
[*] Dumping 0x01c00000 bytes from base address 0x00000000 at offset 0x00000000...
[*] [ 00 % ] Downloaded 0x000010a4 of 0x01c00000 bytes...

Update: Wind River Systems indicated that they plan on fixing the weak password hashing vulnerability in VxWorks 6.9, which has not yet been released.