Last updated at Mon, 05 Feb 2024 20:04:18 GMT

Merry HaXmas, everyone! As the holidays are winding down it is time to start thinking about how to re-purpose some of the gifts you were less than enthusiastic about. Perhaps some socks will find their way to a cousin next year, or a fruitcake will lie in wait for a nosy neighbor. Completely unrelated to that, the Metasploit team has a confession to make: we have been taking random Python scripts off the internet and passing them off as modules. Well, sorta. Let me explain. No, there is too much: let me sum up.

Modules as Individual Programs

Most people who have tried to run a Metasploit module over a large number of targets or have tried to run several modules at the same time know that the module system is reaching the limits of its scalability. In fact, most of Metasploit Framework's startup time is spent initializing every module, often multiple times. Every module is read off of disk, has a magical meta-programming stub prepended, and is evaluated in the framework context, giving it full access to the running console.

Or that is how all the modules used to work, rather. Because module loading and running is tied so deeply into how Metasploit is built, many of our efforts to fix this are still ongoing, but in April we got the ball rolling by landing support for running modules in a separate process. We currently have two such modules in tree: an exploit for the Haraka SMTP server and a denial of service module for Slowloris, with more on the way. They communicate to Metasploit Framework via JSON-RPC over stdin/stdout, and the rumblings are true, they are both written in Python.

Since this new style of module runs as its own program, you can script it without ever needing to start msfconsole (or an rpc server) at all. Also, all the tools we write around using them will be able to work with any language, and since we will be using existing standards wherever possible, some tools like the proxy can work with other tools already in your toolbox. Any language that can talk JSON over stdin/stdout and make connections through a SOCKS5 proxy will be able to full take advantage this interface, but we are starting with a focus on Python because, well, let's talk about that.

Intermission: Real Talk about Choosing Python

I tend to get a mix of gleaming eyes (mostly) and scowls (not that many) when the prospect of Python modules comes up. The scowls tend to be caused by valid concerns from people who know the project well and whose opinions I deeply respect, so let's take a quick aside to get some of the tension out of the air by addressing some of these concerns PEP-style before continuing.

Won't running external programs actually be slower?
Latency when starting a module might be a little higher, especially on platforms like macOS that take longer to start programs due to syscall auditing, but I don't think it will be very perceptible since there are some expensive operations that Metasploit does now that we can skip (for the interested, search for module.replicant). Once a module gets going it has its own set of file descriptors and doesn't have to deal with UI threads or whatever other modules are running, so it is able to do its thing faster.

Why add another language to the complexity that you already have with Ruby?
Most of the new security code that isn't written by us is being written in Python (for remote things) or PowerShell (for local Windows stuff). Most researchers I come across are more skilled at Python than they are at Ruby, so to keep Metasploit hackable for the next generation of researchers and pen testers embracing Python (at least a little bit) makes sense. As the first target for this interface, it allows us to quickly get coverage for things that were slightly out of reach before.

Won't you lose all the work done for protocol support in Metasploit?
No. That code isn't going anywhere soon, and if that's what you need you can still write a module like you are used to. On the other hand, if you need something that only, or more conveniently, exists in Python you are free to use it. Even when all modules are converted to standalone programs eventually, our current projects like ruby-smb are designed to be lightweight dependencies that don't need Metasploit plumbing to function and could easily be included in an external Ruby module. In theory, all the protocol libraries in Rex could be extracted in a similar manner.

The proxy built into Metasploit is... lacking. Will these modules have to use it in order to pivot?
No. Currently under development and to be released soon after HaXmas is a modular SOCKS5 proxy that will be able to connect through proxy chains like the current Metasploit PROXIES option, but will support TCP and UDP and domain resolution deferral (like with SOCKS4a) and will have support for a new pivoting msfrpc interface (also under development). Anything that can send connections through a SOCKS5 proxy will be able to pivot with ease. For external modules that might look like PySocks or socksify-ruby.

How are you going to review all this code in a different language?
Many of us on the Metasploit team are programming polyglots with some experience in Python. Some of us have a lot of experience and will not have a problem reviewing code and teaching the rest of the team what to look for. There are also Python experts in other areas of Rapid7 that have been keen to help with Metasploit for a while, and testing/reviewing PRs is always a great first step.

If you still have questions or concerns feel free to reach to us on IRC or Slack, and depending on who is around you might get more detailed pros and cons or maybe find people who agree with you. Such are the miracles of HaXmas.

How It Works

One of the problems of the current module system is that it takes a large amount of tribal knowledge to write sensible and efficient code. To fix that, the top goal of the new system is that modules are simple to write (for someone reasonably familiar with the host language and libraries), and simple to understand (ditto). One of the decisions we made to make that happen is that new modules are stateless. You run it, it does a thing, gives you the results, and then exits; if you need it to do something else, you run it again with different input. This way a large class of modules look like the proof of concept with some added metadata and a wee bit of boilerplate to invoke the library we provide to covert the well-defined JSON-RPC into function calls.

To get the metadata from a module (which includes options), the call sequence looks a bit like:

+------------+
| Metasploit |
|            |  Describe yourself  +-------------------+
|            +-------------------> |  some_module.py   |
|            |                     |                   |
|            |                     |                   |
|            |   Some metadata     |                   |
|            | <-------------------+                   |
|            |                     |                   |
|            |                     +-------------------+
|            |
|            |
+------------+

A module run might look like:

+------------+
| Metasploit |  Do a thing with
|            |   these options     +-------------------+
|            +-------------------> |  some_module.py   |
|            |                     |                   |
|            |                     |                   |
|            |   A bit of status   |                   |
|            | <-------------------+                   |
|            |                     |                   |
|            |  Moar status        |                   |
|            | <-------------------+                   |
|            |                     |                   |
|            |  I found a thing    |                   |
|            | <-------------------+                   |
|            |                     |                   |
|            |                     +-------------------+
|            |
+------------+

If you didn't want the run all of Metasploit to run the module, you have a simpler program that just prints the content of the messages to the screen. Or, since the messages are just normal data, you could have a module run on a remote host with its stdin/stout proxied to a local Metasploit instance, say via meterpreter.

How You Can Help

This HaXmas regift does come with some assembly required.

For module authors: currently, the (hopefully temporary) kluge layer between the new modules and the old module loader requires some tinkering when adding a new module type of external module (here are the changes we had to make to use denial of service modules). We have support for command stager remote exploits and denial of service modules and some scanner modules are on the way, but we need input on what to add next. Have a module you always wanted to write or stumble across a cool PoC in Python? Let us know (or ask us about) what you need in IRC, Slack, or GitHub and we'll do our best to get that part done next.

For Metasploit users: testing and bug reports are always appreciated. The modules currently written in Python should just work if you have Python 2 or 3 installed. The selection of Python modules to test isn't too exciting right now, but keep your eyes out for more content in the coming weeks.