Last updated at Tue, 16 Apr 2024 19:08:40 GMT

NOTE: Tom Sellers, Jon Hart, Derek Abdine and (really) the entire Rapid7 Labs team made this post possible.

On the internet, no one may know if you're of the canine persuasion, but with a little time and just a few resources they can easily determine whether you're running an open “devops-ish” server or not. We're loosely defining devops-ish as:

  • MongoDB
  • CouchDB
  • Elasticsearch

for this post, but we have a much broader definition and more data coming later this year. We use the term “devops” as these technologies tend to be used by individuals or shops that are emulating the development and deployment practices found in the “DevOps” — https://en.wikipedia.org/wiki/DevOps — communities.

Why are we focusing on about devops-ish servers? I'm glad you asked!

The Rise of Ransomware

If you follow IT news, you're likely aware that attackers who are focused on ransomware for revenue generation have taken to the internet searching for easy marks to prey upon. In this case the would-be victims are those running production database servers directly connected to the internet with no authentication.

Here's a smattering of news articles on the subject:

The core reason why attackers are targeting devops-ish technologies is that most of these servers have a default configurations which have tended to be wide open (i.e. they listen on all IP addresses and have no authentication) to facilitate easy experimentation  exploration. Said configuration means you can give a new technology a test on your local workstation to see if you like the features or API but it also means that — if you're not careful — you'll be exposing real data to the world if you deploy them the same way on the internet.

Attackers have been ramping up their scans for these devops-ish services. We've seen this activity in our network of honeypots (Project Heisenberg):

We'll be showing probes for more services, including CouchDB, in an upcoming post/report.

When attackers find targets, they often take advantage of these open configurations by encrypting the contents of the databases and leaving little “love notes” in the form of table names or index names with instructions on where to deposit bitcoins to get the keys back to your data.  In other cases, the contents of the databases are dumped and kept by the attacker but wiped from the target, then demanding a ransom for the return of the kidnapped data. In other cases, the data is wiped from the target and not kept by the attackers, making anyone who gives in to these demands in for a double-whammy – paying the ransom and not getting any data in return.

Not all exposed and/or ransomed services contain real data, but attackers have automated the process of finding and encrypting target systems, so it doesn't matter if they corrupt test databases which will just get deleted as it hasn't cost them any more time or money to do so. And, because the captive systems are still wide open, there have been cases where multiple attacker groups have encrypted systems — at least they fight amongst themselves as well as attack you.

Herding Servers on the Wide-Open Range Internet

Using Project Sonar — https://opendata.rapid7.com/about/ — we surveyed the internet for these three devops databases. NOTE: we have a much larger ongoing study that includes a myriad of devops-ish and “big data” technologies but we're focusing on these three servers for this post given the timeliness of their respective attacks. We try to be good Netizens, so we have more rules in place when it comes to scanning than others do. For example, if you ask us not to scan your internet subnet, we won't. We will also never perform scans requiring credentials/authentication. Finally, we're one of the more profound telemetry gatherers which means many subnets choose to block us. I mention this, first, since many readers will be apt to compare our numbers with the results from their own scans or from other telemetry resources. Scanning the Internet is a messy bit of engineering, science and digital alchemy so there will be differences between various researchers.

We found:

  • ~56,000 MongoDB servers
  • ~18,000 Elasticsearch servers
  • ~4,500 CouchDB servers

Of those 50% MongoDB servers were captive, 58% of Elasticsearch were captive and 10% of CouchDB servers were captive:

A large percentage of each of these devops-ish databases are in “the cloud”:

and several of those listed do provide secure deployment guides like this one for MongoDB from Digital Ocean: https://www.digitalocean.com/community/tutorials/how-to-securely-configure-a-pro duction-mongodb-server. However, others have no such guides, or have broken links to such guides and most do not offer base images that are secure by default when it comes to these services.

Exposed and Unaware

If you do run one of these databases on the internet it would be wise to check your configuration to ensure that you are not exposing them to the internet or at the very least have authentication enabled and rudimentary network security groups configured to limit access. Attackers are continuing to scan for open systems and will continue to encrypt and hold systems for ransom. There's virtually no risk in it for them and it's extremely easy money for them, since the reconnaissance for and subsequent attacking of exposed instances likely often happens from behind anonymization services or from unwitting third party nodes compromised previously.

Leaving the configuration open can cause other issues beyond the exposure of the functionality provided by the service(s) in question. Over 100 of the CouchDB servers are exposing some form of PII (going solely by table/db name) and much larger percentages of MongoDB and Elasticsearch open databases seem to have some interesting data available as well. Yes, we can see your table/database names. If we can, so can anyone who makes a connection attempt to your service.

We (and attackers) can also see configuration information, meaning we know just how out of date your servers, like MongoDB, are:

So, while you're checking how secure your access configurations are, it may also be a good time to ensure that you are up to date on the latest security patches (the story is similarly sad for CouchDB and Elasticsearch).

What Can You Do?

Use automation (most of you are deploying in the cloud) and within that automation use secure configurations. Each of the three technologies mentioned have security guides that “come with” them:

It's also wise to configure your development and testing environments the same way you do production (hey, you're the one who wanted to play with devops-ian technologies so why not go full monty?).

You should also configure your monitoring services and vulnerability management program to identify and alert if your internet-facing systems are exposing an insecure configuration. Even the best shops make deployment mistakes on occasion.

If you are a victim of a captive server, there is little you can do to recover outside restoring from backups. If you don't have backups, it's up to you do decide just how valuable your data is/was before you consider paying a ransom. If you are a business, also consider reporting the issue to the proper authorities in your locale as part your incident response process.

What's Next?

We're adding more devops-ish and data science-ish technologies to our Sonar scans and Heisenberg honeypots and putting together a larger report to help provide context on the state of the exposure of these services and to try to give you some advance notice as to when attackers are preying on new server types. If there are database or server technologies you'd like us to include in our more comprehensive study, drop a note in the comments or to research@rapid7.com.

Burning sky header image by photophilde used CC-BY-SA