Last updated at Mon, 04 Dec 2017 19:49:59 GMT

One cold winter afternoon as I sat in my office, cursing the air several degrees warmer around me due to slow internet connectivity, I thought to take a look at exactly the issue was. I had recently installed a new system of wireless access points which should be blanketing the entire house with a strong enough signal to make the air glow well out into the yard.

I logged into the controller for the APs, which helpfully provided all manner of statistics regarding the different devices connected, who was connected to to which AP, how the bandwidth was being used, and a dizzying array of other statistics (one of the selling points of these devices to a data nerd). Being a relatively technology-prone household, I was reasonably sure I would find the internet connection being choked out by several devices streaming at once, some large download running, or something similar, and would castigate the offender and perhaps adjust the rate limiting rules a bit to keep things a bit more sane in the future.

On digging into the data, I found an unfamiliarly-named device sending and receiving unusually large amounts of data almost continuously. Curious, I copied the MAC address from the wireless controller and looked it up on the internet  -- Ubiquiti Networks Inc. This was one of the wireless video cameras set up to watch the house. Uh oh.

The IoT Problem

Why uh oh? In the fall of 2016, compliments of the Mirai botnet, hundreds of thousands of compromised Internet of Things (IoT) devices were used in distributed denial of service attacks against a variety of targets. These attacks were sufficiently powered to take down large swaths of the internet and even cause internet access issues across entire countries.

The issue that Mirai, and surely others to follow, took advantage of is that many of these IoT devices ship with a default set of credentials that is common across all devices of a model, or even all devices of a manufacturer. Mirai searches for these devices and recruits them into its botnet, using them to attack other targets on the internet.

This sort of widespread issue is rife for abuse, and this is exactly what happened. Worse yet, the source code for Mirai was publically released. Now we have an example of a very successful DDoS attack, source code for the attack’s tool, and a huge number of unpatched devices.

How huge? Some estimates say that in 2016 there were slightly less than 18 billion IoT devices. Yes, that’s billion with a b. Now we begin to see where the problem lies.

Worse yet, the DDoS attacks that were carried out using Mirai only had a bit over 100,000 devices to work with, according to some estimates. This is a small fraction of the devices out there and had a very significant impact.

A Little Research

The first stop in figuring out if my overly-chatty device was to do a bit of research. Not all IoT devices are problematic, and the list of devices being co-opted by Mirai and its variants was a relatively short one. From a little searching on the topic, there are a couple ways to go about searching for which devices are problematic; we can try interrogating the network for devices that that have issue, hoping to find them directly, or we can research a bit to find out if we might have a problem based on the particular hardware in general.

From a purely research perspective, finding the devices that might be problematic isn’t terribly difficult. A bit of brief googling turned up a list from the Krebs on Security website that were targeted by Mirai. The list did turn up a device from the same vendor as mine, but not the exact same device. The listed hardware did, however, have the same default credentials as mine. I always change the credentials on my devices when I stand them up, so this shouldn’t be a problem.

No really, I’m totally positive I changed them. I was standing several up at a time, surely I changed them all? Uh oh?

The next research bit before diving into the devices would be try out a couple of the multitude of IoT scanners that have popped up, all claiming to be able to suss out problem devices. After a bit of googling, I decided to try a couple of them; one a web-based scanner and one a script.

The web-based scanner was from Imperva, a well known security tool company. It has a simple ‘press go’ interface and automatically scans the address you are browsing from.

Its results, however, are not what I would call conclusive:

I take a certain amount of umbrage to statements, like the above, which imply that the absence of information means that the results are negative. Well, perhaps the next scanner will be better.

The second scanner, also from a well-known security company, is a script-based tool from Rapid7[1] and hosted on GitHub. This is a little more my speed. Furthermore, the tool is written in Perl, so cracking it open was like running into an old friend. Running the tool required a quick download of a few modules from CPAN, and away we go.

Unfortunately, no great results here either. After poking about in the configuration file that the script uses, it turns out that it was looking for devices from a specific list of vendors with the default administrative credentials set.

One of the devices in question was indeed still using the default login and password, but the script failed to figure out what the device was due to looking for a very specific bit of text in the source of the login screen. Nonetheless, I had a device on my network that very neatly fit the pattern for being abused by an IoT botnet, and this bore further investigation.

Investigating

There are a number of routes that we can take to investigate a specific device and see what it is up to. In the absence of historical log data, we can scan it externally, try to get onto the device itself and see what we can see, and we can look at the traffic inbound to and outbound from the device.

Nmap

One of the easiest and most informative tools that we can use to look at our device is Nmap. We can do a quick nmap -A against the device, which will give us OS detection, version detection, script scanning, and traceroute, resulting in:

So here we can see the expected web server, running lighthttpd on ports 80 and 443, rtsp running on port 554, presumably to support streaming video from the device, and Dropbear running an ssh server. One thing to notice here is the lack of a Telnet server immediately visible on port 23. It could be on another port, but this does put us out of the immediately dangerous spot as far as Mirai and other variants go.

Internally

If we can get on the system itself (where system = tiny golfball-sized IP camera running linux), we should be able to interrogate it and find out exactly what it is that is using up so much bandwidth.

SSH

Since we saw an SSH server running from the nmap scan, we can try to connect to that. Using the still default vendor credentials of ubnt for a username and password we can ssh ubnt@10.0.0.197 and voilà:

So here we have a standard BusyBox linux, common to many embedded systems. This gives us a somewhat limited but common set of tools that we can use to check things out a bit further.

Netstat

First and foremost, we can make use of netsat. Running netstat -anputw will give us a solid idea of who we are presently talking to. These options will list all UDP (u), TCP (t) and RAW (w)  connections on all sockets (a) with numeric hosts (n) and includes the associated program (p):

Nothing much of interest here. We can see the SSH connection from 10.0.0.117 and the various camera specific connections to the recording back end on 10.0.0.197. Nothing unusual talking or listening.

Poking About the System

While logged in, we have the opportunity to poke around the system in general. This device is a very typical stripped-down linux BusyBox. In /var/log/ there are system logs, as we might expect, which are largely full of the camera specific bits doing camera-specific things. We can also look at dmesg and learn a bit more about the specific hardware and environment:

One interesting thing to discover here, for gee whiz purposes, is that the system appears to be running on OpenWrt, which is somewhat cool for a consumer IP camera. Overall, nothing too interesting or unusual.

It is worth noting, for purposes of looking for signs of Mirai, that there would also not be anything here. Mirai maintains a connection through the open telnet server and does not, at this point, have a mechanism for persistence. It is also, however, worth noting, that putting such a mechanism in place would be fairly trivial and we should expect its arrival/discovery at any point.

Examining the Traffic

There are a couple bits that we can look at for purposes of examining the specific traffic externally to the device. We can look at packet captures and we can look at netflow data.

Packet Captures

In this case, I asked the pfSense box to pull a packet capture of all traffic destined to or from our questionable camera. Since the camera was attached to a wireless network and, from an infrastructure perspective, it would have been a bit of a pain to get monitoring in place an isolate traffic out for the particular device, this was by far the easiest route.

In our case, unfortunately (or fortunately, rather), nothing of interest. While this doesn’t mean that there would never be anything of interest, it does mean that the device is not spending all of its time actively attacking and/or communicating with odd devices on the internet.

NetFlow

In order to get a better idea of what exactly the device is doing, we can also look at the NetFlow data. We can pull NetfFow from a variety of places on the network. NetFlow was originally developed by Cisco and gives us  variety of information about the source and destination traffic on the network.

In my case, I have a pfSense firewall and can make use of the softflowd package to collect the netflow data. From there, I need to send it somewhere to aggregate and analyze the data. There are a huge number of tools available for this task, many of them commercial and either free or with a free trial period. I do, however like to use open source tools whenever possible, so we’ll go with ntop.

After configuring pfSense to send the data and analyzing it with nopt, again, a whole lot of… nothing. This is not unexpected after the data from the packet capture, but it does give us a greater level of detail on the nothing that we previously saw. Graphically detailed nothing is, on occasion, very useful when presenting investigative results to management, so this is a worthwhile exercise in learning how to collect data in this format.

Given a much longer period of time in which to collect the netflow data, we would likely see the odd unexpected connection occur, possibly for firmware updates or a phone home to the vendor. Nonetheless, we again have confirmation that the device is not doing anything terribly unexpected.

Conclusion

Ultimately, my device ended up not doing anything terribly suspicious, but I feel better for giving it the once over. The large amount of traffic that I was seeing was likely rtsp streaming back to the Network Video Recording (NVR) software on one of the other systems on my network.

Yes, it was overly chatty, yes it was bogging down the network, no it was not a compromised device trying to take down Amazon. I was reasonably sure that this was the case relatively early on, but working through the process and tooling to run down suspicious activity is a valuable exercise in and of itself.

The IoT world is a very dark and scary place with twisty and crooked little paths through it. We can find internet enabled anything, from toasters, to light bulbs, to cameras, and, being a relatively new phenomenon, the security around these devices is practically nil.

While the IoT security problem is clearly bigger than a breadbox, and we can expect to see DDoS attacks and the like from swarms of these devices for the foreseeable future, these issues do bring to mind the growing pains of another similar set of devices -- wireless access points.

The early days of wireless access points saw them shipping with common default passwords, disabled encryption settings, and firmware rife with vulnerabilities, manufacturer backdoors, and other such bugaboos. While it did take some time for the industry to mature sufficiently for these problems to be mitigated across the majority of vendors, it did eventually happen. Now all we need to do is hold on long enough for this to happen for IoT, and hope the wheels don’t come off in the meanwhile.

For more security deep dives, check out a few of our other articles: