Last updated at Tue, 05 Dec 2017 21:40:04 GMT
Welcome to Defender Spotlight! In this weekly blog series, we interview cybersecurity defenders of all varieties about their experience working in security operations. We’ll inquire about their favorite tools, and ask advice on security topics, trends, and other know-how.
Today, we're talking with Ryan Huber. Currently at Slack, Ryan has previously held positions at companies such as Orbitz and Risk I/O, doing security, engineering, or a combination of both. He enjoys computers, and can often be found working with other people that enjoy computers.
Let's get down to business.
Tell us about yourself, and your history working in security operations.
My life story: I started off using BBSes in 1992. I was curious about systems, and despite a few near misses, was never caught doing anything I shouldn’t have been doing.
I happened into my first “real” job, running the *nix servers for a large-ish dialup ISP, in 1997. I had done some network stuff and knew Linux pretty well, but they were using SCO OpenServer 5.05 (the reason I nearly quit the week I started). That job was probably the best learning experience of my career, because until then if I broke something it only affected me. At an ISP, it meant lots of people were unable to establish a PPP connection.
I had a few small scale development and security jobs until moving to Chicago in 2001, where I joined Orbitz.com. I had a number of different roles at Orbitz, and did security related work in most of them, but didn’t formally join the security team until 2006, when I moved to London to head up our EU security efforts. I spent a total of 11 years working for Orbitz (2001-2012), and am forever grateful for the opportunity.
In December 2013, the company where I was working started using Slack, and being a neckbeard, I decided to write a plugin for the command line IRC client I like, weechat. The docs were sparse, so I reverse engineered the web client to create my plugin, and thanks to a chance meeting learned that they were looking for someone to join in a security role. I started at Slack in 2014, and still work there today.
What you are working on these days?
I spend a lot of my time developing security tools and thinking about how to spot malicious activity. I’m also mildly active in security communities and some of my time chatting with people in about 20 different security-related Slack groups.
Around the end of last year, I wrote the first version of an auditd alternative in golang. We now use the result of that work, go-audit, on all of the servers at slack. I just finished our implementation of a backend service we are calling “uberproxy”, which has similar functionality to the systems Google talks about in their BeyondCorp papers. Next up, I’m wranging PKI. Pray for mojo.
Can you tell us about a moment in your career where you were proud to be working as a defender?
Is it a cop out to say I’m always proud of being a defender? Security and defense are what I love to do, and it is an amazing privilege to do what you love every day. I think sharing tools and information are among my favorite professional activities, so getting together with other folks and having a chance to show them what I’ve been working on is a source of pride.
In your opinion, what are the most important elements of implementing a successful security operations center capability?
I haven’t worked in a SOC, but I have opinions about managing security events and alerting. I wrote a short blog post about it, so if you have a few minutes, have a read.
What are some of your favorite OSS or commercial software, products, or tools that you use on a daily basis? How do they make your job easier?
I love my ChromeOS devices. I have a ridiculous number of Chromebooks and a couple of desktop Chromeboxes. They make my life easier via:
- Not having to think too much about things like persistent malware.
- Having my work environment automatically synchronized between boxes.
- Updates are automatic and fast.
I run ChromeOS in stock (not dev) mode and do all of my development on remote Linux instances with mosh+tmux. If you leave mosh running, it feels like you are just using a local shell. I used to do a lot in Python, but golang has stolen my heart in the past year. The net/http lib so handy that I miss it when I am using other languages.
I have a love/hate with ElasticSearch, but I couldn’t do any better, so I mostly love it. It makes a huge amount of data searchable, doesn’t cost a billion dollars, and that makes my life much easier. `grep -F` only goes so far.
There aren’t many security products I rate, but one worth of mention is Thinkst’s canary.tools service(+devices). Simple managed honeypots that you sprinkle around your environment. You should take a few minutes to check them out.
What are some of the trends in the security industry that you find encouraging?
I like what Alex Stamos (and many others) have been saying about building platforms as opposed to solutions. As a defender, you quickly learn that the solutions security vendors sell are not always what they claim to be. Splunk is a great example of a this model. It wasn’t created as a security tool, but it became an important tool for many security teams.
I’m also happy to see interest in defensive talks going beyond lip service. The O’Reilly Security Conference is one to watch in this space. The SecOps/SecDevOps/DevSecOps/whatever world is probably approaching what the DevOps world was 5 years ago. I think we’re at the beginning of this and I’m excited to see where we will be after we’ve had a few years to share ideas and techniques for security operations and automation.
What are the top 3 things defenders should be worrying about today? What worries you the most personally?
Credential theft will probably be my #1 for a very long time. There is no amount of training that can make your users perfect at security, so it is important to think of credentials as just one facet of security. I think of usernames and passwords as the equivalent of a firewall. We still use firewalls, but no mature company believes a firewall makes them secure.
#2 is snakes because snakes are scary.
#3 is missing some useful indicator because you aren’t looking. I talk about this a lot, but the thought of missing a small breach and it becoming a large breach is the thing that should keep most of us up at night.
What advice would you give to someone getting started in security?
Learn development and write code. Even if you never use your development skills, it gives you empathy for people who write software. It is easy to find bugs in software. If you want to write security software, being able to write legible code is a must. No one exists in a vacuum.
What do successful security teams look like? What qualities and skills do the ideal team members have? What do teams tend to struggle with the most?
A mix of disciplines and mutual respect. You will disagree with other humans, but generally they are trying to make things better, so keep that in the back of your head. Ideal team members are people who are skilled and are comfortable saying “I don’t know”. In security, I think teams struggle hugely with diversity of all kinds, which can lead to unhealthy groupthink. Security is also full of egos, and we’ve allowed that to be okay for far too long.
What are some of the best industry events to attend and why?
So far, I’ve had the best experiences at smaller conferences. At some scale, security conferences become performance art, which is rather counterproductive. I don’t have a favorite event, but I am holding out hope that the folks doing good work on defense will make this space interesting in near term.
That's it for today! Have anything else to end on?
Thanks for inviting me to do this. Talking this out gave me a chance to think about security more broadly, which is always a useful exercise!
Thanks, Ryan! We enjoyed hearing what you had to say, but especially your thoughts on snakes. 🐍
If there are any other questions you have for Ryan or you just want to say hi, you can find him on Twitter.
