Great Barrier Grief: How to Break Through Bottlenecks with Automated AppSec

Last updated at Fri, 29 Dec 2023 20:29:31 GMT

Most security teams don’t want to slow down or delay engineering or innovation, but the nature of the job can sometimes make this inevitable. Zate Berg, senior manager of security for Indeed.com, was able to avoid his team becoming a bottleneck in their software engineering team’s high-velocity process. How did he do it? In our brand-new episode of our podcast, Security Nation, Zate explains his challenges and successes in building an application security program for Indeed.com, and six takeaways every company can leverage to accomplish what Zate has done.

Zate’s journey from remote farm to infosec leader

Zate grew up on a million-acre farm (yes, million!) in Australia that was so isolated that he attended school every day over the radio and roamed the outback in his spare time. In his teenage years, he moved to the city, discovered technology, and instantly fell in love with it. He’s spent 20 years as a technology professional, 15 of which have been in security. Along the way, he was hired at Rapid7, and he later went on to become the head of the information security team at Indeed.com.

A large technology company, Indeed.com’s software engineering team moves fast. Their goal is to iterate and try new things to continuously deliver value to customers. This means the security team can’t get in the way of speed and progress, which put Zate on a mission to build security into the engineering process. With thousands of sub-products and developers doing hundreds of production deploys a week, this was a feat full of challenges.

Building an integrated application security pipeline

Zate and his team had to ensure they were catching issues to reduce risk without slowing down or blocking engineering. Without a 500-person security team to do all of this, they turned to dynamic application security testing to integrate with the build process and scan all external-facing applications within a 24-hour period. When an engineer pushes code, their appsec tool pulls a copy of it to the side, detects the changes, and runs a dynamic analysis.

A project that took two years to complete, it’s gone from two pieces of code to 16 pieces of code that work together to scan everything in their GitLab through cross-scripting, open redirects, SQL injections, etc. Zate and his team started out by building the first parts of the process (a now open-source) and over the next year and a half, added ways to do authenticated scans and have a controller that schedules things for developers and engineers to look at when their application is being scanned.

They’ve also built this process into their bug bounty program so they can take a finding from a bug bounty on one piece of software and look for it across their entire environment to find all possible vulnerabilities. This way, they’re not relying on a bug bounty as their only source of truth—it’s more of a safety net. Plus, it enables engineering to see what code repository and line of code caused the issue so they can understand and fix it. With an authenticated API checker running simultaneously to this, it can identify methods and URL endpoints and parameters to pinpoint what someone from the outside could touch that they should be monitoring.

Zate and his team also built a piece called NOMAD, which is their own version of a spider that uses Selenium and headless chrome to build an outside view of an application. Combined with the rest of their appsec tools, they created a product called ERGO that combines all the in findings into one object that can be fed through a scheduling app into their vulnerability scanner (called AVA). This process takes an input of everything you need to know about an application, runs through all of the findings, and scans the rest of the environment for them.

They’re now at a point where they’re looking to take pieces of their appsec tool and bundle it into smaller pieces that either an engineer can run on their laptop while they’re building things or a QA person can use to automatically tell them about vulnerabilities. This way, engineering’s existing QA checks can be run through a proxy that helps the security team build the view of what the app is.

Challenges with building a scalable, progressive appsec process

The biggest challenge they faced early on was everything took exceptionally longer than anticipated to figure out and create because of the scale in which they were working. What they thought would take three months to build took two years since they opted not to use off-the-shelf solutions and tie them together. To work for their infrastructure, it had to be modular, able to detect new things quickly, and run through thousands of potential endpoints every 24 hours.

Another challenge they faced was needing to rewrite AVA, as it was taking too long to do certain types of scans. Part of this was due to the fact that they were trying to do too much with it.

Six takeaways for building an application security program

In the podcast interview, Zate offered six key takeaways for security teams:

1. Address the low-hanging fruit

Find the low-hanging fruit you can knock out first that will make a measurable impact and build from there. This way, you don’t have to wait for a bug bounty to get a random hit.

2. Working with engineering is far more productive than blocking it

You can’t stop people from pushing code to production, so as a security team, it’s your job to gain visibility into what is happening across your environment. This may involve crawling your DNS or looking for new repos. When the engineering team at Indeed.com creates a new repo, it has metadata that explains what kind of service it is (internal or external), who owns it, etc., so security can quickly track down the team that owns it when they find something so that it can get fixed quickly.

3. Get team buy-in through education

Getting engineering buy-in for security tools often doesn’t go well. The way Zate’s team approached it, though, is something all security teams can learn from. When they first began building their appsec tools, they pretended that nobody would want it. Once engineering started to see it, however, they saw they could gain insight into endpoints and applications, a functionality they never had before. Naturally, wanted to know what else they could see, so Zate’s team educated them about what they could do with it, and the word spread. This also gave the engineers exposure into how security could be built into their existing process, and not be a disruption.

4. Standardize for easier integrations

It can be easy to want to integrate all the things, but this leads to complexity and often isn’t useful. Instead, pick the two or three things that just about everybody on the team uses and build security processes to work with them.

5. Keep the team focused to build momentum

If someone isn’t focused on it every day, it’s going to fall to the bottom of the pile. As Zate realized certain elements of the security program were going to take time to build, he designated one person who was really passionate about the project to oversee it. By investing in this, they were able to go from having WES and AVA to a 16-part appsec tool.

6. See the magic in software

Software is magic. You can build big pieces of software, but when you run into bottlenecks, as long as you’re prepared to dig in a little bit, you can solve the bottlenecks, which is the key to making something an engineering team can adopt.

Zate is a wealth of information when it comes to application security, scalable security, and building a team. Want to hear more about his journey? Be sure to subscribe to our podcast for more episodes like this!