Don’t dam upstream: ways to build a feedback loop
I was interviewing a candidate for a security analyst role and asked one of my two favorite questions: “Talk to me about a time… or a project… where, looking back on it you think to yourself: if I never have to do that again, it’ll be too soon. What was that misery, and what made it miserable?”
The candidate had a strong technical background and his experience was right on the mark. He also had an exceptionally relevant response. He described working at a federal SOC. Overall, it was a great learning experience, he said. They were constantly finding bad stuff and he learned a lot from his peers, but neither he nor his co-workers had any ability to influence detection. A separate team handled that. And — for security reasons — neither team could talk to the other. Hah! So, every week he’d see the same false positives he’d flagged the week before… and the week before that. Over time, this bred a feeling of helplessness, boredom and eventually burnout.
Back to the interview. We were hiring for a role where the candidate would be in the exact same position he’d just said he never wanted to repeat. At the time, the feedback loop in our SOC was broken and the required fixes weren’t trivial. Even though he was an exceptionally well qualified candidate, we chose not to proceed because he’d have been miserable.These disconnects aren’t unusual. “Just add a feedback loop” is too simplistic an answer. Solving this problem in security operations is much harder. Many analysts in a SOC lack the experience to effectively drive detection. Those who do have the experience typically don’t work in the SOC (or at least, not on shift) and may have forgotten exactly how frustrating this situation can be. Still, it’s not hopeless. If you find yourself in this situation, here are four options to build in a feedback loop.
1. Align incentives
If the SOC and detection/intel team report into different managers, make it clear to your detection team’s manager that her success is measured by the SOC manager’s enthusiastic support.
2. Get physical
Is your SOC sectioned off from the rest of your security team? Reserve seats for your sister team’s personnel. If there aren’t enough seats, rotate people through.
3. Make the pain transparent
By measuring the time wasted chasing dead ends (or even the volume of dead ends) and tying those to root causes, you’ll make it clear when adjustments are needed upstream.
4. Celebrate improvement
As you use metrics to drive change in your detection methodologies, reward your teams when the needle meaningfully moves in the right direction. Common wins help unify teams.