In hiring for my team I sometimes get questions about on call. I wanted to share some context on how I view on call for our team vs a dedicated engineering team. To the question of whether a role is a responder role or a developer role:
It is both. I feel strongly that successful engineers solve their own problems so I am looking for responder-developers that can play both positions. In the larger more traditional SOC/SIRT teams they will often carve off engineering as a specialization mostly to ease hiring as they scale. The downside is the dedicated tools team can loose touch with the operational aspects of the role and start building things the ops team doesn’t need, even as they get more and more mired in requirements documentation and program oversight. The Netflix model, whether in security or building the UI, is that every engineer owns their program from start to finish; so you develop the requirements, you decide what to work, you build it and you operate it. Everyone is ‘on call’ and this aligns the incentives to write good clean functional code; when it breaks you get paged. I know many folks coming out of a SOC/SIRT role at a more traditional company are looking to leave on call behind, but that is mostly due to the type of on call they had to endure. We aren’t triaging alerts and wading through false positives. We only engage on the high signal incidents escalated to us from the other security and crisis teams.
I manage the team and also take an on call shift. Getting to work some problems and then switch context back into building (the team) is enjoyable for me. I don’t consider on call a burden, but we are working to lower the cognitive load it requires further through automation… and hiring stunning colleagues to build and respond!