AI Handles Incidents, Engineers Lose Touch With Their Systems

AI capabilities were nowhere near what we have today, and that remained a prototype, but this is now a reality. These tools do it all: inspect alerts, form hypotheses, query telemetry, correlate recent deployments, and even implement the fix themselves. As much as I love to see it, I have a major concern: we are losing touch with our systems.

The better these tools become at resolving routine incidents, the less practice human responders will get. And when an ambiguous, high-severity incident comes in that automation cannot solve, responding engineers will be in trouble. These AI-assisted incident response tools, more commonly called “AI SR Es” – a term I don’t particularly like – are fantastic in many ways.

They feel especially magical when they handle a routine incident at night and you don’t have to wake up for a capacity issue. The problem is that routine incidents are also how responders “safely” develop an intuition for how their systems behave and fail. Human-factors researcher Lisanne Bainbridge described this paradox in her famous 1983 paper, The Ironies of Automation.

She explained that automation reduces operators’ opportunities to practice routine work while leaving them responsible for new and abnormal situations. She argues that, therefore, operators need to be more skilled and receive even more training than before automation. In the years to come, I predict that the average MTTR for most incidents will go down – thanks to AI-assisted incident response – but that the resolution time will shoot up for complex incidents because incident responders lost touch with their system and are struggling to investigate.

We can look at the aviation industry for inspiration. Plane automation handles much of the flying, but pilots remain responsible for situations that automation cannot manage: engine failures, unreliable instruments, rejected takeoffs, stalls, and other abnormal conditions. These events are extremely rare.

Modern turbine engines, for example, experience fewer than one in-flight shutdown per 100,000 engine flight hours. In other words, that is rare enough that a commercial pilot may complete an entire career without experiencing one outside a simulator. But when a failure occurs, pilots must react quickly and correctly.

For example, on TransAsia Airways Flight 235, the right engine’s propeller autofeathered shortly after takeoff. And while the aircraft was designed to continue flying on its left engine, the crew misidentified the problem. The aircraft stalled and crashed only 117 seconds after the first warning.

Airline pilots regularly return to simulators to rehearse rare emergencies. Under US FAA rules, captains must complete recurrent training or a proficiency check every six months, including scenarios such as an engine failure during takeoff. While most software incidents do not threaten lives, that is no reason not to perfect our craft.

Turns out the technology that created the issue can also help close it. At Rootly, where I work, we partnered with Uptime Labs to apply this idea through realistic incident simulations. Engineers take the incident commander’s seat during a simulated e-commerce outage, using observability tools while coordinating with LLM-powered stakeholders in Slack.

But what about using AI as a trainer? Responders can ask an agent to explain the steps it took, the signals it examined, and the evidence behind its diagnosis. But explanation and observation are not substitutes for practice.

You might pick up a few things from watching Serena Williams play, but you only learn tennis by getting on the court, and incident response is no different. I spent more than half a decade of my career building a software engineering school around progressive education: learning by doing. It was in-person, but we had no teachers; students worked on projects instead of listening to lectures.