💌 Hey there, it’s Elizabeth from SigNoz!
This newsletter is an honest attempt to conversation concerning all things - observability, OpenTelemetry, open-source and the engineering in between!
&
This part took 6 days, 5 hours to be cooked, anticipation we served. 🌚
There are two famous prophecies floating about tech circles these days.
The archetypal says SRE is the forthcoming of all application engineering, that as AI writes additional and additional code, the humans who remain volition be the ones keeping systems alive. The second says AI volition consume all tech job alive, SREs included. Neither is particularly helpful if you’re an SRE trying to fig out what your Tuesday volition appearance akin in 2027.
Let’s ask a additional restricted inquiry by looking at what’s already happening: When AI handles 95% of your event response, do you get worse at handling the 5% that really matters?
Most of us already use AI for our regular activity (our brains a small fried!), and so do SREs. Today’s conversation agenda is not whether AI replaces SREs, but whether AI is quietly making SREs small capable and whether anyone volition notice item before the next novel outage hits. The foundational example for this complete conversation comes from a 1983 investigation document that’s eerily prescient.
Note from author: Below is a concise precursor to History of Automation, which you power bask if you are into History and Cultures (like me 😉).
In 1983, a cognitive therapist named Lisanne Bainbridge published a document called Ironies of Automation. It became among the most cited document in individual factors research, and its center disagreement is nearly uncomfortably applicable today.
Bainbridge studied what happened whenever factories and manufacturing systems automated the activity of blue-collar operators. The findings were paradoxical and revealed that the additional you automate a process, the additional crucial the individual controller becomes during the rare moments automation fails and the small custom they get, and the worse they rotate into at exactly those interventions. Automation, which was inherently designed to eliminate humans from the loop, remaining them alongside the worst imaginable job, i.e., lengthy stretches of passive monitoring punctuated by rare, high-stakes crises they were increasingly unprepared for.
Ring any bells yet? 🙂
Basically, I’m example a parallel between the AI upheaval and manufacturing automation. Industrial automation reshaped blue-collar activity by taking complete regular bodily tasks, and the workers who remained had to grip exceptions they’d misplaced the muscle recollection for. AI is doing the identical item to cognition workers by taking complete the regular cognitive tasks, the form matching, the triage, the known-issue resolution and leaving humans alongside the rare, complex, ambiguous problems.
The exact problems that necessitate profound expertise, the exact expertise that atrophies whenever you halt practising.
Now we’re replaying this form alongside AI agents, and the stakes in application systems are lone growing.
Let’s obtain inventory of anywhere things remain today in the earth of location reliability engineering.
Alert noise decrease and intelligent grouping, runbook implementation for known issues, log form finding and anomaly flagging, and essential root-cause suggestions from historic event data, and auto-remediation for well-understood nonaccomplishment modes akin restarting a wrecked pod or scaling up a assistance that’s operating hot, are all fairly automated today.
Some contiguous targets contain multi-signal association throughout metrics, logs, and traces, autonomous root-cause inspection for partially understood failures, predictive event finding before users are affected, AI-driven alter hazard appraisal and automated rollbacks.
PagerDuty frames this as a tiered model.
Tier 1 incidents: Known issues alongside known fixes get completely automated.
Tier 2 incidents: Partially understood problems obtain AI recommendations alongside individual validation.
Tier 3 incidents: Novel, complex, cascading failures remain human-led, alongside AI providing supporting context.
But here’s the catch.
If human SREs (okay, now we have to use adjectives akin individual 🥹) lone affect alongside Tier 3 incidents, ie the novel, never-before-seen outages, anywhere do they build the intuition to grip them? Intuition is normally developed from years of hands-on event response, form acknowledgment built through repetition, and the benevolent of gut-level understanding of a scheme that lone develops from painfully waking up at the odd hr to resolve the bug that brought the scheme down.
This is anywhere the image starts getting blurry. Let’s appearance at several emerging investigation on AI-induced deskilling throughout multiple fields, which paints a accordant and concerning picture.
In medicine, a recent study established that endoscopists who used AI aid for polyp finding saw their unassisted finding rates autumn from 28% to 22% following a duration of AI use. They got worse at the item they were expected to be experts in, not since they forgot the theory, but since they stopped exercising the skill.
In aviation, research has shown that long-haul pilots who depend heavily on autopilot systems cognition measurable degradation in situational awareness and manual flying ability. The issue got grave adequate that the FAA now mandates additional manual flying period to counteract the effect.
Somewhere complete the former year, AI stopped being a tool I occasionally reached for and became the archetypal item I attain for, always. My intuition now is to offload as much as imaginable and use my own thinking lone anywhere it’s entirely unavoidable. The issue is that these moments are becoming the lone exercises my brain gets, and I can awareness the rust.
We can diagram a form here. The additional you let the scheme handle, the worse you get at handling things yourself and here’s the really hazardous part, you don’t awareness it happening. It gets masked as hyper-productivity. Cognitive investigation suggests that since AI tools create tasks awareness easier and enhance apparent performance, users are frequently unable to accurately fairness the true position of their own skills. You awareness competent, dashboards appearance green, and afterward on a Wednesday, a novel event hits that doesn’t equivalent any form the AI has seen, and you realise the muscle has atrophied.
For SREs, this manifests in particular ways, akin we halt study raw log streams since the AI summarises them, we halt forming hypotheses during incidents since the AI suggests base causes, we halt construction mental models of scheme architecture since the AI maps requirements for us, and all of these individually looks akin a efficiency win. Collectively, they hollow out the extremely ability that makes an SRE productive whenever things go sideways in ways nobody anticipated.
But there’s item equal additional concerning than deskilling, and researchers have started calling it never-skilling. Deskilling method you formerly had a capability but have since misplaced it. Never-skilling method you never developed it in the archetypal place. For younger SREs entering the site today in an surroundings anywhere AI handles most of the event reply workflow, the opportunities to build foundational intuition and muscle through hands-on custom are vanishing.
The training pipeline itself is damaged and not self-healing.
SREs realise their skills are degrading and lean additional on AI to compensate, which additional degrades their skills, creating a vicious sequence from which escape is difficult.
We are certainly not rejecting AI tooling; we are adopting it and integrating it stronger than always before, since that’s the lone way forward.
A few approaches value considering:
Deliberate inefficiency. Just as the FAA mandates manual flying period equal whenever the autopilot is absolutely capable, SRE teams can designate certain incidents, equal the ones the AI could handle, as human-practice opportunities. This can be considered as a long-term funding to keep skills fresh, although it power arrive at the disbursal of a super-fast solution
Build for human-in-the-loop, not human-on-the-side. There’s a meaningful difference between a scheme anywhere a individual approves an AI’s advice and one anywhere a individual actively engages alongside the issue alongside AI. The erstwhile keeps humans in a supervisory function that Bainbridge (the female who wrote that investigation document concerning 40 years ago) showed leads to vigilance decay, and the second keeps them cognitively engaged.
Let’s zoom out and obtain a appearance at the bigger picture.
Everything we’ve discussed here, the ironies of automation, the deskilling risk, the never-skilling problem, collectively applies fine beyond SRE. Software engineering as a entire is navigating the identical tension. As AI writes additional code, reviews additional PRs, and handles additional debugging, the identical questions apply.
We’re talking concerning SREs specifically since that’s the earth we live in at SigNoz. We build an open-source observability platform, the benevolent of tool that gives SREs the metrics, traces, and logs they need to comprehend their systems deeply. For us, this deskilling inquiry is not a rhetorical fad; it immediately shapes how we’re construction AI into our product.
Our method is to commencement alongside an AI aide that helps SREs leverage the power of LLMs during keeping humans firmly in control. Eventually, we’ll allow additional autonomy but inside apparent guardrails, and lone as rely is earned.
One advantage we have in this area is that, as an observability platform, we sit on the data itself, the metrics, traces, and logs that SREs depend on. Most AI SRE products today merge alongside observability tools through APIs, which method they’re operating alongside a limited, second-hand perspective of your systems. Because we own the data layer, we can build much deeper, additional context-aware AI capabilities that comprehend your scheme the way an informed SRE would.
And to answer the burning inquiry in your head, our goal isn’t AI that replaces SREs. It’s AI that supercharges SREs. Unlike ongoing lore, we accept humans volition remain essential for the decisions that matter most, particularly those that effect manufacturing infrastructure.
The forthcoming of SRE is individual with AI intentionally designed to keep humans sharp, engaged, and prepared for the 5% that really counts.
Here’s the LinkedIn post our originator posted a few days ago, which inspired me to compose this.
