production agent trust and precision
Google's on-call LLM agents optimize for precision over coverage to earn operator trust DevOpsDays ZurichTL;DW
- Prioritize precision over coverage in agent-based ticket triage—teams request more coverage only after trusting high-precision automation, not before.
- Run agent actions in dry-run mode for extended periods before production deployment to build operator confidence and avoid rogue comments that worsen workload.
- Overfitting is a major risk when autogenerating skills for few tickets; keep humans in the loop during skill creation and maintain continuous feedback loops from live handling.
- Use cron jobs aligned with on-call shift starts so engineers see pre-filtered, relevant ticket queues rather than accumulated noise from the previous shift.
- Agents should only read production data (logs, monitoring) and create change lists—never mutate infrastructure without human oversight and monitoring.
- Cultural shift needed: on-callers must validate agent responses are adequate, not just fix alerts; responsibility expands from alert management to response quality.
- Frame ticket automation as temporary band-aid that frees engineering capacity to fix root causes, not as permanent noise-handling solution.
- Start slow, iterate quickly on team feedback, and continuously deliver small wins to maintain adoption momentum without losing trust through speed or slowness.
- Prepare for adoption success early with self-service approach and canned response templates—four engineers spent mornings answering adoption requests unprepared.
- Collaboration across teams and alignment on shared values (eliminating soul-crushing work) drives better results than individual "winning" in AI implementation space.
TL;DW
- Prioritize precision over coverage in agent-based ticket triage—teams request more coverage only after trusting high-precision automation, not before.
- Run agent actions in dry-run mode for extended periods before production deployment to build operator confidence and avoid rogue comments that worsen workload.
- Overfitting is a major risk when autogenerating skills for few tickets; keep humans in the loop during skill creation and maintain continuous feedback loops from live handling.
- Use cron jobs aligned with on-call shift starts so engineers see pre-filtered, relevant ticket queues rather than accumulated noise from the previous shift.
- Agents should only read production data (logs, monitoring) and create change lists—never mutate infrastructure without human oversight and monitoring.
- Cultural shift needed: on-callers must validate agent responses are adequate, not just fix alerts; responsibility expands from alert management to response quality.
- Frame ticket automation as temporary band-aid that frees engineering capacity to fix root causes, not as permanent noise-handling solution.
- Start slow, iterate quickly on team feedback, and continuously deliver small wins to maintain adoption momentum without losing trust through speed or slowness.
- Prepare for adoption success early with self-service approach and canned response templates—four engineers spent mornings answering adoption requests unprepared.
- Collaboration across teams and alignment on shared values (eliminating soul-crushing work) drives better results than individual "winning" in AI implementation space.
Maria Henrika Peetz details how Google automated repetitive ticket triage by targeting only well-understood ticket types where high precision is achievable—fetching logs, checking monitoring—while ignoring the rest. Dry-run periods showed premature agent actions eroded trust, making precision the primary metric over speed or coverage.