AI Support Triage — a production agent

What I built

I spearheaded an AI agent that took over support-ticket triage — work that was consuming roughly one in seven hours of the team's time. It runs the full loop in production: classify and route the ticket, investigate the live system, then draft the fix and the reply — with a human approving every action.

How it works

A two-call classifier decides what to do first — deliberately blind to the routing list — then, separately, who owns it. A gated investigator agent reads the live system, code, telemetry, and docs to draft an evidence-cited fix and reply. A person approves every call before anything goes out; it was trained on two years of resolved history and improves with every correction.

How I built it

Architecture: a rulebook is the single source of truth, served over both HTTP and MCP so humans and agents read the same rules. A two-key, fail-closed write policy gates every external write (chat / tickets) behind a human/agent switch — nothing auto-sends. The agent runs as a looped Claude-CLI process against that tool surface (FastAPI + MySQL + Docker).

How I got the accuracy up: a persona pod supplied routing expertise and independent judging, and I ran an optimization loop — blind-triage a locked set, score each call with a three-lens consensus against ground truth, then meta-optimize the rulebook — on 594 labeled tickets (80/20 train/holdout, the test set kept untouched). A negative result (adding a rich routing reference raised routing but collapsed branch accuracy) was found and reverted rather than shipped.

The whole thing is eval-driven: a golden set is the gate, and every human correction becomes a fresh training example.

Questions this project answers

How do you know an AI agent is actually good, and not just plausible-sounding? Against held-out ground truth, not vibes. I built a golden set of previously-resolved tickets the agent had never seen and measured handling accuracy against the known-good resolutions (~84%); routing accuracy improved from 54% to the mid-70s on a locked holdout. Every human correction during the beta became new evaluation data, so quality was tracked continuously. Operationally I instrument per-turn cost, latency, error rates, and traces, so failures are diagnosable.

When an agent is confidently wrong, what stops it from doing damage? A human sign-off gate on every external write, two-key and fail-closed, so nothing auto-sends. Autonomy expands only as the eval numbers earn it, an observe-then-enforce rollout rather than turning it loose. The hard part was never the model, it was trust: making the agent's behavior legible, catching the cases where it was confidently wrong, and feeding corrections back in.

Is this real production AI, or a proof of concept? In production. It ran the full loop, classify, investigate, draft the fix and the reply, in a team-wide, month-long beta, reclaiming roughly one in seven hours of the team's time with a human in control.

How do you keep an agent's cost and latency in check? The expensive part is gated. The investigator only fires when the classifier's confidence is low, so the deep evidence pass runs on the minority of tickets; the common path stays a cheap two-call classification.

Results

• 84% handling accuracy on golden-set evals.
• Routing accuracy climbed from 54% to the mid-70s on a locked holdout.
• Reclaimed roughly one in seven hours of team time, with a human in control.
• Reached a team-wide, month-long production beta running the full loop.