A self-improving customer-support agent: it learns from resolved tickets, hands off what it should not answer and reviews its own mistakes every week
A mid-size software-enabled services company of about 300 employees, a leader in its segment.
Reviewed by Christopher Clayton, CEO
- Scale
- ~2,200 tickets a month, 1 agent, 6 ticket categories, 3 escalation rules, 1 weekly mistake review
- Engagement
- 5 months, with a three-week shadow period before it answered anyone
The challenge
The support team handled about 2,200 tickets a month, and most were repeats: password resets, invoice copies, how-to questions and status checks. A first chatbot from a vendor answered from a static knowledge base, went out of date within weeks and sometimes answered confidently and wrongly. The company wanted an agent that improves with use, but not one that could promise refunds, change accounts or give an answer nobody had checked.
Support work is repetitive until it is not. Most of the 2,200 tickets this company received each month were questions it had answered many times, and a few were the kind that could cost a customer or a contract. The first chatbot treated them all the same: it answered from a static knowledge base that drifted out of date and sometimes sounded sure while being wrong. The company asked for an agent that gets better with use and knows what it must not answer.
The project began with a three-week shadow period. The agent drafted replies to live tickets, the team compared them with what was actually sent, and the first accuracy figure came from real cases before any customer saw an answer. After that, each closed ticket with a human-approved reply became a candidate article, and a support lead reviewed it before the agent could rely on it. The agent had read access to the knowledge base and ticket history and could draft or hand off, but it had no access to billing or account changes.
Escalation was designed rather than hoped for. The agent hands off whenever a customer asks for a refund, raises a legal or security matter, asks for a person, or when its own confidence is low, and the handoff carries a summary so nobody repeats the story. Once a week the agent reviews its own mistakes, listing reopened tickets, down-rated replies and human corrections, explaining what went wrong and proposing a fix to an article or a rule. A support lead approves or rejects each fix.
After five months about 71 percent of tickets were resolved without a human, up from 62 percent, and the reopen rate stayed under 6 percent. The less obvious gain was speed of correction: an outdated invoice instruction that would once have lingered for months was caught in the next weekly review. This illustrative case shows the pattern: a narrow agent, a person who approves what it learns and a routine that makes its mistakes visible.
What our team delivered
- A shadow period of three weeks in which the agent drafted replies to live tickets and the team compared them with what was sent, so the first accuracy figure came from real tickets before any customer saw an answer
- Learning from resolved tickets: each closed ticket with a human-approved answer becomes a candidate knowledge article, reviewed by a support lead before the agent may rely on it
- Narrow tools: read access to the knowledge base and ticket history, a draft-reply tool and a handoff tool; no direct access to billing or account changes, which stay with staff
- Three escalation rules: the agent hands off when the customer asks for a refund, a legal or security matter, or a person, and when its own confidence on the answer is low, always with a summary so the customer does not repeat the story
- A weekly self-review: the agent lists tickets where a customer reopened, rated the reply down or a human corrected it, explains what it got wrong and proposes a fix to an article or rule; a support lead approves or rejects each proposal
- Reporting for the support lead: resolution rate, handoff reasons, reopen rate and which articles are used most, so the team sees where the product, not the agent, is the problem
Result
After five months the agent resolved about 71 percent of tickets without a human, up from 62 percent in the first month, while the reopen rate stayed below 6 percent and handoffs arrived with a summary. The weekly review caught recurring errors, such as an outdated invoice instruction, within days instead of months. Every change to what the agent knows passed a person first. This is an illustrative engagement that describes the kind of work our team does; details are composite.
Get this for your company
Related services with fixed prices and a written scope.
AI agent bot, standard
One bot for a task that crosses two systems or handles varied documents, with exception handling and a review queue for your team.
About 3 to 4 weeks
$5,900 per bot
Microsoft Copilot and AI readiness
Before you turn Copilot on, find out what it will be able to see, and fix the permissions first.
Report within 8 business days
$2,400 per tenant
Scanning of software codebase
Static analysis and secret detection across one codebase, reviewed by an engineer so you get findings, not noise.
Report within 3 business days of read-only access
$500 per codebase
Show 1 more
Discovery and requirements workshop
Workshops with your team to map every system, process, user group and integration, and to agree what the new platform must do.
Written requirements and a system inventory in 2 to 3 weeks
$6,500 fixed fee
After you purchase, we contact you within 1 hour, any time of day or night, to start onboarding.
No payment is taken online. When we call you within the hour we confirm the scope and send an invoice (ACH, card or crypto).
Five million events, three escalations
What a managed SOC does to a day of log volume for a 250‑seat organisation. Collection produces millions of events; correlation rules turn them into a thousand or so alerts; triage merges and dismisses most of those into dozens of incidents; only a few need a human on the customer side. Bar length is the logarithm of the count, so each step is orders of magnitude. Illustrative figures.
Three approval tiers for AI agents
Every action an agent can take falls into one of three tiers: it runs on its own, a named person approves it first, or a human always does it. The examples are illustrative; each firm draws its own lines during scoping.
Every agent tool call passes one checkpoint
Agents reach MCP servers only through a gateway that checks policy, enforces a per-agent scope allow-list and logs every call to the SIEM. A call outside the allow-list is blocked. A local stdio server started on a laptop skips the gateway, so it needs its own control.
Facing something similar?
Tell us about your environment. We will tell you plainly what we would do and what it would take.