We recently announced our launch as the first Agentic Support Platform, built to help humans and agents work together to resolve tickets. Today, we’re taking a closer look at Support Agent, the AI that works to fully resolve an issue before it reaches a human.

Everything below is a retro from our own production queue earlier this year, mostly February and March. Pylon runs customer support on Pylon, so the agent in this post is the same one we ship. It answers real tickets from real customers, most of whom are support teams that know exactly what an AI agent should have been able to do. That makes our production queue an unforgiving test.

Unlike simpler B2C cases that can be handled with a quick lookup or deterministic runbook, B2B issues often require more complex action. We had a deadline at the end of March to meaningfully improve our own resolution rate. We categorized every issue Support Agent could not resolve, then tackled the categories in order of frequency.

The biggest category was the expected one. Knowledge gaps accounted for more than half of escalations. The second was feature requests. That category was interesting because no article can solve it.

What our own queue is actually made of

A resolution rate tells you how you're doing, but it doesn't tell you what to build. So we started a weekly report that took every escalated ticket from the previous seven days and asked a different question: Why did this one get handed to a human?

In our baseline week (ending Feb 28), we categorized roughly 200 escalated tickets by reason. Two things stood out.

1. Most of it was a supply problem. More than half of these escalations happened because our knowledge base simply didn't cover the topic. That's fixable.
2. A quarter of these were never winnable. Bug reports need an engineer to read a stack trace. Account actions need privileged access. When a customer opens a thread by tagging a specific teammate, escalating isn't a failure; it’s the correct behavior.

Escalation reasons, not the resolution rate, are the actionable metric.

Lever one: The missing knowledge

Knowledge gaps went first, and the fix was unglamorous. We generated bite-sized articles from the past ninety days of already-resolved tickets, answers our team had already written, restructured so an agent could retrieve them. We backtested them against held-out issues, then shipped them. Knowledge-gap escalations fell from about 100 a week to a low of 26, on flat-to-rising ticket volume.

"We backtested by running a new week's issues against an agent with the newly generated articles. This gave a more accurate predictor of future resolution than if we tested on the exact same resolved tickets, which would just be "memorizing the test."

Why generate articles at all, instead of pointing the agent at past tickets directly? The Support Agent replies to customers on its own, so by design it doesn’t have access to other customers' conversations. Reviewed articles let that knowledge reach the Support Agent after an initial sanitization pass.

Our human-assisting flows work differently. Assist Agent references conversations indexed across all accounts in Pylon, and exposure is limited because the person reading them is a user on our own tenant. We'll dive deeper into agentic security in a future post.

Lever two: The category no document could fix

What happens when you fix your biggest problem? The second-biggest one becomes the new biggest problem.

Feature requests were 15% of escalations, and no amount of knowledge base content was going to help. There’s no help center article that answers “are you building a Google Chat integration?” Even if we wrote one for each missing feature, it would go stale the moment the roadmap changes. Writing one is how you teach your AI to confidently tell customers something untrue.

So we shipped one tool, add_evidence_to_feature_request, that works alongside Pylon's Product Intelligence suite. When the agent recognizes a feature request, it searches our existing feature requests, attaches the customer's ticket to the matching one as evidence, and tells the customer their request has been logged against real demand. What used to need a teammate to read the ticket, find the right feature request, and write back now happens automatically.

In its first full week the tool resolved 22 tickets, 16.5% of everything the agent resolved that week. Escalated feature requests fell from 28 to 15.

Some are a logged request. Some are a straight no, with a reason and a workaround. You can't document a roadmap, but with Pylon's Support Agent, you can route a request.

The results over time

Over seven weeks, our own Support Agent, running in production on Pylon's live support queue, nearly doubled its resolution rate.

Throughout this period, “resolved” means the agent closed the ticket without a human. That's either a confirmed resolution or a case where the customer stopped replying and it auto-closed, which is most of it.

We can also adjust the denominator to ignore tickets no AI should be closing (bug reports, tagged teammates, vendor auto-replies to our own maintenance email). Doing that puts our final week's rate at 142 of 283 addressable tickets, or 50.2%.

What's next

We hit our goal. The resolution rate crossed 40% the week ending March 28 and settled at 40.9% by mid-April, nearly double where we started. Not bad for the first line of defense before a ticket reaches our human support team. The more useful result was that the queue itself changed shape. Knowledge gaps fell from more than half of all escalations to 29%, and future iterations like continuous knowledge updates should push that further.

What rose in their place was bug reports, up from 8.5% of escalations to 19%. Those need an engineer. Account actions need someone with permission to change a setting. So our attention moved to the other half of the handoff, making the human faster rather than removing them. That work is handled by our new Assist Agent, in beta, and it's worth its own deep-dive series.

To recap:

1. Categorize your escalations before you build anything. A large share were never yours to win, and knowing which ones stops you from optimizing a number that can't move.
2. Auto-fill knowledge gaps, but gate the output behind human review. Generation multiplies fast. Your agent will happily read a thousand mediocre articles, but your team has to live in there.
3. For the categories knowledge can't fix, give the agent an action, not a better answer.