2025 · evals, prompt engineering, cost

Support triage,
57% ? 96%

The support team had turned the AI triage off because it was wrong too often. Three support staff were sorting every ticket by hand. This is how it came back, how it was measured, and what it freed up.

The problem

Fifteen triage categories. The earlier classifier agreed with humans 57% of the time, basically a coin flip. So the team didn’t trust it, and looked at every single ticket. That meant three full time employees, plus the heads of multiple departments.

The eval set first

Before touching the prompt: a golden set of 50 human labeled tickets for fast iteration, then 200 for sign off, stored and scored in Langfuse.

What changed

Major prompt iteration, plus few shot examples chosen from the misclassified tickets. “No response necessary” was a huge challenge, because it was so broad.

Later added automatic categorization of uploaded images including bodyweight scales and prescriptions.

Each change ran against the 50 set; anything that held went to the 200 set.

Cost

After accuracy held, I slimmed the input removing irrelevant IDs and data. Token use fell 25.9% with no regression on the golden set.

What I would tell another team
Build 2+ evals before the prompt: small important set, larger broader set. Keep testing small set every iteration, and larger set at least daily.