A QA program built from the rubric up
Designed the QA framework for a 24/7 omnichannel operation: rubrics, weights and scoring philosophy, running at up to 500 reviews a week on AI QA platforms, with a weekly coaching loop.
Latest rubric revision lifted quality scores about 5%; frameworks adopted company-wide
The knot
A QA scorecard is only useful if it rewards the right behaviors and its scores predict something. Otherwise agents learn to game it, and the scores stop meaning anything. The multilingual accounts added another problem: how do you score quality consistently across 4 to 16 languages when the reviewers speak two or three of them?
What I did
The rubric
- Started from a written scoring philosophy: QA exists to show the team what good service looks like, and how to get there.
- Designed the rubrics on that basis. I’ve used both “tri-nary” scoring (yes, no, and not-applicable, so items that don’t apply drop out of the count) and graded scales.
- Grouped the items into three sections, customer experience, issue resolution and internal work (notes, profiles, contact reasons), and weighted issue resolution highest, 40% against 30% and 30%. Policy lives inside issue resolution. Within a section every item weighs the same.
- Wrote every item as a short checklist of plain questions. If any answer is wrong, the item is marked “no”. Two reviewers reading the same checklist should land on the same score.
- Moved critical fails from single items to a judgment at section level. Instead of one missed box zeroing a review, the reviewer looks at everything already scored in that section and decides whether the problems add up to a fail.
- Kept time management separate from AHT. AHT depends on queue volume, case mix and luck of the draw. The rubric asks a different question: was the time spent on this case used well, however long it was?
- Showed agents a band (exceeds, meets, needs improvement, unsatisfactory) rather than a bare percentage.
Running it
- Scaled the program to up to 500 reviews a week across chat, tickets and voice, on Observe AI, Level AI and Klaus.
- For the multilingual problem, interviewed native speakers of each language and distilled their input into a per-language rulebook of quirks and conventions. Reviewers then score an AI-assisted English translation against that rulebook. Punctuation was the hardest thing to score fairly across languages; the rulebooks are what made it possible. Those templates stayed in use after I moved on from the accounts.
- Built the coaching loop. Every week, each person got a side-by-side list: areas where they were excellent and areas to improve, with a concrete action and a plain explanation for each improvement item. Once a month there was a live call to talk it through.
- Added LLM-assisted scoring, plus first drafts of rubrics and coaching summaries, all of which I reviewed and finalized.
What changed
The latest rubric revision lifted quality scores by about 5%. The frameworks were adopted company-wide.
What I’d tell someone doing the same
Write the scoring philosophy first and the rubric second. If two reviewers can read the same rubric and score the same ticket differently, the rubric needs more work.