A QA program built from the rubric up
Designed the QA framework for a 24/7 omnichannel operation: rubrics, weights and scoring philosophy, running at up to 500 reviews a week on AI QA platforms, with coaching built on the results.
Latest rubric revision lifted quality scores 5%; frameworks adopted company-wide
The knot
Most QA programs start from a scorecard somebody found online. They reward the wrong behaviours, the scores don’t predict anything useful, and agents learn to game them within a month. The multilingual accounts added another problem: how do you score quality consistently across 4 to 16 languages when the reviewers speak two or three of them?
What I did
- Designed the account’s QA framework from scratch: rubrics, weights, and a written scoring philosophy, so that two reviewers looking at the same ticket land on the same score.
- Scaled the program to up to 500 reviews a week across chat, tickets and voice, on Observe AI, Level AI and Klaus.
- Built the coaching loop. QA data drives structured coaching for 11 team leads and performance management across 100+ agents.
- Earlier, as the company’s multilingual SME, designed language-specific QA methodologies for programs in languages the reviewers didn’t speak. Those templates stayed in use after I moved on from the accounts.
- Added LLM-assisted scoring, plus first drafts of rubrics and coaching summaries, all of which I review and finalise.
What changed
The latest rubric revision lifted quality scores by 5%. The frameworks were adopted company-wide.
What I’d tell someone doing the same
Write the scoring philosophy first and the rubric second. If two reviewers can read the same rubric and score the same ticket differently, the rubric needs more work.