Untangle

All work

CSAT up 20 points after a survey redesign

Redesigned the survey methodology to remove cultural bias, then measured the effect with A/B tests on how and when the survey was delivered.

CSAT up 20 points, sustained at 75–80%

Role
Designed the methodology, ran the tests, owned the number with the client
When
TODO: year
Tools
CSAT, Survey design, A/B testing, Kustomer

The knot

CSAT on the account had been flat for a long time, and the usual response to a flat CSAT is to push the agents harder. Before doing that, I wanted to know what the number was measuring.

CSAT tracks how customers feel about an interaction. Internal quality tracks whether the agent did the work correctly. The two are related but they move independently, and the survey we were using had a cultural bias built into it: customers from different backgrounds were reading the same question and answering on different scales.

What I did

  • Separated the two measures. Internal quality is scored by the QA program. CSAT is designed and tracked on its own, as a measure of customer perception.
  • Redesigned the survey methodology to remove the bias.
  • Ran controlled A/B tests on survey delivery, including timing and how the survey was introduced, so the change was measured rather than assumed.
  • Rebuilt the survey flow in Kustomer to match.

What changed

CSAT rose 20 points and has held at 75–80% since.

What I’d tell someone doing the same

Before coaching anyone on a CSAT number, check what the survey is measuring and who it’s reaching. In our case a good part of the gap was in the instrument.