CSAT up 20 points after a survey redesign
The survey was asking a question that meant different things to customers and to the team. Rewrote it, renamed the teams customers were being sent to, and measured the effect with A/B tests.
CSAT from 55–60% to around 75–80%
The knot
CSAT on the account sat at 55–60% for a long time, and the usual response to a flat CSAT is to push the agents harder. Before doing that, I wanted to know what the number was measuring.
The survey asked “Was your issue solved?” It turns out “solved” means two different things. Inside the team, a case is solved when it’s handled correctly and by policy. To the customer, it’s solved when they got what they wanted. A refund request handled perfectly, and declined because the policy didn’t allow it, was a “no” on the survey every time.
The answer options made it worse. Some of them implied outcomes, like a refund, that weren’t possible in that customer’s case, so the survey itself was setting expectations we couldn’t meet. And the survey offered several ways to say an interaction was bad but only one way to say it was good, which meant we knew what went wrong but never what to reproduce.
We had to learn to speak the customers’ language.
What I did
- Rewrote the question and the answer options around the interaction with the agent, rather than an outcome the customer may not have been entitled to.
- Added the answer “I was expecting a different answer”. Customers who were unhappy with the outcome, like a refund we couldn’t give, now had their own box, and we analyzed those responses separately.
- Added multiple positive options, so a good rating tells us what went right, not just that something did.
- Renamed the teams customers were transferred to. “Tier 2” and “Tier 3” became “Billing Specialists” and “Technical Support Specialists”. A customer being handed to a specialist feels helped; a customer being moved up a tier feels shuffled around an org chart.
- Asked chat agents to mention the survey during the conversation.
- Ran A/B tests on delivery timing, and on whether agents mentioned the survey or not. The agents who mentioned it scored noticeably higher.
- Separated the two measures for good. Internal quality was scored by the QA program. CSAT was designed and tracked on its own, as a measure of how customers felt.
What changed
CSAT went from 55–60% to around 75–80%, and stayed there for the rest of my time on the account.
Roughly 5 of those 20 points came from chat agents mentioning the survey, because chat carried most of our volume. The rest came from the new question, the new answers and the renamed teams.
What I’d tell someone doing the same
Before coaching anyone on a CSAT number, read the survey as a customer would. In our case a good part of the gap was in the question.