Our survey scores look fine. Can I trust them?
Surveys and Their Limits
Response rates, safety, framing and timing all bias the result — usually in the reassuring direction. How to read a survey sceptically.
Procedure
Surveys are the standard instrument and they are systematically biased towards good news. Knowing how helps.
Why results skew positive
The most affected people are least likely to respond. Exhausted and disengaged staff skip surveys.
Those who have already decided to leave have no reason to participate.
People who do not believe it is anonymous answer safely.
People who have seen previous surveys produce nothing stop bothering, and their absence removes the strongest negative signal.
Selection compounds. Each round, the sample becomes more favourable while the underlying reality does not.
Reading the response rate
A falling response rate is a finding, frequently more informative than the scores.
Segment it. Low response in one team is a signal about that team.
Below a threshold, treat the results as unreliable rather than as good news. Where that threshold sits depends on your size, and it should be decided before you see the results.
Making the instrument better
Short. Six to ten questions, answered in three minutes. Long surveys reduce response and quality simultaneously.
Specific. "My workload is manageable most weeks" is actionable. "I am satisfied at work" is not.
Consistent wording between rounds, or you cannot compare.
Free text, which carries most of the useful content. Read it rather than running sentiment analysis on it.
Credible anonymity. State the minimum group size for reporting, who sees raw responses, and what is done with free text. Then hold to it.
Timing
Avoid the week after a reorganisation unless the reorganisation is what you are measuring.
Avoid the peak season, which measures the peak rather than the baseline — unless that is deliberate.
Same time each year for comparison.
Frequency matched to your ability to act. Quarterly surveys with annual action produce fatigue and cynicism.
Pulse surveys
Advantages: faster signal, shorter, easier to sustain.
Risks: survey fatigue, and an implied promise of rapid response that most organisations cannot meet.
Use them for a small number of tracked questions, not as a substitute for the fuller instrument.
Stop them if nothing is being done with the results, which is more honest than continuing.
What surveys cannot tell you
Why. They locate the problem, not the cause. The follow-up conversation does that.
What individuals experience. Averages conceal the person who is struggling most.
Whether a specific manager is the issue, reliably, in small teams.
The severity of anything. A score of six out of ten does not distinguish mild dissatisfaction from serious distress.
The complement
Talk to people. Structured conversations with a sample of staff, run by someone they can be candid with, produce more usable information than any survey.
Use the survey to decide who to talk to and what to ask about.
Combine with the leading indicators, which are behavioural and not subject to response bias. Where the survey says workload is fine and the hours data says otherwise, believe the hours data.
Reading free text properly
The free-text responses carry most of the useful content and are usually the least well handled.
Read them. All of them, by a person, at least once.
Do not run sentiment analysis instead. A score of negative-0.4 discards the entire content.
Code them by the six areas, which converts a pile of comments into a distribution.
Count the specific and the repeated. One person's detailed account of a broken process is worth more than fifty generic comments.
Quote them back, anonymised and with permission where identifying, when reporting results. Nothing demonstrates that the survey was read like accurate quotation.
Protect anonymity carefully, since free text is identifiable in a way ratings are not. State how you will handle it before people write.
Deciding the threshold before you look
A discipline that prevents a common self-deception.
Decide, in advance, what response rate makes the results usable.
Decide what score would trigger action, before seeing any.
Write both down and share them with whoever commissioned the survey.
Then hold to them. The temptation to reinterpret a disappointing result is strong and it is visible to everyone who participated.
A survey with a pre-agreed action threshold is a different instrument from one where the response is decided after the numbers arrive.