Pilot survey options before asking people to answer

A survey question can sound neutral and still force people into answers that do not describe their situation. Its follow-up questions can introduce another problem by asking somebody to rate an experience they never had. Before distribution, walk a small set of invented respondent situations through the actual options and proposed branches.

Define the question's coverage

Write the information the question should collect and who is meant to answer it. Separate eligibility from the behavior you want to understand. Decide whether the question concerns everyone invited, everyone with access, or only people who attended. That choice affects which options and follow-ups make sense.

Fictional practice example: a survey asks how often someone joins a weekly session and offers every week, occasionally, and rarely. Invent reviewers who were never invited, were invited but never attended, attended once, and attended regularly. These are test situations, not real participants or survey results. Do not add invented personality details that distract from whether the question can represent their circumstances.

Answer using only the supplied options

Have reviewers select an option for each situation without assistance from the question's author. Ask whether the choice is honest, merely the closest available answer, or impossible. Record the question, persona state, selected option, and reason for uncertainty in an answerability register.

In the example, rarely may force never-attendees to imply some attendance. A never-invited person may need a separate eligibility response rather than a lower frequency. If regularly could mean different things, decide whether named intervals would communicate the intended distinction better. Also state whether one answer or several answers are permitted; overlapping options can cause trouble even when every situation has somewhere to fit.

Trace each follow-up path

Write where each answer leads and which information that next question assumes. Follow the entire path until its exit, including any skipped sections. Check that someone who never attended is not asked to rate the host's explanations, and that a person who attended once is not forced to describe a recurring pattern.

Record unreachable questions, circular routes, missing exits, and paths that contradict an earlier answer. A good first question does not repair a branch that collects inapplicable ratings later. If the survey platform has not been chosen, a written branch map can still establish the intended behavior without claiming that routing is implemented.

Revise categories and branches together

Add or clarify responses that represent the intended population, then update the follow-up map to match. Distinguish a substantive response such as never invited from a choice to decline answering. Neither should silently become a negative rating. Keep question wording and the category definitions together so a later editor can understand why an option exists.

Changes trade detail against effort. Too many narrowly separated options can make a simple question difficult to answer, while too few can collapse important situations. Retain distinctions that change interpretation or routing. Keep undecided cases visible rather than adding an other option and assuming every problem is solved.

Repeat the previously failed situation

Give the revised questions to another reviewer and rerun the unrepresentable persona from the original pilot. Ask them to complete the branch without explaining the intended route. Check at least one previously working situation too, because a new option can alter an existing path.

Close the register only when the selected situations have honest answers and appropriate exits. Keep unresolved cases alongside their next decision. This small exercise checks answerability and routing; it does not establish population coverage, response rates, or the reliability of a real survey's conclusions.