In the first (and longest) of three separate blog posts about how to get better answers from asking better questions, I’m going to look at eight key characteristics that make up a good question.
The list comes from reflecting on why some questions achieve more insight, and why others do not. Through numerous digital projects I know that better questions form the starting point which leads to creating more effective and innovative digital products and services.
It is by no means intended as a definitive list. However, I find it a useful reminder and checklist when writing and reviewing questions to ask in research sessions. The characteristics work with all kinds of research activities from writing tasks for usability testing, when coming up with stakeholder interview questions, in facilitating workshops and conducting user research.
A good question requires time to think about it, to write and to iterate upon. The extra effort is worth putting in for the greater information you will gain from your user experience research activities.
What makes a good question? Eight factors to consider:
1. A good question ends in a question mark.
It may sound obvious but when reviewing facilitation guides (and listening to interviewers on the radio) I’m amazed by how often non-questions creep in. A statement with a question mark appended to the end does not magically turn it into a question.
Check your facilitation guide / survey / interview questions to make sure they start with or include either a: Why, How, Where, Who, What or When.
As a rule of thumb:
- Ask ‘what, when, where, who’ to gather fact and context.
- Ask ‘how do you’ to uncover processes and flows.
- Ask ‘how might we’ to investigate opportunities.
- Ask ‘why’ to find underlying reasons and motivations.
2. A good question has a purpose.
You can only evaluate how good your questions are if you have something to benchmark them against. Write out and share the purpose for the research (a problem statement, a hypothesis, a list of information to gather) and then check your questions are helping you answer it.
If you don’t know why you are asking a question, delete it. Not all questions have to directly support the research outcomes. For example, you might ask questions to ease a participant into the session. However, knowing why you are asking these types of questions will help you limit the time you spend on them.
The time for carrying out research and analysing results is always limited. Knowing your most essential question(s) will focus your time where you feel it is most valuable.
Another advantage of knowing your key question(s) is that you’ll be ready to ask it to the right person should the occasion arise. I’ve had times where I’ve asked my key question to a CEO whilst sharing a lift. I’ve got some great answers from being been in a queue and starting to talk to someone who matched the target customer group I was researching.
3. A good question gives insight that is actionable.
It’s easy to take up too much time in a research session enjoying a chat. Remember why you are there. Spend your time getting answers that inform your thinking and can be shared to empower the whole project team.
It may sound counter-intuitive but don’t ask interviewees for design solutions. This is unfair on the them. They are not designers. They haven’t got the chance to think deeply about the issue. They don’t know any of the context or constraints of the project.
Instead use their time to help you more deeply understand the tasks they need to get done, how they currently go about tackling them and what pain points they encounter.
4. A good question opens up a conversation.
How you ask something will impact on how it is answered. In general, ask divergent questions that open up ideas rather than convergent ones that close down the conversation.
Closed and close-ended questions aren’t wrong. Sometimes you need to establish facts and to clarify. Just make sure you are not inadvertently using them.
Closed questions — ones that can be answered with a ‘yes’ or ‘no’ or with a direct answer, will stop the flow of any conversation. For example, ‘Do you like to drink tea in the morning?’ or ‘What was the last book you read?’ will get you facts but little understanding without further questioning.
Asking a close-ended question such as ‘Do you prefer tea or coffee?’ might make it seem like you are getting good data on drinking habits, but it doesn’t allow you to discover any of the alternative possibilities.
5. A good question is neutral and free of bias.
Watch out for leading phrases or charged words. Starting a question with ‘Would you agree . . ?’ or ‘Do you think . . ?’ tells the interviewee how you would like them to answer.
Likewise, be on the look out for charged words that give the interviewee a clue as to how to respond. If you ask: ‘Tell me what you think of our exciting new design?’, you are making it difficult for the interviewee to offer feedback that disagrees with your assumption that the design is exciting and new.
It is easy to accidentally introduce bias into your questions. Un-spotted, it proves toxic for the validity of your findings. Look out for phrases and words that are charged with bias and re-write your questions to be more neutral.
6. A good question is interesting.
Ask these simple questions of your questions: is it worth asking? Is it worth answering?
As you have to listen to multiple people respond to the same questions, make sure they are both interesting to ask and to answer.
You’ll know if you have a boring question because you’ll have a little internal sigh before you ask it; the interviewee won’t get physically or vocally animated as they answer, and you’ll write fewer notes about it.
If this happens you should re-write the question or drop it from your script. If the opposite happens and an interviewee says ‘that’s a good question’, take note and re-use in other projects.
7. A good question is short.
Being interviewed is difficult. You have to listen, comprehend and then respond on the spot to questions which we interviewers have often thought about for a while.
Short questions are easier to listen to and understand (and analyse, as the answers tend to be more focused). One way to make questions shorter and easier to answer is to look out for some key warning signs:
- Too many connectives such as ‘and’
- Commas
- Too many words
When you have a compound sentence — the ones with multiple thoughts connected by an ‘and’ — interviewees tend to just answer the second part.
Multiple commas in a question are an indicator of subordinate clauses. These are hard to read and prove even harder to listen to.
The longer the sentence, the greater the mental strain in listening to it. As a guideline you should review any question containing over 12 words to see if it can be simplified.
The writing guru Ann Wylie has carried out a study to evaluate the effect the number of words in a sentence has with reader comprehension. Although the study investigated reading words, the findings offer a guide that can be used for aural comprehension. The study for the American Press Institute was based on articles from 410 newspapers. It reported:
- When the average sentence length was fewer than eight words readers understood 100 percent of the story.
- With 9–14 word sentences readers could understand more than 90 percent of the information.
- With 29 word sentences comprehension dropped to less than 50 percent.
Of course, other factors such as the complexity of the words and the knowledge of the reader come into play. However, in my experience shorter questions tend to get better answers.
8. A good question can be answered.
Again, it sounds obvious but I’ve seen many experienced interviewees get tripped up with this in a number of ways.
Ask questions based on the interviewees’ experiences and behaviours.
It is very tempting when designing for the future to ask the interviewee to imagine it for us. You’ll leave the research session thinking you have answers. However, when you re-visit your notes the ideas captured often don’t stand up to scrutiny or help you understand the why behind the what. Avoid asking for conjecture or future predictions. Base questions in ‘What do you do?’ over ‘What would you do?’.
And one last point: get people to show you rather than tell you.
The best questions move away from opinion and allow you to observe behaviour. This is the strength and essence of observed user testing. But even in interviews it is possible to base responses in reality by finding questions that allow people to give you examples from real situations.
Be careful of jargon in any form (client, sector, UX & digital words).
I’ve had sessions where participants look confused when I’ve asked them to ‘return to the homepage’ or to ‘use the main navigation’. It is amazing how quickly you adopt words from the digital world and your client’s business. Be careful as any time you make the interviewee feel stupid, you reduce the chances of them opening up and giving you deeper insight.
What factors do you think make a good question?
I hope you enjoyed the list and you find it useful when thinking about your next research session. I’d be interested in reading what factors or advice you would add. Please use the comments below to join in the conversation.
This article was originally written by Chris How and first published on the blog of The Unit, a digital agency based in Brighton.
