One of the main ways of finding out what
users like and don't like about a product is by asking them,
which can be done using interviews or questionnaires as
we discussed in Chapter 13.
One of the benefits of using questionnaires is that the
same questions can be given to many people at the same time.
Questionnaires are also relatively inexpensive to distribute
and analyze, particularly web-based questionnaires, which
are becoming increasingly common. Web-based questionnaires
also have the advantage that data can be stored directly
into a database for analysis.
Gary Perlman's website (www.acm.org/~perlman/question.html)
provides eight online questionnaires that you can use. These
questionnaires are based on the work of experts in human-computer
There are two ways of using the questionnaires:
- The most straightforward way is to use
the questionnaires as they are without changing any code.
You simply select the questionnaire that best suits your
needs and either ask the participants to complete it online
and email their data to you or print the questionnaire
and distribute it on paper.
- You can customize the questionnaires
by changing the cgi code and there are instructions for
doing this on the website. The advantage of this approach
is that you can decide what the questions should be and
you can distribute the questionnaire from your own university
server, which gives you more control over the process.
To do this you'll need some technical expertise.
This activity encourages you to review the
questionnaires before using them for an evaluation. Start
by going to the website (www.acm.org/~perlman/question.html)
and explore and compare the eight questionnaires. Try completing
some of them and send the data to yourself. How do they
differ? Here are some things that we noticed, you may find
- The questionnaires are derived from different
sources but they all use various forms of Likert scales.
- The number of questions varies from 3
to 100. When might you want only 3 questions and when
might you want more? If you are planning an evaluation,
which are the most appropriate questions for your evaluation
and what is the ideal number?
- Notice that you can add comment boxes
by clicking on the little icons or by requesting that
comment boxes are added to all the questions. The QUIS
questionnaire also has 6 blank spaces for questions.
- There are differences between the scales;
some are 7-point and some are 9-point scales. Most scales
start from 1 but the QUIS scale starts from zero. Most
provide a NA (not applicable) option too. The higher the
number on the scale the more positive the rating. Some
of the questionnaires also have semantic labels such as
'bad' and 'good' to show the direction of the scale. QUIS
uses pairs of adjectives and places one word at each end
of the scale. All the questionnaires repeat the scale
for each group of questions so that users don't get confused.
- Some of the questionnaires invite users
to write additional comments in a text box at the end.
Also notice that there are useful links
to papers or abstracts written by the questionnaire authors
and to additional information about questionnaire design.
For example, take a look at the link to 'Frequently
Asked Questions about Questionnaires in Usability Engineering'
provided by Jurek Kirakowski.
When you've finished examining the questionnaires
scroll to the bottom of the page and read about how to modify
the questionnaire and look at the example of the data format
that is emailed for one of the questionnaires
In this activity you will compare two different
questionnaires. Select two questionnaires of your choice
to compare and then ask 3 participants to complete each
questionnaire for an evaluation of one of the following
items: a VCR, a computerized toy, a clock, the home page
of a website, or something else of your choice. In order
to make sure that the participants experience the product,
you should suggest a few tasks for them to do with it before
completing the questionnaires. For example, insert and play
a video, rewind it, set the VCR to record a program. When
the questionnaires have been completed compare the data
that each provides. Which questionnaire provides the most
useful data about the product's usability and why do you
think this? Which questionnaire did the participants prefer,
In this activity you will compare the efficacy
of the Nielsen Heuristic Evaluation (NHE) questionnaire
with standard heuristic evaluation. To do this, start by
selecting a product, such as those mentioned in activity
2, and ask at least 3 typical users to evaluate it using
the questionnaire. As in activity 2, you should suggest
tasks that the participants should do with the product so
that they experience it. Then ask one or two expert evaluators
to use the same heuristics to evaluate the same product
but this time give them just the list heuristics (i.e.,
not in questionnaire form) and ask them to follow the process
for heuristic evaluation described in Chapter 13. You can
use peers from your class as experts but if you do this
try to get people outside of your class to participate in
the questionnaire part of the activity. Do the two groups
(users and experts) report on the same issues or are their
responses quite different? What are the advantages and limitations
of heuristic evaluation verses collecting data with the