Chapter 14: Introducing Evaluation
| Web Resources
| In-Depth Activity Comments
| Teaching Materials
The main goals of this chapter are to accomplish the following:
- Explain the key concepts and terms used in evaluation.
- Introduce a range of different types of evaluation methods.
- Show how different evaluation methods are used for different purposes at different
stages of the design process and in different contexts of use.
- Show how evaluation methods are mixed and modified to meet the demands of evaluating
- Discuss some of the practical challenges of doing evaluation.
- Illustrate through short case studies how methods discussed in more depth in Chapters
8, 9, and 10 are used in evaluation and describe some methods that are specific to
- Provide an overview of methods that are discussed in detail in the next two chapters.
Imagine that you designed an app for teenagers to share music, gossip, and photos. You prototyped
your first design and implemented the core functionality. How would you find out
whether it would appeal to them and whether they will use it? You would need to evaluate
it—but how? This chapter presents an introduction to the main types of evaluation and the
methods that you can use to evaluate design prototypes and design concepts.
Evaluation is integral to the design process. It involves collecting and analyzing data
about users’ or potential users’ experiences when interacting with a design artifact such as
a screen sketch, prototype, app, computer system, or component of a computer system. A
central goal of evaluation is to improve the artifact’s design. Evaluation focuses on both the
usability of the system (that is, how easy it is to learn and to use) and on the users’ experiences
when interacting with it (for example, how satisfying, enjoyable, or motivating the
Devices such as smartphones, iPads, and e-readers, together with the pervasiveness of
mobile apps and the emergence of IoT devices, have heightened awareness about usability
and interaction design. However, many designers still assume that if they and their colleagues
can use a product and find it attractive, others will too. The problem with this assumption
is that designers may then design only for themselves. Evaluation enables them to check that
their design is appropriate and acceptable for the target user population.
There are many different evaluation methods. Which to use depends on the goals of the
evaluation. Evaluations can occur in a range of places such as in labs, people’s homes, outdoors,
and work settings. Evaluations usually involve observing participants and measuring
their performance during usability testing, experiments, or field studies in order to evaluate
the design or design concept. There are other methods, however, that do not involve participants
directly, such as modeling users’ behavior and analytics. Modeling users’ behavior
provides an approximation of what users might do when interacting with an interface; these
models are often done as a quick way of assessing the potential of different interface configurations.
Analytics provide a way of examining the performance of an already existing
product, such as a website, so that it can be improved. The level of control on what is evaluated
varies; sometimes there is none, such as for studies in the wild, and in others there is
considerable control over which tasks are performed and the context, such as in experiments.
In this chapter, we discuss why evaluation is important, what needs to be evaluated, where
evaluation should take place, and when in the product lifecycle evaluation is needed. Some
examples of different types of evaluation studies are then illustrated by short case studies.