Griffiths - Lecture
Usability Evaluation by Query Techniques
These notes are based on the the description of query technique given in
Dix et al.
Query techniques involve questioning the user directly about their experience
of using the system under evaluation. This may be done face to face
as an interview, or in writing as a questionnaire.
Get user's viewpoint directly.
Reveal issues not considered by the designer.
Relatively simple (but note problems with questionnaire construction below)
and cheap to administer.
Information received is subjective, users may ‘rationalize’ events.
Difficult to get feedback on alternative designs if user has not experienced
Useful as supplementary method.
Effective for high level evaluation - preferences, impressions, attitudes.
Often structured in a top down style, starting with general questions
about the task, and moving on to more leading questions. (Why ...?
What if ...?)
Must be well planned in advance, with central questions prepared.
This may be important to ensure consistency between interviews from different
interviewees, and by different interviewers.
It is crucial that the designer of the system being evaluated does not
conduct the interview! They will be unable to avoid giving off 'body
language' signals which are likely to bias responses.
Level of questioning can be varied to suit the context.
Interesting issues can be probed as they arise.
Depending on degree of structure, encoding of results may be problematic.
Personality and style of the interviewer may effect response. (Respondent's
will generally seek to ‘please’ the interviewer.)
Design of the questionnaire is crucial. This requires that the following
must be clearly established.
"A questionnaire is a method for the elicitation, and
recording, and collecting of information." [Jurek Kirakowski
Purpose: what information is sought
Analysis: Measurable feedback, or subjective responses.
Compared with other usability evaluation techniques, questionnaires:
Reach a wider subject group.
Takes less time to administer.
Can be rigorously analysed.
Can be administered throughout the design process.
lists these additional advantages:
Gives feedback from the point of view of the user (and if based on a trustworthy
sample) the result will hold for the total population.
Measures gained are largely independent of the system, users or tasks to
which it was applied.
Less flexible than interview - questions fixed in advance.
Likely to be less probing.
lists these additional disadvantages:
Tells you only the user's reaction as perceived by the user.
Questionnaires are good for subjective measures, but performance measures
should be obtained by other methods.
The development of a reliable questionnaire is expensive, so a general
purpose questionnaire will not completely fit a specific situation.
The numbers produced from analysis need interpretation to be valuable.
An overriding disadvantage is that surveys that collect subjective data
are extremely time-consuming (and expensive) to develop. For them
to be meaningful, they have to be demonstrably reliable and valid.
The reliability of a questionnaire is its ability to give consistent
results when completed by people holding the same views under similar circumstances.
Reliability may be assessed using statistical methods, and is usually
expressed as a numeric value between zero (extremely unreliable) to one
The validity of a questionnaire is the level of confidence held
that it is actually collecting data (and measuring) what you believe it
should. This is not easy to determine!
Confidence in the validity of a questionnaire can be developed by demonstrating
its consistency with other measures for the quality that it is intended
to measure, e.g., other questionnaires or standards checklists.
For a through demonstration of how the issues of reliability and validity
can be addressed in questionnaire development, see Kirakowski's paper on
the development of SUMI.
Types of questions
According to Kirakowski (FAQ)
, for usability evaluation questionnaires there are three types of question,
relating to the information sought:
Factual-type questions: for objective data that it would be
uneconomic to obtain another way.
Opinion-type questions: directing the respondent's thought
outwards, basically determining the popularity of something.
Attitude questions: directing the respondent's thought inwards,
determining their satisfaction with an artefact or design detail.
categorizes attitudes towards use of IT systems the five categories:
The user's feeling of being efficient
The degree to which the user likes the system
How helpful the user feels the system is
To what extent the user feels in control of the interactions
Does the user feel they can learn more about the system by using it.
Additionally, questions can be categorized by their construction as:
Ask the user for unprompted opinion on a topic, e.g., “Can you suggest
any improvements on the layout?”
Useful for gathering subjective information, but difficult to rigorously
analyse, or compare - so should be viewed as supplementary. May also
be used in the exploratory phase of research.
May identify issues missed by the designer.
Objective factual information may be obtained.
May be analysed by the use of 'content analysis'.
Narrowly focused, with a factual or categorical (yes/no) reply, or chooses
from pre-set responses (which makes for easy coding).
Judge degree of agreement or disagreement of a statement against a scale,
The important information is easy to find?
Granularity of the scale may vary.
1-3 gives obvious interpretation of the numbers. A finer scale
give room for expression of levels of agreement etc.
A very fine scale (1-10) makes consistency problematic.
1-5 and 1-7 often used. Interpretation of intermediate values
may be given.
Use of an even number forces user to make a choice, but this may simply
result in random answers being given.
Respondent offered a choice of explicit responses to select from, e.g.:
How frequently do you use the system? (Tick
May be required to tick all that apply.
At least once a week
At least once a fortnight
At least once a month
Less frequently than once a month [ ]
Useful for gathering information on users’ previous experience.
User is asked to place an ordering on a list, and are useful for capturing
Please rank the way that you would like to obtain
help with using this package. (1 most preferred, 2 next etc.)
On-line help system [ ]
Commercial Usability Evaluation Questionnaires
There are two leading general usability evaluation questionnaires available
as commercial products: SUMI, developed by the Human Factors Group (HFG)
at University College, Cork, and QUIS, developed by the University of Maryland.
Additionally, HFG and Nomos Management AB, Sweden, have developed WAMMI
specifically for web site testing.
The Software Usability Measurement Inventory (SUMI)
has been normalized against a substantial international database of results.
The questionnaire currently consists of 50 scalar question statements
with a scale of three: Agree, Don't Know, Disagree.
It comes with a software analysis too which allows comparison with the
database on these scales:
The Questionnaire for User Interaction Satisfaction (QUIS)
is designed in a modular way, so that only sections dealing with particular
issues of interest may be administered.
It is constructed of scalar question statements with a scale of 0..9,
the zero being anchored with a negative adjective, and the nine with a
It is designed to measure of overall system satisfaction along six scales,
and contains hierarchically organized measures of eleven specific interface
terminology and system feedback,
An early version of this questionnaire is given in Shneiderman.
Web site Analysis and MeasurMent Inventory (WAMMI)
is a service for evaluating web sites on line. A button is placed
on the site to be evaluated. Users clicking on the button get the
questionnaire downloaded as a form. When completed, the data is stored
The data from questionnaires is analysed against a standardized database,
and normalized scores for the following scales are presented:
Dix, A., Finlay, J., Abowd, G. &
Beale, R., Human-Computer Interaction (2nd. ed.), Prentice Hall Europe,
Human Factors Research Group, University College
Cork, SUMI Home Page, http://www.ucc.ie/hfrg/questionnaires/sumi/
Human Factors Research Group, Ireland, and Nomos
Management AB, Sweden, WAMMI Home Page, http://www.nomos.se/wammi/
Kirakowski, Jurek, The Use of Questionnaire
Methods for Usability Assessment http://www.ucc.ie/hfrg/questionnaires/sumi/sumipapp.html
Kirakowski, Jurek, Questionnaires
in Usability Engineering: A list of Frequently Asked Questions (3rd ed.),
Shneiderman, Ben, Designing the User
Interface : strategies for effective human-computer interaction (3rd ed.),
University of Maryland, Office of Technology Commercialization,
QUIS Home Page, http://www.lap.umd.edu/QUIS/