Agenda

• Introduction
• Usability Evaluation:
  • Heuristic Evaluation (HE)
  • Thinking Aloud Testing (TA)
  • Formal Experiments (FE)

Full Course Notes

• Human-Computer Interaction Course Notes [242 pages PDF]
  https://courses.isds.tugraz.at/hci/
https://courses.isds.tugraz.at/hci/hci.pdf
• Highlights from Chapters:

  1  Human-Computer Interaction
  2  The Psychology of Usable Things
  3  Usability Engineering
  
  8  Usability Inspection Methods (HE)
  
  9  Usability Testing (TA + FE)
  

Human Factors and Ergonomics

Traditionally:

• Ergonomics = design of work for human abilities and limitations.
• Human factors = human abilitites, performance, limitations
  (also non-work, more North American).
• Originally focussed more on hardware interfaces.

Shape Coding for Aircraft Control Knobs

• Dozens of gear-up accidents when landing B-17 aircraft.
• Pilots frequently retracted the landing gear after touching down.
• In 1943, Alphonse Chapanis investigated.
• Many identical control knobs next to one another.

Shape Coding for Aircraft Control Knobs 2

• “Pilot errors” were really “design errors”.
• Quick fix: small rubber wheel glued to landing gear control, wedge shape to flap control.
• Gear-up landings ceased.
• After war, shape-coded wheel and flap controls were standardised.

Shape Coding in Nuclear Power Station Control Room

Affordances

Affordances are the range of possible (physical) actions by a user on an artefact:

• Perceived Affordances are the actions a user perceives to be possible.
• Real Affordances are the actions which are actually possible.

• A chair affords (suggests) sitting.
• Knobs afford turning.
• A button affords pushing.

Don Norman, The Psychology of Everyday Things, 1988.

Affordances in Door Designs

• Ambiguous door designs:
  • A knob affords turning, but do you push or pull?
  • A horizontal bar affords pushing, but which side do you push on?
• Good use of affordances in door designs:
  • A vertical handle affords grasping and pulling.
  • A flat panel affords pushing and the broadness indicates which side to push.

Video: Norman Doors

• Norman Doors video [Posner 2016]
• Video: https://youtu.be/yY96hTb8WgI

Mappings

UI Fails

Images from the Interface Hall of Shame, used with permission.

Human-Computer Interaction

Usability Engineering Lifecycle

Usability Evaluation Methods

Modified Soup Analogy

Extending Robert Stake’s soup analogy [Stake 1976, page 19] to usability evaluation methods:

• When the cook tastes other cooks’ soups, that’s exploratory.
• When the cook assesses a certain recipe, that’s predictive.
• When the cook tastes the soup while making it, that’s formative.
• When the guests (or food critics) taste the soup, that’s summative.

Heuristic Evaluation (HE)

• Formative usability inspection method.
• First described by Nielsen and Molich [1990].
• Small team of evaluators (in practice 3 to 4 usability specialists) systematically checks interface design against small set of recognised usability principles (the “heuristics”).
• Often called an expert review.

How to Conduct a Heuristic Evaluation;
https://nngroup.com/articles/how-to-conduct-a-heuristic-evaluation/

Usability Heuristics

• Small set of recognised usability principles.
• Used to help inspect an interface.
• Original nine defined by Nielsen and Molich [1990]
• Some renamed and tenth added by Nielsen [1994].
• Names in square brackets are new names, where different.

10 Usability Heuristics for User Interface Design;
https://nngroup.com/articles/ten-usability-heuristics/

Classic 10 Generic Usability Heuristics

1. Feedback [Visibility of System Status]

   Keep users informed about what is going on, appropriate feedback within reasonable time. For example: busy cursor [1–10s], progress indicator [>10s].

2. Speak the Users’ Language [Match Between System and the Real World]

   Use words, phrases, and concepts familiar to the user, rather than system-oriented terms. Follow real-world conventions. Match users’ mental model. Beware of misleading metaphors.

3. Clearly Marked Exits [User Control and Freedom]

   Users often select functions by mistake, need clearly marked “emergency exit”, so users can explore without penalty. Support undo and redo.

4. Consistency [Consistency and Standards]

   Users should not have to wonder whether different words, situations, or actions mean the same thing. Follow platform conventions.

5. Error Prevention

   Prevent a problem from occurring in the first place. For example: select file from menu rather than typing in name, confirmation before dangerous actions, beware of modes, avoid similar command names, warning if Caps Lock is activated when entering a password, etc.

6. Recognition rather than Recall

   Make objects, actions, and options visible. Provide examples, default values, easily retrievable instructions.

7. Accelerators [Flexibility and Efficiency of Use]

   Accelerators, unseen by novice user, can make expert user more efficient. For example: abbreviations, command keys, type-ahead, edit and reissue previous commands, menu of most recently used files, macros.

8. Minimalist Design [Aesthetic and Minimalist Design]

   Every extra unit of information competes with the relevant units of information and diminishes their relative visibility. “Less is more”.

9. Good Error Messages [Help Users Recognise, Diagnose, and Recover from Errors]

   Error messages should be in plain language (no codes or jargon), precise, constructive, and defensive.

10. Help and Documentation

    Easy to search, focused on the user’s tasks. Liberal use of recipes and examples.

Performing a Heuristic Evaluation

Step-by-step recipe for performing a HE, in five phases:

1. Preparation
2. Individual Evaluations
3. Aggregation
4. Severity Ratings
5. Debriefing and Report

1. Preparation

• The design may be verbal description, paper mock-up, working prototype, or running system.
• Provide evaluators with checklist of usability heuristics.
• Optionally provide evaluators with some domain-specific training.

2. Individual Evaluations

• Each evaluator works alone (≈ 1–2 hours).
• Interface is often examined in two passes: first pass focuses on general flow, second on particular elements in detail.
• Notes taken either by evaluator or evaluation manager.
• Make a list of (potential) problems and a list of positive findings.

  Not all problems fall under a heuristic. It is OK to include problems which are not covered by any heuristic.

• Use screen video capture software such as OBS Studio, Camtasia, or AZ Screen Recorder to record each evaluation session (including the evaluator’s voice if possible), since some problems may not be reproducible later on.

• Re-create a short video clip to illustrate each finding, concisely describing the finding in the audio track.

  Where a finding cannot be recreated, extract a video clip from your evaluation recording. If the existing audio commentary does not describe the finding adequately, record a new audio commentary to describe the issue.

  In some cases, a static image (still frame) may suffice to illustrate a finding.

3. Aggregation

• Independent findings are now aggregated into one large list (by the evaluation manager) using a spreadsheet:
  • Start by copying the longest individual list of problems over into the spreadsheet. Then look through the other individual lists and compare each problem to see if it is already in the aggregated list.
  • If two or more evaluators find the same (or very similar) problem, combine them into one.
  • List many small related problems (such as 27 individual typos) as one problem (rather than 27).
• Proceed analogously to create an aggregated list of positive findings.

4. Severity Ratings

• Proceed analogously to create an aggregated list of positive findings.

• Each evaluator now assigns severity ratings individually to each problem in the large list, unseen by the other evaluators.

  For example, on a simple 0 (not a problem) to 4 (catastrophic problem) scale.

• The individual severity ratings are averaged to obtain the final mean severity rating for each problem.
• The aggregated list is sorted in decreasing order of average severity.
• Proceed analogously for the positive findings and positivity ratings.

5. Debriefing and Report

• If possible, organise a group debriefing session to recommend possible solutions.
• Write up all of your results into a report.

Examples of HE

• EnigUsab;
  https://projects.isds.tugraz.at/enigusab/he/he.html
• kastner-oehler.at;
  https://courses.isds.tugraz.at/hci/reports/ss2023/g5-09-kastner/he/
• Other examples:
  https://courses.isds.tugraz.at/hci/reports/
• Materials:
  https://courses.isds.tugraz.at/hci/practicals/materials/

Thinking Aloud Testing (TA)

• Formative usability testing method.
• Test users verbalise their thoughts (“think aloud”) while performing tasks.
• A running commentary of what they are seeing, thinking, and doing.
• Provides insight into their thought processes and why things go wrong (process data).
• Relatively small number of test users (say 3 to 5 per user group).
• Often combined with post-test interview and questionnaire.

Steve Krug; Rocket Surgery Made Easy; New Riders, 2009. ISBN 0321657292.

Carol Barnum; Usability Testing Essentials; 2^nd Edition, Morgan Kaufmann, 2020.
ISBN 0128169427.

Test Setup

Test Setups in 2002 and 2019

Test Checklist

1. Preparation
   • Reset interface for new user.
   • Check that everything is ready in test room.
2. Opening
   • Greet the participant.
   • Go through orientation script and set the stage.
   • Background questionnaire: facilitator asks and fills out form.
   • Ask user to read and sign consent and non-disclosure forms.
3. Test Session
   • Move over to testing area (computer).
   • Start computerised session recording.
   • Provide any prior training.
   • Provide training of thinking aloud.
   • User begins with tasks.
   • User finishes last task.
4. Closing
   • Interview: how was it?
   • Structured interview questions.
   • Individual interview questions arising from test.
   • Feedback questionnaire: user fills out form.
   • Thank participant, provide any remuneration, show participant out.
5. Clean-Up
   • Summarise thoughts about this test.
   • Organise data sheets and notes.
   • Check and archive session recordings.

Examples of TA

• EnigUsab;
  https://projects.isds.tugraz.at/enigusab/ta/ta.html
• kastner-oehler.at;
  https://courses.isds.tugraz.at/hci/reports/ss2023/g5-09-kastner/ta/
• Other examples:
  https://courses.isds.tugraz.at/hci/reports/
• Materials:
  https://courses.isds.tugraz.at/hci/practicals/materials/

Formal Experiments (FE)

• Summative usability testing method.
• Controlled experiment with representative test users.
• Measurement and collection of quantitative data (bottom-line data).
• Both objective (performance, success rate) and subjective (ratings, overall preference) measures.
• Followed by more or less rigorous statistical analysis.
• Needs larger number of test users for statistical accuracy (sometimes around 16 to 20, but often 50 or 100 or more).
• Never ask users to think aloud during a formal experiment!

  Thinking aloud slows users down and can change their behaviour.

• Two main uses:
  • Assess the absolute performance of an interface.
  • Objectively compare two (or more) alternative interface designs.

Bill Albert and Tom Tullis; Measuring the User Experience, 3^rd Ed; Morgan Kaufmann, 2022.
ISBN 0128180803.

Andy Field et al; Discovering Statistics Using R; Sage, 2012. ISBN 1446200469.

Video: Saving Lives by Design

• Harold Thimbleby; Saving Lives by Design; Swansea University, 2011.
• Video: https://youtu.be/3LiGsPR34w8?t=04m50s [04:50-07:50]

Thank You

Web site:
https://isds.tugraz.at/keith/

These slides:
https://keithandrews.com/talks/2024/2024-01-11-frequentis/

Human-Computer Interaction Course Notes [242 pages PDF]
https://courses.isds.tugraz.at/hci/hci.pdf