Well-Tested, Easy to Use: How to Measure Any Product's Usability (w/ Philip Kortum)

Yeah, so let's start off with getting into, you know, you've worked with tons of human factor systems in the 90s, 2000s, and so on.

Before we get into the importance of the system usability scale, give listeners a backdrop of kind of the nature of the work, the nature of the problems, and what gave rise to SUS and the need eventually to kind of validate it empirically.

Sure.

So I think for those of us in the usability community, one of the big things that we try and do is we try and figure out, and it's a simple question, is this system or device or product usable? And for really simple systems, that's pretty easy.

If I give you a can opener and say, hey, is this easy to use? That's easy for us to assess.

As I get more and more complex systems, it becomes more difficult.

And so if I'm judging a cockpit, how can I judge the usability of that? If I'm judging an entire battleship or guided missile frigate, how do I determine if that's usable? And so typically, we try and use measures that are defined by ISO 9241 Part 11, which are effectiveness, efficiency, and satisfaction.

And the two that we really hang our hats on from a data standpoint are effectiveness and efficiency.

And effectiveness just says, can I do the task? If I ask you to use the can opener to open a can, can you do that? And then I can simply measure, were you able to do it? I can also take other effectiveness measures like how many errors did you make? How many attempts did you have to do before you were able to open the can? I can measure other things such as how many times you have to call the help desk? How many times did you reach for the manual? Efficiency is also an important measure for lots of products, and that's, could I do it in a timely way? And so almost always we just put a stopwatch to that and say, it took you X number of minutes to complete this task.

That number by itself is not particularly meaningful, but it's meaningful as a comparison.

If you're doing iterative work, right, I have version one, version two, version three, or if I'm trying to compare your can opener and my can opener, am I faster with one or the other? But the raw number doesn't mean anything.

It might take me two minutes to do a task that's something simple, and it might take me an hour to do a complex task, And in both cases, those might be considered very efficient numbers.

We just don't know.

And then satisfaction is a little bit more subjective.

We're just trying to understand if people find the experience enjoyable.

I think what happened is a lot of people wanted to tie some numbers together.

to the satisfaction number.

And we started looking for surveys or instruments that we could use that would measure the subjective usability that we would use as a proxy for satisfaction.

And so this started to happen in the late 80s, early 90s, when people started trying to put some numbers to that.

And the reason that subjective usability numbers were beneficial is it allowed you to make more direct comparisons across products.

It allowed you to understand how things were changing over time.

And it allowed you to think about being able to measure without specific reference to a task.

So if I, let's say, I'm trying to measure Microsoft Word.

And so the task that I might measure would be create a new paragraph and then bullet that paragraph.

If then I want to compare that task to adding the table of contents, it gets much more confusing about whether the time metrics and the success metrics match.