A Harvard Professor Puts Smartphone Usability to the Test

By  |  Tuesday, August 9, 2011 at 11:39 am

[UPDATE: Upon further reflection, this seems to be a student project created for the class, not research by Galletta himself. And as I said, it’s not clear how serious a test it was or what the methodology was. (I do note that the end credits list a “cast.” My bad for jumping to conclusions after reading this story.]

Professor Dennis Galletta has been teaching a summer course at Harvard on Human Factors in Information Systems Design. As part of it, he conducted some usability testing of the iPhone 4, Samsung’s Windows Phone 7-based Focus, HTC’s Android-based Thunderbolt, and RIM’s BlackBerry Storm. He had people who hadn’t used any of the phones try to make a call, add a contact, and send a text message, and videotaped their attempts to do so.

Here’s a video summary of what transpired:

The overall results are what you might guess if you’ve used all these platforms: iPhone and Windows Phone did the best, Android lagged behind, and BlackBerry befuddled people. When folks struggle in the video, it’s not always obvious why–but it looks like Windows Phone’s animated flourishes may make the interface less intuitive, at least to a newcomer. For all of Windows Phone’s virtues, I’ve always thought it would be even better if it didn’t scream “LOOK AT ME!” quite so insistently; this might be evidence to support that theory.

The video doesn’t spell out the study’s methodology. I’d be interested to learn more details–what sort of people tried the phones, and how many of them were there?–and see a more extended cut of the video that shows all the gory details and skips the musical accompaniment. (I love Yakkety Sax as much as the next Benny Hill fan, but it doesn’t exactly enhance the seriousness of the experiment.)

Of course, the experiences of a handful of clueless newbies aren’t a definitive verdict on these operating systems. Personally, I’m less interested in how well neophytes fare with a phone than how efficient and intuitive it is for a more experienced user over the long haul. BlackBerry phones are utterly perplexing when you pick one up for the first time–but once you get the knack, they make sense. But I’ve now used Android for years and still fumble with certain basic tasks…

The iPhone isn’t perfect, but it strikes a sensible balance between approachability for beginners and efficiency for long-term users. I wonder how a phone based on HP’s WebOS would have fared?

It’s interesting that the tests focused on using phones as, well, phones. Do you think that the conclusions would have been different if e-mail, browsing, app management, and other more PC-like activities had been included?


Read more: , , , , ,

7 Comments For This Post

  1. lothinator Says:

    What makes iPhone so difficult is if you want to do anything *other than* make a call or launch Safari. Then it gets hard.

  2. Andres Iriarte Says:

    Hi! My name is Andres Iriarte. I was part of the team who did this usability study. You are right, I wasn't a official study made by Galletta. He was our professor (BTW a really good one) in the Human Factors in Information Systems Design. We wanted to study the usability of different phone for first time users. Also, we wanted to have use some humor in the video (like the Benny Hill theme..)
    I am glad you guys did not follow the other articles saying that it was a Harvard study. It was just a student project. I hope you like it, and please contact me if you have any questions about it. Thank you!

  3. Harry McCracken Says:

    Thanks for the info, Andres.

  4. Andres Iriarte Says:

    No problem! And I'm sorry if the video wasn't very descriptive

  5. Guest Says:

    why a storm? you are comparing top notch devices (latest and greatest) with a very old blackberry device (what about a torch or at least the storm 2?)

  6. Bryan Says:

    A blackberry Storm? seriously? that phone came out 3 years ago. wanna make it a fair fight use the Blackberry Torch.

  7. Marc E Says:

    Andres, such a shame that you got the vido pulled…