[UPDATE: Upon further reflection, this seems to be a student project created for the class, not research by Galletta himself. And as I said, it’s not clear how serious a test it was or what the methodology was. (I do note that the end credits list a “cast.” My bad for jumping to conclusions after reading this story.]

Professor Dennis Galletta has been teaching a summer course at Harvard on Human Factors in Information Systems Design. As part of it, he conducted some usability testing of the iPhone 4, Samsung’s Windows Phone 7-based Focus, HTC’s Android-based Thunderbolt, and RIM’s BlackBerry Storm. He had people who hadn’t used any of the phones try to make a call, add a contact, and send a text message, and videotaped their attempts to do so.

Here’s a video summary of what transpired:

The overall results are what you might guess if you’ve used all these platforms: iPhone and Windows Phone did the best, Android lagged behind, and BlackBerry befuddled people. When folks struggle in the video, it’s not always obvious why–but it looks like Windows Phone’s animated flourishes may make the interface less intuitive, at least to a newcomer. For all of Windows Phone’s virtues, I’ve always thought it would be even better if it didn’t scream “LOOK AT ME!” quite so insistently; this might be evidence to support that theory.

The video doesn’t spell out the study’s methodology. I’d be interested to learn more details–what sort of people tried the phones, and how many of them were there?–and see a more extended cut of the video that shows all the gory details and skips the musical accompaniment. (I love Yakkety Sax as much as the next Benny Hill fan, but it doesn’t exactly enhance the seriousness of the experiment.)

Of course, the experiences of a handful of clueless newbies aren’t a definitive verdict on these operating systems. Personally, I’m less interested in how well neophytes fare with a phone than how efficient and intuitive it is for a more experienced user over the long haul. BlackBerry phones are utterly perplexing when you pick one up for the first time–but once you get the knack, they make sense. But I’ve now used Android for years and still fumble with certain basic tasks…

The iPhone isn’t perfect, but it strikes a sensible balance between approachability for beginners and efficiency for long-term users. I wonder how a phone based on HP’s WebOS would have fared?

It’s interesting that the tests focused on using phones as, well, phones. Do you think that the conclusions would have been different if e-mail, browsing, app management, and other more PC-like activities had been included?

 
7 Comments


Read more: Apple. iPhone, Google Android, Harvard, Microsoft Windows Phone 7, RIM BlackBerry, Smartphones