In his Alertbox column of 20 July 2009, Jakob Nielsen shares the results of a usability testing experiment they ran with 48 people who were asked to complete some tasks using several websites on their mobile phones. Nielsen determined the average success rate at 59%. When they broke down that average figure by the type of phone participants used during the test, they found the following:

  • Participants using feature phones (i.e. something like a Nokia 6300, with a smallish colour screen, a numeric keypad and no third-party native applications) were able to complete 38% of the tasks.
  • Participants using smartphones (i.e. something like a Blackberry Bold, with a bigger colour screen, a qwerty keyboard and third-party native applications) were able to complete 55% of the tasks.
  • Participants using what Nielsen calls “touch-screen” phones (i.e. something like an iPhone, with an even bigger screen than a smartphone incorporating touch technology) were able to complete 75% of the tasks.

The moral of this story is that handset usability affects test results. A wonderfully designed website will feel difficult and cumbersome when used with a phone plagued by usability issues. Not that feature phones are badly designed (some are, some aren’t), but they are not optimised for web browsing or application usage.

Similarly, not all touch-screen phones are built equal, and some of them will perform better than others. In any case, we must find a way to minimise the effect of handset (and browser) usability on test results, and here is how:

whenever possible, test with participants’ own phones.

They might be terrible as handsets go, but participants are probably accustomed to their flaws and have developed tricks and workarounds. If this is not possible, and sometimes it won’t be, schedule training time on your test plan to explain participants how to use the phone, and include some “warm-up” tasks participants can attempt to become familiar with the handset.