The question of where to run the tests, in a usability laboratory or in the field, is important in the mobile world. The busy, noisy, distracting mobile context of use couldn’t be more different from the calm, quiet and focused laboratory environment. Comparative studies about the importance of field testing in mobile software have reached contradictory conclusions. [INSERT LINK TO ACADEMIC REFERENCES PAGE] A. Kaikkonen et al. concluded: “There was no difference in the number of problems that occurred in the two test settings. Our hypothesis that more problems would be found in the field was not supported” (Usability Testing of Mobile Applications: A Comparison between Laboratory and Field Testing in Journal of Usability Studies Issue 1, Vol. 1, November 2005, pp. 4-16). C.M. Nielsen et al., however, found that “evaluations conducted in field settings can reveal problems not otherwise identified in laboratory evaluations” (It’s Worth the Hassle! The Added Value of Evaluating the Usability of Mobile Systems in the Field in NordiCHI papers 2006). What both papers agree on is the fact that testing in the field is a messy affair. A. Kaikkonen et al. observe it takes “double the time in comparison to the laboratory”; C.M. Nielsen et al. describe it as “complex and time-consuming”.
The truth is that most industry projects lack the time, budget, personnel and expertise to run tests in the field. For most of us, field testing is not even an option, and we must find consolation in the fact that testing in the lab is better than no testing at all.
If field testing must be done (and for certain types of software, it must: think of data collection applications, geo-location related software or mobile payments), it should be done late in the design cycle as a validation mechanism. It should be carefully planned and rehearsed with pilot runs, and the team should be ready to handle unexpected problems on test day.