What is Good Enough for Mobile System Evaluation: A Comparison of Laboratory and Field Setting.

摘要

There is no consensus on whether field tests are necessary in evaluating mobile systems. Empirical studies have compared evaluations in the laboratory environment and field environment, but they provide different results due to the lack of consistent testing environments. Contributing to mobile system evaluation exploration, this study conducts a comparative analysis between laboratory study and field study by recreating the natural context in the laboratory and using recording software embedded in the mobile device. The results show that involving natural use context in a controlled testing environment identifies more usability problems than a complex field test, when the objective is to uncover mobile system design details. If the goal of a usability evaluation is to involve users' natural behaviors and examine context-dependent usability problems, a field test is necessary. We further analyze how to effectively simulate the natural context in the laboratory environment. The conclusion shows that the results might be different when using different user pools.