Multistep planning for crowdsourcing complex consensus tasks

摘要

Crowdsourcing receives massive vote information from non-expert workers, for finishing tasks that can hardly be handled by current technology of machine intelligence. Massive vote information and non-expert workers bring serious issues of labor costs and the efficiency of crowdsourcing. This paper focuses on the tasks, classifying objects in images or videos into a set of given candidates by letting workers vote on a set of options that characterize these candidates. Designing a good asking strategy, i.e., setting up the order of presenting the options to a worker and asking the worker whether an option is true or false, is one starting point to save labor costs and enhance efficiency of deciding the correct answer from the candidates. We propose the problem of determining the time steps of vote collection before stopping to set up the asking strategy. In terms of this problem, we establish a single-step collection based partially observable Markov decision process (POMDP) to analyze how a vote influences the whole system, for instance, influences the belief over each option. Formally define the multistep collection problem as the timed decision (TD) problem. We propose the MC-EVA algorithm based on Monte Carlo sampling to solve the TD problem. Evaluate the MC-EVA algorithm over three simple but typical cases and a real-world Galaxy Zoo 2 project. Experiments show MC-EVA's great superiority in runtime over the state-of-the-art single-step collection algorithm, and its superiority in effectiveness than other multistep collection algorithms; show its labor cost saving and enhanced efficiency with the use of calculated asking strategies. (C) 2021 Elsevier B.V. All rights reserved.