BluePrint for Human Intelligence Task processing (a.k.a. "Job Jar")

Motivation

During a disaster, people on the scene may report via social media or text messages. Emergency managers might find useful information there, but it's buried in large quantities of incomplete pieces, possibly not in a language known to the EM personnel. Much of this work is not easily automated, e.g. determining whether a message is a request for aid (and specifically what and how much and where). In order to clean up this crowdsourced information, we need...another crowd, but one that's trained, or at least learns as they go, and whose work is cross-checked for accuracy. A similar need holds for data gathering done by volunteers.

These are called "human intelligence tasks" (HITs), and what we want to produce is a system for managing them. Examples of HIT platforms are Amazon's Mechanical Turk and Crowdflower. We want a system tailored to the needs of emergency managers, and to working with crowdsourced data during emergencies.

See also: ​http://www.playsourcing.org/ (only s/Ushahidi/Sahana Eden/ ;-) And in June, see article on Playsourcing in: ​http://www.crcpress.com/product/isbn/9781439853498

We're suggesting this for GSoC, but the whole thing would be too big for one GSoC project -- one or more subsets can be split out into GSoC projects.

Overview

We would like to manage human-intelligence tasks, e.g. data entry, cleaning crowdsourced data. This involves:

• Figure out how to specify a task, and provide a UI for defining tasks.
• Allow reading data for tasks from specified sources.
• Provide tasks to users in a web form.
• Assign tasks to workers based on skills and / or measured performance, or let workers select tasks.
• Compare results of multiple workers on the same task.
• Decide when a task is sufficiently complete.
• Insert tasks with known results as tests.
• Evaluate worker performance.
• Administer training and testing for new workers.

Project breakdown

These are some suggested ways to split the project into somewhat independent pieces.

Task definition

Each overall job to accomplish one purpose will have its own specific set of input data, its own instructions to workers, its own way for workers to enter results, and its own needs for how much verification is needed. Rather than hard-wire a few sorts of task types, we'd like to let the people who need the work done specify these things. To permit this, we might need to...

Figure out what information is needed to specify a task. This might include:

• Specify the task input data source(s).
  • Note: The actual connection to the data source, and the process of reading from it, should be isolated. This is not specific to this project -- reading from a remote feed can be used for many purposes. Here, we could assume that the data is being placed in one or more database tables, or in files on the server.
  • Most commonly each task will operate on one item of data -- can assume this to start.
• Provide instructions for workers.
• Specify what worker skills are needed.
• Specify how many workers should receive each task, and how many need to agree on the result for it to be accepted.
• Specify the web form tools that the worker will need to enter results, e.g.:
  • Text input.
  • Radio buttons or exclusive select from a list.
  • Multi-select boxes or list.
  • Combo-box, i.e. either exclusive or multi-select, but also allow adding a new option.
  • Selection of locations on a map.
• Specify constraints on the results, especially ones that can be checked in the form, such as:
  • Data types for text fields (date, number with range, ...)
  • Multiple selections that make sense together.
• Specify how to compare results from different workers -- when do the results match?
• Assign experts who can handle difficult cases and verify a sampling of results.

Decide how the task setup should be stored, that is:

• What database schema is appropriate?
• How does it fit in with other Eden components?

Then, provide a web form or wizard that allows the administrator to enter the task setup information.

Managing the task queue

The overall process will be:

• Get items of data from the data source.
• Pick one or more appropriate workers for each.
• Present the tasks to the workers.
• Allow workers to refuse tasks.
• Receive their results.
• Do any automated checking -- push back obviously invalid results.
• Store results.
• Record completion of tasks when there is enough consensus on the results.
• Record task failure if there's not enough agreement.
• Dispatch failed tasks and a sample of successful tasks to experts.

The task queue will be driven by the input data.

Assigning tasks to users

Workers will have different skills, and different levels of ability and training on those skills. Some skills will be hard to learn in the short term (e.g. languages) while others can be learned quickly (e.g. classifying messages by topic). When we're training new workers, we might want to give them tasks to practice on even if their level of ability isn't high. But for production work, we'd want to assign tasks that are appropriate for each worker's skills and skill level.

Presenting tasks to workers

Collating, comparing, verifying results

Evaluating worker performance

Providing feedback

---

BluePrints