Changes between Version 12 and Version 13 of BluePrintHumanIntelligenceTasks


Ignore:
Timestamp:
04/05/11 15:41:40 (11 years ago)
Author:
Pat Tressel
Comment:

--

Legend:

Unmodified
Added
Removed
Modified
  • BluePrintHumanIntelligenceTasks

    v12 v13  
    1414http://www.crcpress.com/product/isbn/9781439853498
    1515
    16 We're suggesting this for GSoC, but it's too big for one summer -- one or more subsets can be split out into GSoC projects.
     16We're suggesting this for GSoC, but the whole thing would be too big for one GSoC project -- one or more subsets can be split out into GSoC projects.
    1717
    1818=== Overview ===
     
    3535==== Task definition ====
    3636
    37 Each overall job that needs to be done to accomplish one purpose will have its own specific set of input data, its own instructions to workers, its own way for workers to enter results, and its own needs for how much verification is needed.  Rather than hard-wire a few sorts of task types, we'd like to let the people who need the work done specify these things.  To permit this, we might need to...
     37Each overall job to accomplish one purpose will have its own specific set of input data, its own instructions to workers, its own way for workers to enter results, and its own needs for how much verification is needed.  Rather than hard-wire a few sorts of task types, we'd like to let the people who need the work done specify these things.  To permit this, we might need to...
    3838
    3939Figure out what information is needed to specify a task.  This might include:
     
    4747 * Provide instructions for workers.
    4848 * Specify what worker skills are needed.
     49 * Specify how many workers should receive each task, and how many need to agree on the result for it to be accepted.
    4950 * Specify the web form tools that the worker will need to enter results, e.g.:
    5051  * Text input.
     
    5354  * Combo-box, i.e. either exclusive or multi-select, but also allow adding a new option.
    5455  * Selection of locations on a map.
    55  * Advanced setup:
    56   * Allow specifying form layout.
    57   * Provide convenient access to helpful tools from within the form.
     56 * Specify constraints on the results, especially ones that can be checked in the form, such as:
     57  * Data types for text fields (date, number with range, ...)
     58  * Multiple selections that make sense together.
     59 * Specify how to compare results from different workers -- when do the results match?
     60 * Assign experts who can handle difficult cases and verify a sampling of results.
    5861
    59 Decide how this info should be stored.
     62Decide how the task setup should be stored, that is:
     63 * What database schema is appropriate?
     64 * How does it fit in with other Eden components?
    6065
    61 Then, provide a web form or wizard that allows the administrator to enter that information.
     66Then, provide a web form or wizard that allows the administrator to enter the task setup information.
    6267
    63 ==== Task assembly and presentation ====
     68==== Managing the task queue ====
     69
     70The overall process will be:
     71 * Get items of data from the data source.
     72 * Pick one or more appropriate workers for each.
     73 * Present the tasks to the workers.
     74 * Allow workers to refuse tasks.
     75 * Receive their results.
     76 * Do any automated checking -- push back obviously invalid results.
     77 * Store results.
     78 * Record completion of tasks when there is enough consensus on the results.
     79 * Record task failure if there's not enough agreement.
     80 * Dispatch failed tasks and a sample of successful tasks to experts.
     81
     82The task queue will be driven by the input data.
    6483
    6584==== Assigning tasks to users ====
     85
     86Workers will have different skills, and different levels of ability and training on those skills.
     87Some skills will be hard to learn in the short term (e.g. languages) while others can be learned
     88quickly (e.g. classifying messages by topic).
     89When we're training new workers, we might want to give them tasks to practice on even if their
     90level of ability isn't high.  But for production work, we'd want to assign tasks that are appropriate
     91for each worker's skills and skill level.
     92
     93==== Presenting tasks to workers ====
     94
     95
    6696
    6797==== Collating, comparing, verifying results ====