Changes between Version 15 and Version 16 of BluePrintHumanIntelligenceTasks

04/05/11 22:27:36 (10 years ago)
Pat Tressel



  • BluePrintHumanIntelligenceTasks

    v15 v16  
    7373The overall flow would be:
    74  * Get items of data from the data source.
     74 * Identify new items of data that have not been processed.
    7575 * Pick one or more appropriate workers for each.
    7676 * Present the tasks to the workers.
    8585The task queue will be driven by the input data. Actually reading the data from a remote
    8686source is "beyond the scope" of this project -- that would be a common process needed by
    87 other Eden components as well.
     87other Eden components as well. If we don't have a component for reading from a particular
     88source that we'd like to support, we can mock reading from it (e.g. by downloading a batch
     89of data or generating fake data and having a cron job add it to appropriate source data
    8992==== Assigning tasks to users ====
    91 Workers will have different skills, and different levels of ability and training on those skills.
    92 Some skills will be hard to learn in the short term (e.g. languages) while others can be learned
    93 quickly (e.g. classifying messages by topic).
    94 When we're training new workers, we might want to give them tasks to practice on even if their
    95 level of ability isn't high.  But for production work, we'd want to assign tasks that are appropriate
    96 for each worker's skills and skill level.
     94There are several conditions for assigning tasks to workers.
     95 * They should be signed up to work on tasks, and maybe on specific kinds of tasks.
     96   That is, there will be other people registered as users, so we need to identify those
     97   who are HIT workers.
     98 * We should let HIT workers say when they're available to work -- that might not be
     99   the instant they log in, and they'll want to take breaks.
     100 * An inactivity timer might be needed to detect whether the user has dropped offline or
     101   is stuck on something.
     102 * Workers will have different skills, and different levels of ability and training on those
     103   skills. For production work (not training), we'd want to assign tasks that are appropriate
     104   for each worker's skills and skill level. The requirements for skills will come from the
     105   task definition.
     106 * If the task definition calls for it, we'll need to dispatch each task to multiple workers.
     107 * We need to track who has worked on what task, even if they declined it, so that we don't
     108   assign a task to a worker more than once.
     110Sometimes the assignment might be adjusted (these are beyond the basics):
     111 * When we're training new workers, we might want to give them tasks to practice on even if
     112   their level of ability isn't high. We might not even count their results toward completion
     113   of a task, but just compare them against the other assigned workers.
     114 * Experienced workers with higher performance ratings on a task might need fewer other
     115   workers assigned to the same task. Eventually a worker might be promoted to "expert",
     116   and become a reviewer for other workers.
     117 * If the task instructions change, we could inject a notification. Could also use this to
     118   prompt the worker to take breaks or deliver performance "rewards".
     120There is a volunteer management component in Eden that is currently under revision. It has
     121simple support for specifying worker skills, and can also serve to identify HIT workers.
     122So that work on this project is not affected by changes, we can fork the volunteer component
     123and modify it as needed. Any useful or necessary mods can be fed back in to the new version
    98126==== Presenting tasks to workers ====
     128The task will need a form that displays the data to the worker and accepts their answer.
     130 * The form should be assembled based on what's specified in the task definition.
     131  * Appropriate means of displaying the data.
     132  * Widget that implements the required input method.
     133  * A way to make instructions and help info available without distracting an experienced worker.
     134 * The worker should be allowed to decline a task in case they don't think they can do it.
     135 * Letting them annotate the task with comments or concerns could be helpful in diagnosing
     136   problems with the input data, or with the HIT system.
     137 * Customizing the layout or look-and-feel might improve usability.
     138 * A means of contacting the user via the form could help in determining if they're still
     139   online and active.
    102141==== Collating, comparing, verifying results ====
    104 ==== Evaluating worker performance ====
     143==== Evaluating worker performance and providing feedback ====
    106145==== Providing feedback ====