|Version 20 (modified by 10 years ago) ( diff ),|
Table of Contents
- Project breakdown
- Receive tweets and / or SMS messages from the public.
- Dispatch these to online workers to classify and geocode.
- Display on a map.
During the Haiti earthquake of Jan 2010, people trapped in buildings sent SMS messages to a designated shortcode. These were classified, translated, and geocoded by online workers using Amazon's Mechanical Turk, then provided to emergency managers.
During the Kenya 2013 general election, citizens and trained election monitors reported election-related incidents via SMS and twitter. These were automatically entered into a map database, then vetted by online workers to remove spam and contact the sender for clarification, before making the information public. See: https://uchaguzi.co.ke/
During a Random Hacks of Kindness hackathon in 2010, a variant of this project was implemented using a Sahana Eden as the back end and a custom web page (not automatically generated by Eden) as the front end. This was designed as a training game -- workers got "experience points" and were awarded badges. See: http://gwob.org/101010-hackathon-winners/
This project is intended to be easy to subdivide into tasks that can be worked on somewhat independently and in parallel, given the choice of a few naming conventions for new database tables and fields.
In order to keep our work together, and distinct from other work, we'll add a new module. This is the first step in added "human intelligence task" processing, in which results are verified by sending the same task to multiple workers, and comparing the results. So let's call our new module "hit". That means the controller file will be:
The model will be:
The view pages will be in the directory:
Everyone may find it useful to refer to:
- The lesson on "making a new module" in the Eden book:
- The index of Eden APIs:
Set up incoming messages
Read the information on how messages are received by Eden from Twitter or SMS, and get test messages into an Eden instance.
- UserGuidelines/Messaging (very much in need of clarification and concrete examples)
User interface: http://demo.eden.sahanafoundation.org/eden/msg/
Fill in required "new module" boilerplate
Look at the lesson on "making a new module" in the Eden book:
That puts the model file in the
eden/models directory, but that is just to avoid complication.
eden/models are loaded on every http request, whether they're needed or not.
Most Eden models are in
eden/modules/s3db, and are only loaded by http requests that need them.
Since our message processing won't be used by most types of requests, we want it in
Add the new module to the list of enabled modules. This is normally specified in a "template" that has the
customizations for a particular site. Here, we will "cheat" and just add the new module to our configuration
eden/models/000_config.py. Get the default module list from
Copy it to
models/000_config.py and add an entry for the hit module.
Add database tables for message processing tasks
Received messages are stored in the "message log" table, msg_message.
(This is a special kind of table called (in Eden terminology) a "superentity". This is like a superclass but for database tables. Records in multiple specialized tables have "parent" records in a shared superentity table, so other tables can refer to any of the specialized tables without needing a foreign key field for every one, by instead linking to the superentity record. References to ordinary non-superentity tables are simpler.)
We want to add a table that joins to this to hold the data entered by a worker for a message. That table will need fields for:
- A foreign key reference to the msg_message table. Here is another table with such a reference:
- A category that the worker will assign. This can be just a text field for now. Here's an example of a
text field (the from address in a message):
(The category will be empty til the worker fills it in, so we can't require that it be non-empty.)
- A location that the worker will enter either by filling out a form and / or clicking on a map.
There is a standard widget for selecting locations that will be included automatically if the
location is specified as in this example:
Here is where the function that generates the foreign key reference is defined:
The name for the function used outside the gis module includes the "gis_" prefix to avoid ambiguity.
Why do we want a separate table? Why not just add a category and location to the msg_message table? Eventually, we want to do "human intelligence task" processing, in which results are verified by sending the same task to multiple workers, and comparing the results. So we may have more than one set of results for each message. We want to include which worker did each task, so we can check the quality of their work and refer them to more training if needed.
Add a controller function to generate task pages for workers
- Other controllers in
- The documentation for the controller helper function:
- The documentation for a custom controller:
Eden will automatically generate pages that correspond to database tables (list forms) or individual records (read or edit forms), or empty forms for adding new records (create forms). However, when a worker requests a task, there is not yet any database record for the task. Instead, we want to:
- Select a new, not-yet-processed message.
- Create a record in the (new) hit_task table (being worked on by the model team).
- Return the standard form for the new record to the user.
Add a view that presents a task to the worker and submits their work
Generate a list of categories from the database
We would like to encourage workers to use existing categories when there is a close enough match, but be able to add new ones if not. So, we want to give the worker a menu of categories to choose from, consisting of all the current categories found in the category field being added by the team working on adding the new tables, and also let the worker add a new category.