Personal Details

Name: Ashwyn Sharma

Email: ​ashwyn1092@gmail.com

Freenode IRC Nickname: ashwyn

Skype: ashwyn sharma

Age: 19

Education: Currently pursuing B.E.(Bachelor in Engineering) from N.S.I.T, New Delhi.

Country: India

Timezone: GMT +0530

Github Branch: ​https://github.com/ashwyn/eden

Project Abstract

The essential requirement for this project is to parse inbound messages, with an initial focus to SMS. The project is specifically aimed at the CERT usecase where they wish to process responses to deployment notifications. Or in other words, to handle replies to deployment requests. Currently the message parsing is done in the core code i.e. modules/s3/s3msg.py, to be particular, in the parse_message() method. The parsing rules will be defined in private/templates/default/parser.py which allows for hosting of multiple profile options in the main code. This also enforces the on-going work in the development of the Profile Layer, in which deployment-specific files are separated from core code.The parsing module utilizes a data model "msg_workflow” to link the source and the workflow to schedule tasks. Processing of OpenGeoSMS encoded messages is also an important area to work on especially for the existing Android Client, for which it will be of real use. Also to provide robustness and extend the existing code, the pyparsing Parser module can be incorporated or any other parsing generator; which will be subjective to the parsing needs.

Project Plan

Project Deliverable: The project aims at parsing inbound messages such as SMS from CERT responders after deployment. It enables the processing of responses to deployment notifications; which is essentially controlled by the module to which the message is routed.

Project Justification: Parsing of inbound messages is a critical utility for a trained volunteer group such as CERT(Community Emergency Response Teams) where communication between various deployments and volunteers play a vital role. As this will be a deployment-specific option, the functionality becomes an important component for Sahana Eden.

Implementation Plan:

• Keeping the development of Profile Layer in mind and the functionality being a part of deployment-specific options, the rules for parsing are contained in private/templates/default/parser.py.

• The module contains a class S3ParsingModel which contains the “msg_workflow” data model (See ​https://docs.google.com/document/d/1Y9dDCshurrZSw33r-RC_uVQ_Va6_LEZM-2aLcaT2Krc/edit?pli=1) and another class S3Parsing in which the parsing routines are defined which decide the various parsing workflows.

The current parsing rules implement the functionality in the following manner:

1. The inbound message text is passed as an argument to the parse_message() method in the s3msg.py module.
2. The text is matched with a predefined list of primary and contact keywords after splitting with whitespace as the delimiter.
3. A database query is generated to the concerned database according to the matched keywords.
4. The query retrieves the relevant field values and generates a reply to the inbound message query.
5. Also these parsing rules have been implemented only for modules – ‘Person’ , ‘Hospital’ and ‘Organisation’.

Extending these rules to other modules can be in scope of the project. *One of the main issues will be identifying the messages that belong to a particular source, so it could have its own processing.Now, that here is handled by the data model which defines a ‘msg_workflow' table in the database which links the Source to the Workflow with any required args.So the essential features of this approach have been listed below:

1. The Parser workflow table links 'Source X' to 'Workflow Y'.
2. Now, designing the details of the Workflow Y would be a developer task.
3. Whereas linking ‘Source X’ to ‘Workflow Y’ will be a configurable option.
4. So essentially,the Parser Table links Source to Workflow with any other required args & this acts like a Template for the schduler_task table.

• Now, a task process_log() is defined in tasks.py , where the objective of process_log() is to scan through all the messages in msg_log; and process those for parsing which are flagged as unparsed (is_parsed=False).The task is scheduled in zzz_1st_run.py where it is chained to the concerned parsing task(this is achieved by the msg_workflow table, the ‘source_task_id’ field in msg_log will help retrieve the respective parsing workflow_task_id from msg_workflow).
• Also,this allows for chaining of workflows where a source for a workflow could be another workflow instead of an Incoming source.We can have 2nd-pass Parser workflows which don't start from the Source direct but can plugged as output from a 1st-pass one.

Source -> process_log() ->1st pass parser -> detailed Parser ---> Module

• Here,the 1st pass parser is customized per-deployment;and decides which email source goes to a particular workflow (simple msg_workslow link) or decides based on other factors such as keywords to which main workflow the messages should be passed.
• The data model is integrated with the templates folders (or a sub-folder say private/templates/parsing) which serves as the initial UI.The post-install UI will consist of a CRUD interface admin panel, a simple s3_rest_controller().However, eventually this is planned to be the part of the WebSetup.
• We want to be able to direct the message to the appropriate module to handle the data.This could be done either by launching a real REST request or else simulating one via the API.

  resource = s3mgr.define_resource("module", "resourcename")
  

• Messages which are routed to a specific resource can be subscribed to by the user.For this purpose,we can use the existing Save Search and Subscription functionality where the user can subscribe to new messages for a specific resource using a resource filter.The msg_log can be made a component for the resources.Now,if it's a component, then when someone opens the resource, messages will be there in a tab.Also, if the message has to be tied to multiple resources, then we can use a relationship (link) table.
• Implementing/extending the utility for other modules especially the IRS module will be of real use, where enabling to log reports through SMS will be vital, which can also use the OpenGeoSMS encoding standards(LatLon generates a google-maps URL) for integration with our Android Client. A dedicated routine to generate OpenGeoSMS URLs already exists in prepare_opengeosms() in s3msg.py itself. So integration with the parsing routine won’t be difficult. Other modules for which this can be implemented are : ‘Request’ and ’Inventory’.
• Finally the code will be tested on the system and the bugs (if any ;-) ) will be fixed.

Future Options:

• Though the parsing rules will be generic , a few minor tweaks for other processes such as Email and Twitter will have to be performed to maintain its generic nature.
• One of the most valuable functionality that can be added here is to make the SMS communication more interactive. e.g. the text body received does not match any of the expected keywords , the API dispatches a reply stating the expected format of the message.
• Adapting the parsing rules to cover as wide a base of inbound messages as possible. This will involve making a wider collection of keywords to be searched for every concerned module.Linking different labels across the DB to module-specific keywords will be really helpful.Also the list of primary keywords to be matched can also be made a deployment-specific option.

Data Model

Data Model Blueprint: msg_workflow

Field Name	Purpose	Datatype/table
source_task_id	Inbound Email/SMS Source(Another workflow in case of chained workflows).	FK referenced to scheduler_task.
workflow_task_id	Parsing Workflow	FK referenced to scheduler_task.
s3.meta_fields()		metadata

Changes in msg_log:

Field Name	Purpose	Datatype/table
is_parsed	Parsing Status of Inbound Messages	Boolean
source_task_id	Inbound Email/SMS Source(Another workflow in case of chained workflows).	FK referenced to scheduler_task.

Project Status

First trimester:

SMART Goal	Status
The msg_log data model is tweaked.	Completed
process_log() is put in place to schedule the parsing of inbound messages.	Completed
Parsing Workflow layout is drafted	Completed

Second Trimester:

SMART Goal	Status
Parsing Rules moved from modules/s3msg.py to private/templates/default/parser.py	Completed
msg_workflow data model defined.	Completed
Import parsing workflow tasks to s3msg.py	Completed
deployment_settings to define enabled parsers.	Completed
Link the different inbound sources to workflows	Completed

Timeline

First Trimester:

Due Date	SMART Goal	Measure	Status
(24th April - 7th May)	1.Design of the workflow handling mechanism. 2.process_log() method is defined. 3.Tweaks in msg_log datamodel.	1.Community bonding period. 2.Decision to outline the template.	Completed
(8th May – 21st May)	1.The msg_workflow data model is developed. 2.S3ParsingModel starts to take shape.	Code committed locally.	Completed

Second Trimester:

Due Date	SMART Goal	Measure	Status
28th May -*(won’t be available due to university exams )
4th June	1.Parsing Rules moved to private/templates/default/parser.py. 2.Scheduler logic for parsing workflows is developed.	SMS response processing starts to take shape.	Completed
11th June	1.API designed to link source-->workflow 2.Bug fixing in fetch_inbound_email() 3.parse_import() replaces process_log()	1.Code committed locally. 2.Tested on local system.	Completed
17th June	1.Scheduler logic moved to s3msg.py form zzz_1st_run.py. 2.deployment_setting.parser.parser_enabled defined.	Code committed to trunk.	Completed
2nd July	1.Message Parsing UI designed. 2.UI for Email settings. 3.deployment_setting.parser.template defined and parser_enabled removed. 4.Data model msg_workflow redefined.	Code committed to trunk.	Completed
8th July	1.Integration of the IRS module for incident reporting through SMS.	1.Commits so far tested thoroughly. 2.Bug fixing. 3.Code committed to trunk.[Mid-Term evaluations :-) ]	In Progress

Third Trimester:

Due Date	SMART Goal	Measure
15th July	1.Integration with templates folders. 2.Development of post-install UI (CRUD interface admin panel).	Code committed locally.
22nd July	1.Routing mechanism to resources. 2.Save search and subscription implemented for msg_log.	Code committed to trunk.
29th July	1.OpenGeoSMS (process_opengeosms() ) routine is tweaked and linked with the respective parsing methods. 2.The existing functionality in the Android Client is tested .	Code committed to trunk.
5th August	1.Clickatell functionality is developed for eden. 2.Clickatell allows for a more robust testing mechanism. 2.The functionality is extended to other modules (if needed). 3.Feedback from mentors.	Code committed locally.
13th August	1.System testing & Bug fixing. 2.Final changes to the code are applied.	1.Project reaches final stage. 2.Bug fixes 3.Final Code committed to trunk.
20th August	PENCILS DOWN	Project Completed. :-)

Weekly Meeting

Fridays 13:00 UTC