[[TOC]]
= Blueprint for the Messaging Module =
The Messaging Module is a modular system for getting Messages in & out of Sahana Eden & routed around it.

== Core Architecture ==
=== Inbound Messages ===
 * When a message is picked up by the handler, it decides whether it's an XForm or not.
 * If it is, then it is routed direct to the relevant resource.
 * If not then it is added to the Master Message Log ({{{msg_log}}})
 * When new messages arrive in the Master Message Log, then the Parser looks for keywords & tries to Route automatically - either to a resource or to a {{{pe_pentity}}} (NB 'routing' to a resource is actually just 'tagging').
=== Routing within Eden ===
 * Users can view the Master Message Log via the 'Ticketing Module' UI & do manual Routing - either to a resource or a {{{pe_pentity}}}.
 * Users can Subscribe to Resources in order to receive copies of all messages which are routed to a resource.
  * This is done by pressing the 'Subscribe' button on the page for that Resource.
   * Button becomes 'Unsubscribe' if user already subscribed.
 * When a user visits a Resource via it's Web UI, then the [wiki:RESTController] scans the {{{msg_log}}} & provides access to associated messages in the views (display/update views have a list of associated messages in a table below the main record. list views have an extra column showing the number of messages tagged to this resource & number unread by this user ({{{unread(read)}}}) which is hyperlinked to a view which displays these records.
=== Outbound Messages ===
 * If a pentity has a message sent to them (they subscribed or routed to them automatically or manually) then as well as being tagged for them, a copy is placed in the Outbox with them as recipient. Their list of Contacts is browsed & the appropriate contact method is used, depending on the priority of the message (e..g. a user may want to be sent an SMS for a high priority message, such as a security alert, but just be sent an email if it's just an update to one of their projects). If there are multiple methods available for a priority level then all methods are used at once - each having a separate entry in the outbox.
 * XML exports (such as RSS feeds) include tagged messages as linked resources:
   * S3ResourceController has a hook for a function that retrieves all message URLs for a resource: s3xrc = s3xrc.S3ResourceController(..., messaging=<function>)
   * This function takes the the resource name (=full tablename incl. prefix) and the record ID as arguments, and returns a list of URLs to messages for this resource and user, where each list item is (url, timestamp), with timestamp refering to the message timestamp
   * The XML exporter inserts one element <message url="..." unread="True/False"/> per message and resource element as child of the <resource> tag
   * This can be used for any feeds that are produced of the XML exporter
   * Requires: ~~make RSS feeds an XML export template~~

=== Message Flows ===
[[Image(Message Flows.png)]]

=== How do we achieve this? ===
 * Table {{{msg_log}}} is used as a replacement for the current {{{msg_inbox}}} & {{{ticket_log}}}.
  * Fields: sender, subject, body, verified, actionable, assignee (multiple?), actioned, priority
   * Q: Do we move some of these 'ticketing' fields to a separate, linked {{{ticket_msg}}} table?
 * Table {{{msg_tag}}} is used to tag messages for resources.
  * Fields: message_id, resource, record (uid)
   * Q: How to display this nicely?
    * A: Give the message a tab with all distribution resources (db.tables). Once a table is selected, a second selector with all UUIDs (represented, of course) in that table appears, so you can choose the target. Use s3xrc.model.configure to add a name_nice for each table.
 * Table {{{pe_subscription}}} is used to maintain subscriptions for a {{{pe_pentity}}}.
  * Fields: pr_pe_id, resource, record (uid)
  * Subscriptions can be added/deleted from the linked resource
   * Add to Views (layout.html)
    * http://myserver.org/eden/msg/subscribe?resource=or_org&uid=XXX
    * Controller needs writing
  * Subscriptions can be viewed from the user's subscriptions page
   * ~~Add to 01_menu.py & pr.py~~
 * Table {{{msg_outbox}}} is used to identify messages which need to be routed externally & store delivery statuses.
  * Fields: message_id, pe_contact_id, status
 * Table {{{msg_read_status}}} is used to show that a message has been read by a user.
  * Fields: message_id, auth_user_id ('read' is assumed)
  * Q: Do we mark as read if pe_contact_id status suggests person has received message? (Fran thinks not)
  * Good to have UI to 'Mark Unread'
 * Parser has hooks for modules to plug-into?
 * Q: Do we treat a pentity the same way as any other resource?
  * Just that when a message is routed to them, it not only tags but also creates an entry in {{{msg_outbox}}}, which is scanned by the Cron task for 'pending' messages to be sent out.
 * {{{shn_rest_controller()}}} is modified to check msg_log for messages tagged to the resource (list) or record (display/update)
  * Views modified as above
 * 'Ticketing Module' UI
  * default view is all un-actioned messages (assigned to this user & unassigned?)
  * If a message is routed to a resource/pentity, this results in a 'create' form popup with as many fields auto-populated as possible (subject into 'name' field & body into 'comments' field, if nowhere better)
   * Q: If a message is routed to a resource, does this automatically mark ticket as 'actioned'? Provide a popup to ask this?
 * [BluePrintAuthorization#SpecificExamples Auth restrictions]

== Parser ==
[http://pyparsing.wikispaces.com Pyparsing] is included in the modules folder.

This can decode messages, especially from compressed formats such as Tweets or SMS but also free-from formats such as email.

We could add further text processing such as our implementation of [http://en.wikipedia.org/wiki/Levenshtein_distance Levenshtein].
==== Micro-Syntax ====
 * [http://en.wikipedia.org/wiki/Geohash GeoHashes]
  * [http://pypi.python.org/pypi/Geohash/ Python lib]
 * Tweets
  * [http://dev.twitter.com/doc/get/geo/:id Geotagging] where hardware/software combination available: place_id from http://twitter.com/{username}/statuses/{tweet_id}.json -> http://api.twitter.com/1/geo/id/{place_id}.json
  * otherwise use WW place WW message:
   * http://www.weather.gov/stormreports/
   * http://www.srh.noaa.gov/media/epz/Twitter/TwitterStormReportProgram.wmv
  * EPIC's [http://epic.cs.colorado.edu/tweak-the-tweet/helping_haiti_tweak_the_twe.html Tweak the Tweet]
 * SMS codes looked up on a paper wheel:
  * http://mekongict.org/2010/06/user-oriented-design/ (slide 23 on)
  * http://ndt.instedd.org/2010/05/it-without-software.html
== User Interface ==
The home page should be a common Inbox view which, by default, shows all new unread Items (the 'Ticketing Module').

Users/Groups should be able to subscribe to incoming messages of a certain type (Project / Location / Organisation) & have these messages forwarded via SMS / Email / Twitter / etc.

Ideally there should be the ability to launch a popup/ticker to notify of new items & a portlet which can be on a user's dashboard when they login.

The 'Send Message/Alert' screen should have a small box to enter the Plain-text message which is sent to all recipients (using all available SMS/Email addresses in the Group specified).

There should be a 2nd box to create a Rich-text message, with the option of adding attachments. This content just gets sent to those members of the group with an Email address specified.

If a user is added to a group we warn about any missing details, but still allow addition (the ref is by pe_pentity.id so addresses can be added/changed later anyway).

When sending we need to provide warning messages about users for whom details weren't available.

== Transports ==
=== Email ===
{{{gluon/tools.py}}} has mail settings:
{{{
mail=MailS3()
# These settings should be made configurable as part of the Messaging Module
mail.settings.server='mail:25'
mail.sender='sahana@sahanapy.org'
}}}
& a function:
{{{
mail.send(to, subject='None', message='None')
}}}

Can use this to send emails with dynamic data inserted into templates (like HTML pages):
 * http://groups.google.com/group/web2py/browse_thread/thread/f6c903f9c67cfe2e

There is a Cron job to check the contents of the Outbox & send pending messages. On Success it marks the message as 'Sent':
 * http://web2py.com/examples/default/cron

==== Rich Text ====
We'd like to be able to send HTML mails for rich-text support (including attachments)
This requires the use of smtplib to send MIME-encoded files:
 * http://groups.google.com/group/web2py/browse_thread/thread/3d8ed693dd2f29bc
 * https://web2py.com/wiki/default/page/98b7448f-059a-47f7-82da-dce4728aa4dd
 * Simple app for sending mails from a form (using smtplib): http://web2py.com/appliances/default/show/10

=== SMS ===
Support both online gateways (such as Clickatell) & local hardware such as:
 * a Bluetooth mobile phone
 * [http://www.mobigater.bg MobiGater] – Takes 1 sim card. Cost approx EUR 70.
 * [http://www.2n.cz/products/gsm_gateways/voip_gsm_gateway/voiceblue_voip_gsm_gateway.html VoiceBlue Lite] – Takes 4 sim cards and can handle 4 concurrent calls. Cost approx EUR 900.
 * [http://www.google.com/products?client=safari&rls=en-us&q=multimodem+mtcba-g-f4&oe=UTF-8&um=1&ie=UTF-8&ei=H-6BSpO1DJS-NpC6gaAL&sa=X&oi=product_result_group&ct=title&resnum=4 Multi-Tech MultiModem GPRS MTCBA-G-F4]

The SMS Handler Daemon to handle Inbound messages is a separate Python script.
 * Routes incoming message according to whether it's using specialist micro-syntax (e.g. submitting an XForm to another controller)
 * Non-XForms messages get sent to the [wiki:BluePrintTicketing Ticketing] module's Master Message Log
 * Tagged messages also get Routed to the appropriate area (Project / Location / Organisation)

It can distinguish between simple messages (added to the Inbox of the Messaging Module) & those which are encoded in Binary XML (the output of XForms from the J2ME client).

Implementation details are here: http://wiki.sahanafoundation.org/doku.php/dev:pythonsms

We use Cron to check Outbox & signal the daemon to send Pending messages (as for email)

SMS alerts (security alerts more common than natural disasters):
 * Being able to trigger an SMS alert broadcast upon reception of an SMS
  * Is this just an XForm to a Group? (Can we pre-populate the XForm to do this whenever a certain number is called or just a single word routes here?)

SMS login / data entry / reports
 * [wiki:BluePrintMessagingModuleJ2ME J2ME client]

Would be good to add an LCR (Least-Cost Routing engine) for SMS deliveries (so that e.g. local messages routed via modem, but International via Gateway).

Toolkits we can base on:
 * SlingshotSMS (PyGSM-based with !CherryPy REST interface): http://code.developmentseed.org/slingshotsms/dashboard
  * http://developmentseed.org/blog/2009/aug/14/slingshotsms-alpha-code-released-lightweight-sms-gateway-stick
 * RapidSMS (PyGSM-based): http://rapidsms.org
 * !PySerial: http://pyserial.wiki.sourceforge.net/pySerial
 * Gammu: http://www.gammu.org (Python-gammu: http://www.gammu.org/wiki/index.php?title=Gammu:Python-gammu)
 * Gnokii: http://www.gnokii.org (!PyGnokii: http://mobilehacking.org/index.php/PyGnokii)
 * Kannel: http://kannel.org (!PyKannel: http://mobilehacking.org/index.php/PyKannel)
 * SMSTools: http://smstools3.kekekasvi.com

Hardware compatibility databases:
 * UserGuidelinesMsg#WhatHardwaredoIneed

=== APRS ===
Automatic Packet Reporting System used by Amateur Radio networks:
 * http://en.wikipedia.org/wiki/Automatic_Packet_Reporting_System

=== CAP ===
Common Alerting Protocol:
 * http://xml.coverpages.org/OASIS-CAPv11-200510.pdf
  * Use an XSLT?
 * Subset of EDXL-DE: http://docs.oasis-open.org/emergency/edxl-de/v1.0/EDXL-DE_Spec_v1.0.pdf
  * http://code.google.com/p/pyedxl/source/browse/trunk/edxl/edxl.py
 * http://talksahana.com/2009/03/04/firefox-browser-cap-alerting-plugin-sahana-idea-for-gsoc2009/

=== Twitter ===
Adding support for sending out/receiving alerts via Twitter would be nice.
 * [http://www.web2py.com/AlterEgo/default/show/224 Simple Web2Py example of posting/reading]
 * Hashtags can be parsed using  [wiki:pyparsing].
 * [http://dev.twitter.com/doc Twitter API]

=== XForms ===
We have {{{controllers/xforms.py}}} to generate XForms automatically out of Sahana models.

These can be used by the [wiki:BluePrintMessagingModuleJ2ME J2ME client] for data collection.

This can be used to create [wiki:BluePrintOCR OCR]-able Printed Forms.


=== XMPP ===
Instant Messaging & Presence including !GoogleTalk:
 * http://code.google.com/apis/talk/open_communications.html

== Alternate implementations ==
 * Agasti design: http://wiki.sahanafoundation.org/doku.php?id=dev:msg_archi
 * Agasti User Guide http://wiki.sahanafoundation.org/doku.php?id=doc:message:english
 * [http://instedd.org/geochat GeoChat]
 * [http://www.frontlinesms.com Frontline SMS] (used by Ushahidi)

----
BluePrints