[[TOC]] = Blueprint for the Messaging Module = The Messaging Module is a modular system for getting Messages in & out of Sahana Eden & routed around it. == Core Architecture == === Inbound Messages === * When a message is picked up by the handler, it decides whether it's an XForm or not. * If it is, then it is routed direct to the relevant resource. * If not then it is added to the Master Message Log ({{{msg_log}}}) * When new messages arrive in the Master Message Log, then the Parser looks for keywords & tries to Route automatically - either to a resource or to a {{{pe_pentity}}} (NB 'routing' to a resource is actually just 'tagging'). === Routing within Eden === * Users can view the Master Message Log via the 'Ticketing Module' UI & do manual Routing - either to a resource or a {{{pe_pentity}}}. * Users can Subscribe to Resources in order to receive copies of all messages which are routed to a resource. * This is done by pressing the 'Subscribe' button on the page for that Resource. * Button becomes 'Unsubscribe' if user already subscribed. * When a user visits a Resource via it's Web UI, then the [wiki:RESTController] scans the {{{msg_log}}} & provides access to associated messages in the views (display/update views have a list of associated messages in a table below the main record. list views have an extra column showing the number of messages tagged to this resource & number unread by this user ({{{unread(read)}}}) which is hyperlinked to a view which displays these records. === Outbound Messages === * If a pentity has a message sent to them (they subscribed or routed to them automatically or manually) then as well as being tagged for them, a copy is placed in the Outbox with them as recipient. Their list of Contacts is browsed & the appropriate contact method is used, depending on the priority of the message (e..g. a user may want to be sent an SMS for a high priority message, such as a security alert, but just be sent an email if it's just an update to one of their projects). If there are multiple methods available for a priority level then all methods are used at once - each having a separate entry in the outbox. * XML exports (such as RSS feeds) include tagged messages as linked resources: * S3ResourceController has a hook for a function that retrieves all message URLs for a resource: s3xrc = s3xrc.S3ResourceController(..., messaging=) * This function takes the the resource name (=full tablename incl. prefix) and the record ID as arguments, and returns a list of URLs to messages for this resource and user, where each list item is (url, timestamp), with timestamp refering to the message timestamp * The XML exporter inserts one element per message and resource element as child of the tag * This can be used for any feeds that are produced of the XML exporter * Requires: ~~make RSS feeds an XML export template~~ === Message Flows === [[Image(Message Flows.png)]] === How do we achieve this? === * Table {{{msg_log}}} is used as a replacement for the current {{{msg_inbox}}} & {{{ticket_log}}}. * Fields: sender, subject, body, verified, actionable, assignee (multiple?), actioned, priority * Q: Do we move some of these 'ticketing' fields to a separate, linked {{{ticket_msg}}} table? * Table {{{msg_tag}}} is used to tag messages for resources. * Fields: message_id, resource, record (uid) * Q: How to display this nicely? * A: Give the message a tab with all distribution resources (db.tables). Once a table is selected, a second selector with all UUIDs (represented, of course) in that table appears, so you can choose the target. Use s3xrc.model.configure to add a name_nice for each table. * Table {{{pe_subscription}}} is used to maintain subscriptions for a {{{pe_pentity}}}. * Fields: pr_pe_id, resource, record (uid) * Subscriptions can be added/deleted from the linked resource * Add to Views (layout.html) * http://myserver.org/eden/msg/subscribe?resource=or_org&uid=XXX * Controller needs writing * Subscriptions can be viewed from the user's subscriptions page * ~~Add to 01_menu.py & pr.py~~ * Table {{{msg_outbox}}} is used to identify messages which need to be routed externally & store delivery statuses. * Fields: message_id, pe_contact_id, status * Table {{{msg_read_status}}} is used to show that a message has been read by a user. * Fields: message_id, auth_user_id ('read' is assumed) * Q: Do we mark as read if pe_contact_id status suggests person has received message? (Fran thinks not) * Good to have UI to 'Mark Unread' * Parser has hooks for modules to plug-into? * Q: Do we treat a pentity the same way as any other resource? * Just that when a message is routed to them, it not only tags but also creates an entry in {{{msg_outbox}}}, which is scanned by the Cron task for 'pending' messages to be sent out. * {{{shn_rest_controller()}}} is modified to check msg_log for messages tagged to the resource (list) or record (display/update) * Views modified as above * 'Ticketing Module' UI * default view is all un-actioned messages (assigned to this user & unassigned?) * If a message is routed to a resource/pentity, this results in a 'create' form popup with as many fields auto-populated as possible (subject into 'name' field & body into 'comments' field, if nowhere better) * Q: If a message is routed to a resource, does this automatically mark ticket as 'actioned'? Provide a popup to ask this? * [BluePrintAuthorization#SpecificExamples Auth restrictions] == Parser == [http://pyparsing.wikispaces.com Pyparsing] is included in the modules folder. This can decode messages, especially from compressed formats such as Tweets or SMS but also free-from formats such as email. We could add further text processing such as our implementation of [http://en.wikipedia.org/wiki/Levenshtein_distance Levenshtein]. === Geo === * [http://en.wikipedia.org/wiki/Geohash GeoHashes] * [http://pypi.python.org/pypi/Geohash/ Python lib] * Tweets * [http://dev.twitter.com/doc/get/geo/:id Geotagging] where hardware/software combination available: place_id from http://twitter.com/{username}/statuses/{tweet_id}.json -> http://api.twitter.com/1/geo/id/{place_id}.json * otherwise use WW place WW message: * http://www.weather.gov/stormreports/ * http://www.srh.noaa.gov/media/epz/Twitter/TwitterStormReportProgram.wmv == User Interface == The home page should be a common Inbox view which, by default, shows all new unread Items (the 'Ticketing Module'). Users/Groups should be able to subscribe to incoming messages of a certain type (Project / Location / Organisation) & have these messages forwarded via SMS / Email / Twitter / etc. Ideally there should be the ability to launch a popup/ticker to notify of new items & a portlet which can be on a user's dashboard when they login. The 'Send Message/Alert' screen should have a small box to enter the Plain-text message which is sent to all recipients (using all available SMS/Email addresses in the Group specified). There should be a 2nd box to create a Rich-text message, with the option of adding attachments. This content just gets sent to those members of the group with an Email address specified. If a user is added to a group we warn about any missing details, but still allow addition (the ref is by pe_pentity.id so addresses can be added/changed later anyway). When sending we need to provide warning messages about users for whom details weren't available. == Transports == === Email === {{{gluon/tools.py}}} has mail settings: {{{ mail=MailS3() # These settings should be made configurable as part of the Messaging Module mail.settings.server='mail:25' mail.sender='sahana@sahanapy.org' }}} & a function: {{{ mail.send(to, subject='None', message='None') }}} Can use this to send emails with dynamic data inserted into templates (like HTML pages): * http://groups.google.com/group/web2py/browse_thread/thread/f6c903f9c67cfe2e There is a Cron job to check the contents of the Outbox & send pending messages. On Success it marks the message as 'Sent': * http://web2py.com/examples/default/cron ==== Rich Text ==== We'd like to be able to send HTML mails for rich-text support (including attachments) This requires the use of smtplib to send MIME-encoded files: * http://groups.google.com/group/web2py/browse_thread/thread/3d8ed693dd2f29bc * https://web2py.com/wiki/default/page/98b7448f-059a-47f7-82da-dce4728aa4dd * Simple app for sending mails from a form (using smtplib): http://web2py.com/appliances/default/show/10 === SMS === Support both online gateways (such as Clickatell) & local hardware such as: * a Bluetooth mobile phone * [http://www.mobigater.bg MobiGater] – Takes 1 sim card. Cost approx EUR 70. * [http://www.2n.cz/products/gsm_gateways/voip_gsm_gateway/voiceblue_voip_gsm_gateway.html VoiceBlue Lite] – Takes 4 sim cards and can handle 4 concurrent calls. Cost approx EUR 900. * [http://www.google.com/products?client=safari&rls=en-us&q=multimodem+mtcba-g-f4&oe=UTF-8&um=1&ie=UTF-8&ei=H-6BSpO1DJS-NpC6gaAL&sa=X&oi=product_result_group&ct=title&resnum=4 Multi-Tech MultiModem GPRS MTCBA-G-F4] The SMS Handler Daemon to handle Inbound messages is a separate Python script. * Routes incoming message according to whether it's using specialist micro-syntax (e.g. submitting an XForm to another controller) * Non-XForms messages get sent to the [wiki:BluePrintTicketing Ticketing] module's Master Message Log * Tagged messages also get Routed to the appropriate area (Project / Location / Organisation) It can distinguish between simple messages (added to the Inbox of the Messaging Module) & those which are encoded in Binary XML (the output of XForms from the J2ME client). Implementation details are here: http://wiki.sahanafoundation.org/doku.php/dev:pythonsms We use Cron to check Outbox & signal the daemon to send Pending messages (as for email) SMS alerts (security alerts more common than natural disasters): * Being able to trigger an SMS alert broadcast upon reception of an SMS * Is this just an XForm to a Group? (Can we pre-populate the XForm to do this whenever a certain number is called or just a single word routes here?) SMS login / data entry / reports * [wiki:BluePrintMessagingModuleJ2ME J2ME client] Would be good to add an LCR (Least-Cost Routing engine) for SMS deliveries (so that e.g. local messages routed via modem, but International via Gateway). Toolkits we can base on: * SlingshotSMS (PyGSM-based with !CherryPy REST interface): http://code.developmentseed.org/slingshotsms/dashboard * http://developmentseed.org/blog/2009/aug/14/slingshotsms-alpha-code-released-lightweight-sms-gateway-stick * RapidSMS (PyGSM-based): http://rapidsms.org * !PySerial: http://pyserial.wiki.sourceforge.net/pySerial * Gammu: http://www.gammu.org (Python-gammu: http://www.gammu.org/wiki/index.php?title=Gammu:Python-gammu) * Gnokii: http://www.gnokii.org (!PyGnokii: http://mobilehacking.org/index.php/PyGnokii) * Kannel: http://kannel.org (!PyKannel: http://mobilehacking.org/index.php/PyKannel) * SMSTools: http://smstools3.kekekasvi.com Hardware compatibility databases: * UserGuidelinesMsg#WhatHardwaredoIneed === APRS === Automatic Packet Reporting System used by Amateur Radio networks: * http://en.wikipedia.org/wiki/Automatic_Packet_Reporting_System === CAP === Common Alerting Protocol: * http://xml.coverpages.org/OASIS-CAPv11-200510.pdf * Use an XSLT? * Subset of EDXL-DE: http://docs.oasis-open.org/emergency/edxl-de/v1.0/EDXL-DE_Spec_v1.0.pdf * http://code.google.com/p/pyedxl/source/browse/trunk/edxl/edxl.py * http://talksahana.com/2009/03/04/firefox-browser-cap-alerting-plugin-sahana-idea-for-gsoc2009/ === Twitter === Adding support for sending out/receiving alerts via Twitter would be nice. * [http://www.web2py.com/AlterEgo/default/show/224 Simple Web2Py example of posting/reading] * Hashtags can be parsed using [wiki:pyparsing]. * [http://dev.twitter.com/doc Twitter API] === XForms === We have {{{controllers/xforms.py}}} to generate XForms automatically out of Sahana models. These can be used by the [wiki:BluePrintMessagingModuleJ2ME J2ME client] for data collection. This can be used to create [wiki:BluePrintOCR OCR]-able Printed Forms. === XMPP === Instant Messaging & Presence including !GoogleTalk: * http://code.google.com/apis/talk/open_communications.html == Alternate implementations == * Agasti design: http://wiki.sahanafoundation.org/doku.php?id=dev:msg_archi * Agasti User Guide http://wiki.sahanafoundation.org/doku.php?id=doc:message:english * [http://instedd.org/geochat GeoChat] * [http://www.frontlinesms.com Frontline SMS] (used by Ushahidi) ---- BluePrints