Version 156 (modified by 12 years ago) ( diff ) | ,
---|
Blueprint for the Messaging Framework
Table of Contents
Core Architecture
Inbound Messages
- When a message is picked up by the handler, it decides whether it's an XForm or not.
- If it is, then it is routed direct to the relevant resource.
- If not then it is added to the Master Message Log (
msg_log
) - When new messages arrive in the Master Message Log, then the Parser looks for keywords & tries to Route automatically - either to a resource or to a
pe_pentity
(NB 'routing' to a resource is actually just 'tagging').
Routing within Eden
- Users can view the Master Message Log via the 'Ticketing Module' UI & do manual Routing - either to a resource or a
pe_pentity
. - Users can Subscribe to Resources in order to receive copies of all messages which are routed to a resource.
- This is done by pressing the 'Subscribe' button on the page for that Resource.
- Button becomes 'Unsubscribe' if user already subscribed.
- This is done by pressing the 'Subscribe' button on the page for that Resource.
- When a user visits a Resource via it's Web UI, then the RESTController scans the
msg_log
& provides access to associated messages in the views (display/update views have a list of associated messages in a table below the main record. list views have an extra column showing the number of messages tagged to this resource & number unread by this user (unread(read)
) which is hyperlinked to a view which displays these records.
Outbound Messages
- If a pentity has a message sent to them (they subscribed or routed to them automatically or manually) then as well as being tagged for them, a copy is placed in the Outbox with them as recipient. Their list of Contacts is browsed & the appropriate contact method is used, depending on the priority of the message (e..g. a user may want to be sent an SMS for a high priority message, such as a security alert, but just be sent an email if it's just an update to one of their projects). If there are multiple methods available for a priority level then all methods are used at once - each having a separate entry in the outbox.
- XML exports (such as RSS feeds) include tagged messages as linked resources:
- S3ResourceController has a hook for a function that retrieves all message URLs for a resource: s3xrc = s3xrc.S3ResourceController(..., messaging=<function>)
- This function takes the the resource name (=full tablename incl. prefix) and the record ID as arguments, and returns a list of URLs to messages for this resource and user, where each list item is (url, timestamp), with timestamp refering to the message timestamp
- The XML exporter inserts one element <message url="..." unread="True/False"/> per message and resource element as child of the <resource> tag
- This can be used for any feeds that are produced of the XML exporter
- Requires:
make RSS feeds an XML export template
- From address for emails can be set by changing mail.settings.sender under the hood.
- If using gmail as Server for Outbound Email messages then this will override mail.settings.sender, so need to integrate this
- Grey out the setting & explain why in Help if this option is set
- Allow a per-user Outbound server override
- Could use GoogleVoice for Outbound SMS: http://code.google.com/p/pygooglevoice/
Message Flows
How do we achieve this?
- Table
msg_log
is used as a replacement for the currentmsg_inbox
&ticket_log
.- Fields: sender, subject, body, verified, actionable, assignee (multiple?), actioned, priority
- Q: Do we move some of these 'ticketing' fields to a separate, linked
ticket_msg
table?
- Q: Do we move some of these 'ticketing' fields to a separate, linked
- Fields: sender, subject, body, verified, actionable, assignee (multiple?), actioned, priority
- Table
msg_tag
is used to tag messages for resources.- Fields: message_id, resource, record (uid)
- Q: How to display this nicely?
- A: Give the message a tab with all distribution resources (db.tables). Once a table is selected, a second selector with all UUIDs (represented, of course) in that table appears, so you can choose the target. Use s3xrc.model.configure to add a name_nice for each table.
- Q: How to display this nicely?
- Fields: message_id, resource, record (uid)
- Table
pe_subscription
is used to maintain subscriptions for ape_pentity
.- Fields: pr_pe_id, resource, record (uid)
- Subscriptions can be added/deleted from the linked resource
- Add to Views (layout.html)
- http://myserver.org/eden/msg/subscribe?resource=org_org&uid=XXX
- Controller needs writing
- Add to Views (layout.html)
- Subscriptions can be viewed from the user's subscriptions page
Add to 01_menu.py & pr.py
- Table
msg_outbox
is used to identify messages which need to be routed externally & store delivery statuses.- Fields: message_id, pe_contact_id, status
- Table
msg_read_status
is used to show that a message has been read by a user.- Fields: message_id, auth_user_id ('read' is assumed)
- Q: Do we mark as read if pe_contact_id status suggests person has received message? (Fran thinks not)
- Good to have UI to 'Mark Unread'
- Parser has hooks for modules to plug-into?
- Q: Do we treat a pentity the same way as any other resource?
- Just that when a message is routed to them, it not only tags but also creates an entry in
msg_outbox
, which is scanned by the Cron task for 'pending' messages to be sent out.
- Just that when a message is routed to them, it not only tags but also creates an entry in
s3_rest_controller()
is modified to check msg_log for messages tagged to the resource (list) or record (display/update)- Views modified as above
- 'Ticketing Module' UI
- default view is all un-actioned messages (assigned to this user & unassigned?)
- If a message is routed to a resource/pentity, this results in a 'create' form popup with as many fields auto-populated as possible (subject into 'name' field & body into 'comments' field, if nowhere better)
- See 'Create Report' in Ushahidi for an example of the workflow here.
- Q: If a message is routed to a resource, does this automatically mark ticket as 'actioned'? Provide a popup to ask this?
- Auth restrictions
- Message settings (Gateway) is a 1-1 component of Organisation, Office & Person
- When Messages are sent, Person settings are checked, if not found then Office settings are checked, if not found then Organisation settings are checked, if not found then Default settings are used. For speed, all records are read when daemon launched & stored in session.s3.msg. An oncreate/onupdate/ondelete callback forces a refresh of this session data.
- Gateway usage should be auditable for billing purposes (i.e. allow a recharge)
Parser
See BluePrint/Messaging/Parser
User Interface
The home page should be a common Inbox view which, by default, shows all new unread Items (the 'Ticketing Module').
Users/Groups should be able to subscribe to incoming messages of a certain type (Project / Location / Organisation) & have these messages forwarded via SMS / Email / Twitter / etc.
Ideally there should be the ability to launch a popup/ticker to notify of new items & a portlet which can be on a user's dashboard when they login.
The 'Send Message/Alert' screen should have a small box to enter the Plain-text message which is sent to all recipients (using all available SMS/Email addresses in the Group specified).
There should be a 2nd box to create a Rich-text message, with the option of adding attachments. This content just gets sent to those members of the group with an Email address specified.
If a user is added to a group we warn about any missing details, but still allow addition (the ref is by pe_pentity.id so addresses can be added/changed later anyway).
When sending we need to provide warning messages about users for whom details weren't available.
Transports
The email settings should be made web-configurable as part of the Messaging Module
- however also need to be used by Auth (order issues)
Rich Text
We'd like to be able to send HTML mails for rich-text support (including attachments) This requires the use of smtplib to send MIME-encoded files:
- http://groups.google.com/group/web2py/browse_thread/thread/3d8ed693dd2f29bc
- https://web2py.com/wiki/default/page/98b7448f-059a-47f7-82da-dce4728aa4dd
- Simple app for sending mails from a form (using smtplib): http://web2py.com/appliances/default/show/10
SMS
Support both online gateways (such as Clickatell, AWS SES or SMPP-compatible) & local hardware such as:
- a Bluetooth mobile phone
- MobiGater – Takes 1 sim card. Cost approx EUR 70.
- VoiceBlue Lite – Takes 4 sim cards and can handle 4 concurrent calls. Cost approx EUR 900.
- Multi-Tech MultiModem GPRS MTCBA-G-F4
The SMS Handler Daemon to handle Inbound messages is a separate Python script.
- Routes incoming message according to whether it's using specialist micro-syntax (e.g. submitting an XForm to another controller)
- Non-XForms messages get sent to the Ticketing module's Master Message Log
- Tagged messages also get Routed to the appropriate area (Project / Location / Organisation)
It can distinguish between simple messages (added to the Inbox of the Messaging Module) & those which are encoded in Binary XML (the output of XForms from the J2ME client).
Implementation details are here:
- http://wiki.sahanafoundation.org/doku.php/dev:pythonsms
- MobileAndroid#SMSStructure
- BluePrint/Messaging/OpenGeoSMS
We use Cron to check Outbox & signal the daemon to send Pending messages (as for email)
SMS alerts (security alerts more common than natural disasters):
- Being able to trigger an SMS alert broadcast upon reception of an SMS
- Is this just an XForm to a Group? (Can we pre-populate the XForm to do this whenever a certain number is called or just a single word routes here?)
SMS login / data entry / reports
Would be good to add an LCR (Least-Cost Routing engine) for SMS deliveries (so that e.g. local messages routed via modem, but International via Gateway).
Toolkits we can base on:
- SlingshotSMS (PyGSM-based with CherryPy REST interface): http://code.developmentseed.org/slingshotsms/dashboard
- RapidSMS (PyGSM-based): http://rapidsms.org
- PySerial: http://pyserial.wiki.sourceforge.net/pySerial
- Nuntium: http://code.google.com/p/nuntium/
- Gammu: http://www.gammu.org (Python-gammu: http://www.gammu.org/wiki/index.php?title=Gammu:Python-gammu)
- Gnokii: http://www.gnokii.org (PyGnokii: http://mobilehacking.org/index.php/PyGnokii)
- Kannel: http://kannel.org (PyKannel: http://mobilehacking.org/index.php/PyKannel)
- Can be used as an SMPP gateway: http://www.kannel.org/faq.shtml#3.4
- SMSTools: http://smstools3.kekekasvi.com
Hardware compatibility databases:
SMPP
- http://en.wikipedia.org/wiki/Short_message_peer-to-peer_protocol
- http://code.google.com/p/smpplib/ (2009/Mar 2010)
- http://sourceforge.net/projects/pysmpp2/ (2006)
- http://pysmpp.sourceforge.net (old: 2003)
APRS
Automatic Packet Reporting System used by Amateur Radio networks:
CAP
See the 2012 GSoC CAP project, CAP usage, and CAP Broker BluePrint.
Common Alerting Protocol:
Originating CAP alerts is one need, receiving them and using them to actuate alerting systems, be they SMS drivers or sirens or whatever is another. Another is a message broker that can interoperate with the commercial boxes. Authentication systems are also a factor, especially where cross-jurisdictional reciprocity is required
Ideally the CAP XML should be signed end-to-end, especially in a federated system where there's more than one authority running servers/aggregators. Our current implementations sign the alerts at the originating server, so they're at least traceable to the source server but not necessarily the individual operator. Current servers check integrity when they get a message from another server, but there isn't a regional PKI.
- http://docs.oasis-open.org/emergency/cap/v1.2/CAP-v1.2.html
- Use an XSLT
- Subset of EDXL-DE: http://docs.oasis-open.org/emergency/edxl-de/v1.0/EDXL-DE_Spec_v1.0.pdf
- http://talksahana.com/2009/03/04/firefox-browser-cap-alerting-plugin-sahana-idea-for-gsoc2009/
- Sahana CAP/EDXL Broker specification - http://lirneasia.net/wp-content/uploads/2009/05/Sahana-CAP-Msg-Mod-v0.2.pdf
- Model definitions have been started in models/msg.py
Wireframe (ported to eden)
- The wireframe built using Python and Eden frameworks. The code is here. Anyone can use this as the basis to continue the CAP Broker developments.
- This ticket contains some of the screen shots and initial GUI layouts with some descriptions.
Adding support for sending out/receiving alerts via Twitter & mining a hashtag for potential value:
- store tweets from 1 or more hastags e.g. #eqnz #canterburyquake to build up a local repo of all relevant tweets
- be able to reply/retweet to these messages e.g. provide an official response
- provide analytics on things like - who are the most active users and retweeters e.g. which to target for sending out messages and who is most likely to retweet
- if possible try and cluster similar/identical tweets to reduce the amount of noise - making it faster and simpler to monitor what is going on. As well as automatic clustering, it would be nice to provide users the ability to manually drag a tweet to an existing cluster so that it is removed from the master timeline (so as to simplify the timeline viewed by the user)
- be able to link tickets/actions to specific tweet
- it should be possible to automatically load a hashtag that is included in all outgoing messages e.g. we would have setup #eqnz to be included in all outgoing tweets. this may change during the event e.g. the Ministry of Civil Defence Emergency Management started with #CanterburyQuake but this was soon shortened and overtaken by #eqnz as being shorter and more useable.
Status
- Tropo has been integrated for a basic Twitter bot support:
- OAuth support has been added using Tweepy
ToDo
msg.send_tweet_by_pe_id
using tweepy- Usecases:
- User subscribes to notifications via Twitter
- We send out alerts to a Hashtag
- Usecases:
- Process Inbound messages to (one of) the Sahana instance's Twitter IDs
- Generic messages for the InBox
- Structured message meant for an application (create a new record or update an existing one)
- Hashtag Harvesting: A User wants to search public hashtag(s) for tweets of interest
- User has a UI to create a filter of which hashtags (& keywords?) they're interested in
- A real-time view is returned of the Tweets which match the filter (probably using JSON)
- # of matches
- 1st 10(-50?) records are browseable to see what they look like
- If there are too many/insufficient matches then user may refine the search & repeat until they're happy
- User saves the query to a msg_twitter_search table
5.
- Option A: A single table for all queries' search results is created
- Option B: A Table is created to hold the results of the search for each record in this search table (similar to the survey module's instances)
- Cron queries Twitter & adds new records to the table as they are found
- When user opens the search results table, they can route messages to applications, e.g.
- 'This is a Request'
- 'This is an Assessment'
- 'This is a Shelter'
Links
- Simple Web2Py example of posting/reading
- OAuth support in Web2Py
- Twitter API
- Python module
- http://timmcnamara.co.nz/post/1349821774/preliminary-thoughts-on-a-real-time-web-crisis
- http://epic.cs.colorado.edu/tweak-the-tweet/helping_haiti_tweak_the_twe.html
- http://blogs.wsj.com/digits/2010/08/18/twitter-after-a-disaster-is-it-reliable/?mod=wsj_share_twitter
- http://eddy.stamen.com
XForms
We have controllers/xforms.py
to generate XForms automatically out of Sahana models.
These can be used by the Mobile clients for data collection.
This can be used to create OCR-able Printed Forms.
XMPP
Instant Messaging & Presence including GoogleTalk:
Alternate implementations
- Agasti:
- Nuntium - generic Channels for both Application-Terminated & Application-Originated Messaging (Ruby code from InSTEDD)
- GeoChat
- Frontline SMS (used by Ushahidi)
- RapidSMS - also uses XForms so could be easy to integrate?
- Microsoft Vine
Use Cases
Security Management
- Allow users to subscribe for SMS security alerts
- Ideally allow subscription via SMS too (with optional confirmation by admin staff)
- Allow security staff to broadcast security alerts by sending the alert to a single number & have it propagate on
- Security for this managed by either pre-registering their numbers as valid or by logging-in 1st
- End-users should be able to report back their status via SMS
- Security staff should receive a report via SMS of who hasn't reported in (ideally with last known Location).
Volunteer Management
- Allow team leaders to be able to send messages to their team members.
- Allow team members to select which communications means (Email/SMS/Twitter) to use for both normal & priority messages.
- Allow users to submit reports via SMS/Email/Twitter/iPadio.
- Allow users to subscribe to updates via SMS/Email/RSS.
Attachments (1)
-
Message Flows.png
(20.8 KB
) - added by 15 years ago.
Message Flows
Download all attachments as: .zip