= Blueprint for the Messaging Framework = [[TOC]] == Current Status == See DeveloperGuidelines/Messaging See also UserGuidelines/Messaging == Core Architecture == See BluePrint/Messaging/DataModel Also see BluePrint/Messaging/Old == Parser == See BluePrint/Messaging/Parser == Channels == === APRS === We would like to be able to connect to the Automatic Packet Reporting System used by Amateur Radio networks: * http://en.wikipedia.org/wiki/Automatic_Packet_Reporting_System === CAP === See the [wiki:BluePrint/CAPBroker CAP Broker BluePrint] === Email === The email settings should be made web-configurable as part of the Messaging Module - however also need to be used by Auth (order issues) ==== Rich Text ==== We'd like to be able to send HTML mails for rich-text support (including attachments) This requires the use of smtplib to send MIME-encoded files: * http://groups.google.com/group/web2py/browse_thread/thread/3d8ed693dd2f29bc * https://web2py.com/wiki/default/page/98b7448f-059a-47f7-82da-dce4728aa4dd * Simple app for sending mails from a form (using smtplib): http://web2py.com/appliances/default/show/10 === SMS === Support both online gateways (such as Clickatell, [http://www.web2pyslices.com/slices/take_slice/123 AWS SES] or SMPP-compatible) & local hardware such as: * a Bluetooth mobile phone * [http://www.mobigater.bg MobiGater] – Takes 1 sim card. Cost approx EUR 70. * [http://www.2n.cz/products/gsm_gateways/voip_gsm_gateway/voiceblue_voip_gsm_gateway.html VoiceBlue Lite] – Takes 4 sim cards and can handle 4 concurrent calls. Cost approx EUR 900. * [http://www.google.com/products?client=safari&rls=en-us&q=multimodem+mtcba-g-f4&oe=UTF-8&um=1&ie=UTF-8&ei=H-6BSpO1DJS-NpC6gaAL&sa=X&oi=product_result_group&ct=title&resnum=4 Multi-Tech MultiModem GPRS MTCBA-G-F4] The SMS Handler Daemon to handle Inbound messages is a separate Python script. * Routes incoming message according to whether it's using specialist micro-syntax (e.g. submitting an XForm to another controller) * Non-XForms messages get sent to the [wiki:BluePrintTicketing Ticketing] module's Master Message Log * Tagged messages also get Routed to the appropriate area (Project / Location / Organisation) It can distinguish between simple messages (added to the Inbox of the Messaging Module) & those which are encoded in Binary XML (the output of XForms from the J2ME client). Implementation details are here: * http://wiki.sahanafoundation.org/doku.php/dev:pythonsms * MobileAndroid#SMSStructure * [wiki:BluePrint/Messaging/OpenGeoSMS] We use Cron to check Outbox & signal the daemon to send Pending messages (as for email) SMS alerts (security alerts more common than natural disasters): * Being able to trigger an SMS alert broadcast upon reception of an SMS * Is this just an XForm to a Group? (Can we pre-populate the XForm to do this whenever a certain number is called or just a single word routes here?) SMS login / data entry / reports * [wiki:BluePrintMobileClient Mobile clients] Would be good to add an LCR (Least-Cost Routing engine) for SMS deliveries (so that e.g. local messages routed via modem, but International via Gateway). Toolkits we can base on: * SlingshotSMS (PyGSM-based with !CherryPy REST interface): http://code.developmentseed.org/slingshotsms/dashboard * http://developmentseed.org/blog/2009/aug/14/slingshotsms-alpha-code-released-lightweight-sms-gateway-stick * RapidSMS (PyGSM-based): http://rapidsms.org * !PySerial: http://pyserial.wiki.sourceforge.net/pySerial * Nuntium: http://code.google.com/p/nuntium/ * Gammu: http://www.gammu.org (Python-gammu: http://www.gammu.org/wiki/index.php?title=Gammu:Python-gammu) * Gnokii: http://www.gnokii.org (!PyGnokii: http://mobilehacking.org/index.php/PyGnokii) * Kannel: http://kannel.org (!PyKannel: http://mobilehacking.org/index.php/PyKannel) * Can be used as an SMPP gateway: http://www.kannel.org/faq.shtml#3.4 * SMSTools: http://smstools3.kekekasvi.com Hardware compatibility databases: * UserGuidelinesMsg#WhatHardwaredoIneed ==== SMPP ==== * http://en.wikipedia.org/wiki/Short_message_peer-to-peer_protocol * http://code.google.com/p/smpplib/ (2009/Mar 2010) * http://sourceforge.net/projects/pysmpp2/ (2006) * http://pysmpp.sourceforge.net (old: 2003) === Twitter === Adding support for sending out/receiving alerts via Twitter & [BluePrintMessagingModule#Micro-Syntax mining] a hashtag for potential value: * store tweets from 1 or more hastags e.g. #eqnz #canterburyquake to build up a local repo of all relevant tweets * be able to reply/retweet to these messages e.g. provide an official response * provide analytics on things like - who are the most active users and retweeters e.g. which to target for sending out messages and who is most likely to retweet * if possible try and cluster similar/identical tweets to reduce the amount of noise - making it faster and simpler to monitor what is going on. As well as automatic clustering, it would be nice to provide users the ability to manually drag a tweet to an existing cluster so that it is removed from the master timeline (so as to simplify the timeline viewed by the user) * be able to link tickets/actions to specific tweet * it should be possible to automatically load a hashtag that is included in all outgoing messages e.g. we would have setup #eqnz to be included in all outgoing tweets. this may change during the event e.g. the Ministry of Civil Defence Emergency Management started with #CanterburyQuake but this was soon shortened and overtaken by #eqnz as being shorter and more useable. ==== !ToDo ==== * {{{msg.send_tweet_by_pe_id}}} using tweepy * Usecases: * User subscribes to notifications via Twitter * We send out alerts to a Hashtag * Process Inbound messages to (one of) the Sahana instance's Twitter IDs * Generic messages for the !InBox * Structured message meant for an application (create a new record or update an existing one) * Hashtag Harvesting: A User wants to search public hashtag(s) for tweets of interest 1. User has a UI to create a filter of which hashtags (& keywords?) they're interested in 2. A real-time view is returned of the Tweets which match the filter (probably using JSON) * # of matches * 1st 10(-50?) records are browseable to see what they look like 3. If there are too many/insufficient matches then user may refine the search & repeat until they're happy 4. User saves the query to a msg_twitter_search table 5. * Option A: A single table for all queries' search results is created * Option B: A Table is created to hold the results of the search for each record in this search table (similar to the survey module's instances) 6. Cron queries Twitter & adds new records to the table as they are found 7. When user opens the search results table, they can route messages to applications, e.g. * 'This is a Request' * 'This is an Assessment' * 'This is a Shelter' ==== Links ==== * [http://www.web2py.com/AlterEgo/default/show/224 Simple Web2Py example of posting/reading] * [http://groups.google.com/group/web2py/browse_thread/thread/56e916dc8784e20a OAuth support in Web2Py] * [http://dev.twitter.com/doc Twitter API] * [http://code.google.com/p/python-twitter/ Python module] * http://timmcnamara.co.nz/post/1349821774/preliminary-thoughts-on-a-real-time-web-crisis * http://timmcnamara.co.nz/post/1364736217/real-time-web-crisis-monitoring-responding-to * http://epic.cs.colorado.edu/tweak-the-tweet/helping_haiti_tweak_the_twe.html * http://blogs.wsj.com/digits/2010/08/18/twitter-after-a-disaster-is-it-reliable/?mod=wsj_share_twitter * http://eddy.stamen.com === XForms === We have {{{controllers/xforms.py}}} to generate XForms automatically out of Sahana models. These can be used by the [wiki:BluePrintMobileClient Mobile clients] for data collection. This can be used to create [wiki:BluePrintOCR OCR]-able Printed Forms. === XMPP === Instant Messaging & Presence including !GoogleTalk: * http://code.google.com/apis/talk/open_communications.html == Alternate implementations == * Agasti: * Design: http://wiki.sahanafoundation.org/doku.php?id=dev:msg_archi * User Guide: http://wiki.sahanafoundation.org/doku.php?id=doc:message:english * [http://code.google.com/p/nuntium/ Nuntium] - generic Channels for both Application-Terminated & Application-Originated Messaging (Ruby code from InSTEDD) * [http://ndt.instedd.org/2010/08/make-your-sms-apps-scale.html Blog Post on how to use it] * [http://instedd.org/geochat GeoChat] * [http://www.frontlinesms.com Frontline SMS] (used by Ushahidi) * [http://nyaruka.github.com/rapidsms-xforms/ RapidSMS] - also uses XForms so could be easy to integrate? * [http://www.vine.net Microsoft Vine] == Use Cases == Security Management * Allow users to subscribe for SMS security alerts * Ideally allow subscription via SMS too (with optional confirmation by admin staff) * Allow security staff to broadcast security alerts by sending the alert to a single number & have it propagate on * Security for this managed by either pre-registering their numbers as valid or by logging-in 1st * End-users should be able to report back their status via SMS * Security staff should receive a report via SMS of who hasn't reported in (ideally with last known Location). Volunteer Management * Allow team leaders to be able to send messages to their team members. * Allow team members to select which communications means (Email/SMS/Twitter) to use for both normal & priority messages. http://www.ch16.org/reports/ * Allow users to submit reports via SMS/Email/Twitter/[http://ipadio.com iPadio]. * Allow users to subscribe to updates via SMS/Email/RSS.