[[TOC]] = Blueprint - Visualisation and Reporting = == Current Status == See [wiki:S3Report S3Report] == Description == Sahana Eden can be used to collect and manage a large variety of data. For this data to add value it needs to be processed and shared in a way in which it can present the information people need to make decisions and plan activities. To achieve this Sahana Eden should be able to produce reports which analyse and visualize data in a flexible, user friendly and configurable way. == Requirements == === Types of Reports === * List * Pivot Tables * Bar Charts * Pie Charts * Scatter Charts * Time Series / Histogram * Maps * Timeline * [wiki:BluePrint/MediaFeed] * http://www.simile-widgets.org/timeline/ * Motion Chart * http://code.google.com/apis/chart/interactive/docs/gallery/motionchart.html What else? === Functionality === * A user friendly wizard to define what data you want to see and how you want to see it to generate different types of reports. * The ability to easily switch between displaying the same data on various types of reports. eg: * List -> Map * Pivot Table -> PIe Chart * Map -> Graph ( for a particular location on the map ) * Drill down within an report to set search criteria, eg: * Select a single bar in a chart * Select a point, box, polygon on a map * Select a wedge in a pie chart * Export Reports * Tables -> XLS * Maps/Graphs -> PNG * All report types should be fully integrated with the [wiki:BluePrintSearch S3Search] module, to allow results to be filtered. * Provide documentation on how to integrate data from Sahana Eden with other reporting & visualisation tools & APIs * Provide some reports suitable for displaying inside popups on Map Polygons [wiki:BluePrintGISFeatureLayers] === Report Gallery === No matter how easy we make it for people to generate their own information products, it will still be too hard or time consuming for some people. I think that it is best to allow for a configurable "gallery" of different reports to allow users to quickly and easily access the reports that they need. * The Report Gallery could be configured to have: * an index page which includes a tumb nail of each report * a menu item * sub-menu items directly to each report * An instance of Sahana could contain more than one Report Gallery * This functionality could be an extension of [wiki: DeveloperGuidelines/SaveSearchAndSubscription Save Search and Subscription] work and allow users to configure their own Report Gallery. === Integration with Survey Module === There is a lot of need for data analysis, visualisation and reporting within the Survey Module and it would be good if the framework level functionality could support this. This will be slightly tricky, because the Survey Module data isn't standard S3XRC layout. Could the Survey Modul just pass XML and then the framework could take it from there? If they supported XML input this could be much more powerful. == Use-Cases == ''Please add any that come to mind'' == Design == === Design Options === ==== Custom Method ==== Instructions: [wiki:S3/S3Report Custom Resource Report] The custom method allows reporting for Sahana Eden resources. Vars could be be used to specify how the report is generated, eg: * What type of report to generate. * What fields would be represented on what graph axises. ==== Chart Module ==== ''OLD - this section may now be redundant, but my provide some ideas''[[BR]] This module could provide the user with the ability to create and embed charts in Sahana Eden. The idea is to provide the user with the ability display any set of data; it need not be tied to a resource or may be a more complex combination of variables than provided in the Sahana Eden framework. The graphs could be in svg format allowing for interactivity. The files themselves could also be stand alone so that they could be easily exported and saved locally. The basic workflow would be: * Create a charts with a web-form or file upload using report/scatter/create. There numerous settings that could be altered to create different a looks and feel. * Or it can be created with a call to the url with the data as vars, such as report/scatter/create?data={"x": [1, 2, 3], "y": [3, 4, 3], "title": "Test Title"}&settings={"width": 300, "height": 200} * An entry gets created in the database for the graph. It can be retrieved an and inserted into a web-page in an object or embed tag, such as * These could appear inside popups on the map * Or, the graphs can be represented without an entry in the database using report/scatter/create?view.svg={"x": [1, 2, 3], "y": [3, 4, 3], "title": "Test Title"}&settings={"width": 300, "height": 200} === Design Notes === * It may be useful to identify a common set of variables which are needed to generate different Types of Reports || '''Variable''' || List || Pivot Tables || Bar Charts || Pie Charts || Scatter Charts || Maps || Timeline || Motion Chart || || Data (list) || n || 1 || 2 || 1 || - || n (layers) || 1 (although a record can be represented || 2 (Color & Size) || || X Axis || || x || || || x || || || x || || Y Axis || || x || || || x || || || x || || || || || || || || || || || == Implementation == === Current === Sahana Eden's current reporting functionality is within the [wiki:S3Report] module. There is also search functionality within the [wiki:BluePrintSearch S3Search] module. === Implementation Plan === The current focus is on migrating S3Report from a full-page design into AJAX-updateable widgets which are linked to [wiki:S3/FilterForms S3Filter] forms. It is desired to provide exports in JSON, CSV, PDF and XLS. Note this isn't simple because: * Users expect that they will export what they see on the screen so the back-end needs to be told the current status as the page is highly interactive. * The back-end codecs need to be extended to support processing this data structure to generate * PDF Graphs will need a server-side method to render images (probably based on matplotlib via S3Chart) * XLS would ideally include the raw Data on 1 or more sheets, the Pivot table on another sheet & Graphs on 1 or more additional sheets. The Pivot Table & Graphs should be linked to the raw data using XLS functionality. There will be a lot of work to implement a complete Data Analysis, Visualisation and Reporting framework, so this could be divided into the following steps: 1. Develop functionality to generate a variety of report types 1. Develop Report Gallery support 1. Allow switching between displaying the same data on various types of reports 1. Report Wizard to generate reports == Tools / Libraries / APIs == === Charts === * Current the following chart tools are included in Sahana Eden: * [http://www.jqplot.com jqPlot]. * [http://matplotlib.sourceforge.net/faq/howto_faq.html#matplotlib-in-a-web-application-server Matplotlib] in S3Chart * The [http://healthscapes.org Healthscapes] !SaVaGe library is included to easily output SVG graphs. See: [wiki:DeveloperGuidelinesSVG] * Outputs SVG graphs * We should have a recommended tested SVG plugin for Internet Explorer < 9 * http://www.savarese.com/software/svgplugin/ (not 64-bit) * http://www.adobe.com/svg/viewer/install/ (Not maintained - e.g. no Windows 7 support) * Background page: http://www.w3.org/Graphics/SVG/IG/wiki/SVG_Plugin_for_IE * Interactive Graphs with matplotlib: http://www.dalkescientific.com/writings/diary/archive/2005/04/24/interactive_html.html * http://www.reynoldsftw.com/2009/02/6-jquery-chart-plugins-reviewed/ * Recommends [http://code.google.com/p/flot/ Flot]: "''From an interaction perspective, Flot by far will get you as close as possible to Flash graphing as you can get with jQuery.''" * http://code.google.com/apis/chart/interactive/docs/gallery.html * http://www.tableausoftware.com/public/gallery * [http://www.web2pyslices.com/main/slices/take_slice/12 Cairo] can generate nice charts from the back-end * T2 includes a basic barchart function: {{{ def barchart(data,width=400,height=15,scale=None, label_width=50,values_width=50): if not scale: scale=max([m for n,c,m in data]) if not scale: return None return TABLE(_class='barchart', *[TR(TD(n,_width=label_width,_style="text-align: right"), TABLE(TR(TD(_width=int(m*width/scale),_height=height, _style='background-color:'+c))),TD(m,_width=values_width), _style="vertical-alignment: middle") for n,c,m in data]) }}} === Spreadsheets === * Excel files are currently written using [http://pypi.python.org/pypi/xlwt xlwt]. * [http://sourceforge.net/projects/pyexcelerator PyExcelerator] * Docs on Python Excel libraries: http://www.python-excel.org/ * Thread recommending xlwt: http://groups.google.com/group/web2py/browse_thread/thread/375307e92df40ca8 * [http://appyframework.org/pod.html Pod] to create Open Document Format === Flash === Could be something to use Flash? (although best-avoided, if possible, to minimise client support) * PyAMF: http://pyamf.org * Web2Py integration: http://mdp.cti.depaul.edu/AlterEgo/default/show/22 * !BirdEye: http://code.google.com/p/birdeye/ * !FusionCharts Free: http://www.fusioncharts.com/Free/ * Open Flash Chart: http://teethgrinder.co.uk/open-flash-chart-2/ * from Python: http://code.google.com/p/open-flash-chart-python/ === Other === * http://www.pentaho.com/ * http://processing.org/ == How To Help! == Implementing Data Analysis, Visualization and Reporting within Sahana Eden will be a ongoing process requiring a diverse range of skills. We'd really welcome input from anyone who's interested! * If you have experience in working on solutions in this area before your input would be very valuable! Please feel free to add notes to this pages or contact the MailingList with your insight! * If you're a (potential) user of Sahana Eden, you can add Use-Stories - these will *really* help guide us to to ensure that we meet your needs, just tell us: * Who the user is? * What Data Analysis / Visualization / Reporting do they want? * Why do they want it for? (Purpose) * If you've got some UX/graphic design talent, you could design the interface and some of the reports * Help with implementing the code! Contact the MailingList to find out how! == References == === Sahana Phase 2 === http://wiki.sahanafoundation.org/phase2/doku.php/doc:reports:english ---- BluePrint