Changes between Version 46 and Version 47 of BluePrint/Synchronisation


Ignore:
Timestamp:
01/19/10 19:04:46 (15 years ago)
Author:
Fran Boon
Comment:

--

Legend:

Unmodified
Added
Removed
Modified

  • BluePrint/Synchronisation

    v46 v47  
    1 Implementation can be found here: SynchronisationImplementation
    2 
    31= Blueprint for Synchronization =
    42
    53We need to implement a system performing automatic synchronization between Sahana instances.
    64
    7 For synchronisation with other systems, we should be able to talk in open standards, such as [wiki:BluePrintEDXL EDXL]
    8 
    9 Currently !SahanaPy data exporting module exports data in CSV (web based: not for autonomous process). We can add support for XML and JSON. JSON is a modern, light-weight alternative to XML which is appropriate to our low-bandwidth operating environment. XML export should be done using XSD stylesheets so that it's easy to export in different formats.
     5For synchronisation with other systems, we can talk in open standards, such as [wiki:BluePrintEDXL EDXL] since we use [wiki:S3XRC] to import/export data via XSLT in either XML or JSON formats (we can also handle CSV, XLS).
     6JSON is a modern, light-weight alternative to XML which is appropriate to our low-bandwidth operating environment.
    107
    118Each syncable record has a UUID field to uniquely identify it across instances.
    129
    13 Automatic synchronization is different from manual data export / import module present in Sahana. Automatic process should run continuously as daemon.
     10Can sync data using single-table URLs like:
     11 * http://localhost:8000/sahana/gis_location/create?format=xml&fetchurl=http://haiti.sahanafoundation.org/prod/gis_location.xml
     12 * http://localhost:8000/sahana/pr_person/create?format=xml&fetchurl=http://haiti.sahanafoundation.org/prod/pr_person.xml
     13 * http://localhost:8000/sahana/or_organisation/create?format=xml&fetchurl=http://haiti.sahanafoundation.org/prod/or_organisation.xml
     14 * http://localhost:8000/sahana/or_office/create?format=xml&fetchurl=http://haiti.sahanafoundation.org/prod/or_office.xml
     15 * http://localhost:8000/sahana/or_contact/create?format=xml&fetchurl=http://haiti.sahanafoundation.org/prod/or_contact.xml
    1416
    15 Currently we are using database dump for exporting which is definitely not optimal way for synchronization of databases. A paper written by Leslie Klieb ( http://hasanatkazmi.googlepages.com/DistributedDisconnectedDatabases.pdf ) discusses various ways for this. In the light of this research, we can implement synchronization as following:
     17We can also use Joined Component URLs like:
     18 * tbc
     19
     20What we need next is:
     21 * A way to do whole DB syncs which maintain full references
     22 * Cron Job
     23 * UI to configure this
     24
     25=== Old ===
     26Current Sync Implementation based on JSON-RPC can be found here: SynchronisationImplementation
     27
     28Automatic synchronization is different from the manual data export / import module present in Sahana. Automatic process should run continuously as daemon.
     29
     30Previously we were using database dump for exporting which is definitely not optimal way for synchronization of databases. A paper written by Leslie Klieb ( http://hasanatkazmi.googlepages.com/DistributedDisconnectedDatabases.pdf ) discusses various ways for this. In the light of this research, we can implement synchronization as following:
    1631 * we need to put time stamp as additional attribute in each table of database (tables which has data like names of missing people etc, we do not need to sync internally required tables which an instance of Sahana installation uses for saving internal information).
    1732
    18 Data deleted from Sahana should stay available but with a deleted flag. This would then not be visible during normal DB queries, but is accessible for audit purposes if required. We can make this a reusable field in {{{models/00_db.py}}} & then add it to each table definition (well, all real, syncable data - no need for internal settings). Delete flag will be Boolean represented if tuple has been deleted or not.
     33Data deleted from Sahana should stay available but with a deleted flag. This would then not be visible during normal DB queries, but is accessible for audit purposes if required. We made this a reusable field in {{{models/00_db.py}}} & then added it to each table definition (well, all real, syncable data - no need for internal settings). Delete flag will be Boolean represented if tuple has been deleted or not.
    1934
    2035When new tuple is added: new date is entered, when tuple is updated: date is modified to present one. If tuple is deleted, we set delete flag as true for that tuple (and do not delete it for real)