Changes between Initial Version and Version 1 of SynchronisationImplementation


Ignore:
Timestamp:
08/19/09 15:37:47 (12 years ago)
Author:
Fran Boon
Comment:

Move from Synchronisation & clean-up

Legend:

Unmodified
Added
Removed
Modified
  • SynchronisationImplementation

    v1 v1  
     1This is an implementation of: BluePrintSynchronisation
     2
     3== Evaluators guide ==
     4=== Current Status ===
     5
     6All webservices are implemented. They can be consumed using JSON and JSON-RPC.
     7
     8Daemon which automates synchronization is functional though its status can not be confirmed without testing Zeroconf with it.
     9
     10== To Do ==
     11 * Daemon as cronjob is not initialized automatically. see [http://groups.google.com/group/web2py/browse_thread/thread/b8a55655c7a651b8/910af539915e3c9d?lnk=gst&q=hasanatkazmi]
     12 * Zeroconf doesn't seems to be very promising. Extensive testing should be done.[[BR]]
     13 * Automatic Synchronization is limited to Sahanapy. SahanaPHP port should be implemented.[[BR]]
     14
     15== Evaluation Guide ==
     16
     170. Install python 2.5 or 2.6[[BR]]
     181. Install json-rpc from [http://json-rpc.org/wiki/python-json-rpc][[BR]]
     192. Install Bazaar[[BR]]
     203. bzr branch lp:~mdipierro/web2py/devel web2py[[BR]]
     214. cd web2py/applications[[BR]]
     225. bzr branch lp:~hasanatkazmi/sahana/p2psync[[BR]]
     236. cd p2psync/cron[[BR]]
     247. start web2py[[BR]]
     258  start daemonX (python deamonX.py)[[BR]]
     269. Replicate same procedure on another machine[[BR]]
     2710. Synchronization module should list other server in Sync Partners. After sometime, synchronization history should list sync activity. Logs are also maintained in cron/synclog.[[BR]]
     28
     29== WEB SERVICES API ==
     30
     31(This API is also used by daemonX which drives automatic p2p syncing between clients)
     32
     33=== Supported web services ===
     34 * JSON
     35 * JSON-RPC
     36
     37=== ToDo (partial work has been done) ===
     38 * XML
     39 * XML-RPC
     40
     41'''Service Proxy''':
     42        JSON: http://localhost:8000/sahana/admin/call/json
     43                e.g. http://localhost:8000/sahana/admin/call/json/getdata?timestamp=0&.........
     44        JSON-RPC: http://server's-ip:port/sahana/admin/call/jsonrpc
     45        replace json and jsonrpc with xml and xmlrpc respectively for xml
     46
     47=== Available functions ===
     48'''putdata(uuid, username, password, nicedbdump):'''
     49        This function is used to insert data in the system.
     50        args:
     51                uuid (required):
     52                        uuid of the machine which is calling, uuid of machine is 16 character unique string. In the case when web services client is also a Sahana instance, uuid will be generated and stored by deamonX.
     53                Username, password (required):
     54                        Used for authentication purposes. Both are strings. A user must be registered at the host machine with data alteration privileges. e.g. Administrator of the system can put data in the system.
     55                nicedbdump (required):
     56                        nicedbdump can be best illustrated using diagrammatically representation. If [] represents a python list then:
     57                        nicedump = [   //each element of this list is another list representing a database table[[BR]]
     58                                [[[BR]]
     59                                        table name,[[BR]]
     60                                        [ comma separated table attributes as string][[BR]]
     61                                        [   //each element in this list is a list which represents a row in table[[BR]]
     62                                                [comma separated row values][[BR]]
     63                                                [][[BR]]
     64                                                ..[[BR]]
     65                                                ..      [[BR]]                         
     66                                        ][[BR]]
     67                                ]       [[BR]]
     68                                [][[BR]]
     69                                [][[BR]]
     70                                ..[[BR]]
     71                                ..[[BR]]
     72                                ..[[BR]]
     73                        ]
     74       
     75        Note that if you pass a table using nicedbdump which is not present in database, it will be simply ignored.
     76        If nicedbdump is not formated properly then an error string will be returned. Following situations will raise an error:
     77        If nicedbdump is not a list.
     78        If nicedbdump is not list of lists.
     79        Each list in nicedbdump represents a table, say n:
     80                If n does not exactly has 3 elements.
     81                If first of these three elements is not a string data type.
     82                If second of these three elements, say s,  is not a list:
     83                        if s in this list is not a string
     84                If third of these three elements, say t, is not a list:
     85                        if each element in t is not a list, let such an element be r:
     86                                if number of elements in r is not equal to number of elements in s
     87                               
     88        If a table (s in the case described above) is not having 'id', 'uuid' and 'modified_on' as attribute. 'id' is unique id for each row in table. This 'uuid' is different from the 'uuid' which daemonX maintains, this is row uuid. 'modified_on' represents the last time data was modified (or created it not altered after creation)
     89        Note that only that data (referenced by row uuid) which has never life then the one in database will be added. In case of absence of that uuid in database, that data is be added.
     90        return:
     91                If user is authenticated and nicedbdump is successfully parsed, data will be added to the database and True will be returned. On the other hand, in case of error, error message will be returned as String.
     92
     93
     94'''getdata(uuid, username, password, timestamp = None):'''
     95        returns data as nicedbdump defined in putdata. Data after timestamp time will returned, if None is passes as timestamp, then that data which has been added to the system after last getdata call from uuid will be returned.
     96        Args:
     97                uuid (required):
     98                        uuid of the machine which is calling, uuid of machine is 16 character unique string. In the case when web services client is also a Sahana instance, uuid will be generated and stored by deamonX.
     99                username, password (required except for local machine):
     100                        Both are strings. used for authentication, user must have privileges for reading the database. If service is called from local machine (i.e. with IP 127.0.0.1) username and password are ignored and user is given access. e.g. deamonX accesses this function locally without providing username and password.
     101                Timestamp (optional):
     102                        timestamp is of string type. It should be like “YYYY-MM-DD HH:MM:SS”. If timestamp = null is passes, system will automatically return data after last getdata operation between uuid and machine. deamonX uses this setting
     103
     104        return:
     105                In case of error (like failure to authenticate), error message as string will be returned. If successful, then nicedbdump will be returned which is described above.
     106
     107=== Example code ===
     108{{{
     109from jsonrpc import ServiceProxy, JSONRPCException
     110#jsonrpc needs simplejson which we have to install first
     111s = ServiceProxy("http://localhost:8000/sahana/admin/call/jsonrpc")
     112try:
     113        nicedbdump = s.getdata("machinename12345","email@lums.edu.pk", "myPassword")
     114        if type(result) == str:
     115                #it means there is an error, now result has error messege
     116                pass
     117        else:
     118                #result is list type for sure,
     119                #result is nicedbdump type
     120               
     121        putit = s.putdata("machinename12345", "email@lums.edu.pk", "myPassword", nicedbdump)
     122        if putit == True:
     123                #data sucessfully sent, parsed and processes at server (but in this case no data will be added because you just queried data and sent it back)
     124                pass
     125        else:
     126                #its an error
     127                pass
     128except JSONRPCException, e:
     129        print repr(e.error)
     130}}}
     131
     132Note: This example is in Python, but you can write a client in any language of your choice.
     133
     134=== Choosing !ZeroConf for Network discovery ===
     135Automatic synchronization between servers require automatic service discovery. We had two major options to choose from:
     1361) ZeroConf
     1372) Mesh4x
     138
     139!ZeroConf & Mesh4x solve different problems. They don't overlap in functionality at all:
     140 * !ZeroConf provides a solution to automatic discovery.
     141 * Mesh4x provides a solution to the data sync.
     142
     143We were more interested in Zeroconf because:
     1441) !ZeroConf has Python library but Mesh4x doesn't. I means double work was required if we go with Mesh4x.
     1452) We just needed automatic discovery of service because we wanted to use web services, so that foreign developers can also use Restful API
     1463) Mesh4x required java daemon, which meant adding jre in the package which would double Sahana package size.
     147
     148
     149
     150
     151=== daemonX: Daemon which runs automatic synchronization ===
     152We created a daemon which calls web services listed above. DaemonX uses ZeroConf libraries available at http://www.amk.ca/python/zeroconf
     153Note that ZeroConf is not being maintained after Dec 2006.
     154daemonX also requires installing jsonrpc libraries from http://json-rpc.org/wiki/python-json-rpc for processing JSON.
     155
     156Very initial tests of daemonX using Zeroconf are below expectations. Using a GPRS moderm as network source, Zeroconf library has thrown errors. More testing needs to be done before making any final statement.