= !BluePrint: Mobile Client Peer-to-Peer Synchronization = [[TOC]] == Introduction == Peer-to-peer sync will enable !EdenMobile users to synchronize data between mobile devices without being connected to an Eden server. This will allow users to operate and collaborate more easily in areas where internet access is unreliable or unavailable. Peer-to-peer sync capabilities exist in [http://developer.servalproject.org/dokuwiki/doku.php?id=content:servalmesh:main_page Serval Mesh] and several messaging apps including [https://briarproject.org/ Briar], [https://github.com/casific/murmur Murmur] and [http://whispercomm.org/shout/ Anonymouse]. == Stakeholders == !EdenMobile users will be the primary stakeholders. The data subjects will also be stakeholders, and appropriate access controls to protect the confidentiality, integrity and authenticity of data must be built in. Trainers who teach !EdenMobile users how to use the application will need to be aware of the new functionality. If the data being synchronized is ultimately delivered to a server then the people deploying and operating the server will also be stakeholders. == User Stories == See the [https://code.briarproject.org/akwizgran/briar/uploads/655c21456bbc36a83193bdb5e0473f07/SahanaEDENBrambleProjectUseCases.odt user stories document]. == Requirements == === Functional requirements=== ==== 1. Establishing peer relationships ==== ===== 1.1. Peer visibility ===== The P2P sync component '''must''' be able to make the device it is running on visible to nearby devices in order to establish a peer relationship. ===== 1.2. Peer discovery ===== The P2P sync component '''must''' be able to discover nearby devices that have made themselves visible. ===== 1.3. Peer selection ===== When peers have been discovered, the P2P sync component '''must''' allow the user to choose a device to establish a peer relationship with. ===== 1.4. Peer authentication ===== When the user has chosen a device to peer with, the P2P sync component '''must''' allow the user to authenticate the peer before giving the peer access to any data. The P2P sync component '''may''' allow the user to continue without authenticating the peer. ===== 1.5. Peer persistence ===== When a peer has been authenticated, the P2P sync component '''may''' allow the user to store information about the peer, including information that will allow the peer to be authenticated automatically during subsequent connections. If information about a peer has been stored, the P2P sync component '''must''' allow the user to delete the information in order to prevent the peer from being authenticated automatically during subsequent connections. ==== 2. Destination servers ==== ===== 2.1. Destination server configuration ===== The P2P sync component '''may''' allow the user or an administrator to enter the details of a '''destination server''' for the data being synced. If a destination server has been configured, the P2P sync component '''may''' deliver data to the server automatically when internet access is available. ==== 3. Data synchronization ==== ===== 3.1. Data selection ===== The P2P sync component '''may''' allow the user to choose which data to synchronize. When data is received from a peer, the P2P sync component '''may''' automatically choose it for synchronization with other peers. ===== 3.2. Data deletion ===== The P2P sync component '''must''' allow the user to delete data from the device, regardless of whether that data has been chosen for synchronization. The P2P sync component '''may''' automatically delete data that has been received from peers, for example on receiving confirmation that the data has been delivered to a destination server. ==== 4. Managing conflicts ==== ===== 4.1. Conflict detection ===== The P2P sync component '''must''' assign an identifier to each item of data chosen for synchronization, such that two peer devices performing synchronization can tell whether they are using the same identifier to refer to the same data. ===== 4.2. Conflict resolution ===== If a conflict is detected during synchronization, the P2P sync component '''may''' resolve the conflict automatically, either by merging the conflicting items or by assigning them distinct identifiers. If a conflict cannot be resolved automatically, the P2P sync component '''must''' be able to continue synchronizing other items with the peer. The P2P sync component '''may''' inform the user of conflicts that cannot be resolved automatically. === Non-functional requirements === ==== 1. Security ==== ===== 1.1. Protection of data in transit ===== The P2P sync component '''must''' ensure the confidentiality, integrity and authenticity of data in transit between peers. If a destination server has been configured, the P2P sync component '''must''' ensure the confidentiality, integrity and authenticity of data in transit between mobile devices and the destination server. ===== 1.2. Protection of data at rest ===== The P2P sync component '''must''' ensure the confidentiality, integrity and authenticity of data at rest on mobile devices, to at least the same level of protection as other components of the !EdenMobile application. ===== 1.3. Access control ===== The P2P sync component '''must''' authenticate peers, or allow the user to authenticate them, before granting them access to data. If a destination server has been configured, the P2P sync component '''must''' authenticate the destination server before granting it access to data. If a delivery confirmation from a destination server is received, either directly from the server itself or from a peer, the P2P sync component '''must''' authenticate the delivery confirmation before acting on it or synchronizing it with peers. To protect the privacy of data subjects, the P2P sync component '''may''' encrypt data before synchronizing it with peers, such that it can only be decrypted by the destination server. ==== 2. Performance ==== ===== 2.1. Power efficiency ===== The P2P sync component '''must not''' significantly reduce the battery life of the device when P2P sync is not being used. ===== 2.2. Communication efficiency ===== The P2P sync component '''must not''' significantly increase the mobile data usage of the device, regardless of whether P2P sync is being used. === Interoperability === For a given deployment, the same version of the !EdenMobile app will be installed on all mobile devices, which should minimize interoperability issues. The protocols used between peers, and between mobile devices and the destination server, should use version negotiation so that any interoperability issues are detected as early as possible. Interoperability with other applications is not required. === Standards === Communication between mobile devices and the destination server should use HTTPS. If the P2P sync component protects the privacy of data subjects by encrypting data between the originating device and the destination server, this should be done using OpenPGP. If the destination server creates delivery confirmations, these should be signed using OpenPGP. === System Constraints === The P2P sync component must operate on mobile devices with limited battery life, storage, and network bandwidth. It must be usable with minimal training and background knowledge. It must tolerate intermittent and unreliable connections between peers, and between mobile devices and the destination server. == Design == There are two design options: ==== 1. Synchronize immutable documents ==== In this option, the data to be synced consists of immutable documents, each of which originates on one mobile device and is synced to other devices without being modified. For example, a completed form or a photo can be treated as an immutable document. Detecting and resolving conflicts is simple in this option: each data item is identified by its cryptographic hash, which ensures that if two items have the same identifier then they also contain the same data. This option allows documents to be encrypted on the originating device and decrypted on the destination server. The destination server confirms delivery of a document by creating a delivery confirmation containing the document's identifier, signed with the server's OpenPGP key. The confirmation can be synced between mobile devices, which can delete the confirmed document if they no longer need a local copy. ==== 2. Synchronize changes to database records ==== In this option, the data to be synced consists of database records that may be modified independently on any device. This provides maximum flexibility for the application, as anything stored in the database can potentially be synced, but it also requires the application to handle conflicts that may arise from concurrent modification of a given record on multiple devices. Detecting and resolving conflicts in this option follows a similar approach to [http://docs.couchdb.org/en/2.0.0/intro/consistency.html CouchDB]. Non-conflicting updates are applied automatically. If a conflict occurs, the P2P sync component stores all conflicting versions of the record and asks the application to resolve the conflict, which may require user intervention, either on the mobile device or on the destination server. This option does not allow records to be encrypted on the originating device and decrypted on the destination server, as mobile devices need access to the contents of updates. Individual fields can be encrypted, but if a conflict occurs in an encrypted field it can only be resolved on the destination server. The destination server confirms delivery of a record by creating a delivery confirmation containing the record's identifier and [https://wiki.apache.org/couchdb/Replication_and_conflicts revision], signed with the server's OpenPGP key. The confirmation can be synced between mobile devices, which can delete the confirmed record if they are storing the same revision and no longer need a local copy (they must keep the record if they are storing a different revision). Multiple revisions of a given record may be confirmed by the destination server. ==== Comparison of the options for each use case ==== The immutable documents option is well suited to the "Situation Reporting and Assessment" and "Self-Organized Canvassing" use cases, which are primarily concerned with collecting data and delivering it to a central server. The "Aid Distribution" use case involves updating information as well as collecting it: for example, recording the fact that an individual who was previously eligible for aid has now received it. The database records option is well suited to this use case, but the use case can also be handled with immutable documents by using a log structure: for example, a log of documents representing events is attached to a document representing an individual. The "Logistics Management" use case is primarily concerned with managing mutable information. This can be handled with immutable documents using a log structure, but for efficiency it may also be necessary to maintain a representation of the current state. The complexity of maintaining two representations of the data may make the database records option preferable for this use case. === Data Model === TODO === Workflows === ==== 1. Peering between two mobile devices ==== ===== 1.1. Peering without stored peer identities ===== * Users: U1 and U2, with devices D1 and D2, respectively * U1 and U2 start peer discovery * D1 discovers D2 and/or D2 discovers D1 * U1 chooses to peer with D2 and/or U2 chooses to peer with D1 * D1 connects to D2 and/or D2 connects to D1 * D1 and D2 exchange authentication information * D1 and D2 display each other's authentication information * U1 and U2 confirm the authentication information or skip authentication * D1 and D2 become peers * U1 and/or U2 may choose to store the peer's identity persistently ===== 1.2. Peering with a stored peer identity on one device ===== * Users: U1 and U2, with devices D1 and D2, respectively * D1 has stored peering information about D2 * U1 and U2 start peer discovery * D1 discovers D2 and/or D2 discovers D1 * U1 chooses to peer with D2 and/or U2 chooses to peer with D1 * D1 connects to D2 and/or D2 connects to D1 * D1 and D2 exchange authentication information * D1 automatically authenticates D2 * D2 displays D1's authentication information * U2 confirms the authentication information or skips authentication * D1 and D2 become peers * U2 may choose to store the peer's identity persistently ===== 1.3. Peering with stored peer identities on both devices ===== * Users: U1 and U2, with devices D1 and D2, respectively * D1 has stored peering information about D2 * U1 and U2 start peer discovery * D1 discovers D2 and/or D2 discovers D1 * U1 chooses to peer with D2 and/or U2 chooses to peer with D1 * D1 connects to D2 and/or D2 connects to D1 * D1 and D2 exchange authentication information * D1 and D2 automatically authenticate each other * D1 and D2 become peers ==== 2. Configuring a destination server ==== * User or administrator enters the URL and OpenPGP key of a destination server * The URL and OpenPGP key are automatically validated * User or administrator may choose to test connectivity to the server * User or administrator saves the configuration ==== 3. Activating and deactivating P2P sync ==== * P2P sync is automatically activated whenever the user starts peer discovery * P2P sync is automatically deactivated when peer discovery is not in progress and no peer connections are open * User can see whether P2P sync is currently active * User can manually deactivate P2P sync, which cancels peer discovery and closes any open peer connections === Technologies === The P2P sync component will use the [https://code.briarproject.org/akwizgran/briar/tree/master Bramble library], which will be packaged as a Cordova plugin. The component will only be available on Android. The Bramble protocols [https://code.briarproject.org/akwizgran/briar-spec/blob/master/protocols/BTP.md BTP] and [https://code.briarproject.org/akwizgran/briar-spec/blob/master/protocols/BSP.md BSP] will be used for syncing data between peers. In Bramble terminology, the P2P sync component will be a BSP client. New modules for discovering, authenticating and storing peers will be developed, or the existing Bramble modules will be generalised. The structure of the BSP client, and the interface between the P2P sync component and the !EdenMobile app, depend on which design option is chosen. In both options, the P2P sync component exposes the following capabilities to the !EdenMobile app: * Activate and deactivate P2P sync. * Discover and authenticate peers. * Optionally store peer identities in the component's database. * Optionally configure a destination server and store the configuration in the component's database. * Optionally encrypt the component's database with a password-derived key. The P2P sync component also implements the following capabilities internally: * Connect to the destination server when internet access is available. * Synchronize data with peers and the destination server. * Authenticate delivery confirmations received from peers and the destination server. The component has the following dependencies: * Bramble * OpenPGP ==== 1. Synchronize immutable documents ==== In this option, the P2P sync component exposes the following extra capabilities to the app: * Store immutable documents in the component's database. * Optionally encrypt documents with the destination server's public key before storing them. * Choose documents from the component's database to be synchronized. The P2P sync component implements the following extra capabilities internally: * Delete documents that were received from peers and have been confirmed by the server. ==== 2. Synchronize database records ==== In this option, the P2P sync component exposes the following extra capabilities to the app: * Choose records from the app's database to be synchronized. This option requires the app's database to be ported from the [https://github.com/nolanlawson/cordova-plugin-sqlite-2 Cordova SQLite plugin] to [https://pouchdb.com/ PouchDB], which [https://pouchdb.com/adapters.html wraps] the SQLite plugin. The [https://github.com/pouchdb-community/pouchdb-replication-stream PouchDB Replication Stream] module extracts changes from the database to be synced via Bramble. This option has the following extra dependencies: * PouchDB * PouchDB Replication Stream == Current Implementation == N/A == Planned Implementation == TODO == Future Extensions == TODO == Outstanding Questions == TODO == References == * [http://developer.servalproject.org/dokuwiki/doku.php?id=content:servalmesh:main_page Serval Mesh] * [https://briarproject.org/ Briar] * [https://github.com/casific/murmur Murmur] * [http://whispercomm.org/shout/ Anonymouse] * [https://code.briarproject.org/akwizgran/briar/uploads/655c21456bbc36a83193bdb5e0473f07/SahanaEDENBrambleProjectUseCases.odt User stories] * [http://docs.couchdb.org/en/2.0.0/intro/consistency.html CouchDB: Consistency] * [https://wiki.apache.org/couchdb/Replication_and_conflicts CouchDB: Replication and conflicts] * [https://code.briarproject.org/akwizgran/briar/tree/master Bramble] * [https://code.briarproject.org/akwizgran/briar-spec/blob/master/protocols/BTP.md BTP] * [https://code.briarproject.org/akwizgran/briar-spec/blob/master/protocols/BSP.md BSP] * [https://github.com/nolanlawson/cordova-plugin-sqlite-2 Cordova SQLite Plugin] * [https://pouchdb.com/ PouchDB] * [https://pouchdb.com/adapters.html PouchDB: Adapters] * [https://github.com/pouchdb-community/pouchdb-replication-stream PouchDB Replication Stream] ---- BluePrint