Changes between Version 22 and Version 23 of S3XRC/S3XML


Ignore:
Timestamp:
06/11/14 09:56:36 (7 years ago)
Author:
MattS
Comment:

Removing out of date content - keep redirect here to avoid broken links

Legend:

Unmodified
Added
Removed
Modified
  • S3XRC/S3XML

    v22 v23  
    1 '''THIS PAGE IS OUT OF DATE - See: [wiki:S3/S3XML]'''
    2 = S3XML - XML/JSON Format =
    3 [[TOC]]
    4 
    5 S3XML is the generic data format used by S3 to export and import data.
    6 
    7 Typically, S3XML would not be used for data exchange directly (though it could be used), but as source/target format for [wiki:S3XRC/S3XML/Transformation On-the-fly Transformation] using XSLT stylesheets. Through this mechanism it is possible to support a variety of XML and JSON formats, and even CSV formats for data exchange.
    8 == XML Format ==
    9 
    10 These can be converted using [wiki:XsltTemplates XSLT Stylesheets].
    11 
    12 === Document Structure ===
    13 
    14 S3XML defines 4 element types:
    15 
    16   - [#s3xml s3xml]
    17   - [#resource resource]
    18   - [#data data]
    19   - [#reference reference]
    20 
    21 ==== s3xml ====
    22 ||'''Parent elements'''||''none (root element)''||
    23 ||'''Child elements'''||[#resource resource]||
    24 ||'''Contents'''||''empty''||
    25 
    26 The root element.
    27 
    28 **Attributes:
    29 ||'''Name'''||'''Type'''||'''Description'''||'''mandatory?'''||
    30 ||'''domain'''||string||the domain name of the data repository||''no''||
    31 
    32 ==== resource ====
    33 ||'''Parent elements'''||[#s3xml s3xml], [#resource resource], [#reference reference]||
    34 ||'''Child elements'''||[#resource resource]||
    35 ||'''Contents'''||''empty''||
    36 
    37 Represents a record.
    38 
    39 **Attributes:
    40 ||'''Name'''||'''Type'''||'''Description'''||'''mandatory?'''||
    41 ||'''name'''||string||the name of the resource, usually the DB table name||''yes''||
    42 ||'''uuid'''||string||a unique identifier for the record||''no''*||
    43 ||'''tuid'''||string||a temporary unique identifier for the record||''no''*||
    44 
    45 (*) Records will be identified within the input file by their {{{uuid}}}, or, if no {{{uuid}}} is specified, by their {{{tuid}}}.
    46 
    47 The {{{uuid}}} will be stored in the database together with the record. If {{{uuid}}} is present and matches an existing record in the database, then this record will be updated. If there's no match or no {{{uuid}}} specified in the {{{resource}}} element, then the importer will create a new record in the database (and automatically generate a {{{uuid}}} if required).
    48 
    49 ==== data ====
    50 ||'''Parent elements'''||[#resource resource]||
    51 ||'''Child elements'''||''none (leaf element)''||
    52 ||'''Contents'''||Text||
    53 
    54 Represents the value of a single field in the record.
    55 
    56 **Attributes:
    57 ||'''Name'''||'''Type'''||'''Description'''||'''mandatory?'''||
    58 ||'''field'''||string||the field name in the record||''yes''||
    59 ||'''value'''||JSON value||the native field value||''no''||
    60 ||'''url'''||URL||the URL to download the contents from*||''no''||
    61 ||'''filename'''||filename||the filename of the attached contents*||''no''||
    62 
    63 The text node in the {{{data}}} element provides a human-readable representation of the field value. If this representation is different from the original value in the database, then the original value must be provided by the ''value'' attribute.
    64 
    65 (*) If the field is for file upload, a {{{url}}} attribute should be provided to specify the location of the file. The importer will try to download and store the file (file transfer) from that URL (''pull''). It is also possible to send the file with the HTTP request - in this case the {{{filename}}} must be specified instead of {{{url}}} (''push''). The ''push'' variant for uploads is meant for peers which do not support pulling for some reason (e.g. mobile phones). Normal servers would always provide a URL for download in order to allow the consuming site decide which files to download and when (saves bandwidth).
    66 
    67 ==== reference ====
    68 ||'''Parent elements'''||[#resource resource]||
    69 ||'''Child elements'''||[#resource resource]||
    70 ||'''Contents'''||Text||
    71 
    72 Represents a foreign key reference.
    73 
    74 **Attributes:
    75 ||'''Name'''||'''Type'''||'''Description'''||'''mandatory?'''||
    76 ||'''field'''||string||the field name in the record||''yes''||
    77 ||'''resource'''||string||the name of the referenced resource, usually the tablename||''yes''||
    78 ||'''uuid'''||string||the unique identifier of the referenced record (foreign key)*||''(yes)''!**||
    79 ||'''tuid'''||string||a temporary identifier for a referenced record (foreign key)*||''(yes)''!**||
    80 
    81 (*) Referenced records would always be exported in the same output file. If a referenced record is found in the same input file, then it will be automatically imported.
    82 
    83 (!**) Records will be identified within the input file by their {{{uuid}}}, or, if no {{{uuid}}} is specified, by their {{{tuid}}}.
    84 
    85 If the referenced record is enclosed in the ''reference'' element, then {{{uuid}}} and {{{tuid}}} can be omitted:
    86 
    87 {{{
    88 <s3xml>
    89    <resource name="xxxyyy">
    90        <reference field="xy" resource="aaabbb">   <!-- the reference element, uuid/tuid can be omitted if -->
    91           <resource name="aaabbb">                <!-- the referenced record is enclosed in the reference -->
    92           </resource>
    93        </reference>
    94    </resource>
    95 </s3xml>
    96 }}}
    97 
    98 === Example ===
    99 
    100 {{{
    101 <s3xml>
    102 
    103   <resource                                                 <-- a record in the database -->
    104       created_on="2009-10-02 08:55:11"                      <-- date/time when the record was created -->
    105       modified_on="2009-10-02 08:56:03"                     <-- date/time when the record was last modified -->
    106       uuid="6e6e76dc-8ed7-408c-bb09-54476e3944ae"           <-- UUID of the record (if present in DB) -->
    107       created_by="None"                                     <-- Author -->
    108       modified_by="Dominic"                                 <-- Last Author -->
    109       name="pr_person">                                     <-- Resource Name -->
    110 
    111     <reference                                              <-- Reference Field (foreign key) in the record -->
    112       field="pr_pe_id"                                      <-- Field name -->
    113       resource="pr_pentity"                                 <-- Name of the referenced resource -->
    114       uuid="6e6e76dc-8ed7-408c-bb09-54476e3944ae"/>         <-- UUID of the referenced entry -->
    115 
    116     <data field="pr_pe_label">730421</data>                 <-- A field in the record -->
    117     <data field="first_name">Dominic</data>
    118     <data field="middle_name"/>
    119     <data field="last_name">König</data>
    120     <data field="preferred_name"/>
    121     <data field="local_name"/>
    122     <data field="opt_pr_gender" value="3">male</data>
    123     <data field="opt_pr_age_group" value="5">Adult (21-50)</data>
    124     <data field="email">dominic@nursix.org</data>
    125     <data field="mobile_phone"/>
    126     <data field="date_of_birth">1973-04-21</data>
    127     <data field="opt_pr_nationality" value="65">Germany</data>
    128     <data field="opt_pr_country" value="167">Sweden</data>
    129     <data field="opt_pr_religion" value="1">none</data>
    130     <data field="opt_pr_marital_status" value="3">married</data>
    131     <data field="occupation">Nurse</data>
    132     <data field="comment"/>
    133 
    134     <resource                                               <-- A sub-resource (component) of the record -->
    135       created_on="2009-10-02 11:34:34"
    136       modified_on="2009-10-02 11:34:34"
    137       uuid="89217054-3c10-4f5d-959a-420254243498"
    138       name="pr_address">
    139 
    140       <data
    141         field="opt_pr_address_type"                         <-- field name -->
    142         value="1">                                          <-- original value in the database -->
    143           Home Address                                      <-- value represented for human readability -->
    144       </data>
    145       <data field="co_name"/>
    146       <data field="street1">Lundgatan</data>
    147       <data field="street2"/>
    148       <data field="postcode">38031</data>
    149       <data field="city">Läckeby</data>
    150       <data field="state"/>
    151       <data field="opt_pr_country" value="167">Sweden</data>
    152       <data field="lat">56.78042</data>
    153       <data field="lon">16.27914</data>
    154       <data field="comment"/>
    155     </resource>
    156   </resource>
    157 </s3xml>
    158 }}}
    159 
    160  * [wiki:UUID] - how we handle Unique IDs for records across heterogeneous systems
    161 
    162 == JSON Format ==
    163 
    164 The data structure of the native S3JSON format is equivalent to the XML format (=element trees) - except that markup elements are represented by prefixes:
    165 
    166 {{{
    167 {
    168     "@domain": "yana",                                             // Server name
    169     "@url": "http://127.0.0.1:8000/eden"                           // Server URL
    170     "$_pr_person": {                                               // Resource, prefix: $_
    171         "@uuid": "44fc762e-02df-44e0-8bd1-9b58e3132894",           // Resource attribute, prefix: @
    172         "@url": "http://127.0.0.1:8000/eden/pr/person/1",
    173         "@created_on": "2009-11-16 22:33:35",
    174         "@created_by": "None",
    175         "@modified_on": "2009-11-19 21:32:19",
    176         "@modified_by": "Dominic",
    177         "first_name": "Dominic",                                   // Data field, no prefix
    178         "last_name": "K\u00f6nig",
    179         "email": "dominic@nursix.org",
    180         "opt_pr_age_group": {"@value": "1", "$": "unknown"},       // Data field with textual representation:
    181         "opt_pr_religion": {"@value": "1", "$": "none"},           // @value=Value, $=TextualRepresentation
    182         "opt_pr_gender": {"@value": "1", "$": "unknown"},
    183         "opt_pr_nationality": {"@value": "999", "$": "unknown"},
    184         "opt_pr_country": {"@value": "999", "$": "unknown"},
    185         "opt_pr_marital_status": {"@value": "1", "$": "unknown"},
    186         "$k_pr_pe_id": {                                           // External Reference (Key), prefix: $k_
    187             "@resource": "pr_pentity",                             // Key resource name
    188             "@uuid": "a2a945bd-4f43-41da-bcdb-e2e638a987ea",       // UUID of the key record
    189             "$": "Dominic K\u00f6nig [no label] (Person)"          // Textual representation of the reference
    190         },
    191         "$_pr_presence": {                                         // Sub-resource (Component):
    192             "@uuid": "14af2751-7277-4e90-b42b-0d0430684561",       // appears as component within the resource
    193             "@created_on": "2009-11-19 19:42:46",
    194             "@modified_on": "2009-11-19 19:42:46"
    195             "@url": "http://127.0.0.1:8000/eden/pr/person/1/presence/1",
    196             "opt_pr_presence_condition": {"@value": "4", "$": "Found"},
    197             "time": {"@value": "2009-11-19 18:42:00 +0000", "$": "2009-11-19 20:42:00"},
    198             "$k_reporter": {
    199                 "@resource": "pr_person",
    200                 "@uuid": "44fc762e-02df-44e0-8bd1-9b58e3132894",
    201                 "$": "Dominic K\u00f6nig"
    202             },
    203         }
    204     },
    205 }
    206 }}}
    207 
    208 '''JSON format characteristics:'''
    209 
    210   * The JSON output contains _no_ whitespace between elements, it's just added here by hand for better readability
    211 
    212   * The outermost structure is always a JSON object (not a list)
    213   * All data is represented as strings (for security reasons)
    214 
    215   * If @value is sent for a field, it overrides the element text ($) at import
    216   * however, the use of @value is not mandatory, data can simply be placed instead of element text
    217   * Note that there is no automatic data encoding: data must be sent in DB-encoded format
    218   * @resource, @name and @uuid attributes are mandatory at input, other attributes can be omitted
    219 
    220   * Multiple records of the same resource will be aggregated as lists like:
    221 
    222 {{{
    223 {
    224     $_my_resource: [
    225         {
    226             // record1 of my_resource
    227         }
    228         {
    229             // record2 of my_resource
    230         }
    231     ]
    232 }
    233 }}}
    234 
    235 ----
    236 
    237 DeveloperGuidelines
     1See [wiki:S3/S3XML]