Changes between Initial Version and Version 1 of S3/S3Resource


Ignore:
Timestamp:
10/22/10 11:18:44 (11 years ago)
Author:
Dominic König
Comment:

--

Legend:

Unmodified
Added
Removed
Modified
  • S3/S3Resource

    v1 v1  
     1[[TOC]]
     2= S3XRC - S3Resource =
     3
     4  * [wiki:S3XRC]
     5  * [wiki:RESTController]
     6
     7== Introduction ==
     8
     9Resources are dynamic, document-alike representations of data objects on the Eden server. As such, they can represent both single instances as well as structured sets of data objects.
     10
     11    ''Note: in the database context, a single instance of a data object is typically called a record (or row), with fields (or columns) as its atomic elements. However, even though S3Resources are typically bound to a relational database and therefore the record/field terminology is often used in this regard, they are '''''not''''' intended to provide any object-relational mapping (ORM).''
     12
     13S3Resources implement an extensible, multi-format [http://en.wikipedia.org/wiki/REST REST]ful API to retrieve and manipulate data on the Eden server by [http://en.wikipedia.org/wiki/HTTP HTTP] requests, where every resource can be addressed by an individual [http://en.wikipedia.org/wiki/URL URL].
     14
     15In the context of the Model-View-Controller (MVC) architecture, S3Resources are controller-generated objects. Typically, the controller generates a resource in response to an incoming HTTP request and returns a view of it (''output'') to the client:
     16
     17    [[Image(arch3.png)]]
     18
     19To parse the URL and the incoming HTTP request and to generate a corresponding resource, the controller can make use of the [wiki:S3XRC_S3Request S3Request] helper class.
     20
     21== Terminology ==
     22
     23A S3Resource consists of elements which can be either ''resource''s themselves (container type) or ''data'' (atomic types).
     24
     25Container type elements within a resource can be:
     26
     27  - primary resources (independent data objects)
     28  - component resources (objects which are part of a primary object)
     29  - referenced resources (objects which are referenced by primary or component resources)
     30
     31== REST Interface ==
     32
     33This interface is used by the ''shn_rest_controller()'' function:
     34
     35{{{
     36res, req = s3xrc.parse_request(module, resource, session, request, response)
     37output = res.execute_request(req, **attr)
     38}}}
     39
     40where {{{res}}} is the S3Resource, and {{{req}}} the [wiki:S3XRC_S3Request S3Request].
     41
     42=== Flow Diagram ===
     43
     44[[Image(s3rest.png)]]
     45
     46=== Method Handlers ===
     47
     48The default method handlers of ''shn_rest_controller()'' are implemented in models/01_crud.py:
     49
     50  - shn_read(r, **attr)
     51  - shn_list(r, **attr)
     52  - shn_create(r, **attr)
     53  - shn_update(r, **attr)
     54  - shn_delete(r, **attr)
     55  - shn_search(r, **attr)
     56
     57where ''r'' is the respective [wiki:S3XRC_S3Request S3Request], and ''**attr'' contains all further arguments passed to shn_rest_controller().
     58
     59During the [#Pre-andPost-Hooks pre-process], you can re-configure which handlers shall be used by
     60
     61{{{
     62r.resource.set_handler(action, handler)
     63}}}
     64
     65where:
     66
     67  - '''action''' is the name of the action, i.e. one of 'read', 'list', 'create', 'update', 'delete', or 'search'
     68  - '''handler''' is the handler function/lambda
     69
     70=== Custom Methods ===
     71
     72You can resource-specific custom methods by:
     73{{{
     74  s3xrc.model.set_method(prefix, name, method, action)
     75}}}
     76
     77where:
     78
     79  - '''prefix''' is the module prefix of a resource
     80  - '''name''' is the name of the resource (without prefix)
     81  - '''method''' is a string representing the name of the method (e.g. "search_simple")
     82  - '''action''' is the method handler function/lambda
     83
     84The ''action'' method has to take the same arguments as the default handlers: ''r'' (S3Request) and ''**attr''.
     85
     86This features URLs like:
     87
     88  /prefix/name/<record_id>/<method>
     89
     90=== Pre- and Post-Hooks ===
     91
     92You can hook in a '''preprocessing''' function into the REST controller (as response.s3.prep) which will be called ''after'' the controller has parsed the request, but ''before'' it gets actually executed - with the current S3RESTRequest as argument (which includes the primary resource record, if any).
     93
     94This allows you to easily make changes to resource settings (e.g. access control, list fields etc.), or even to the REST controller configuration (e.g. custom methods) depending on the request type, its parameters or the addressed resource, without having to parse the web2py request manually. You can even bypass the execution of the request and thus hook in your own REST controller - with the advantage that you don't need to parse the request anymore.
     95
     96'''NB: Hooks should be added at the top of your controller function, especially pre-processing hooks.'''
     97
     98A simple example:
     99
     100Original code fragment:
     101{{{
     102    if len(request.args) == 0:
     103        # List View - reduce fields to declutter
     104        table.message.readable = False
     105        table.categories.readable = False
     106        table.verified_details.readable = False
     107        table.actioned_details.readable = False
     108
     109    response.s3.pagination = True #enable SSPag here!
     110
     111    return shn_rest_controller(module, resource, listadd=False)
     112}}}
     113
     114Using the pre-processor hook instead:
     115{{{
     116    def log_prep(jr):
     117        if jr.representation == "html" and \
     118           jr.method is None and \
     119           jr.component is None:
     120            # List View - reduce fields to declutter
     121            table.message.readable = False
     122            table.categories.readable = False
     123            table.verified_details.readable = False
     124            table.actioned_details.readable = False
     125        return True
     126
     127    response.s3.prep = log_prep
     128    response.s3.pagination = True #enable SSPag here!
     129
     130    return shn_rest_controller(module, resource, listadd=False)
     131}}}
     132
     133The return value of the preprocessor function can simply be True, in which case the REST request will be executed as usual. Returning False would lead to a HTTP400 "Invalid Request" exception being raised.
     134
     135The return value of the preprocessor function can also be a dict for more granular control - containing the following elements (all optional):
     136
     137 - '''success''': boolean (default: True)
     138 - '''output''': dict (default: None)
     139 - '''bypass''': boolean (default: False)
     140
     141If ''bypass'' is True, then the REST controller does not execute the request (the post-hook is executed, though). ''output'' must not be None in this case - it will be returned from the REST controller.
     142
     143If ''success'' is False, and ''output'' is not None, then the REST controller does not execute the request, but just returns "output" (post-hook will ''not'' be executed in this case).
     144
     145If ''success'' is False and ''output'' is None, a HTTP400 "Invalid Request" will be raised instead.
     146
     147Examples:
     148
     149In most cases, you will just return "True" - in some cases you might want to raise an error, e.g.:
     150{{{
     151    response.error = "This request cannot be executed"
     152    return dict(
     153        success=False,
     154        output=dict(title="My Pagetitle", item="Sorry, no data..."))
     155}}}
     156
     157There is also a '''post-processing hook''' (response.s3.postp) that allows you to execute something directly after the REST request has been executed, but before the shn_rest_controller returns. The post-hook function will be called with the current S3RESTRequest and the output dict of its execution as arguments.
     158
     159PostP Examples:
     160{{{
     161def user_postp(jr, output):
     162    # Replace the ID column in List views with 'Action Buttons'
     163    shn_action_buttons(jr)
     164    return output
     165response.s3.postp = user_postp
     166}}}
     167
     168{{{
     169def postp(r, output):
     170   # Redirect to read/edit view after create rather than to list view
     171   if r.representation == "html" and r.method == "create":
     172       r.next = r.other(method="", record_id=s3xrc.get_session(session,
     173module, resource))
     174   return output
     175response.s3.postp = postp
     176}}}
     177==== Passing information between main Controller & Prep ====
     178Scope normally means that these 2 sections can only talk to each other via globals or the Request object.
     179
     180If you need to pass data between them, you can use this trick:
     181{{{
     182vars = {} # the surrounding dict
     183def prep(r, vars):
     184    vars.update(x=y) # the actual variable to pass is x
     185    return True
     186
     187response.s3.prep = lambda r, vars=vars: prep(r, vars)
     188
     189output = shn_rest_controller(module, resource)
     190
     191x = vars.get(x, None)
     192}}}
     193
     194An example usage is in {{{controllers/gis.py}}} for location()
     195
     196== Export behavior ==
     197
     198The XML export function supports HTTP/GET. If no record ID is specified in the request, this will get a ''list'' attempt, otherwise a ''read'' attempt to the specified record. This is the same as for the Joined Resource Controller in general, except the following behaviour:
     199
     200  - when attempting to read a joined resource, you will get both the primary record and all belonging records in '''this''' joined resource.
     201  - when attempting to read a primary resource, you will get both the primary record and all belonging records in '''all''' joined resources.
     202
     203== Import behavior ==
     204
     205The XML import function supports HTTP/PUT, HTTP/POST as well as HTTP/GET with explicit ''create'' and ''update''.
     206
     207''However, the use of POST is actually wrong here and is therefore handled like GET. The only way for list+create with ExtJS, though.''
     208
     209The behaviour is similar to the XML Export function:
     210
     211  - when there is no join in the request, resources will be joined automatically:
     212     - the resource-element of the primary will be imported
     213     - when joined resources for that element are also found in the XML source, they will be imported as well (all)
     214  - when a joined resource is specified in the request, only elements for that joined resource will be imported - no other joined resources and not the primary record either
     215
     216The import function can read from the request body (by default), but also pull from files or URL's:
     217
     218  - to import XML data from files, append a ?filename=<full_path_to_xml_file> variable to the request URL.
     219  - to fetch XML data from URL's, append a ?fetchurl=<fully_qualified_url> variable to the request URL. Sources can be HTTP as well as FTP sources, as long as they export XML data.
     220
     221This allows you to transfer resources directly from one Sahana Eden server to the other, e.g.:
     222
     223{{{
     224http://localhost:8000/eden/pr/person/create?format=xml&fetchurl=http://vita.sahanafoundation.org/eden/pr/person.xml
     225}}}
     226
     227fetches all person data (including all joined resources) from vita.sahanafoundation.org and creates or updates corresponding records on localhost.
     228
     229=== Validation ===
     230
     231Imported data is validated using the ''requires'' validators as specified in the models, before comitting them to the database.
     232
     233In case of any validation error, no data import will happen at all. Instead, the import data tree with error attributes added to the erroneous elements (see JSON reponse format) will be returned.
     234
     235To override this, you may specify "ignore_errors=True" in the URL. In this case the import just skips the erroneous records and always returns a success message (error messages are stored, but not returned). Note that "ignore_errors" is not recommended to be represented in regular a user interface, but just used manually if at all necessary (e.g. in manual pre-population of data).
     236
     237'''Note:''' In contrast to validation errors, !IntegrityError and IOError exceptions during data import do not prevent or roll back any data import that happened before the exception, and the returned element tree does not contain any error attributes, and these exceptions can not be overridden by "ignore_errors" either.
     238
     239=== Onvalidation/Onaccept Callbacks ===
     240
     241Import of records happens almost as if they were entered in HTML forms, i.e. ''onvalidation''- and ''onaccept'' callbacks are executed as usual. They receive a pseudo-form as parameter, which is ''Storage()'' instead of ''Form()''. Other than ''Form()'' objects, the pseudo-forms contain only '''form.vars''' (as usual, so most of the callbacks should work without change) and a '''form.method''' (which contains either "create" or "update") to indicate the action (which, in case of XML imports, can '''not''' be determined from the request).
     242
     243NOTE: Never redirect from onvalidation or onaccept functions - this would break the XML import! Instead, set a flag in response.s3, and let the calling controller redirect upon this flag.
     244=== Authentication/Auditing ===
     245
     246XML imports/exports are fully ''Auth''-enabled, i.e. all actions are checked for permission and support auditing as specified in the Settings.
     247
     248=== Response Format ===
     249
     250The response object to ''create'', ''update'' or ''delete'' requests in both XML as well as JSON representations always contains a JSON body like:
     251
     252{{{
     253{
     254    "status": "success",                 // "success" or "failed"
     255    "statuscode": "200",                 // HTTP Status Code
     256    "message" : "Ok",                    // Message as clear text (optional)
     257    "tree" : {                           // Tree object in the JSON response indicates an error
     258                                         // object containing the originally submitted data tree as described above
     259        "$_my_resource": {
     260
     261            "myfield": {
     262                "@value": "xxxx",
     263                "$": "Bullshit",
     264                "@error": "Validation Error: myfield must be integer!"    // @error indicates an error for this field
     265            }
     266        }
     267    }
     268}
     269}}}
     270
     271  * ''tree'' is only returned in case of an error during Create or Update actions
     272  * ''@error'' attributes can be re-sent (will be removed by the controller)
     273
     274=== UUID Mapping and Matching ===
     275
     276On data export, all references are mapped from internal id's to uuid's - given that uuid's are present in the referenced table, otherwise the reference field is not represented in XML or JSON at all.
     277
     278On data import, all references are mapped back from uuid's to internal id's, provided that the referenced record (with that uuid) exists in the database, otherwise the reference field is not imported (get's a default value).
     279
     280On data import with create method, records with matching UUID's will automatically get updated instead of newly created.
     281
     282Therefore, for resources that have to be exchanged, the use of UUID's is highly recommended.
     283
     284=== In-line transformation with XSLT ===
     285
     286[[Image(s3xml.png)]]
     287
     288The XML/JSON interface uses XSLT to transform data from/to the raw XML/JSON format into foreign formats.
     289
     290{{{
     291<?xml version="1.0"?>
     292<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version="1.0">
     293    <xsl:output method="xml"/>
     294    <xsl:template match="/">
     295        <xsl:copy-of select="."/>
     296    </xsl:template>
     297</xsl:stylesheet>
     298}}}
     299
     300This is just an example template for import/export, but it is not used. Instead, when you specify the native formats "xml" or "json", the raw formats described above are used.
     301
     302To enable other formats, you have to:
     303
     304For import:
     305
     306  - put an XSLT template for import into static/xsl/import and name the template <format>.xslt (e.g. pfif.xslt)
     307  - in the file models/01_crud.py add the <format> name to this line:
     308{{{
     309shn_xml_import_formats = ["xml", "pfif"]
     310}}}
     311
     312For export analogous:
     313
     314  - put an XSLT template for export into static/xsl/export and name the template <format>.xslt (e.g. pfif.xslt)
     315  - in the file models/01_crud.py add the <format> name to this line:
     316{{{
     317shn_xml_export_formats = ["xml", "pfif"]
     318}}}
     319
     320From there, you can use that format name as either extension or ?format= option in requests:
     321
     322Export:
     323{{{
     324http://localhost:8000/eden/pr/person/1.pfif
     325}}}
     326
     327Import:
     328{{{
     329http://localhost:8000/eden/pr/person/1.pfif/create
     330}}}
     331
     332and the corresponding XSL template will now be used at import/export from/into that format.
     333
     334Note: "json" and "xml" '''must''' be in the format lists in models/01_crud.py!
     335
     336 * [http://free.editix.com/features/xslt_debugger.html EditiX] - free GUI XSLT debugger (Java-based, so x-platform)
     337 * [http://xalan.apache.org Xalan] - free CLI XSLT debugger (Java-based, so x-platform)
     338 * Docs: [wiki:DeveloperGuidelinesTips#XSLT]
     339
     340----
     341DeveloperGuidelines