s3_rest_controller

Introduction

The so-called REST Controller (function s3_rest_controller()) is a helper function to easily apply the RESTful API of the S3Resource class to your controller.

s3_rest_controller does:

• parse and execute the incoming HTTP request on the specified resource
• populate and hand-over view variables
• choose and set the response view template (response.view)

Using s3_rest_controller, a basic RESTful controller for the pr_image table can look like:

def image():

    """ RESTful CRUD controller """

    return s3_rest_controller("pr", "image")

This exposes all standard URLs and methods for this table, including:

• interactive create, read, update, delete and list views
• non-interactive data export/import (GET/POST/PUT/DELETE) in multiple formats

Basic Syntax

output = s3_rest_controller(prefix, resourcename)

• prefix is the application prefix of the resource
• resourcename is the name of the resource (without prefix)

• output contains the result of the request and can be returned from the controller as-is
  • in interactive view formats, this is a dict of view variables

Basic Options

All of the following options are set using:

s3mgr.configure(table, key=value)

where:

• table is the respective DB table
• key is the setting key
• value is the configuration value

You can specify multiple settings at once:

s3mgr.configure(table, key1=value1, key2=value2, ...)

configure overrides any settings that have been made for that table before (e.g. in the model), where only the specified keys are changed while all others are left untouched.

You can also delete a particular setting by:

s3mgr.model.clear_config(table, "key")

where "key" must be the respective key as string.

CRUD Options

You can define which of the CRUD functions are allowed for your resource. By default, they all are True.

  s3mgr.configure(table, editable=True,             # Records can be updated
                         insertable=True,           # New records can be added
                         deletable=True,            # Records can be deleted
                         listadd=True)              # to have a add-record form in list view

Note:

• these settings are completely independent from the user's permission to perform the respective actions. If you have set insertable=False, then adding records to this table will be impossible even if the user has the permission to add records.
• in standard views, the add-form in list views is hidden and can be activated by clicking the respective "Add Record" button. If listadd=False, then the "Add Record" button will redirect to the create-form view instead.
• these settings do not affect non-interactive representations

List-Fields

By default, all readable fields of a table appear in list views.

To control which fields are to be included in list views, to include virtual fields or fields in referenced tables, or to change the order of the columns, use the list_fields CRUD setting.

This setting takes a list of field names of the fields to be displayed:

s3mgr.configure(table,
                list_fields=["id", "name", "location_id"])

NOTE: If your view uses dataTables (which all default view templates do), then you must include "id" at the first place to have it working properly.

For virtual fields, you should provide a tuple of (T("FieldLabel"), "fieldname"), because virtual fields do not have a .label setting (as they are functions and not Field instances):

s3mgr.configure(table,
                list_fields=["id",
                             "name",
                             "location_id",
                             (T("Total costs"), "total_costs")]

You can use the tuple notation for any other field as well, in order to override the field label for this list view.

To include a field from a referenced table, insert the field name as "<foreign_key>$<fieldname>", e.g.:

s3mgr.configure(table,
                list_fields=["id",
                             "name",
                             "location_id$lat",
                             "location_id$lon"]

List-add

By default, list views will contain a hidden create-form and an "Add Record"-button to activate the create-form (provided that the user is authorized to add records to this resource).

If the resource is set to insertable=False or if the user is not permitted to create records in this table, no "Add Record"-button will be available.

NOTE: If for some reason the default embedded create-form is not desired, you can set s3xrc.configure(table, listadd=False, addbtn=True), which will render the "Add Record"-button as link to the create method instead.

Redirection

Default destination of the redirection after a create or update is the read view of the processed record, after a delete it is the list view of the respective table.

The redirection destination can be configured per DB table, using:

s3mgr.configure(table, create_next=url)

s3mgr.configure(table, update_next=url)

s3mgr.configure(table, delete_next=url)

where:

• table is the respective DB table
• url is the URL to redirect to

If, for create_next or update_next, url contains the string "[id]" (or its URL-encoded equivalent), then this string will be replaced by the ID of the updated/newly created record before redirection.

Note:

• redirection does not happen after non-interactive data imports!

Callbacks

For every DB table, you can define functions to be invoked upon certain CRUD events. Those "callbacks" can be:

• a single callable (function, lambda, callable object)
• a list of callables, which are invoked in list order
• a dict of callables, where the tablename is used as key to find the callable to be invoked
• a dict of lists of callables, where the tablename is used as key to find the list of callables to be executed in list order

The return value of the callables, if any, is ignored.

Important: Callbacks are invoked in the same manner during non-interactive data imports, where usually multiple records will be processed in the same request. Therefore, callback functions must not redirect, nor commit or roll back the current transaction!

Validation Callbacks

You can define extra form validation methods to be invoked after a create/update form has successfully passed the indivdual field validation, by using:

s3db.configure(tablename, create_onvalidation=callback)

s3db.configure(tablename, update_onvalidation=callback)

where:

• table is the respective DB table
• callable is the callback setting, see Callbacks

If either of create_onvalidation or update_onvalidation is not set, then the onvalidation setting is tried:

s3db.configure(tablename, onvalidation=callback)

This allows you to define a common onvalidation callback for both create and update.

Onvalidation callbacks are meant to allow additional form data validation (beyond individual field validators). The callback functions receive the form as first and only parameter, while their return values will be ignored.

Any validation errors are to be reported directly into the form as:

form.errors[fieldname] = error_msg

where:

• fieldname is the field containing the invalid value
• error_msg is the error message to be displayed in the form close to that field

If after the execution of the onvalidation callback any messages are found in form.errors, then no data are being imported and instead, the process will return to the input view with the messages displayed close to the respective form fields.

In non-interactive data imports, the error message will be added to the import tree as extra attribute of the invalid element. The XML importer will however process all records in the import tree in order to find all validation errors before reporting the invalid tree to the sender, and in case ignore_errors is used, all valid records will be imported in the first attempt.

On-accept Callbacks

You can define methods to be invoked after a record has been created/updated, by using:

s3db.configure(tablename, create_onaccept=callback)

s3db.configure(tablename, update_onaccept=callback)

where:

• table is the respective DB table
• callable is the callback setting, see Callbacks

If either of create_onaccept or update_onaccept is not set, then the onaccept setting is tried:

s3db.configure(tablename, onaccept=callback)

This allows you to define a common onaccept callback for both create and update.

The onaccept callbacks are meant to perform extra post-processing of the newly created/updated record (e.g. to update dependent records). The callback functions receive the respective FORM instance (with the input data being in form.vars) as first and only parameter, while their return value will be ignored.

On-Delete-Cascade Callback

You can specify callbacks to be invoked when a record is to be deleted:

s3db.configure(tablename, ondelete_cascade=callback)

where:

• table is the respective DB table
• callable is the callback setting, see Callbacks

The ondelete_cascade callback is meant to perform cascade actions before deleting a record (e.g. to update or remove dependend records, or to release constraints that cannot be introspected using the "ondelete" field setting). The hook will receive the record as its only parameter, its return value will be ignored.

Note: the record will only contain the record ID but no details

On-Delete Callback

You can also define callbacks to be invoked after a record has been deleted:

s3db.configure(tablename, ondelete=callback)

where:

• table is the respective DB table
• callable is the callback setting, see Callbacks

The ondelete callbacks are meant to perform extra post-processing of the deleted record (e.g. updates to counters or other aggregates). The callback function will receive the respective record as its only parameter, while its return value will be ignored.

Note: at the time when the callback is invoked, the record is already deleted from the database.

Note: the record will only contain the record ID but no details

Note: soft-deleted (archived) records hold their former foreign keys as a JSON object in the deleted_fk field, whilst all foreign key fields will be set to None.

Important: Do not delete a record by simply settings its deleted field to True. This would neither perform the necessary cascading, nor invoke any callbacks nor does it properly store and release any foreign key constraints. Always use S3Resource.delete() to delete records!

Pagination

The default pagination method is server-side (SSPag), meaning, in list views the client will receive only the first of the available rows, and then retrieve more rows as needed by subsequent Ajax calls.

In contrast to that, in client-side pagination (CSPag) mode all available rows of the list are retrieved and send to the client at once. For most tables, though, this will probably be a huge data set and take a long time to extract and transmit, while mostly being unnecessary when the user only needs to see the first 20 rows to find what he's looking for.

However, some tables may by their nature only contain one or few rows, and then server-side pagination is not needed (in fact, inefficient). In these cases, the respective controller can turn it off by:

response.s3.no_sspag = True

View Control

In web2py, the default view is chosen after the name of the controller, i.e. if the controller is person(), then the default view is person.html.

s3_rest_controller() modifies this schema in order to allow you to create method-specific views for the same controller.

Default View

The default view is chosen in a fallback cascade, which is:

1. <prefix>/<resource-name>_<component-name>_<method>.html
2. <prefix>/<resource-name>_<method>.html
3. <method>.html

where:

• prefix is the application prefix of the resource name (e.g. "pr")
• resource-name is the name of the resource (e.g. "person")
• component-name is the name of the component resource (e.g. "address", only if the request targets a component resource)
• method is one of:
  • display
  • list
  • list_create (to be deprecated)
  • create
  • update
  • delete
  • search

Example:

http://localhost:8000/eden/pr/person/create.html

is looking for one of pr/person_create.html, pr/create.html or finally create.html as default view.

If none of these is found, "default.html" serves as catch-all fallback.

Custom View

To choose a custom view, you can easily override the default setting after s3_rest_controller returns:

  output = s3_rest_controller(prefix, resourcename)

  response.view = "myview.html"

  return output

Additional View Variables

In interactive view formats, any additional named arguments in the s3_rest_controller argument list will be added to the view variables:

output = s3_rest_controller(prefix, resourcename, **attr)

• attr: additional view variables

• any callable argument will be invoked with the S3Request as first and only argument, and its return value will be added to the view variables
• any non-callable argument will be added to the view variables as-is
• any argument that gives None will remove this key from the view variables

A typical use-case is rheader:

def my_rheader(r):
  if r.interactive and r.component:
    # Code producing the rheader...
    return rheader
  else:
    return None

output = s3_rest_controller(prefix, name, rheader=my_rheader)

If my_rheader(r) gives something else than #!python None, then this value is added as rheader to the view variables.

Advanced Options

Filtering Lists

You can filter lists in the controller by setting response.s3.filter to a filter query:

    # This filters for females:
    response.s3.filter = (db.pr_person.gender == 2)

    return s3_rest_controller("pr", "person")

Note that this only takes effect in the main controller (not in prep or postp).

Note that response.s3.filter affects both, the primary resource and components!

In prep, you can also add filter queries using the add_filter method:

    def prep(r):
        resource = r.resource
        query = (db.pr_address.type == 1) # Home addresses only
        resource.add_filter(query)
        return True
    response.s3.prep = prep

    return s3_rest_controller("pr", "person")

However, add_filter again affects both, primary and component records - so this example would:

• only retrieve person records which have a type 1 address record
• only retrieve the address records with type 1.

This can be an unwanted side-effect.

To have the primary resource unfiltered, and filter only records in a particular component, you can use add_component_filter:

    def prep(r):
        resource = r.resource
        query = (db.pr_address.type == 1) # Home addresses only
        resource.add_component_filter("address", query)
        return True
    response.s3.prep = prep

    return s3_rest_controller("pr", "person")

In this case, all person records would be selected - while only address records of type 1 would be retrieved.

Pre-populating Create-Forms

Create-forms can be pre-populated with data by one of these 3 methods:

1. model defaults (standard)
2. values from another record in the database
3. values provided by the controller

Model defaults are defined per field and can be set as db.my_table.field.default = value at any time before the REST method is applied (even in the pre-process).

Values from another record in the database can be used via a URL query like:

• /my/resource/create?from_record=id&from_fields=fieldname1,fieldname2,...

The id of the original record can also be specified as tablename.id, if the original record is in another table. Additionally, fieldnames can be specified as fieldname$original_fieldname to map between different fieldnames.

To pre-populate Create-forms from the controller, you can specify the variable populate in the arguments of s3_rest_controller:

output = s3_rest_controller(prefix, resourcename,
                            populate=dict(fieldname1=value1,
                                          fieldname2=value2))
return output

Instead of a dict, you can also pass a callable object as populate. This will be executed with the current S3Request and the named arguments of s3_rest_controller in order to produce the field/value dict:

def populate(r, **attr):
    """
    Helper function to pre-populate create-forms

    """

    # some code to produce the data
    # ...

    # return the dict
    return dict(fieldname1=value1,
                fieldname2=value2)

output = s3_rest_controller(prefix, resourcename,
                            populate=populate)

return output

Note that populate will only be applied in GET requests and only if no record_id is specified. That means, if it uses a separate form to generate the data, you need to revert the request into GET in order to have the create-form pre-populated:

data = None
form = FORM(...some form...)
output = dict(helper_form=form)
if form.accepts(request.vars, session, formname="helper_form"):
    output = dict() # remove the helper form
    data = Storage(...) # some code to extract the data from the helper form
if data:
    request.env.request_method = "GET" # revert to GET if data available

_output = s3_rest_controller(prefix, resourcename, populate=data)

if isinstance(_output, dict):
    output.update(_output)
else:
    output = _output

return output

NB: This construction could be used e.g. to loop in an OCR client into a REST controller. The helper form would then be a file upload form which is displayed alongside with the normal create-form - so the user can either enter data manually and submit the create-form, or first submit a file to pre-populate the create form, and then edit the data and submit the create-form.

Pre-Process

A prep hook would allow you to change a handler configuration in certain situations, e.g. testing a URL variable:

def myresource():

   """ RESTful CRUD controller """

   # Define pre-processor as local function:
   def prep(r):
       mylist = request.vars.get("mylist")
       if mylist:
           r.set_handler("list", my_list_controller)
       return True # do not forget to return True!

   # Hook pre-processor into REST controller:
   s3db.prep = prep

   output = s3_rest_controller(modulename, resourcename)
   return output

This example would switch to my_list_controller instead of s3_list in case there is a ?mylist= in the URL. In all other cases, the default handlers are executed as usual, you still have a RESTful API for your resources.

While you can define prep's and postp's as local functions (as in the example above) or even lambdas, it is also very well possible to create more generic, reusable prep and postp functions (e.g. to implement different method handler configurations, or to catch certain situations to bypass the CRUD, or to manipulate the output dict in a certain way).

A very important structure during prep and postp is the S3Request object (usually instantiated as "r"). This object contains all necessary information to process the current REST request, and it is passed to both prep and postp. See S3RESTful API for a list of attributes and methods.

Passing information between main controller & pre-processor

Scope normally means that these 2 sections can only talk to each other via globals or the Request object.

If you need to pass data between them, you can use this trick:

vars = {} # the surrounding dict
def prep(r, vars):
    vars.update(x=y) # the actual variable to pass is x
    return True

response.s3.prep = lambda r, vars=vars: prep(r, vars)

output = s3_rest_controller(module, resource)

x = vars.get(x, None)

An example usage is in controllers/gis.py for location()

Post-Process

Method Handlers

The s3_rest_controller has multiple hooks,Look at how s3_rest_controller is actually defined in models/00_utils.py:

def s3_rest_controller(prefix=None, resourcename=None, **attr):

    set_handler = r.set_handler
    set_handler("barchart", s3_barchart)
    set_handler("compose", s3base.S3Compose)
    set_handler("copy", lambda r, **attr: \
                               redirect(URL(args="create",
                                            vars={"from_record":r.id})))
    set_handler("deduplicate", s3base.S3Merge)
    set_handler("filter", s3base.S3Filter)
    set_handler("import", s3base.S3Importer)
    set_handler("map", s3base.S3Map)
    set_handler("profile", s3base.S3Profile)
    set_handler("report", s3base.S3Report)
    set_handler("import", S3PDF(),
                    http = ["GET", "POST"],
                    representation="pdf")


    # Execute the request and apply crud functionality or execute other method handler
    output = r(**attr) 
    return output

This is nothing else than a wrapper for the global r object (an instance of S3Request). You don't have to use s3_rest_controller to have a RESTful API, you can make your own handler configuration and call r directly.

But even when using s3_rest_controller, you can take control over what actually happens. A very comfortable (and recommended) way to get control over s3rest when using s3_rest_controller is to hook in prep and postp functions. Look above to find out when prep and postp hooks are invoked.

look S3Method to see how method handler can be created using S3Method

Custom Methods

If you have a resource "warehouse" and want to implement a function "report" that generates a report about the current status of the warehouse.

Now you're gonna provide this report in a RESTful way, i.e. you want to provide it as a resource that can be addressed via URLs like:

• http://site.myserver.org/eden/wm/warehouse/report

and is provided in several different data formats beyond HTML (the interactive view), let's say - XLS and XML:

• http://site.myserver.org/eden/wm/warehouse/report.xls
• http://site.myserver.org/eden/wm/warehouse/report.xml

you can use s3_rest_controller for this!

This could be your "warehouse" CRUD controller:

def warehouse():

   """ RESTful CRUD controller """

   return s3_rest_controller(module, resource, ...)

At first you implement your report function (in the controller file). This function takes the argument r (=S3Request) and a dict of named arguments (just the same named arguments from the s3_rest_controller call above). This function returns the report. Then, in your controller, you plug in this function to your resource - together it would look like that:

def warehouse():

    """ RESTful CRUD+Reports controller """

    # Plug warehouse_report into warehouse resource:
    s3db.set_method(module, resource, method="report", action=warehouse_report)

    return s3_rest_controller(module, resource, ...)

def warehouse_report(r, **attr):

    """ Warehouse report generator """

    # Code to produce the report goes here
    report = ...

    return report

Note that if "report" is a dict, then the REST controller automatically adds the S3Request as "r" to that dict before returning it. Thus, "r" is available to the view templates. That also means: do not use "r" as key in that dict.

However, there may be other formats than HTML - to implement other formats, you might need to check for the representation of the request. This can be found in r:

def warehouse_report(r, **attr):

    """ Warehouse report generator """

    if r.representation in ("html", "popup"):
        # Code to produce the report items:
        title = T("Warehouse Report")

        # Assemble the report items in a dict:
        report = dict(title=title, ...)

    elif r.representation == "xls":
        # Code to produce the XLS report goes here
        ...

    elif r.representation in s3_xml_export_formats:
        # Code to produce the XML report goes here
        ...

    else:
        r.error(501, r.ERROR.BAD_FORMAT)

    return report

See S3Request to find out more about "r".

To produce the XML report, it is probably sufficient to just export the requested warehouse information in S3-XML, and then use XSLT stylesheets to produce the finally desired XML formats. That's pretty easy:

def warehouse_report(r, **attr):
    ...
    elif r.representation in s3_xml_export_formats:
        report = export_xml(xrequest)
    ...

Perhaps you want to add an RSS feed:

def warehouse_report(r, **attr):
    ...
    elif r.representation == "rss":
        # Code to produce the RSS report goes here
        report = ...
    ...

Getting the data from the warehouse_report.

Your implementation already supports a variety of URLs:

This URL addresses the report for all warehouses:

• http://site.myserver.org/eden/wm/warehouse/report

This URL addresses the report for the warehouse record with ID=1.

• http://site.myserver.org/eden/wm/warehouse/1/report

This URL addresses the report in XLS format for the warehouse record with the UUID=123654278 (assuming that you have UUID's in your warehouse table).

• http://site.myserver.org/eden/wm/warehouse/report.xls?warehouse.uid=123654278

The S3Request provides the resource information to your warehouse_report function.

In case a specific record has been requested, you can access it as:

    record = r.record

If r.record is None, then the request is targeting all warehouse records, so you'd take:

    table = r.table
    records = db().select(table.ALL)
    for record in records:
       ...

instead.

NOTE: representation in r is always all lowercase, there is no differentiation between the ".XML" and ".xml" extension in the URL.

And...not to forget: your warehouse_report is still a controller function, that means you can implement forms as usual (e.g. operated with form.accepts or web2py Crud).

Re-Routing Controllers

There may be a time when you want to bypass the default controllers, or else add a custom controller which can have different options.

This allows re-routing of any controller/function combination to the custom/rest controller - which is a vanilla s3_rest_controller, i.e. without any prep, postp or attr. The latter can then be added in customise_controller.

To use this, a template will configure e.g.:

settings.base.rest_controllers = {("org", "organisation"): ("org", "organisation")}

...which re-routes org/organisation to custom/rest, and runs it as a s3_rest_controller with prefix="org" and name="organisation". This re-routing is completely transparent, i.e. the s3_rest_controller (and anything called from there) sees a normal org/organisation request. Consequently, to customise this call, one would use customise_org_organisation_controller (not customise_custom_rest_controller!), as usual.

As this allows re-routing of any controller/function combination, one can also "add" controllers that do not exist, e.g.:

settings.base.rest_controllers = {("org", "suppliers"): ("org", "organisation")}

This would though still be customised via customise_org_organisation_controller (=>can inspect r.function in prep to distinguish), but page access authorisation would now be through org/suppliers rather than org/organisation (so can have different permissions for those).

When adding a route with a custom module prefix:

settings.base.rest_controllers = {("my", "organisation"): ("org", "organisation")}

...then a corresponding module entry for that must be added to settings.modules - though this is useful anyway if menus are to be added automatically etc. Again, page authorisation would go through my/organisation here instead of org/organisation, so there could be different permission sets.

Note: this feature is orthogonal to default/index/*, which is still useful for implementing non-REST custom controllers, and should therefore be retained.

---

DeveloperGuidelines