= s3_rest_controller =
[[TOC]]

== Introduction ==

The so-called '''REST Controller''' (function {{{s3_rest_controller()}}}) is a helper function to easily apply the RESTful API of the [wiki:S3/S3Resource S3Resource] class to your controller.

{{{s3_rest_controller}}} does:

  - parse and execute the incoming HTTP request on the specified resource
  - populate and hand-over view variables
  - choose and set the response view template ({{{response.view}}})

Using {{{s3_rest_controller}}}, a basic RESTful controller for the {{{pr_image}}} table can look like:

{{{#!python
def image():

    """ RESTful CRUD controller """

    return s3_rest_controller("pr", "image")

}}}

This exposes all standard [wiki:S3XRC/RESTfulAPI/URLFormat URLs] and [wiki:S3XRC/RESTfulAPI methods] for this table, including:

  - interactive create, read, update, delete and list views
  - non-interactive data export/import (GET/POST/PUT/DELETE) in multiple formats

== Basic Syntax ==

{{{#!python
output = s3_rest_controller(prefix, resourcename)
}}}

  - '''prefix''' is the application prefix of the resource
  - '''resourcename''' is the name of the resource (without prefix)

  - '''output''' contains the result of the request and can be returned from the controller as-is
    - in interactive view formats, this is a {{{dict}}} of view variables

== Basic Options ==

All of the following options are set using:

{{{#!python
s3db.configure(table, key=value)
}}}

where:
  - '''table''' is the respective DB table
  - '''key''' is the setting key
  - '''value''' is the configuration value

You can specify multiple settings at once:
{{{#!python
s3db.configure(table, key1=value1, key2=value2, ...)
}}}

'''configure''' overrides any settings that have been made for that table before (e.g. in the model), where only the specified keys are changed while all others are left untouched.

You can also delete a particular setting by:

{{{#!python
s3db.model.clear_config(table, "key")
}}}

where "key" must be the respective key as ''string''.

=== CRUD Options ===

You can define which of the CRUD functions are allowed for your resource. By default, they all are True.

{{{#!python
  s3db.configure(table, editable=True,             # Records can be updated
                        insertable=True,           # New records can be added
                        deletable=True,            # Records can be deleted
                        listadd=True)              # to have a add-record form in list view
}}}

Note:
  - these settings are completely independent from the user's permission to perform the respective actions. If you have set {{{insertable=False}}}, then adding records to this table will be impossible even if the user has the permission to add records.
  - in standard views, the add-form in list views is hidden and can be activated by clicking the respective "Add Record" button. If {{{listadd=False}}}, then the "Add Record" button will redirect to the create-form view instead.
  - these settings do not affect non-interactive representations

=== List-Fields ===

By default, all readable fields of a table appear in list views.

To control which fields are to be included in list views, to include virtual fields or fields in referenced tables, or to change the order of the columns, use the {{{list_fields}}} CRUD setting.

This setting takes a list of field names of the fields to be displayed:

{{{#!python
s3db.configure(table,
               list_fields=["id", "name", "location_id"])
}}}

  NOTE: If your view uses dataTables (which all default view templates do), then you '''must''' include {{{"id"}}} at the first place to have it working properly.

For virtual fields, you should provide a tuple of {{{(T("FieldLabel"), "fieldname")}}}, because virtual fields do not have a {{{.label}}} setting (as they are functions and not {{{Field}}} instances):

{{{#!python
s3db.configure(table,
               list_fields=["id",
                            "name",
                            "location_id",
                            (T("Total costs"), "total_costs"),
                            ]
}}}

You can use the tuple notation for any other field as well, in order to override the field label for this list view.

To include a field from a referenced table, insert the field name as "<foreign_key>$<fieldname>", e.g.:

{{{#!python
s3db.configure(table,
               list_fields=["id",
                            "name",
                            "location_id$lat",
                            "location_id$lon",
                            ]
}}}

=== List-add ===

By default, list views will contain a hidden create-form and an "Add Record"-button to activate the create-form (provided that the user is authorized to add records to this resource).

If the resource is set to {{{insertable=False}}} or if the user is not permitted to create records in this table, no "Add Record"-button will be available.

  NOTE: ''If for some reason the default embedded create-form is not desired, you can set {{{s3xrc.configure(table, listadd=False, addbtn=True)}}}, which will render the "Add Record"-button as link to the create method instead.''
=== Redirection ===

Default destination of the redirection after a ''create'' or ''update'' is the ''read'' view of the processed record, after a ''delete'' it is the ''list'' view of the respective table.

The redirection destination can be configured per DB table, using:

{{{#!python
s3db.configure(table, create_next=url)
}}}
{{{#!python
s3db.configure(table, update_next=url)
}}}
{{{#!python
s3db.configure(table, delete_next=url)
}}}

where:
  - '''table''' is the respective DB table
  - '''url''' is the URL to redirect to

If, for create_next or update_next, '''url''' contains the string "[id]" (or its URL-encoded equivalent), then this string will be replaced by the ID of the updated/newly created record before redirection.

Note:
  - redirection does not happen after non-interactive data imports!
=== Callbacks ===

For every DB table, you can define functions to be invoked upon certain CRUD events. Those "callbacks" can be:

  - a single callable (function, lambda, callable object)
  - a list of callables, which are invoked in list order
  - a dict of callables, where the tablename is used as key to find the callable to be invoked
  - a dict of lists of callables, where the tablename is used as key to find the list of callables to be executed in list order

The return value of the callables, if any, is ignored.

  '''Important:''' Callbacks are invoked in the same manner during non-interactive data imports, where usually multiple records will be processed in the same request. Therefore, callback functions '''must not''' redirect, nor commit or roll back the current transaction!
==== Validation Callbacks ====

You can define extra form validation methods to be invoked after a create/update form has successfully passed the indivdual field validation, by using:

{{{#!python
s3db.configure(tablename, create_onvalidation=callback)
}}}
{{{#!python
s3db.configure(tablename, update_onvalidation=callback)
}}}

where:

  - '''table''' is the respective DB table
  - '''callable''' is the callback setting, see [#Callbacks Callbacks]

If either of {{{create_onvalidation}}} or {{{update_onvalidation}}} is not set, then the {{{onvalidation}}} setting is tried:

{{{#!python
s3db.configure(tablename, onvalidation=callback)
}}}

This allows you to define a common onvalidation callback for both ''create'' and ''update''.

Onvalidation callbacks are meant to allow additional form data validation (beyond individual field validators). The callback functions receive the '''form''' as first and only parameter, while their return values will be ignored.

Any validation errors are to be reported directly into the form as:
{{{#!python
form.errors[fieldname] = error_msg
}}}

where:

  - '''fieldname''' is the field containing the invalid value
  - '''error_msg''' is the error message to be displayed in the form close to that field

If after the execution of the onvalidation callback any messages are found in {{{form.errors}}}, then no data are being imported and instead, the process will return to the input view with the messages displayed close to the respective form fields.

In non-interactive data imports, the error message will be added to the import tree as extra attribute of the invalid element. The XML importer will however process all records in the import tree in order to find all validation errors before reporting the invalid tree to the sender, and in case {{{ignore_errors}}} is used, all valid records will be imported in the first attempt.

==== On-accept Callbacks ====

You can define methods to be invoked after a record has been created/updated, by using:

{{{#!python
s3db.configure(tablename, create_onaccept=callback)
}}}
{{{#!python
s3db.configure(tablename, update_onaccept=callback)
}}}

where:

  - '''table''' is the respective DB table
  - '''callable''' is the callback setting, see [#Callbacks Callbacks]

If either of {{{create_onaccept}}} or {{{update_onaccept}}} is not set, then the {{{onaccept}}} setting is tried:

{{{#!python
s3db.configure(tablename, onaccept=callback)
}}}

This allows you to define a common onaccept callback for both ''create'' and ''update''.

The {{{onaccept}}} callbacks are meant to perform extra post-processing of the newly created/updated record (e.g. to update dependent records). The callback functions receive the respective {{{FORM}}} instance (with the input data being in {{{form.vars}}}) as first and only parameter, while their return value will be ignored.

==== On-Delete-Cascade Callback ====

You can specify callbacks to be invoked when a record is to be deleted:

{{{#!python
s3db.configure(tablename, ondelete_cascade=callback)
}}}

where:

  - '''table''' is the respective DB table
  - '''callable''' is the callback setting, see [#Callbacks Callbacks]

The {{{ondelete_cascade}}} callback is meant to perform cascade actions ''before'' deleting a record (e.g. to update or remove dependend records, or to release constraints that cannot be introspected using the "ondelete" field setting). The hook will receive the ''record'' as its only parameter, its return value will be ignored.

  '''Note:''' the ''record'' will only contain the record ID but no details

==== On-Delete Callback ====

You can also define callbacks to be invoked ''after'' a record has been deleted:

{{{#!python
s3db.configure(tablename, ondelete=callback)
}}}

where:

  - '''table''' is the respective DB table
  - '''callable''' is the callback setting, see [#Callbacks Callbacks]

The {{{ondelete}}} callbacks are meant to perform extra post-processing of the deleted record (e.g. updates to counters or other aggregates). The callback function will receive the respective ''record'' as its only parameter, while its return value will be ignored.

  '''Note:''' at the time when the callback is invoked, the record is already deleted from the database.

  '''Note:''' the ''record'' will only contain the record ID but no details

  '''Note:''' ''soft-''deleted (archived) records hold their former foreign keys as a JSON object in the {{{deleted_fk}}} field, whilst all foreign key fields will be set to None.

  '''Important:''' Do not delete a record by simply settings its {{{deleted}}} field to True. This would neither perform the necessary cascading, nor invoke any callbacks nor does it properly store and release any foreign key constraints. Always use S3Resource.delete() to delete records!
=== Pagination ===

The default pagination method is '''server-side''' (SSPag), meaning, in list views the client will receive only the first of the available rows, and then retrieve more rows as needed by subsequent Ajax calls.

In contrast to that, in '''client-side''' pagination (CSPag) mode all available rows of the list are retrieved and send to the client at once. For most tables, though, this will probably be a huge data set and take a long time to extract and transmit, while mostly being unnecessary when the user only needs to see the first 20 rows to find what he's looking for.

However, some tables may by their nature only contain one or few rows, and then server-side pagination is not needed (in fact, inefficient). In these cases, the respective controller can turn it off by:

{{{#!python
response.s3.no_sspag = True
}}}

== View Control ==

In web2py, the default view is chosen after the name of the controller, i.e. if the controller is person(), then the default view is person.html.

s3_rest_controller() modifies this schema in order to allow you to create method-specific views for the same controller.
=== Default View ===

The default view is chosen in a fallback cascade, which is:

  1. <prefix>/<resource-name>_<component-name>_<method>.html
  2. <prefix>/<resource-name>_<method>.html
  3. <method>.html

where:

  - '''prefix''' is the application prefix of the resource name (e.g. "pr")
  - '''resource-name''' is the name of the resource (e.g. "person")
  - '''component-name''' is the name of the component resource (e.g. "address", only if the request targets a component resource)
  - '''method''' is one of:
    - display
    - list
    - list_create (to be deprecated)
    - create
    - update
    - delete
    - search

Example:

{{{
http://localhost:8000/eden/pr/person/create.html
}}}

is looking for one of '''pr/person_create.html'', '''pr/create.html'' or finally ''create.html'' as default view.

If none of these is found, "default.html" serves as catch-all fallback.
=== Custom View ===

To choose a custom view, you can easily override the default setting ''after'' s3_rest_controller returns:

{{{#!python
  output = s3_rest_controller(prefix, resourcename)

  response.view = "myview.html"

  return output
}}}
=== Additional View Variables ===

In interactive view formats, any additional named arguments in the {{{s3_rest_controller}}} argument list will be added to the view variables:

{{{#!python
output = s3_rest_controller(prefix, resourcename, **attr)
}}}

  - '''**attr''': additional view variables


  - any callable argument will be invoked with the {{{S3Request}}} as first and only argument, and its return value will be added to the view variables
  - any non-callable argument will be added to the view variables as-is
  - any argument that gives {{{None}}} will remove this key from the view variables

A typical use-case is '''rheader''':

{{{#!python
def my_rheader(r):
  if r.interactive and r.component:
    # Code producing the rheader...
    return rheader
  else:
    return None

output = s3_rest_controller(prefix, name, rheader=my_rheader)
}}}

If {{{my_rheader(r)}}} gives something else than {{{#!python None}}}, then this value is added as {{{rheader}}} to the view variables.

== Advanced Options ==

=== Filtering Lists ===

You can filter lists in the controller by setting {{{response.s3.filter}}} to a filter query:
{{{#!python
    # This filters for females:
    response.s3.filter = (db.pr_person.gender == 2)

    return s3_rest_controller("pr", "person")
}}}

Note that this only takes effect in the main controller (not in prep or postp).

Note that {{{response.s3.filter}}} affects both, the primary resource and components!

In {{{prep}}}, you can also add filter queries using the {{{add_filter}}} method:
{{{#!python
    def prep(r):
        resource = r.resource
        query = (db.pr_address.type == 1) # Home addresses only
        resource.add_filter(query)
        return True
    response.s3.prep = prep

    return s3_rest_controller("pr", "person")
}}}

However, {{{add_filter}}} again affects both, primary and component records - so this example would:
- only retrieve {{{person}}} records which have a type 1 {{{address}}} record
- only retrieve the {{{address}}} records with type 1.

This can be an unwanted side-effect.

To have the primary resource unfiltered, and filter only records in a particular component, you can use {{{add_component_filter}}}:
{{{#!python
    def prep(r):
        resource = r.resource
        query = (db.pr_address.type == 1) # Home addresses only
        resource.add_component_filter("address", query)
        return True
    response.s3.prep = prep

    return s3_rest_controller("pr", "person")
}}}

In this case, all {{{person}}} records would be selected - while only {{{address}}} records of type 1 would be retrieved.
=== Pre-populating Create-Forms ===

Create-forms can be pre-populated with data by one of these 3 methods:

  1. model defaults (standard)
  2. values from another record in the database
  3. values provided by the controller

'''Model defaults''' are defined per field and can be set as {{{db.my_table.field.default = value}}} at any time before the REST method is applied (even in the pre-process).

'''Values from another record''' in the database can be used via a URL query like:

  - /my/resource/create?from_record=''id''&from_fields=''fieldname1'',''fieldname2'',...

The ''id'' of the original record can also be specified as ''tablename.id'', if the original record is in another table. Additionally, fieldnames can be specified as ''fieldname$original_fieldname'' to map between different fieldnames.

To '''pre-populate''' Create-forms '''from the controller''', you can specify the variable {{{populate}}} in the arguments of {{{s3_rest_controller}}}:

{{{#!python
output = s3_rest_controller(prefix, resourcename,
                            populate=dict(fieldname1=value1,
                                          fieldname2=value2))
return output
}}}

Instead of a {{{dict}}}, you can also pass a callable object as {{{populate}}}. This will be executed with the current {{{S3Request}}} and the named arguments of {{{s3_rest_controller}}} in order to produce the field/value {{{dict}}}:

{{{#!python
def populate(r, **attr):
    """
    Helper function to pre-populate create-forms

    """

    # some code to produce the data
    # ...

    # return the dict
    return dict(fieldname1=value1,
                fieldname2=value2)

output = s3_rest_controller(prefix, resourcename,
                            populate=populate)

return output
}}}

Note that {{{populate}}} will only be applied in {{{GET}}} requests and only if no record_id is specified. That means, if it uses a separate form to generate the data, you need to revert the request into {{{GET}}} in order to have the create-form pre-populated:

{{{#!python
data = None
form = FORM(...some form...)
output = dict(helper_form=form)
if form.accepts(request.vars, session, formname="helper_form"):
    output = dict() # remove the helper form
    data = Storage(...) # some code to extract the data from the helper form
if data:
    request.env.request_method = "GET" # revert to GET if data available

_output = s3_rest_controller(prefix, resourcename, populate=data)

if isinstance(_output, dict):
    output.update(_output)
else:
    output = _output

return output
}}}

NB: This construction could be used e.g. to loop in an OCR client into a REST controller. The helper form would then be a file upload form which is displayed alongside with the normal create-form - so the user can either enter data manually and submit the create-form, or first submit a file to pre-populate the create form, and then edit the data and submit the create-form.
=== Pre-Process ===


A prep hook would allow you to change a handler configuration in certain situations, e.g. testing a URL variable:

{{{#!python
def myresource():

   """ RESTful CRUD controller """

   # Define pre-processor as local function:
   def prep(r):
       mylist = request.vars.get("mylist")
       if mylist:
           r.set_handler("list", my_list_controller)
       return True # do not forget to return True!

   # Hook pre-processor into REST controller:
   s3db.prep = prep

   output = s3_rest_controller(modulename, resourcename)
   return output
}}}

This example would switch to my_list_controller instead of s3_list in case there is a ?mylist= in the URL. In all other cases, the default handlers are executed as usual, you still have a RESTful API for your resources.

While you can define prep's and postp's as local functions (as in the example above) or even lambdas, it is also very well possible to create more generic, reusable prep and postp functions (e.g. to implement different method handler configurations, or to catch certain situations to bypass the CRUD, or to manipulate the output dict in a certain way).

A very important structure during prep and postp is the S3Request object (usually instantiated as "r"). This object contains all necessary
information to process the current REST request, and it is passed to both prep and postp. See [wiki:S3XRC#RESTfulAPI S3RESTful API] for a list of attributes and methods.

==== Passing information between main controller & pre-processor ====

Scope normally means that these 2 sections can only talk to each other via globals or the Request object.

If you need to pass data between them, you can use this trick:
{{{#!python
vars = {} # the surrounding dict
def prep(r, vars):
    vars.update(x=y) # the actual variable to pass is x
    return True

response.s3.prep = lambda r, vars=vars: prep(r, vars)

output = s3_rest_controller(module, resource)

x = vars.get(x, None)
}}}

An example usage is in {{{controllers/gis.py}}} for location()
=== Post-Process ===

=== Method Handlers ===

The s3_rest_controller has multiple hooks,Look at how s3_rest_controller is actually defined in models/00_utils.py:

{{{#!python
def s3_rest_controller(prefix=None, resourcename=None, **attr):

    set_handler = r.set_handler
    set_handler("barchart", s3_barchart)
    set_handler("compose", s3base.S3Compose)
    set_handler("copy", lambda r, **attr: \
                               redirect(URL(args="create",
                                            vars={"from_record":r.id})))
    set_handler("deduplicate", s3base.S3Merge)
    set_handler("filter", s3base.S3Filter)
    set_handler("import", s3base.S3Importer)
    set_handler("map", s3base.S3Map)
    set_handler("profile", s3base.S3Profile)
    set_handler("report", s3base.S3Report)
    set_handler("import", S3PDF(),
                    http = ["GET", "POST"],
                    representation="pdf")


    # Execute the request and apply crud functionality or execute other method handler
    output = r(**attr) 
    return output
}}}

This is nothing else than a wrapper for the global r object (an instance of S3Request).
You don't have to use s3_rest_controller to have a RESTful API, you can make your own handler configuration and call r directly.

But even when using s3_rest_controller, you can take control over what actually happens. A very comfortable (and recommended) way to get control over s3rest when using s3_rest_controller is to hook in prep and postp functions. Look above to find out when prep and postp hooks are invoked.


look [wiki:S3/S3Method] to see how method handler can be created using S3Method

=== Custom Methods ===

If you have a resource "warehouse" and want to implement a function "report" that generates a report about the current status of the warehouse.

Now you're gonna provide this report in a RESTful way, i.e. you want to provide it as a resource that can be addressed via URLs like:

  - '''!http://site.myserver.org/eden/wm/warehouse/report'''

and is provided in several different data formats beyond HTML (the interactive view), let's say - XLS and XML:

  - '''!http://site.myserver.org/eden/wm/warehouse/report.xls'''
  - '''!http://site.myserver.org/eden/wm/warehouse/report.xml'''

you can use s3_rest_controller for this!

This could be your "warehouse" CRUD controller:

{{{#!python
def warehouse():

   """ RESTful CRUD controller """

   return s3_rest_controller(module, resource, ...)
}}}

At first you implement your report function (in the controller file). This function takes the argument r (=S3Request) and a dict of named arguments (just the same named arguments from the s3_rest_controller call above). This function returns the report. Then, in your controller, you plug in this function to your resource - together it would look like that:

{{{#!python
def warehouse():

    """ RESTful CRUD+Reports controller """

    # Plug warehouse_report into warehouse resource:
    s3db.set_method(module, resource, method="report", action=warehouse_report)

    return s3_rest_controller(module, resource, ...)

def warehouse_report(r, **attr):

    """ Warehouse report generator """

    # Code to produce the report goes here
    report = ...

    return report
}}}

Note that if "report" is a dict, then the REST controller automatically adds the S3Request as "r" to that dict before returning it. Thus, "r" is 
available to the view templates. That also means: do not use "r" as key in that dict.

However, there may be other formats than HTML - to implement other formats, you might need to check for the representation of the request. This can be 
found in r:

{{{#!python
def warehouse_report(r, **attr):

    """ Warehouse report generator """

    if r.representation in ("html", "popup"):
        # Code to produce the report items:
        title = T("Warehouse Report")

        # Assemble the report items in a dict:
        report = dict(title=title, ...)

    elif r.representation == "xls":
        # Code to produce the XLS report goes here
        ...

    elif r.representation in s3_xml_export_formats:
        # Code to produce the XML report goes here
        ...

    else:
        r.error(501, r.ERROR.BAD_FORMAT)

    return report
}}}

See [wiki:S3/S3Request] to find out more about "r".

To produce the XML report, it is probably sufficient to just export the requested warehouse information in S3-XML, and then use XSLT stylesheets to produce the finally desired XML formats. That's pretty easy:

{{{#!python
def warehouse_report(r, **attr):
    ...
    elif r.representation in s3_xml_export_formats:
        report = export_xml(xrequest)
    ...
}}}

Perhaps you want to add an RSS feed:

{{{#!python
def warehouse_report(r, **attr):
    ...
    elif r.representation == "rss":
        # Code to produce the RSS report goes here
        report = ...
    ...
}}}

Getting the data from the warehouse_report.

Your implementation already supports a variety of URLs:

This URL addresses the report for all warehouses:
  - '''!http://site.myserver.org/eden/wm/warehouse/report'''

This URL addresses the report for the warehouse record with ID=1.
  - '''!http://site.myserver.org/eden/wm/warehouse/1/report'''

This URL addresses the report in XLS format for the warehouse record with the UUID=123654278 (assuming that you have UUID's in your warehouse table).
  - '''!http://site.myserver.org/eden/wm/warehouse/report.xls?warehouse.uid=123654278'''

The S3Request provides the resource information to your warehouse_report function.

In case a specific record has been requested, you can access it as:
{{{#!python
    record = r.record
}}}
If r.record is None, then the request is targeting all warehouse records, so you'd take:
{{{#!python
    table = r.table
    records = db().select(table.ALL)
    for record in records:
       ...
}}}
instead.

NOTE: representation in r is always all lowercase, there is no differentiation between the ".XML" and ".xml" extension in the URL.

And...not to forget: your warehouse_report is still a controller function, that means you can implement forms as usual (e.g. operated with form.accepts or web2py Crud).

=== Re-Routing Controllers ===
There may be a time when you want to bypass the default controllers, or else add a custom controller which can have different options.

This allows re-routing of any controller/function combination to the custom/rest controller - which is a vanilla s3_rest_controller, i.e. without any prep, postp or attr. The latter can then be added in customise_controller.

To use this, a template will configure e.g.:

{{{
settings.base.rest_controllers = {("org", "organisation"): ("org", "organisation")}
}}}
...which re-routes org/organisation to custom/rest, and runs it as a s3_rest_controller with prefix="org" and name="organisation". This re-routing is completely transparent, i.e. the s3_rest_controller (and anything called from there) sees a normal org/organisation request. Consequently, to customise this call, one would use customise_org_organisation_controller (not customise_custom_rest_controller!), as usual.

As this allows re-routing of any controller/function combination, one can also "add" controllers that do not exist, e.g.:

{{{
settings.base.rest_controllers = {("org", "suppliers"): ("org", "organisation")}
}}}
This would though still be customised via customise_org_organisation_controller (=>can inspect r.function in prep to distinguish), but page access authorisation would now be through org/suppliers rather than org/organisation (so can have different permissions for those).

When adding a route with a custom module prefix:

{{{
settings.base.rest_controllers = {("my", "organisation"): ("org", "organisation")}
}}}
...then a corresponding module entry for that must be added to settings.modules - though this is useful anyway if menus are to be added automatically etc. Again, page authorisation would go through my/organisation here instead of org/organisation, so there could be different permission sets.

Note: this feature is orthogonal to default/index/*, which is still useful for implementing non-REST custom controllers, and should therefore be retained.


----

DeveloperGuidelines