Version 8 (modified by 13 years ago) ( diff ) | ,
---|
Table of Contents
Pre-Populate
Introduction
We need to be able to import data into a Sahana Eden instance for testing, demos and to help with getting up and running.
Types of Data
- Item Catalogue
- Organisation List
- Assets
- Warehouse Supplies
- Human Resources (Staff & Volunteers)
- Locations (Administrative Boundaries)
Methods
On First Run
In models/000_config.py their is a deployment setting which is used to control prepopulate. This is a list of directory names and will import data from a number of CSV files. The directories must be in the private/prepopulate directory tree and must include a file called tasks.cfg.
The prepopulate deployment setting can also be a number which will relate to a single directory, the numerical value is still supported because existing instances of Eden and their maintenance scripts use this number. The use of a number is discouraged in favour of the list of names because this new approach is clearer and more flexible.
Deployment setting examples
Use the Shipment folder as the starting point for the prepopulate.
deployment_settings.base.prepopulate = ["Standard"]
The Standard folder is not in the root directory, which is private/prepopulate, rather it is in demo (private/prepopulate/demo) but this will be found automatically.
Uses the roles to import some special roles and then the user directory which is a special directory which is external to the version control and so private data (potentially sensitive data - such as volunteer details) can be held and then imported.
deployment_settings.base.prepopulate = ["roles", "user"]
The tasks.cfg file comprises a list of import jobs. They can be of two distinct types:
- Basic Importer Jobs
- Special Import Jobs
The tasks.cfg file supports the # character on the first column as a comment.
Basic Importer Jobs
These will use the UI Importer to load in the data and requires the following information:
- The controller
- The function
- The CSV
- The XSL transform file
For example:
"survey","question_list","questionnaire24H.csv","question_list.xsl"
Which will use the questionnaire24H.csv file, which in this case will be in the same directory as the tasks.cfg file, to import the data. Using the question_list.xsl transform file which will be the static/formats/s3csv/survey directory. The importer will then use the survey/question_list function to "manage" the import using any onvalidate or onaccept callbacks that are set up by this controller.
The CSV file does not need to be in the same directory as the tasks.cfg file, if it is held in a different file then the full path needs to be given relative to the private/prepopulate directory.
All transform files are stored in the static/formats/s3csv directory tree.
Special Import Jobs
For some import jobs it is easier to create a purpose built import function rather than use the Importer. These are created by using a asterisk at the start of the line in the tasks.cfg file, as follows:
*,gis_set_default_location,defaultlocation.csv
This uses the csv file default_location.csv and the function gis_set_default_location to set, in this case, the default location. Another special function is import_role which is used to import user permissions (or roles) examples of the roles and what the csv looks like can be found in the private/prepopulate/roles directory.
URL Import
Data can be imported using URL calls:
http://127.0.0.1:8000/eden/supply/item/create.s3csv ?filename=/home/web2py/applications/eden/static/formats/s3csv/eric.csv &transform=/home/web2py/applications/eden/static/formats/s3csv/eric.xsl
However this method isn't easily automated
Also see: UserGuidelinesImporter
Python Script
This offers us better options to automate the import of data.
# Function to handle conflicts s3mgr.configure("supply_item", deduplicate=response.s3.item_deduplicate) resource = s3mgr.define_resource("supply", "item") resource.import_xml(open("/home/web2py/applications/eden/static/formats/s3csv/eric.csv", "r"), format="csv", stylesheet="/home/web2py/applications/eden/static/formats/s3csv/eric.xls")
Should we have a wrapper function for this which is easily accessed from the CLI?
Rational:
- People could create their own "Deployment" pre-populate directories with unique (& contextual) data which could be used to easily create simulations. ( And be a great GCI task )
- We should separate csv from xsl (xsl could very easily confuse semi-technical users)
- CSV templates should be easily downloaded - prepopulate data should not (always) be
- All XLS should be in trunk (even if it is for private data) and have a matching CSV Template
- Then a configuration file can be created where you can just list the CSV file names which you want to be imported when Sahana Eden is initialized. Copies of this files could be saved in the private/prepopulate/<deployment>/ directory then a config setting could select which file to use, to make it easy to set up Sahana Eden to have different types of data in it.