Version 54 (modified by 11 years ago) ( diff ) | ,
---|
Importer
Table of Contents
Spreadsheets
Data is normally imported from spreadsheets.
Currently these need to be converted to CSV format, however there is now support for building .xls[x] templates to wrap around the CSV:
Process:
- File should be encoded in 'UTF-8' (with or without BOM)
- Data needs putting into the correct columns
- Fields containing commas should be wrapped with "" quotes
- There should be no line-breaks in text fields (see cleanCSV.py for help with this)
- Date formats need to be correct: ISO8601
- If the file is very large, then it can be better to split into multiple files to avoid memory exhaustion (see splitCSV.py for help with this)
This can be done to PrePopulate an instance.
After 1st run, there are menu links for many resources, but any REST controller can have the 'import' method invoked if there is a suitable .xsl file as static/formats/s3csv/module/resource.xsl
:
http://host.domain/module/resource/import
Files & Stylesheets can be loaded from alternate sources, such as:
http://host.domain/eden/hrm/person.s3csv/create?fetchurl=http://my.host/HR.csv&transform=/home/web2py/applications/eden/static/formats/s3csv/hrm_import.xsl
Alternately these can be done using CLI scripts:
OpenStreetMap
XSLT available to import Hospitals & Locations into Sahana Eden
- GIS/OpenStreetMap
- http://127.0.0.1:8000/eden/hms/hospital/create.osm?filename=c:\archives\gis-gps\data\americas\haiti\haiti_hospital.osm
ToDo: Provide a UI to select which levels the admin boundary levels correspond to:
Ushahidi
XSLT available to import Ushahidi reports into Sahana Eden's Incident Reporting system
- Access only available to Administrators: http://127.0.0.1:8000/eden/irs/ireport/ushahidi
- URL to enter is http://instancename.crowdmap.com/api/%3Ftask=incidents%26by=all%26resp=xml
See Also
- BluePrint/Importer
- DeveloperGuidelines/PrePopulate
- DataImportCLI
- On-the-fly Transformation using external Stylesheets
- Old User Guidelines
- Can try to use this to clean up data: http://openrefine.org
Attachments (2)
-
cleanCSV.py
(1.1 KB
) - added by 11 years ago.
Remvoe line-breaks within text fields from a CSV file
-
splitCSV.py
(1.2 KB
) - added by 11 years ago.
Spli a large CSV file into multiple files to avoid memory exhaustion when importing
Download all attachments as: .zip