= User Guidelines for GIS Data = [[TOC]] Assumes installation of the relevant [wiki:GIS#Tools tools] == Import Data == === UUIDs === See: [wiki:UUID#Mapping] === CSV === Locations can be imported using the normal import procedure: * [BluePrint/PrePopulate Prepopulate] tasks.cfg: * {{{gis,location,location.csv,location.xsl}}} * After install: * http://127.0.0.1:8000/eden/gis/location/import The CSV needs to have specific columns: * WKT column if we have polygon info (or Lat and Lon for Points, if not) * For L1, we need these columns: Country, ADM1_NAME (& WKT) * For L2, we need these columns: ADM1_NAME, ADM2_NAME (& WKT) [Country can also be used to help separate duplicates] * For L3, we need these columns: ADM2_NAME, ADM3_NAME (& WKT) [ADM1_NAME and Country can also be used to help separate duplicates] * For L4, we need these columns: ADM3_NAME, ADM4_NAME (& WKT) [ADM2_NAME, ADM1_NAME and Country can also be used to help separate duplicates] * For specific lcoations, we need these columns: ADMx_NAME (for appropiate parent level of hierarchy) Name (& Lat/Lon) [ADM2_NAME, ADM1_NAME and Country can also be used to help separate duplicates] * Code columns are used, if-present * ADM1_CODE, ADM1_CODE2, ADM2_CODE, ADM2_CODE2, ADM3_CODE, ADM3_CODE2, ADM4_CODE, ADM4_CODE2, Code, Code2 * Population & Elevation columns are read, if-present * Ensure that names are consistent between Levels * The PROPER() spreadsheet function is useful to get the names in the correct format (then Paste as Text). * The VLOOKUP() spreadsheet function is useful if the different levels of the hierarchy are in different sheets & linked via a code instead of the name (as we need): * If the lookup sheet is called 'lookup' and the lookup table is in B2-C87 then lookup code in C2 to column 2 of the lookup table: =VLOOKUP(C2; lookup.B$2:C$87; 2; 0) * To remove duplicate rows, can create a new column with {{{=IF(A1=A2;1;0)}}}, Paste Special, Sort & Delete 1s * http://wiki.services.openoffice.org/wiki/Documentation/How_Tos/Removing_Duplicate_Rows Basic Hierarchy can often be found from Wikipedia (although currently there's no easy way to download this - a student project to enhance Wikipedia for this would be much appreciated!). For the Polygon data, it is normal to get this from Shapefiles, such as GADM or UN CODS. === Shapefiles === Inspect the data using [http://qgis.org qGIS]. Use ogr2ogr to convert the data to CSV: {{{ ogr2ogr -f CSV CSV TM_WORLD_BORDERS-0.3.shp -lco GEOMETRY=AS_WKT ogr2ogr -f geojson TM_WORLD_BORDERS-0.3.json TM_WORLD_BORDERS-0.3.shp }}} If needing to reproject (e.g. for the [https://www.geoint-online.net/community/haitiearthquake/Geospatial%20Data%20Files/haiti_departments01132010.zip Haiti Departements]): {{{ ogr2ogr -f CSV haiti_departments Haiti_departementes_edited_01132010.shp -s_srs EPSG:32618 -t_srs EPSG:4326 -lco GEOMETRY=AS_WKT }}} NB AS_WKT requires OGR v1.6+ === KML === Can convert a KML to CSV using the attached script: {{{python KML2WKT.py .kml}}} * requires [http://pypi.python.org/pypi/keytree/ keytree] This can then be imported into Sahana by editing the column headers & using the Importer qGIS can be used to convert this into a Shapefile (uses ogr2ogr so can also do using the CLI, if you prefer): give it column headers with 'WKT' for the WKT column name. * This is the easiest way to load into [wiki:InstallationGuidelinesGISDataLinux#PostGIS PostGIS] (using PGAdmin III's Shapefile Importer plugin) to allow !GeoServer to serve as WMS === Geonames === There is an import_geonames() function in S3GIS which downloads/unzips the country file (a TAB-separated list) from http://download.geonames.org/export/dump/ It should be run for the different levels of hierarchy that you wish to import (generally just the lowest level as Geonames just has Point data, so it's best to use other sources for the Polygons 1st, that way the Geonames importer can locate these Points within the correct Polygons of the hierarchy) NB It takes some time to do this import! Pakistan imports 95000 locations! Update: Geonames schema 2.2 supports parentADM(1-4): http://geonames.wordpress.com/2010/09/29/geonames-ontology-2-2/ * will be good for when we only have hierarchy, not polygons * need to check whether much data has this populated though. Python 2.5 doesn't support Zipfile.extract() & Zipfile.read() isn't unicode-safe. Until this is fixed, download the file manually 1st: {{{ cd ~web2py/applications/eden/cache wget http://download.geonames.org/export/dump/PK.zip unzip PK.zip }}} In Web2py CLI: {{{ gis.import_geonames('PK', 'L5') db.commit() }}} Alternate approach: 1. Transform each line in this file into XML by regular expression: {{{ ^(\d*)\t([^\t]*)\t([^\t]*)\t([^\t]*)\t([0-9\.]*)\t([0-9\.]*)\t[^\t]*\t([A-Z]*).* into: $1 $2 $3 $4 $5 $6 $7 }}} This can be done using an RE-capable editor (e.g. Kate), Perl or even Python. Note: Need to replace & with & and to remove any invalid characters 2. Transform into S3XRC-XML using XSLT, stylesheet is available at * [http://pub.nursix.org/eden/geonames/geonames.xsl] === !OpenStreetMap === See below: [wiki:UserGuidelinesGISData#Import] === WFS === It is possible to use the WFS Plugin to get data into qGIS & thence export into other formats. May need to use a Custom CRS (in Settings menu - remember to Save!) such as: * ESRI's Spherical Mercator ([http://www.cadmaps.com/gisblog/?p=81 different] to 900913) http://spatialreference.org/ref/esri/54004/proj4/ Can then go to the Layer Properties & Specify CRS to this User Defined Coordinate System. Can then Save As and change the CRS to something like the standard WGS84. === Yahoo === * http://developer.yahoo.com/geo/geoplanet/data/ * Script to process http://pastie.org/1139680 == Display Data == === !GeoServer === !GeoServer can provide geospatial data in Raster (WMS) or Vector (WFS/KML) formats. Once you have installed in [wiki:InstallationGuidelinesGISDataLinux#GeoServer Linux] or [wiki:InstallationGuidelinesGISDataWindows#GeoServer Windows], then login: * l: admin * p: geoserver [wiki:InstallationGuidelinesGISDataLinux#PostGIS PostGIS] is recommended as the main data store. Configure: * [http://geo.eden.sahanafoundation.org/geoserver/web/?wicket:bookmarkablePage=:org.geoserver.web.security.user.UserPage Admin Password] * [http://geo.eden.sahanafoundation.org/geoserver/web/?wicket:bookmarkablePage=:org.geoserver.web.admin.ContactPage Contact Details] * [http://geo.eden.sahanafoundation.org/geoserver/web/?wicket:bookmarkablePage=:org.geoserver.web.admin.GlobalSettingsPage Disable Global Services] (2.1+ only) * [http://geo.eden.sahanafoundation.org/geoserver/web/?wicket:bookmarkablePage=:org.geoserver.wfs.web.WFSAdminPage WFS Details] * [http://geo.eden.sahanafoundation.org/geoserver/web/?wicket:bookmarkablePage=:org.geoserver.wms.web.WMSAdminPage WMS Details] inc the Limited SRS List - probably to just: {{{4326, 900913}}} * Disable the demo [http://geo.eden.sahanafoundation.org/geoserver/web/?wicket:bookmarkablePage=:org.geoserver.web.data.layer.LayerPage Layers] & [http://geo.eden.sahanafoundation.org/geoserver/web/?wicket:bookmarkablePage=:org.geoserver.web.data.layergroup.LayerGroupPage Layer Groups] * Add [http://geo.eden.sahanafoundation.org/geoserver/web/?wicket:bookmarkablePage=:org.geoserver.web.data.workspace.WorkspacePage Workspace] * Add [http://geo.eden.sahanafoundation.org/geoserver/web/?wicket:bookmarkablePage=:org.geoserver.web.data.store.StorePage PostGIS Store] * http://docs.geoserver.org/stable/en/user/gettingstarted/web-admin-quickstart/index.html ==== Import Shapefiles ==== e.g. Country Outlines: * http://thematicmapping.org/downloads/TM_WORLD_BORDERS-0.3.zip These can be loaded direct into !GeoServer, however there will be better performance by importing into PostGIS: [[BR]](can also use pgAdmin III GUI's Shapefile loader on plugins menu) {{{ su postgres shp2pgsql -s 4326 -I TM_WORLD_BORDERS-0.3.shp public.countries | psql -d gis }}} To reproject the data into 900913 for a slight performance advantage: {{{ drop constraint srid; update table set geomcolumn=transform(geomcolumn,900913); }}} ==== Configure !GeoServer ==== * Add [http://geo.eden.sahanafoundation.org/geoserver/web/?wicket:bookmarkablePage=:org.geoserver.web.data.layer.LayerPage Layer] * Add [http://geo.eden.sahanafoundation.org/geoserver/web/?wicket:bookmarkablePage=:org.geoserver.wms.web.data.StylePage Style] for WMS presentation: * http://docs.geoserver.org/stable/en/user/styling/sld-introduction.html * Publish * Apply Style Colours: * http://www.colorsontheweb.com/colorwizard.asp * http://colorschemedesigner.com/ Zoom Levels (in Spherical Mercator): ||= Zoom Level(s) =||= !MinScale =||= !MaxScale =|| || 1 || 250000000 || n/a || || 2 || 100000000 || 250000000 || || 3 || 50000000 || 100000000 || || 4 || 25000000 || 50000000 || || 5 || 10000000 || 25000000 || || 6 || 5000000 || 10000000 || || 7 || 2500000 || 5000000 || || 8 || 2000000 || 2500000 || || 9 || 1000000 || 2000000 || || 10 || 500000 || 1000000 || || 11 || 250000 || 500000 || || 12 || 100000 || 250000 || || 13 || 50000 || 100000 || || 14-22 || n/a || 50000 || ==== Configure !GeoWebCache ==== The raw WMS server will be slow, so once you've chosen your style, then you should serve via GWC. This caches pre-rendered tiles & also does MetaTiling so that the WMS has less separate requests (at a cost of increased RAM requirements) The version embedded within !GeoServer is great for providing a zeroconfig of the common options, however there are cases where you need to define a layer manually: * Want a Background Colour (bgcolor) * Want to specify an alternate style (& you'd rather not republish the layer on the WMS) * Want to render a set of layers into a single tileset (so that clients don't need to download them separately & merge locally) Example {{{geowebcache.xml}}} in the GADM section: * [wiki:UserGuidelinesGISData#GADM] ==== WFS ==== If you are displaying a complex dataset at zoomed-out resolutions, then you will want to have simplified views. e.g. Hospitals aren't shown at all at low zooms, are shown as Points at medium zooms & shown as Polygons at high zooms. Scale-dependent styling using SLD in !GeoServer: * http://ian01.geog.psu.edu/geoserver_docs/data/naturalearth/naturalearth_physical.html Scale-dependent styling in !OpenLayers: * http://docs.openlayers.org/library/feature_styling.html#custom-rules Simplifying Polygons in PostGIS: * http://linfiniti.com/2011/06/scale-dependent-generalization-in-postgis-and-qgis/ Showing the different layers at different zooms using !GeoServer: * http://docs.geoserver.org/stable/en/user/tutorials/feature-pregeneralized/feature-pregeneralized_tutorial.html {{{ cd wget http://kent.dl.sourceforge.net/project/geoserver/GeoServer%20Extensions/2.1.0/geoserver-2.1.0-feature-pregeneralized-plugin.zip cd /var/lib/tomcat6/webapps/geoserver/WEB-INF/lib/ unzip ~/geoserver-2.1.0-feature-pregeneralized-plugin.zip /etc/init.d/tomcat6 restart }}} ==== Add WMS Layer to Sahana Eden ==== * ''tbc'' === WMS Reprojection === * Have a remote WMS source that you want to access? * Have a desire to keep !OpenStreetMap/Google/Bing layers? * WMS source server doesn't support the 900913 projection? e.g. TRMM Rainfall Monitoring Solution: [wiki:InstallationGuidelinesGISDataLinux#ConfigurationforTRMM MapProxy] === Grid === We have a 'Coordinate Grid' Layer available by default. Other options: * [wiki:BluePrintGISGrid] === Administrative Areas === ==== GADM ==== GADM is the best source of global Administrative Boundaries: * http://gadm.org There are often better local sources for specific countries, although getting hold of these can be difficult. Note that some countries have boundaries which change frequently and so datasets can often be a little out of date. To import into Sahana Eden's gis_location table (for consistency of naming/boundaries across basemap & dynamic data): * Install latest [wiki:InstallationGuidelinesGISDataLinux#GDAL Python GDAL bindings] * Open a web2py CLI: {{{ python web2py.py -S eden -M }}} * Optionally, define a filter for which countries you wish to import data, e.g. for Asia-Pacific (without TL, as that will be imported from UN CODS): {{{ countries = [ "AF", "AU", "BD", "BN", "CK", "CN", "FJ", "FM", "HK", "ID", "IN", "JP", "KH", "KI", "KP", "KR", "LA", "MH", "MM", "MN", "MV", "MY", "NP", "NZ", "PG", "PH", "PK", "PW", "SB", "SG", "SL", "TH", "TO", "TV", "TW", "VN", "VU", "WS"] }}} * Import: {{{ gis.import_admin_areas(countries=countries) }}} This can then be served as separate WMS layers using !GeoServer & !GeoWebCache.[[BR]] You can use GeoServer's [http://docs.geoserver.org/2.1.3/user/data/sqlview.html SQLView] feature.[[BR]] SLD files are attached: * [http://eden.sahanafoundation.org/attachment/wiki/UserGuidelinesGISData/gadm_v1_lev0_base_sld.xml L0 Base] * SQL View: SELECT id, name, area, the_geom FROM gis_location WHERE level='L0' * [http://eden.sahanafoundation.org/attachment/wiki/UserGuidelinesGISData/gadm_v1_lev0_overlay_sld.xml L0 Overlay] * SQL View: SELECT id, name, area, the_geom FROM gis_location WHERE level='L0' * [http://eden.sahanafoundation.org/attachment/wiki/UserGuidelinesGISData/gadm_v1_lev1_overlay_sld.xml L1 Overlay] * SQL View: SELECT id, name, area, the_geom FROM gis_location WHERE level='L1' * [http://eden.sahanafoundation.org/attachment/wiki/UserGuidelinesGISData/gadm_v1_lev2_overlay_sld.xml L2 Overlay] * SQL View: SELECT id, name, area, the_geom FROM gis_location WHERE level='L2' Example [http://eden.sahanafoundation.org/attachment/wiki/UserGuidelinesGISData/geowebcache.xml geowebcache.xml] for GADM attached which provides 3 layers: * L0 Base * L0-L2 Base (merged) * L0-L2 Overlay (merged) {{{ cp geowebcache.xml /var/gis/geoserver_data/gwc /etc/init.d/tomcat6 restart }}} === Population Density === ==== GPWv3 ==== Gridded Population of the World, version 3 (GPWv3) is the standard global dataset for both measured & projected population densities. Download gl_gpwfe_pdens_10_wrk_25.zip from (requires registration): * http://sedac.ciesin.columbia.edu/gpw/global.jsp?file=gpwv3&data=pdens&type=wrk&resolut=25&year=10&version=gpw-v3 {{{ mkdir /home/data/GPWv3 cd /home/data/GPWv3 unzip gl_gpwfe_pdens_10_wrk_25.zip /usr/local/bin/gdal_translate -of GTiff glfedens10/glds10ag/hdr.adf glds10ag.tif ln -s /home/data/GPWv3 /var/lib/tomcat6/webapps/geoserver/data/coverages/GPWv3 }}} Add new GeoTIFF Store to !GeoServer to serve as WMS: {{{ URL: file:coverages/GPWv3/glds10ag.tif }}} Style: * [http://eden.sahanafoundation.org/attachment/wiki/UserGuidelinesGISData/populationDensity.sld populationDensity.sld] ==== GRUMPv1 ==== Global Rural-Urban Mapping Project, version 1 (GRUMPv1) is a newer dataset which combines satellite with census data to locate people within settlements rather than just administrative areas: * http://sedac.ciesin.columbia.edu/gpw/aboutus.jsp === Topographic Maps === ==== WMS ==== Topography can be rendered as WMS using e.g. !GeoServer Download the SRTMs in GeoTIFF format from ftp://xftp.jrc.it/pub/srtmV4/tiff * [http://eden.sahanafoundation.org/attachment/wiki/UserGuidelinesGISData/get.sh get.sh] can assist with this * Beware this is a *lot* of data! (At least 60Gb for a global dataset) * The GeoTIFFs are already well [http://linfiniti.com/2011/05/gdal-efficiency-of-various-compression-algorithms/ compressed] * Partial datasets are, of course, possible (@!ToDo: A script to select an area & download just that area) {{{ mkdir /tmp/SRTMv4 cd /tmp/SRTMv4 sh get.sh }}} Unzip the data into a folder called 'SRTMv4' in the !GeoServer 'coverages' folder (or use a symlink): {{{ cd /var/lib/tomcat6/webapps/geoserver/data/coverages mkdir SRTMv4 unzip -o /tmp/SRTMv4/\*.zip rm /var/lib/tomcat6/webapps/geoserver/data/coverages/SRTMv4/*.hdr rm /var/lib/tomcat6/webapps/geoserver/data/coverages/SRTMv4/*.tfw rm /var/lib/tomcat6/webapps/geoserver/data/coverages/SRTMv4/readme.txt }}} Give Tomcat permission to the folder: {{{ chown tomcat6 /var/lib/tomcat6/webapps/geoserver/data/coverages/SRTMv4 }}} Configure !GeoServer by adding a new Store using the [http://docs.geoserver.org/stable/en/user/tutorials/image_mosaic_plugin/imagemosaic.html Image mosaicking plugin] * URL: file:coverages/SRTMv4 Publish (defaults OK) Styling: * [http://eden.sahanafoundation.org/attachment/wiki/UserGuidelinesGISData/srtm.sld srtm.sld] 'official' * [http://eden.sahanafoundation.org/attachment/wiki/UserGuidelinesGISData/topography.sld topography.sld] 'common usage' Test out using the direct WMS URL: * http://host.domain/geoserver/service/wms Once happy then start using the !GeoCache URL (but don't do this too early as otherwise you have to invalidate the cache to see your changes): * http://host.domain/geoserver/gwc/service/wms ==== !OpenStreetMap ==== Contours can be rendered using OSM tools: * [wiki:InstallationGuidelinesGISDataLinux#Contours] * http://wiki.openstreetmap.org/wiki/Contours * Uses [http://wiki.openstreetmap.org/wiki/SRTM SRTM] ([http://wiki.openstreetmap.org/wiki/ASTER ASTER] is a future possibility) * Alternate approach: [http://wiki.openstreetmap.org/wiki/Srtm2Osm SRTM 2 OSM] http://de.wikipedia.org/wiki/Benutzer:Alexrk2/SRTM-Reliefs - suggests using GIMP's emboss filter! Azimuth = 135, height = 50, depth = 10 === Old Printed Maps === Old Printed Maps can be 'Rectified' to be overlaid on the base maps: * [wiki:BluePrintGISRectifier] == !OpenStreetMap == === Base Map === We have out-of-the-box the ability to use !OpenStreetMap Tiles as base layer. This can include local OSM sites (OSM Taiwan is included as an example) === Vector Overlays === Can have OSM Vectors displayed over the top of other Base Layers (e.g. Satellite Images) * [wiki:BluePrintGISOSMLayers] === Import === We have an XSLT [http://eden.sahanafoundation.org/browser/static/formats/osm/import.xsl stylesheet] to import .osm files e.g for hospitals and clinics: {{{ osmosis --read-xml country.osm --tf accept-nodes amenity=hospital,clinic --tf reject-ways --tf reject-relations --write-xml nodes.osm osmosis --read-xml country.osm --tf reject-relations --tf accept-ways amenity=hospital,clinic --used-node --write-xml ways.osm osmosis --rx nodes.osm --rx ways.osm --merge --wx country_hospitals.osm http://myhost.com/eden/hms/hospital/create.osm?filename=country_hospitals.osm }}} This needs more work to understand the admin hierarchy properly to be able to import Places. * http://wiki.openstreetmap.org/wiki/Map_Features#Places * http://wiki.openstreetmap.org/wiki/Placename_hierachies is_in tag Geofabrik have updated extracts daily for Pakistan: * http://download.geofabrik.de/osm/asia/pakistan.osm.bz2 Otherwise pull a BBOX directly using Osmosis: * http://wiki.openstreetmap.org/wiki/Osmosis Osmosis requires Java. Python options for filtering based on tag, which would be more suitable for integration within Sahana, however we need to add Polygon filtering using Shapely: * http://dev.omniscale.net/imposm.parser/ * [http://pbf.raggedred.net ParsePBF] * http://blog.lifeeth.in/2011/02/extract-pois-from-osm-pbf.html Ruby script to generate KML of recently-added locations by a group of users: * http://github.com/vertis/osm_export/tree/master === Basemap for Garmin GPS === * [wiki:UserGuidelinesGISDataOpenStreetMapGarmin] == PostgreSQL management == * [wiki:InstallationGuidelinesPostgreSQL#Maintenance] === PostGIS functions === * Centroids {{{ SELECT name, iso2, asText(ST_Transform(ST_Centroid(the_geom), 4326)) AS centroid FROM countries; }}} == Data Sources == * Admin 0 & 1, Hydro features, Raster & Vector: http://www.naturalearthdata.com/ === L0 === * http://www.gadm.org * http://www.naturalearthdata.com/ * VMap0: http://en.wikipedia.org/wiki/Vector_Map * http://barendgehrels.blogspot.com/2010/12/free-shapefile-of-countries-of-world.html * http://barendgehrels.blogspot.com/2010/12/free-shapefiles-of-of-world-2.html * http://barendgehrels.blogspot.com/2010/12/free-shapefiles-of-of-world-3.html * http://barendgehrels.blogspot.com/2011/01/free-shapefiles-of-countries-of-world-4.html * http://thematicmapping.org/downloads/world_borders.php * UN (North Africa): http://unosat-sdn.web.cern.ch/unosat-sdn/northafrica/region/ungiwg_level0_region.zip === L1 === * http://www.gadm.org * http://www.naturalearthdata.com/ * http://geonames.org * US: http://www.mapcruzin.com/free-united-states-arcgis-maps-shapefiles.htm === L2 === * SALB (High-quality, country-verified, but not all countries): http://www.unsalb.org/ * http://www.gadm.org * http://geonames.org * California: http://swdb.igs.berkeley.edu/pub/data/GEOGRAPHY/DECENNIAL2010/state_county_2010_shp.zip === OGC (WMS/WFS) === * http://geonetwork-opensource.org/gallery.html#geonetwork-nodes * http://www.ogc-services.net * !BlueMarble & Mosaic (!LandSat7): http://onearth.jpl.nasa.gov/ * Hazard Layers: http://preview.grid.unep.ch:8080/geoserver/ows?service=WMS&request=GetCapabilities * Hazard Layers: http://www.pdc.org/mde/services.jsp * Massachusetts: http://lyceum.massgis.state.ma.us/wiki/doku.php === Aerial Imagery === http://wiki.openstreetmap.org/wiki/Aerial_imagery ---- [wiki:GIS]