Context Navigation

Data

Timestamp:: 01/04/11 07:45:37 (14 years ago)
Author:: Fran Boon
Comment:: --

Legend:

: Unmodified
: Added
: Removed
: Modified

UserGuidelines/GIS/Data

-              v109
+              v110
 e.g. PakistanUnionCouncils
 === UUIDs ===
 See: [wiki:UUID#Mapping]
+=== Geonames ===
+There is an import_geonames() function in S3GIS which downloads/unzips the country file (a TAB-separated list) from http://download.geonames.org/export/dump/
+It should be run for the different levels of hierarchy that you wish to import (generally just the lowest level as Geonames just has Point data, so it's best to use other sources for the Polygons 1st, that way the Geonames importer can locate these Points within the correct Polygons of the hierarchy)
+NB It takes some time to do this import! Pakistan imports 95000 locations!
+Update: Geonames schema 2.2 supports parentADM(1-4): http://geonames.wordpress.com/2010/09/29/geonames-ontology-2-2/
+ * will be good for when we only have hierarchy, not polygons
+ * need to check whether much data has this populated though.
+Python 2.5 doesn't support Zipfile.extract() & Zipfile.read() isn't unicode-safe. Until this is fixed, download the file manually 1st:
+{{{
+cd ~web2py/applications/eden/cache
+wget http://download.geonames.org/export/dump/PK.zip
+unzip PK.zip
+}}}
+In Web2py CLI:
+{{{
+gis.import_geonames('PK', 'L5')
+db.commit()
+}}}
+Alternate approach:
+. Transform each line in this file into XML by regular expression:
+{{{
+^(\d*)\t([^\t]*)\t([^\t]*)\t([^\t]*)\t([0-9\.]*)\t([0-9\.]*)\t[^\t]*\t([A-Z]*).*
+into:
+<location>
+       <id>$1</id>
+       <name>$2</name>
+       <asciiName>$3</asciiName>
+       <localNames>$4</localNames>
+       <lat>$5</lat>
+       <lon>$6</lon>
+       <featureClass>$7</featureClass>
+</location>
+}}}
+This can be done using an RE-capable editor (e.g. Kate), Perl or even Python.
+Note: Need to replace & with &amp; and to remove any invalid characters
+. Transform into S3XRC-XML using XSLT, stylesheet is available at
+   * [http://pub.nursix.org/eden/geonames/geonames.xsl]
+=== !OpenStreetMap ===
+See below: [wiki:UserGuidelinesGISData#Import]
+=== Shapefiles ===
+Inspect the data using [http://qgis.org qGIS].
+Use ogr2ogr to convert the data:
+{{{
+ogr2ogr -f CSV CSV TM_WORLD_BORDERS-0.3.shp -lco GEOMETRY=AS_WKT
+ogr2ogr -f geojson TM_WORLD_BORDERS-0.3.json TM_WORLD_BORDERS-0.3.shp
+}}}
+If needing to reproject (e.g. for the [https://www.geoint-online.net/community/haitiearthquake/Geospatial%20Data%20Files/haiti_departments01132010.zip Haiti Departements]):
+{{{
+ogr2ogr -f CSV haiti_departments Haiti_departementes_edited_01132010.shp -s_srs EPSG:32618 -t_srs EPSG:4326 -lco GEOMETRY=AS_WKT
+}}}
+NB AS_WKT requires OGR v1.6+
+Inspect the CSV & adjust columns as-necessary:
+ * We need a sheet for L1 with columns: ADM0_NAME, ADM1_NAME (& WKT)
+ * We want a sheet for L2 with columns: ADM1_NAME, ADM2_NAME (& WKT) [ADM0_NAME can also be used to help separate duplicates]
+ * We want a sheet for L3 with columns: ADM2_NAME, ADM3_NAME (& WKT) [ADM1_NAME can also be used to help separate duplicates]
+=== CSV ===
+There is a function available in {{{modules/s3/s3gis.py}}} to import from CSV.
+The CSV needs to have specific columns:
+ * WKT column if we have polygon info (or points for L4)
+ * For L1, we need these columns: ADM0_NAME, ADM1_NAME (& WKT)
+ * For L2, we need these columns: ADM1_NAME, ADM2_NAME (& WKT) [ADM0_NAME can also be used to help separate duplicates]
+ * For L3, we need these columns: ADM2_NAME, ADM3_NAME (& WKT) [ADM1_NAME can also be used to help separate duplicates]
+ * Ensure that names are consistent between Levels
  * The PROPER() spreadsheet function is useful to get the names in the correct format (then Paste as Text).
  * The VLOOKUP() spreadsheet function is useful if the different levels of the hierarchy are in different sheets & linked via a code instead of the name (as we need)
+There is a function available in {{{modules/s3/s3gis.py}}} to import from CSV.
+Basic Hierarchy can often be found from Wikipedia (although currently there's no easy way to download this - a student project to enhance Wikipedia for this would be much appreciated!).
+For the Polygon data, it is normal to get this from Shapefiles (see below).
 Example for Pakistan:
 …
 db.executesql("CREATE INDEX %s__idx on %s(%s);" % (field, tablename, field))
 }}}
+=== Shapefiles ===
+Inspect the data using [http://qgis.org qGIS].
+Use ogr2ogr to convert the data to CSV:
+{{{
+ogr2ogr -f CSV CSV TM_WORLD_BORDERS-0.3.shp -lco GEOMETRY=AS_WKT
+ogr2ogr -f geojson TM_WORLD_BORDERS-0.3.json TM_WORLD_BORDERS-0.3.shp
+}}}
+If needing to reproject (e.g. for the [https://www.geoint-online.net/community/haitiearthquake/Geospatial%20Data%20Files/haiti_departments01132010.zip Haiti Departements]):
+{{{
+ogr2ogr -f CSV haiti_departments Haiti_departementes_edited_01132010.shp -s_srs EPSG:32618 -t_srs EPSG:4326 -lco GEOMETRY=AS_WKT
+}}}
+NB AS_WKT requires OGR v1.6+
+=== Geonames ===
+There is an import_geonames() function in S3GIS which downloads/unzips the country file (a TAB-separated list) from http://download.geonames.org/export/dump/
+It should be run for the different levels of hierarchy that you wish to import (generally just the lowest level as Geonames just has Point data, so it's best to use other sources for the Polygons 1st, that way the Geonames importer can locate these Points within the correct Polygons of the hierarchy)
+NB It takes some time to do this import! Pakistan imports 95000 locations!
+Update: Geonames schema 2.2 supports parentADM(1-4): http://geonames.wordpress.com/2010/09/29/geonames-ontology-2-2/
+ * will be good for when we only have hierarchy, not polygons
+ * need to check whether much data has this populated though.
+Python 2.5 doesn't support Zipfile.extract() & Zipfile.read() isn't unicode-safe. Until this is fixed, download the file manually 1st:
+{{{
+cd ~web2py/applications/eden/cache
+wget http://download.geonames.org/export/dump/PK.zip
+unzip PK.zip
+}}}
+In Web2py CLI:
+{{{
+gis.import_geonames('PK', 'L5')
+db.commit()
+}}}
+Alternate approach:
+. Transform each line in this file into XML by regular expression:
+{{{
+^(\d*)\t([^\t]*)\t([^\t]*)\t([^\t]*)\t([0-9\.]*)\t([0-9\.]*)\t[^\t]*\t([A-Z]*).*
+into:
+<location>
+       <id>$1</id>
+       <name>$2</name>
+       <asciiName>$3</asciiName>
+       <localNames>$4</localNames>
+       <lat>$5</lat>
+       <lon>$6</lon>
+       <featureClass>$7</featureClass>
+</location>
+}}}
+This can be done using an RE-capable editor (e.g. Kate), Perl or even Python.
+Note: Need to replace & with &amp; and to remove any invalid characters
+. Transform into S3XRC-XML using XSLT, stylesheet is available at
+   * [http://pub.nursix.org/eden/geonames/geonames.xsl]
+=== !OpenStreetMap ===
+See below: [wiki:UserGuidelinesGISData#Import]
+=== WFS ===
+It is possible to use the WFS Plugin to get data into qGIS & thence export into other formats.
+May need to use a Custom CRS (in Settings menu - remember to Save!) such as:
+ * ESRI's Spherical Mercator ([http://www.cadmaps.com/gisblog/?p=81 different] to 900913) http://spatialreference.org/ref/esri/54004/proj4/
+Can then go to the Layer Properties & Specify CRS to this User Defined Coordinate System.
+Can then Save As and change the CRS to something like the standard WGS84.
 === Yahoo ===
  * http://developer.yahoo.com/geo/geoplanet/data/
  * Script to process http://pastie.org/1139680
+=== WFS ===
+It is possible to use the WFS Plugin to get data into qGIS & thence export into other formats.
+May need to use a Custom CRS (in Settings menu - remember to Save!) such as:
+ * ESRI's Spherical Mercator ([http://www.cadmaps.com/gisblog/?p=81 different] to 900913) http://spatialreference.org/ref/esri/54004/proj4/
+Can then go to the Layer Properties & Specify CRS to this User Defined Coordinate System.
+Can then Save As and change the CRS to something like the standard WGS84.
 == Display Data ==
 === !GeoServer ===