Changes between Version 109 and Version 110 of UserGuidelines/GIS/Data


Ignore:
Timestamp:
01/04/11 07:45:37 (11 years ago)
Author:
Fran Boon
Comment:

--

Legend:

Unmodified
Added
Removed
Modified
  • UserGuidelines/GIS/Data

    v109 v110  
    66
    77e.g. PakistanUnionCouncils
     8
    89=== UUIDs ===
    910See: [wiki:UUID#Mapping]
    10 === Geonames ===
    11 There is an import_geonames() function in S3GIS which downloads/unzips the country file (a TAB-separated list) from http://download.geonames.org/export/dump/
    12 
    13 It should be run for the different levels of hierarchy that you wish to import (generally just the lowest level as Geonames just has Point data, so it's best to use other sources for the Polygons 1st, that way the Geonames importer can locate these Points within the correct Polygons of the hierarchy)
    14 
    15 NB It takes some time to do this import! Pakistan imports 95000 locations!
    16 
    17 Update: Geonames schema 2.2 supports parentADM(1-4): http://geonames.wordpress.com/2010/09/29/geonames-ontology-2-2/
    18  * will be good for when we only have hierarchy, not polygons
    19  * need to check whether much data has this populated though.
    20 
    21 Python 2.5 doesn't support Zipfile.extract() & Zipfile.read() isn't unicode-safe. Until this is fixed, download the file manually 1st:
    22 {{{
    23 cd ~web2py/applications/eden/cache
    24 wget http://download.geonames.org/export/dump/PK.zip
    25 unzip PK.zip
    26 }}}
    27 
    28 In Web2py CLI:
    29 {{{
    30 gis.import_geonames('PK', 'L5')
    31 db.commit()
    32 }}}
    33 
    34 Alternate approach:
    35  1. Transform each line in this file into XML by regular expression:
    36 {{{
    37 
    38 ^(\d*)\t([^\t]*)\t([^\t]*)\t([^\t]*)\t([0-9\.]*)\t([0-9\.]*)\t[^\t]*\t([A-Z]*).*
    39 
    40 into:
    41 
    42 <location>
    43        <id>$1</id>
    44        <name>$2</name>
    45        <asciiName>$3</asciiName>
    46        <localNames>$4</localNames>
    47        <lat>$5</lat>
    48        <lon>$6</lon>
    49        <featureClass>$7</featureClass>
    50 </location>
    51 
    52 }}}
    53 
    54 This can be done using an RE-capable editor (e.g. Kate), Perl or even Python.
    55 Note: Need to replace & with &amp; and to remove any invalid characters
    56 
    57  2. Transform into S3XRC-XML using XSLT, stylesheet is available at
    58    * [http://pub.nursix.org/eden/geonames/geonames.xsl]
    59 
    60 === !OpenStreetMap ===
    61 
    62 See below: [wiki:UserGuidelinesGISData#Import]
    63 
    64 === Shapefiles ===
    65 Inspect the data using [http://qgis.org qGIS].
    66 
    67 Use ogr2ogr to convert the data:
    68 {{{
    69 ogr2ogr -f CSV CSV TM_WORLD_BORDERS-0.3.shp -lco GEOMETRY=AS_WKT
    70 ogr2ogr -f geojson TM_WORLD_BORDERS-0.3.json TM_WORLD_BORDERS-0.3.shp
    71 }}}
    72 If needing to reproject (e.g. for the [https://www.geoint-online.net/community/haitiearthquake/Geospatial%20Data%20Files/haiti_departments01132010.zip Haiti Departements]):
    73 {{{
    74 ogr2ogr -f CSV haiti_departments Haiti_departementes_edited_01132010.shp -s_srs EPSG:32618 -t_srs EPSG:4326 -lco GEOMETRY=AS_WKT
    75 }}}
    76 NB AS_WKT requires OGR v1.6+
    77 
    78 Inspect the CSV & adjust columns as-necessary:
    79  * We need a sheet for L1 with columns: ADM0_NAME, ADM1_NAME (& WKT)
    80  * We want a sheet for L2 with columns: ADM1_NAME, ADM2_NAME (& WKT) [ADM0_NAME can also be used to help separate duplicates]
    81  * We want a sheet for L3 with columns: ADM2_NAME, ADM3_NAME (& WKT) [ADM1_NAME can also be used to help separate duplicates]
     11
     12=== CSV ===
     13There is a function available in {{{modules/s3/s3gis.py}}} to import from CSV.
     14
     15The CSV needs to have specific columns:
     16 * WKT column if we have polygon info (or points for L4)
     17 * For L1, we need these columns: ADM0_NAME, ADM1_NAME (& WKT)
     18 * For L2, we need these columns: ADM1_NAME, ADM2_NAME (& WKT) [ADM0_NAME can also be used to help separate duplicates]
     19 * For L3, we need these columns: ADM2_NAME, ADM3_NAME (& WKT) [ADM1_NAME can also be used to help separate duplicates]
     20 * Ensure that names are consistent between Levels
    8221 * The PROPER() spreadsheet function is useful to get the names in the correct format (then Paste as Text).
    8322 * The VLOOKUP() spreadsheet function is useful if the different levels of the hierarchy are in different sheets & linked via a code instead of the name (as we need)
    8423
    85 There is a function available in {{{modules/s3/s3gis.py}}} to import from CSV.
     24Basic Hierarchy can often be found from Wikipedia (although currently there's no easy way to download this - a student project to enhance Wikipedia for this would be much appreciated!).
     25
     26For the Polygon data, it is normal to get this from Shapefiles (see below).
    8627
    8728Example for Pakistan:
     
    173114db.executesql("CREATE INDEX %s__idx on %s(%s);" % (field, tablename, field))
    174115}}}
     116
     117=== Shapefiles ===
     118Inspect the data using [http://qgis.org qGIS].
     119
     120Use ogr2ogr to convert the data to CSV:
     121{{{
     122ogr2ogr -f CSV CSV TM_WORLD_BORDERS-0.3.shp -lco GEOMETRY=AS_WKT
     123ogr2ogr -f geojson TM_WORLD_BORDERS-0.3.json TM_WORLD_BORDERS-0.3.shp
     124}}}
     125If needing to reproject (e.g. for the [https://www.geoint-online.net/community/haitiearthquake/Geospatial%20Data%20Files/haiti_departments01132010.zip Haiti Departements]):
     126{{{
     127ogr2ogr -f CSV haiti_departments Haiti_departementes_edited_01132010.shp -s_srs EPSG:32618 -t_srs EPSG:4326 -lco GEOMETRY=AS_WKT
     128}}}
     129NB AS_WKT requires OGR v1.6+
     130
     131=== Geonames ===
     132There is an import_geonames() function in S3GIS which downloads/unzips the country file (a TAB-separated list) from http://download.geonames.org/export/dump/
     133
     134It should be run for the different levels of hierarchy that you wish to import (generally just the lowest level as Geonames just has Point data, so it's best to use other sources for the Polygons 1st, that way the Geonames importer can locate these Points within the correct Polygons of the hierarchy)
     135
     136NB It takes some time to do this import! Pakistan imports 95000 locations!
     137
     138Update: Geonames schema 2.2 supports parentADM(1-4): http://geonames.wordpress.com/2010/09/29/geonames-ontology-2-2/
     139 * will be good for when we only have hierarchy, not polygons
     140 * need to check whether much data has this populated though.
     141
     142Python 2.5 doesn't support Zipfile.extract() & Zipfile.read() isn't unicode-safe. Until this is fixed, download the file manually 1st:
     143{{{
     144cd ~web2py/applications/eden/cache
     145wget http://download.geonames.org/export/dump/PK.zip
     146unzip PK.zip
     147}}}
     148
     149In Web2py CLI:
     150{{{
     151gis.import_geonames('PK', 'L5')
     152db.commit()
     153}}}
     154
     155Alternate approach:
     156 1. Transform each line in this file into XML by regular expression:
     157{{{
     158
     159^(\d*)\t([^\t]*)\t([^\t]*)\t([^\t]*)\t([0-9\.]*)\t([0-9\.]*)\t[^\t]*\t([A-Z]*).*
     160
     161into:
     162
     163<location>
     164       <id>$1</id>
     165       <name>$2</name>
     166       <asciiName>$3</asciiName>
     167       <localNames>$4</localNames>
     168       <lat>$5</lat>
     169       <lon>$6</lon>
     170       <featureClass>$7</featureClass>
     171</location>
     172
     173}}}
     174
     175This can be done using an RE-capable editor (e.g. Kate), Perl or even Python.
     176Note: Need to replace & with &amp; and to remove any invalid characters
     177
     178 2. Transform into S3XRC-XML using XSLT, stylesheet is available at
     179   * [http://pub.nursix.org/eden/geonames/geonames.xsl]
     180
     181=== !OpenStreetMap ===
     182
     183See below: [wiki:UserGuidelinesGISData#Import]
     184
     185=== WFS ===
     186It is possible to use the WFS Plugin to get data into qGIS & thence export into other formats.
     187
     188May need to use a Custom CRS (in Settings menu - remember to Save!) such as:
     189 * ESRI's Spherical Mercator ([http://www.cadmaps.com/gisblog/?p=81 different] to 900913) http://spatialreference.org/ref/esri/54004/proj4/
     190
     191Can then go to the Layer Properties & Specify CRS to this User Defined Coordinate System.
     192
     193Can then Save As and change the CRS to something like the standard WGS84.
     194
    175195=== Yahoo ===
    176196 * http://developer.yahoo.com/geo/geoplanet/data/
    177197 * Script to process http://pastie.org/1139680
    178198
    179 === WFS ===
    180 It is possible to use the WFS Plugin to get data into qGIS & thence export into other formats.
    181 
    182 May need to use a Custom CRS (in Settings menu - remember to Save!) such as:
    183  * ESRI's Spherical Mercator ([http://www.cadmaps.com/gisblog/?p=81 different] to 900913) http://spatialreference.org/ref/esri/54004/proj4/
    184 
    185 Can then go to the Layer Properties & Specify CRS to this User Defined Coordinate System.
    186 
    187 Can then Save As and change the CRS to something like the standard WGS84.
     199
    188200== Display Data ==
    189201=== !GeoServer ===