wiki:GIS/OpenStreetMap

Version 12 (modified by Fran Boon, 10 years ago) ( diff )

Mapnik build updated

OpenStreetMap

Wikipedia of Maps

http://openstreetmap.org

Base Map

We have out-of-the-box the ability to use OpenStreetMap Tiles as base layer.

This can include local OSM sites (OSM Taiwan is included as an example)

Local Tile Server

If you need to run offline then you need to set up your own Tile Server:

OpenStreetMap data can be stored in a PostGIS database & rendered using Mapnik.

To support Key/Value pairs:

apt-get install postgresql-contrib-9.3
su postgres
psql -d gis -f /usr/share/postgresql/9.3/contrib/hstore.sql

To support Planet diff files:

su postgres
psql gis < /usr/share/postgresql/9.3/contrib/_int.sql

Osm2pgsql

It is recommended to use the latest version from source rather than an out of date package:

apt-get install build-essential libxml2-dev libgeos-dev libpq-dev libbz2-dev proj autoconf subversion libtool
git clone https://github.com/openstreetmap/osm2pgsql
Optionally investgate updating this outdated patch (updated from original here): http://www.mail-archive.com/dev@openstreetmap.org/msg12768.html
#wget http://eden.sahanafoundation.org/raw-attachment/wiki/GIS/InstallationGuidelines/Linux/osm2pgsql_centroid.patch
cd osm2pgsql
#patch -p0 < ../osm2pgsql_centroid.patch
./autogen.sh
./configure
sed -i 's/-g -O2/-O2 -march=native -fomit-frame-pointer/' Makefile
make
make install
# Fix needed for SVN version 0.70.5
cp default.style /usr/local/share

Mapnik

2.0 is packaged for Wheezy, but we want the latest:

apt-get install -y libboost-dev libboost-filesystem-dev libboost-python-dev libboost-regex-dev libboost-system-dev libboost-thread-dev libpq-dev libfreetype6-dev libxml2-dev libjpeg-dev libpng-dev libtiff-dev libproj-dev
wget http://mapnik.s3.amazonaws.com/dist/v2.2.0/mapnik-v2.2.0.tar.bz2
tar jxvf mapnik-v2.2.0.tar.bz2
cd mapnik-v2.2.0
./configure
make
make test-local
make install

Download the OpenStreetMap extensions for Mapnik:

cd
svn export http://svn.openstreetmap.org/applications/rendering/mapnik

Download/decompress Coastlines:

cd
wget http://tile.openstreetmap.org/world_boundaries-spherical.tgz
wget http://tile.openstreetmap.org/processed_p.tar.bz2
wget http://tile.openstreetmap.org/shoreline_300.tar.bz2
wget http://www.naturalearthdata.com/http//www.naturalearthdata.com/download/10m/cultural/10m-populated-places.zip
wget http://www.naturalearthdata.com/http//www.naturalearthdata.com/download/110m/cultural/110m-admin-0-boundary-lines.zip
cd ~/mapnik
tar zxvf ~/world_boundaries-spherical.tgz
tar jxvf ~/processed_p.tar.bz2 -C world_boundaries
tar jxvf ~/shoreline_300.tar.bz2 -C world_boundaries
unzip ~/10m-populated-places.zip -d world_boundaries
unzip ~/110m-admin-0-boundary-lines.zip -d world_boundaries

Disable autovacuum:

vim /etc/postgresql/8.4/main/postgresql.conf
#autovacuum = on
/etc/init.d/postgresql restart

Download/Import area of interest, e.g.:

#wget http://downloads.cloudmade.com/north_america/haiti/haiti.osm.bz2
#su postgres
#osm2pgsql -s -d gis haiti.osm.bz2
wget http://labs.geofabrik.de/haiti/2010-11-24-14-43.osm.bz2
su postgres
osm2pgsql -s -d gis 2010-11-24-14-43.osm.bz2

Re-enable autovacuum:

vim /etc/postgresql/8.4/main/postgresql.conf
autovacuum = on
/etc/init.d/postgresql restart

Setup XML file:

cd ~/mapnik
./generate_xml.py --accept-none --dbname gis --symbols ./symbols/ --world_boundaries ./world_boundaries/

Render tiles:

cd ~/mapnik
vim generate_tiles.py
render_tiles(bbox, mapfile, tile_dir, 0, 8, "World")
#minZoom = 10
#maxZoom = 16
#bbox = (-2, 50.0,1.0,52.0)
#render_tiles(bbox, mapfile, tile_dir, minZoom, maxZoom)
# Haiti
bbox = (-74.668, 17.884, -71.659, 20.232)
render_tiles(bbox, mapfile, tile_dir, 9, 17, "Haiti")
# Muenchen
# comment all under here

mkdir /var/www/tiles
chown postgres /var/www/tiles
ln -s /var/www/tiles
su postgres
MAPNIK_MAP_FILE="osm.xml" MAPNIK_TILE_DIR="tiles/" ./generate_tiles.py

Contours

apt-get install gdal-bin

SRTM3 data:

mkdir ~/mapnik/srtm
cd ~/mapnik/srtm
wget http://mapnik-utils.googlecode.com/svn@170/sandbox/testing/hillshading/srtm_generate_hdr.sh 
chmod +x srtm_generate_hdr.sh 
cp srtm_generate_hdr.sh /usr/local/bin
vim process_srtm3.sh
#!/bin/bash 
PREP_TABLE="1" 
for X in *.hgt.zip; do 
    yes | srtm_generate_hdr.sh $X 
    rm -f "${X%%.zip}" 

    # Import 10m contours 
    rm -f "${X%%.hgt.zip}.shp" "${X%%.hgt.zip}.shx" "${X%%.hgt.zip}.dbf" 
    gdal_contour -i 10 -snodata 32767 -a height "${X%%.hgt.zip}.tif" "${X%%.hgt.zip}.shp" 
    [ "$PREP_TABLE" ] && shp2pgsql -p -I -g way "${X%%.hgt.zip}" contours | psql -q gis 
    shp2pgsql -a -g way "${X%%.hgt.zip}" contours | psql -q gis 

    rm -f "${X%%.hgt.zip}.shp" "${X%%.hgt.zip}.shx" "${X%%.hgt.zip}.dbf" 
    rm -f "${X%%.hgt.zip}.bil" 
    rm -f "${X%%.hgt.zip}.hdr" 
    rm -f "${X%%.hgt.zip}.prj" 
    rm -f "${X%%.hgt.zip}.tif" 
    unset PREP_TABLE 
done

# Haiti
wget http://dds.cr.usgs.gov/srtm/version2_1/SRTM3/North_America/N17W072.hgt.zip
wget http://dds.cr.usgs.gov/srtm/version2_1/SRTM3/North_America/N18W072.hgt.zip
wget http://dds.cr.usgs.gov/srtm/version2_1/SRTM3/North_America/N18W073.hgt.zip
wget http://dds.cr.usgs.gov/srtm/version2_1/SRTM3/North_America/N18W074.hgt.zip
wget http://dds.cr.usgs.gov/srtm/version2_1/SRTM3/North_America/N18W075.hgt.zip
wget http://dds.cr.usgs.gov/srtm/version2_1/SRTM3/North_America/N19W072.hgt.zip
wget http://dds.cr.usgs.gov/srtm/version2_1/SRTM3/North_America/N19W073.hgt.zip
wget http://dds.cr.usgs.gov/srtm/version2_1/SRTM3/North_America/N19W074.hgt.zip
wget http://dds.cr.usgs.gov/srtm/version2_1/SRTM3/North_America/N19W075.hgt.zip
wget http://dds.cr.usgs.gov/srtm/version2_1/SRTM3/North_America/N20W073.hgt.zip
wget http://dds.cr.usgs.gov/srtm/version2_1/SRTM3/North_America/N20W074.hgt.zip
wget http://dds.cr.usgs.gov/srtm/version2_1/SRTM3/North_America/N20W075.hgt.zip

chown postgres .
su postgres
sh process_srtm3.sh

Since we are non-commercial, we should be able to use the SRTM4.1 data from CGIAR, however the Shapefile production crashed when it reached 2Gb.
Example for Haiti:

cd ~/mapnik/srtm
wget http://srtm.csi.cgiar.org/SRT-ZIP/SRTM_V41/SRTM_Data_GeoTiff/srtm_22_09.zip
unzip srtm_22_09.zip
gdal_contour -i 10 -snodata 32767 -a height "srtm_22_09.tif" "srtm_22_09.shp"
su postgres
shp2pgsql -p -I -g way "srtm_22_09" contours | psql -q gis
shp2pgsql -a -g way "srtm_22_09" contours | psql -q gis

Configure Mapnik:

vim ~/mapnik/osm.xml
# Under line &layer-shapefiles;
&layer-contours;
vim ~/mapnik/inc/layer-contours.xml.inc
<Style name="contours10">
        <Rule>
                &maxscale_zoom14;
                &minscale_zoom17;
                <LineSymbolizer>
                        <CssParameter name="stroke">#9cb197</CssParameter>
                        <CssParameter name="stroke-width">0.5</CssParameter>
                </LineSymbolizer>
        </Rule>
</Style>
<Style name="contours50">
        <Rule>
                &maxscale_zoom14;
                &minscale_zoom17;
                <LineSymbolizer>
                        <CssParameter name="stroke">#9cb197</CssParameter>
                        <CssParameter name="stroke-width">0.6</CssParameter>
                </LineSymbolizer>
        </Rule>
        <Rule>
                &maxscale_zoom12;
                &minscale_zoom13;
                <LineSymbolizer>
                        <CssParameter name="stroke">#747b90</CssParameter>
                        <CssParameter name="stroke-width">0.6</CssParameter>
                </LineSymbolizer>
        </Rule>
</Style>
<Style name="contours100">
        <Rule>
                &maxscale_zoom14;
                &minscale_zoom17;
                <LineSymbolizer>
                        <CssParameter name="stroke">#9cb197</CssParameter>
                        <CssParameter name="stroke-width">0.7</CssParameter>
                </LineSymbolizer>
        </Rule>
        <Rule>
                &maxscale_zoom12;
                &minscale_zoom13;
                <LineSymbolizer>
                        <CssParameter name="stroke">#747b90</CssParameter>
                        <CssParameter name="stroke-width">0.7</CssParameter>
                </LineSymbolizer>
        </Rule>
        <Rule>
                &maxscale_zoom10;
                &minscale_zoom11;
                <LineSymbolizer>
                        <CssParameter name="stroke">#855d62</CssParameter>
                        <CssParameter name="stroke-width">0.7</CssParameter>
                </LineSymbolizer>
        </Rule>
</Style>
<Style name="contours-text50">
        <Rule>
                &maxscale_zoom14;
                &minscale_zoom17;
                <TextSymbolizer name="height" face_name="DejaVu Sans Book" size="8" fill="#747b90" halo_radius="1" placement="line" />
        </Rule>
</Style>
<Style name="contours-text100">
        <Rule>
                &maxscale_zoom14;
                &minscale_zoom17;
                <TextSymbolizer name="height" face_name="DejaVu Sans Book" size="8" fill="#747b90" halo_radius="1" placement="line" />
        </Rule>
        <Rule>
                &maxscale_zoom12;
                &minscale_zoom13;
                <TextSymbolizer name="height" face_name="DejaVu Sans Book" size="8" fill="#855d62" halo_radius="1" placement="line" />
        </Rule>
</Style>
<Layer name="srtm_10" status="on" srs="+proj=latlong +datum=WGS84">
        <StyleName>contours10</StyleName>
        <StyleName>contours-text10</StyleName>
        <Datasource>
                <Parameter name="table">(select way,height from contours WHERE height::integer % 10 = 0 AND height::integer % 50 != 0 AND height::integer % 100 != 0) as "contours-10"</Parameter>
                &datasource-settings;
        </Datasource>
</Layer>
<Layer name="srtm_50" status="on" srs="+proj=latlong +datum=WGS84">
        <StyleName>contours50</StyleName>
        <StyleName>contours-text50</StyleName>
        <Datasource>
                <Parameter name="table">(select way,height from contours WHERE height::integer % 50 = 0 AND height::integer % 100 != 0) as "contours-50"</Parameter>
                &datasource-settings;
        </Datasource>
</Layer>
<Layer name="srtm_100" status="on" srs="+proj=latlong +datum=WGS84">
        <StyleName>contours100</StyleName>
        <StyleName>contours-text100</StyleName>
        <Datasource>
                <Parameter name="table">(select way,height from contours WHERE height::integer % 100 = 0) as "contours-100"</Parameter>
                &datasource-settings;
        </Datasource>
</Layer>

Hillshading

Serve Tiles

Tiles can simply be served by Apache:

vim /etc/apache2/sites-available/mysite
<VirtualHost *:80>
  ServerName mysite.org
  Alias /tiles     /var/www/tiles
  <LocationMatch "/tiles/">
    Order Allow,Deny
    Allow from all
  </LocationMatch>
</VirtualHost>
apache2ctl restart

WMS

In order to serve the OSM data via WMS then can use mod_mapnik_wms:

  • http://wiki.openstreetmap.org/wiki/Mod_mapnik_wms
    gpg --gen-key
    apt-get install pbuilder debsigs apache2-prefork-dev libmapnik-dev libgd2-xpm-dev
    svn export http://svn.openstreetmap.org/applications/utils/mod_mapnik_wms
    cd mod_mapnik_wms
    sh autogen.sh
    vim debian/mapnik_wms.load
    LoadFile /usr/lib/libmapnik.so.0.7
    
    debuild
    

This data can then be converted by GeoServer into a KML SuperOverlay to display in Google Earth:

Rails Port

This is needed to allow editing of the data using Potlatch, JOSM, etc

apt-get install imagemagick libmagick9-dev
apt-get install ruby ruby1.8-dev libxml2-dev libxml-ruby1.8 libxml-parser-ruby1.8 rubygems librmagick-ruby
gem install -v=2.3.8 rails 
#gem install libxml-ruby
#gem install composite_primary_keys
#gem install rmagick
gem install timecop
gem install pg
gem install oauth

#svn co http://railsexpress.de/svn/plugins/sql_session_store/trunk sql_session_store

apt-get install postgresql-contrib libpq-dev
su postgres
createuser openstreetmap -s -P
createdb -E UTF8 -O openstreetmap openstreetmap 
createdb -E UTF8 -O openstreetmap osm_test
createdb -E UTF8 -O openstreetmap osm
psql -d openstreetmap < /usr/share/postgresql/8.4/contrib/btree_gist.sql

apt-get install git
cd /home
git clone git://git.openstreetmap.org/rails.git
cd rails
cp config/postgres.example.database.yml config/database.yml
vim config/database.yml

rake gems:install
rake db:migrate
env RAILS_ENV=production rake db:migrate
rake test

osmosis --read-xml-0.6 file="planet.osm.bz2" --write-apidb-0.6 populateCurrentTables=yes host="localhost" database="openstreetmap" user="openstreetmap" password="openstreetmap" validateSchemaVersion=no

select setval('acls_id_seq', (select max(id) from acls)); 
select setval('changesets_id_seq', (select max(id) from changesets)); 
select setval('countries_id_seq', (select max(id) from countries)); 
select setval('current_nodes_id_seq', (select max(id) from current_nodes)); 
select setval('current_relations_id_seq', (select max(id) from current_relations)); 
select setval('current_ways_id_seq', (select max(id) from current_ways)); 
select setval('diary_comments_id_seq', (select max(id) from diary_comments)); 
select setval('diary_entries_id_seq', (select max(id) from diary_entries)); 
select setval('friends_id_seq', (select max(id) from friends)); 
select setval('gpx_file_tags_id_seq', (select max(id) from gpx_file_tags)); 
select setval('gpx_files_id_seq', (select max(id) from gpx_files)); 
select setval('messages_id_seq', (select max(id) from messages)); 
select setval('sessions_id_seq', (select max(id) from sessions)); 
select setval('user_tokens_id_seq', (select max(id) from user_tokens)); 
select setval('users_id_seq', (select max(id) from users));

cd /home/rails
ruby script/server

cd db/functions
make libpgosm.so

    * Log into PgSQL and execute the CREATE FUNCTION statement from maptile.c's comment: 

 CREATE FUNCTION maptile_for_point(int8, int8, int4) RETURNS int4 
 AS '/path/to/rails-port/db/functions/libpgosm', 'maptile_for_point'
 LANGUAGE C STRICT;

 CREATE FUNCTION tile_for_point(int4, int4) RETURNS int8 
 AS '/path/to/rails-port/db/functions/libpgosm', 'tile_for_point'
 LANGUAGE C STRICT;

Vector Overlays

Can have OSM Vectors displayed over the top of other Base Layers (e.g. Satellite Images)

Import

We have an XSLT stylesheet to import .osm files

e.g for hospitals and clinics:

osmosis --read-xml country.osm --tf accept-nodes amenity=hospital,clinic --tf reject-ways --tf reject-relations --write-xml nodes.osm
osmosis --read-xml country.osm --tf reject-relations --tf accept-ways amenity=hospital,clinic --used-node --write-xml ways.osm
osmosis --rx nodes.osm --rx ways.osm --merge --wx country_hospitals.osm
http://myhost.com/eden/hms/hospital/create.osm?filename=country_hospitals.osm

This needs more work to understand the admin hierarchy properly to be able to import Places.

Geofabrik have updated extracts daily for Pakistan:

Otherwise pull a BBOX directly using Osmosis:

Osmosis requires Java. Python options for filtering based on tag, which would be more suitable for integration within Sahana, however we need to add Polygon filtering using Shapely:

Ruby script to generate KML of recently-added locations by a group of users:

Polygons:

Basemap for Garmin GPS

Mirror

How to set up a local mirror for a country, optionally with just selected Feature types, suitable for use by Import PoI:

Debian stable
apt-get install sudo
visudo
fran ALL = NOPASSWD: ALL

cat << EOF >> "/etc/apt/sources.list.d/pgdg.list"
deb http://apt.postgresql.org/pub/repos/apt/ wheezy-pgdg main
EOF
wget --no-check-certificate https://www.postgresql.org/media/keys/ACCC4CF8.asc
apt-key add ACCC4CF8.asc
apt-get update

apt-get -y install postgresql-9.3 python-psycopg2 postgresql-9.3-postgis ptop postgresql-contrib

# Tune PostgreSQL
cat << EOF >> "/etc/sysctl.conf"
## Increase Shared Memory available for PostgreSQL
# 512Mb
kernel.shmmax = 279134208
# 1024Mb (may need more)
#kernel.shmmax = 552992768
kernel.shmall = 2097152
EOF
sysctl -w kernel.shmmax=279134208 # For 512 MB RAM
#sysctl -w kernel.shmmax=552992768 # For 1024 MB RAM
sysctl -w kernel.shmall=2097152

sed -i 's|#track_counts = on|track_counts = on|' /etc/postgresql/9.3/main/postgresql.conf
sed -i 's|#autovacuum = on|autovacuum = on|' /etc/postgresql/9.3/main/postgresql.conf
# 512Mb RAM:
sed -i 's|shared_buffers = 28MB|shared_buffers = 56MB|' /etc/postgresql/9.3/main/postgresql.conf
sed -i 's|#effective_cache_size = 128MB|effective_cache_size = 256MB|' /etc/postgresql/9.3/main/postgresql.conf
sed -i 's|#work_mem = 1MB|work_mem = 2MB|' /etc/postgresql/9.3/main/postgresql.conf
# If 1Gb+ RAM, activate post-install via pg1024 script

adduser osm
mkdir -p /home/osm/planet/replication
cd /home/osm
wget http://dev.openstreetmap.org/~bretth/osmosis-build/osmosis-latest.tgz
mkdir osmosis
cd osmosis
tar zxvf osmosis-latest.tgz
chown -R postgres /home/osm
su postgres
psql
CREATE USER osm WITH PASSWORD 'planet';
\q
createdb -O osm -E UTF8 osm -T template0
psql -d osm -f /usr/share/postgresql/9.3/extension/postgis--2.1.4.sql
#psql -d osm -f /usr/share/postgresql/9.3/extension/hstore--1.2.sql
psql
\c osm
create extension hstore;
\q
psql -d osm -f /home/osm/osmosis/script/pgsnapshot_schema_0.6.sql
psql
\c osm
ALTER TABLE geography_columns OWNER TO osm;
ALTER TABLE geometry_columns OWNER TO osm;
ALTER TABLE nodes OWNER TO osm;
ALTER TABLE relation_members OWNER TO osm;
ALTER TABLE relations OWNER TO osm;
ALTER TABLE schema_info OWNER TO osm;
ALTER TABLE spatial_ref_sys OWNER TO osm;
ALTER TABLE users OWNER TO osm;
ALTER TABLE way_nodes OWNER TO osm;
ALTER TABLE ways OWNER TO osm;
\q
exit

cd /home/osm/planet
# Either grab snapshot from GeoFabrik or use HOT Export Tool
#wget http://download.geofabrik.de/africa/cameroon-latest.osm.pbf -O extract.osm.pbf
wget http://export.hotosm.org/download/023803/rawdata.osm.pbf -O extract.osm.pbf

apt-get install -y openjdk-7-jre

# If wishing to filter the data to just certain types of object
#mv extract.osm.pbf full.osm.pbf
#/home/osm/osmosis/bin/osmosis --read-pbf /home/osm/planet/full.osm.pbf --tf accept-nodes amenity=hospital,place_of_worship,school --tf accept-ways amenity=hospital,place_of_worship,school --write-pbf /home/osm/planet/extract.osm.pbf
su postgres
/home/osm/osmosis/bin/osmosis --read-pbf /home/osm/planet/extract.osm.pbf --write-pgsql database=osm user=osm password=planet

cd /home/osm/planet/replication
/home/osm/osmosis/bin/osmosis --rrii
sudo -H -u postgres psql -d osm -c "select max(tstamp) from nodes";
# http://osm.personalwerk.de/replicate-sequences/
# (Code: https://github.com/MaZderMind/replicate-sequences)
# Put timestamp into site, select hourly
vi state.txt
# paste in contents from website
vi configuration.txt
baseUrl=http://planet.openstreetmap.org/replication/hour

# If wishing to filter the data to just certain types of object
#/home/osm/osmosis/bin/osmosis --rri workingDirectory=/home/osm/planet/replication/ --simc --read-pbf /home/osm/planet/extract.osm.pbf --ac --tf accept-nodes amenity=hospital,place_of_worship,school --tf accept-ways amenity=hospital,place_of_worship,school --tf reject-relations --write-pbf /home/osm/planet/new.osm.pbf
#/home/osm/osmosis/bin/osmosis --read-pbf /home/osm/planet/new.osm.pbf --read-pbf /home/osm/planet/extract.osm.pbf --dc --write-pgsql-change database=osm user=osm password=planet
/home/osm/osmosis/bin/osmosis --rri workingDirectory=/home/osm/planet/replication/ --simc --read-pbf /home/osm/planet/extract.osm.pbf --ac --write-pbf /home/osm/planet/new.osm.pbf
/home/osm/osmosis/bin/osmosis --read-pbf /home/osm/planet/new.osm.pbf --read-pbf /home/osm/planet/extract.osm.pbf --dc --write-pgsql-change database=osm user=osm password=planet
mv /home/osm/planet/new.osm.pbf /home/osm/planet/extract.osm.pbf
vi /home/osm/replicate.sh
#!/bin/sh
n=`ps -ef | grep -v grep | grep /home/osm/osmosis/ | wc -l`
if [ $n -le 0 ]
then
        if [ -s /home/osm/planet/new.osm.pbf ]
        then
        mv /home/osm/planet/new.osm.pbf /home/osm/planet/extract.osm.pbf
fi
/home/osm/osmosis/bin/osmosis --rri workingDirectory=/home/osm/planet/replication/ --simc --read-pbf /home/osm/planet/extract.osm.pbf --ac --write-pbf /home/osm/planet/new.osm.pbf
/home/osm/osmosis/bin/osmosis --read-pbf /home/osm/planet/new.osm.pbf --read-pbf /home/osm/planet/extract.osm.pbf --dc --write-pgsql-change database=osm user=osm password=planet
mv /home/osm/planet/new.osm.pbf /home/osm/planet/extract.osm.pbf
fi

chmod 0755 /home/osm/replicate.sh
vi /etc/crontab
1 *   * * * root    /home/osm/replicate.sh

Monitor:

sudo -H -u postgres psql -d osm -c "select max(tstamp) from nodes";

Attachments (1)

Download all attachments as: .zip

Note: See TracWiki for help on using the wiki.