Using OpenStreetMap Data - P.1

I can think back and remem­ber strug­gling with being able to uti­lize Open­StreetMap (OSM) data in a project of mine. I wanted a free source of infor­ma­tion and geospa­tial data, where I was not bound to strict and lim­it­ing copy­rights. Open­StreetMap offered a one-stop-shop for free and open geospa­tial data. At the time, I was most inter­ested in uti­liz­ing their very detailed road net­work. How­ever, I ran into many hic­cups along the way, which ulti­mately deterred me from com­plet­ing the project. Some time has passed and I have returned to said project and suc­cess­fully com­pleted the pre­lim­i­nary work, that is, being able to uti­lize OSM data in a GIS. This brings us to the the­sis of this blog-post. Herein lies a post that aims to pro­vide guid­ance for users who are unfa­mil­iar with the process of access­ing, retriev­ing and uti­liz­ing OSM data. By doing so I hope to “give back” to the open geospa­tial com­mu­nity, which with­out com­mu­nity mem­ber involve­ment would cease to exist. Remem­ber that all the appli­ca­tions and data that we will use in this tuto­r­ial are avail­able for free (no cost) to the pub­lic… but we all know that there is no such thing as a free lunch (or in our case, a free GIS).

Briefly...

This tutorial will cover the following:

• access and browse an OSM data repositories
• down­load a sub­set (often called an ‘extract’) of the planet.osm data package
• install Post­greSQL (object-relational data­base system)
• install Post­GIS for use with Post­greSQL (spa­tial data­base exten­sion for PostgreSQL)
• install and uti­lize osm2pgsql (con­verts OSM data for use in the Post­greSQL database)
• install QGIS and its depen­den­cies (GIS pack­age)
• query and add data in QGIS from PostGIS/PostgreSQL

Also, this tutorial makes the following assumptions:

• cur­rently run­ning Mac OS X 10.6 or later (tuto­r­ial ver­i­fied using both 10.6.7 and 10.7)
• access to admin­is­tra­tive priv­i­leges so that we can per­form installations
• plenty of stor­age space (extracts from OSM range from a few MB to many GBs)
• will­ing­ness to try some­thing new!

Lastly, this tutorial will follow the following format:

The tuto­r­ial will fol­low a guided, step by step instruc­tion that will assume the role of a new user installing and pro­cess­ing all data from scratch. The included links and infor­ma­tion above are for back­ground to the project that we are about to begin. Please fol­low all the instruc­tions (don’t skip any steps if you don’t know what you are doing) and down­load all the required data when prompted to.

We will be using Ter­mi­nal in this guide but I don’t expect, nor is it a require­ment that any read­ers of this post have any Terminal.app back­ground or knowl­edge. Ter­mi­nal allows users to inter­act with the com­puter through a com­mand line inter­face. If you have not seen or used the Ter­mi­nal before, you may have come across some instance of command-line inter­faces (per­haps on Win­dows machines, à la Com­mand Prompt). I will try my best to explain what we are doing dur­ing the phase of the tuto­r­ial which uti­lizes the Terminal.app.

If you would like more infor­ma­tion about any of the items men­tioned above, please feel free to visit the respec­tive website/wiki as listed below (but remem­ber to come back!):

OpenStreetMap http://www.openstreetmap.org/
Planet.osm http://wiki.openstreetmap.org/wiki/Planet.osm
PostgreSQL http://en.wikipedia.org/wiki/PostgreSQL
PostGIS http://en.wikipedia.org/wiki/PostGIS
Osm2pgsql http://wiki.openstreetmap.org/wiki/Osm2pgsql
QGIS http://www.qgis.org/wiki/Welcome_to_the_QGIS_Wiki

C'est bon. Allez!

OpenStreetMap & the Repositories

Open­StreetMap is a col­lec­tions of free geo­graphic data that can be viewed within a browser (http://www.openstreetmap.org/). We can also access, down­load and uti­lize the under­ly­ing OSM project data from var­i­ous repos­i­to­ries. What makes this so attrac­tive is the fact that its a com­mu­nity dri­ven project, which means that any­one can con­tribute to it. It is also free to use and dis­trib­ute under the CC-BY-SA license (as long as we attribute OSM and the license itself). The stan­dard pack­age that OSM dis­trib­utes is called the ‘planet.osm’. It is a stan­dard XML-formatted .osm file, which at the time of writ­ing (Aug 22, 2011) is over 220 GB (17 GB com­pressed). The planet.osm is updated weekly (every thurs­day) and includes the lat­est revi­sions of nodes, ways and poly­gons (points, lines and poly­gons for the ESRI crowd). I strongly encour­age you NOT to down­load the entire planet.osm and work with it because of its size and the required com­put­ing power needed to work with such a data­base. We will be explor­ing an extract of the planet.osm, specif­i­cally the Province of Ontario (Canada), which comes in a much smaller/cpu-friendly size (2 GB uncom­pressed / 350 MB compressed).

http://wiki.openstreetmap.org/wiki/Planet.osm pro­vides a list of mir­rors from which the planet.osm file and its extracts can be down­loaded. We will specif­i­cally be using the Cloud­Made direc­tory located on http://downloads.cloudmade.com/

Preface:

This tuto­r­ial will reflect what I am doing on my com­puter. This is done in an effort to address com­monly made mis­takes. Tai­lor the tuto­r­ial to suit your own needs or fol­low my instruc­tions to a tee.

Let us make sure that we stay some­what orga­nized dur­ing this tutorial.

• cre­ate a new folder on the desk­top called ‘osm_tutorial’
• cre­ate two sub­fold­ers within ‘osm_tutorial’ called ‘down­loads’ and ‘data’

We will down­load all of our instal­la­tion files to the ‘down­loads’ direc­tory and we will down­load the OSM data to the ‘data’ directory.

Access, browse & download OSM data

Let’s down­load an extract of the planet.osm file (remem­ber that the planet.osm file is too large to han­dle on its own). We will use the Cloud­Made repos­i­tory which updates their planet.osm and ‘extract’.osm (such that ‘extract’ is the name of a location/place) weekly.

• Go to http://downloads.cloudmade.com/

The repos­i­tory is orga­nized in a hier­ar­chal struc­ture (Region > Sub­Re­gion > Coun­try > Province or State). Note that, Cloud­Made does not have the entire planet.osm parsed into smaller extracts, such that only the most pop­u­lar or demanded areas have been extracted for us to use. Some repos­i­to­ries will extract dif­fer­ent regions than oth­ers. There are also some repos­i­to­ries that extract smaller scale areas (Cities, Towns, etc).

I am going to be using the Ontario extract, which is located in Amer­i­cas > North­ern Amer­ica > Canada > Ontario. Feel free to use any of the other extracts. How­ever, I rec­om­mend that you choose an extract from the lowest-level in the heirar­chy (eg, provinces or cities) and stay away from the larger extracts such as regions or countries.

• Fol­low the folder struc­ture ‘Amer­i­cas > North­ern Amer­ica > Canada > Ontario’ to find the ontario.osm.bz2 extract
• Right click > Save Link As on the ‘ontario.osm.bz2’ link
• Save the file to your ‘/Desktop/osm_tutorial/data’ folder

Do not extract this file. A ben­e­fit of using the tools men­tioned in this tuto­r­ial, is the abil­ity to work with fully com­pressed OSM data. Why is the file named ‘ontario.osm.bz2’? BZIP2 (bz2) is an open-source com­pres­sion tool that OSM uti­lizes to pro­duce small pro­duc­tion pack­ages (eg. a 17 GB ‘planet.osm.bz2’ v. 220 GB uncom­pressed ‘planet.osm’).

Download and install PostgreSQL

The next step is to down­load our data­base client, Post­greSQL. There are many iter­a­tions of Post­greSQL and it can get quite con­fus­ing for the begin­ner user. Luck­ily, Dave Page @ Enter­prise DB main­tains an easy-to-use all-in-one installer that is avail­able for Mac OS X (and other plat­forms as well).

• Go to http://www.enterprisedb.com/products-services-training/pgdownload#osx
• Select ‘Installer ver­sion Ver­sion 9.0.4–1′ (lat­est at the time of writ­ing) for Mac OS X
• Save the file to your ‘/Desktop/osm_tutorial/downloads’ folder

We will now try to install Post­greSQL. I say ‘try’ because *if* this is your first attempt at installing Post­greSQL, you will be prompted in regards to your com­put­ers ‘Shared Mem­ory’ con­fig­u­ra­tion. Not to worry, Post­greSQL han­dles the changes that are nec­es­sary, and yes, it is safe to allow Post­greSQL make these changes. I’ve included a snip­pet from the Post­greSQL ‘readme’.

Shared Memory
-------------
PostgreSQL uses shared memory extensively for caching and inter-process communication. Unfortunately, the default configuration of Mac OS X does not allow suitable amounts of shared memory to be created to run the database server.

• Mount the Post­greSQL diskim­age by double-clicking on ‘postgresql-9.0.4–1-osx.dmg’
• Launch the installer and allow it to make the nec­es­sary changes
• You will need to restart your com­puter. Remem­ber to book­mark me before you go!

Welcome back!

• Re-mount the Post­greSQL diskim­age by double-clicking on ‘postgresql-9.0.4–1-osx.dmg’ (if it is not already mounted on the desktop)
• The installer will now pass the shared mem­ory check and load the setup wizard
• Pro­ceed through the setup wiz­ard, accept­ing all default con­fig­u­ra­tion set­tings. Remem­ber to assign a unique pass­word for your Post­greSQL data­base when prompted too
• Make spe­cial note of port num­ber ‘5432’ that Post­greSQL has been assigned to. We will need to call upon this num­ber later on in this tutorial

The installer will take a minute or so to com­plete and once com­pleted will ask you if you would like to ‘Launch Stack Builder at exit?’ This is required as we will need to install the Post­GIS exten­sion because Post­greSQL can­not han­dle our OSM data on its own. Post­GIS will act as our data­bases fore­run­ner, han­dling the spa­tial infor­ma­tion in the OSM data for our Post­greSQL database

• Make sure the check­box to launch stack builder on exit is ticked
• When stack builder launches, it will ask you to select your Post­greSQL instal­la­tion from the drop-down style menu
• Select ‘Spa­tial Exten­sions > Post­GIS’ (as seen below)

• Pro­ceed through the rest of the instal­la­tion, accept­ing all default con­fig­u­ra­tion options
• When prompted to, input your unique pass­word for the Post­greSQL data­base. By default, the port num­ber (5432) and user­name (post­gres) are correct
• Once the instal­la­tion is com­pleted, launch ‘pgAd­min III’ from the ‘Appli­ca­tions’ folder
• Double-clicking on ‘Post­greSQL 9.0 (localhost:5432)’ will start the data­base (located under ‘Object browser > Servers’ in the left pane)
• Right click > New Data­base… in the cen­tral pane (mak­ing sure not to select the ‘post­gres’ or ‘template_postgis’ items
• Fill out the ‘New Data­base…’ form with the set­tings as shown below
• That’s it, we can exit the application

Download and install osm2pgsql

Our next step is to down­load and install osm2pgsql, which will expe­dite the ‘ontario.osm.bz2’ file into our Post­greSQL database

• Go to http://dbsgeo.com/downloads/#osm2pgsql
• Select and down­load ‘osm2pgsql 0.70.5 — Snow Leop­ard / OS 10.6 (Intel x86_64)’ to your ‘/Desktop/osm_tutorial/downloads’ folder
• Mount the osm2pgsql diskim­age by double-clicking ‘osm2pgsql_0.70.5_snow_intel_r1.dmg’
• You must install GEOS and PROJ frame­works before installing the osm2pgsql tool. Do this now. Pro­ceed through the instal­la­tions, accept­ing all default configurations
• Once GEOS and PROJ are installed, we can move on to osm2pgsql. Accept all default con­fig­u­ra­tions when installing osm2pgsql

Using osm2pgsql and the Terminal.app

We will use osm2pgsql to parse our osm data into the Post­greSQL data­base. Here is where we will encounter the use of the Terminal.app. Like I men­tioned ear­lier, I will try to explain what exactly we are telling Ter­mi­nal to do.

• Launch Ter­mi­nal, you can use the spot­light (mag­nify glass in the top right-hand cor­ner) to search for it

The first line of code (if you haven’t opened ter­mi­nal recently) should say some­thing sim­i­lar to this(replace my com­puter name and user­name with your own). Ter­mi­nal is let­ting us know what{whos} com­puter we are on (michael-markietas-mac), and where are we per­form­ing the tasks (a direc­tory, in this case, michael markieta’s home directory)

• Last login: Mon Aug 22 19:38:44 on console
• Michael-Markietas-MacBook-Pro:~ michaelmarkieta$

We need to change the work­ing direc­tory from our user direc­tory (michael­marki­eta) to the desk­top. We use the ‘cd’ com­mand which intu­itively means ‘change direc­tory’. We also pass in where we would like to change our direc­tory to. In this case, we will change direc­tory to the ‘desk­top’. Type the fol­low­ing into Ter­mi­nal (or copy and paste):

• cd desktop

If done cor­rectly, Ter­mi­nal will switch the cur­rent work­ing direc­tory to the desk­top and the repeat­ing line of code should read some­thing like this:

• michael-markietas-mac:~desktop$>

Now that we know how to change work­ing direc­to­ries, lets nav­i­gate to our ontario.osm.bz2 file which is located in ‘desktop/osm_tutorial/data’. To move more than one folder when chang­ing direc­to­ries, we must insert our path between apos­tro­phes(“”) and sep­a­rate fold­ers with for­ward slashes (/). Type the fol­low­ing into Ter­mi­nal (or copy and paste):

• cd "osm_tutorial/data"

Ter­mi­nal should now be work­ing from within the ‘data’ folder, which is located inside the ‘osm_tutorial’ folder, which is also located inside{on} the ‘desk­top’, which is also part of michael markieta’s user folder. The full path would look some­thing like ‘/Users/Michael Markieta/Desktop/osm_tutorial/data’.

• michael-markietas-mac:~data$>

Now to per­form some osm2pgl magic. The tool offers many con­fig­urable para­me­ters. These can be seen by typ­ing the fol­low­ing into Terminal:

• osm2pgsql --help

We will con­cern our­self with a few of these para­me­ters, but its use­ful to look over what is included with a tool when you instal some­thing new. We will be using –U, user­name, –d, data­base name to guide osm2pgsql in pars­ing the ontario.osm.bz2 file into our Post­greSQL data­base. The code will need to look like this:

• osm2pgsql -d gis -U postgres -P 5432 ontario.osm.bz2

Let's pick this apart one chunk at a time.

• osm2pgsql — in words, we are telling Terminal.app to start the osm2pgsql script
• -d gis — in words, we are con­fig­ur­ing the script to use the data­base called ‘gis’
• -U postgres — in words, we are con­fig­ur­ing the script to access the data­base with user­name ‘postgres’
• -P 5432 — in words, we are con­fig­ur­ing the script to access the data­base on port 5432
• ontario.osm.bz2 — in words, this is the file we would like our script to parse into our database

We will need to let it run for a few min­utes. Make your­self some coffee/tea, some break­fast (if you are work­ing into the wee hours of the night/morning like I am).. We can fin­ish up this tuto­r­ial when its done run­ning the tool.

If every­thing went smoothly, you should have received the fol­low­ing con­fir­ma­tion in Terminal:

• Completed planet_osm_point
• Completed planet_osm_roads
• Completed planet_osm_polygon
• Completed planet_osm_line

This tuto­r­ial is get­ting a bit too long and will be con­tin­ued on another page. But first, lets take a look at what we have com­pleted so far from the list of items we were sup­pose to tackle.

• access and browse an OSM data repositories
• down­load a sub­set (often called an ‘extract’) of the planet.osm data package
• install Post­greSQL (object-relational data­base system)
• install Post­GIS for use with Post­greSQL (spa­tial data­base exten­sion for PostgreSQL)
• install and uti­lize osm2pgsql (con­verts OSM data for use in the Post­greSQL database)
• install QGIS and its depen­den­cies (GIS pack­age)
• query and add data in QGIS from PostGIS/PostgreSQL

Looks like we are almost there! Stay tuned everyone...

Part two is now up: http://www.spatialanalysis.ca/2011/using-openstreetmap-data-part-2/