Re: [OSM-talk] osm2pgsql 1.2.0

[Sarah Hoffmann]

Hi,

On Tue, Nov 05, 2019 at 12:16:30AM +0100, wambac...@posteo.de wrote:
> i'm using osm2pgsql-0.96 right now. Is it possible (and meaningful) to
> switch to osm2pgsql-1.2.0?
> 
> Must i change anything in my database? Runing diff updates using a flatfile.

You can switch from 0.96 to 1.2 any time. The database layout
is compatible between the two versions. The only difference is
that 1.x no longer supports old-style multipolygons. That should
not be an issue with the curent planet anyway.

I'd always recommend using the latest release to get the latest fixes.
However, the main performance improvements for 1.2 are noticable during
import. If you prefer a packaged 0.96 over a self-compiled 1.2, then
it's not too much loss to remain with the packages version.

Sarah


___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

[OSM-talk] osm2pgsql 1.2.0

[wambacher]

Hi,

i'm using osm2pgsql-0.96 right now. Is it possible (and meaningful) to 
switch to osm2pgsql-1.2.0?


Must i change anything in my database? Runing diff updates using a flatfile.

Regards
walter

--
My projects:

Admin Boundaries of the World 
Missing Boundaries 


Emergency Map 
Postal Code Map (Germany only) 
Fools (QA for zipcodes in Germany) 
Postcode Boundaries of Germany 
OSM Software Watchlist 

___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] osm2pgsql diff application with filtered OSM data

[Nick Whitelegg]

>This will leave your site down between the delete and import of new data.

>It's also going to be fragile, because using append mode with a file that 
>isn't a diff isn't supported, and if the area has a lot of footpaths, it'll be 
>slower, since >append has to do more work.

>If you match the SQL delete and osmosis filtering carefully you shouldn't get 
>too many errors, but you've probably got some to do with updates and >changing 
>object types.

>As long as you're aware of these problems and it works for your needs, I'd say 
>to go for it.


Thanks. I have actually used append frequently over several years and never had 
a problem with it; main issue is that memory is expensive on servers so I have 
to do take various less-than-ideal steps to work round low memory.


The intention is to bring the site down over the (say) 2-3 hours the update is 
taking place; this is what I used to do when I last performed weekly updates. 
This would be scheduled for (probably) Wednesday 0100-0400 UK time, a time that 
the server is likely to see little traffic as it's UK and Ireland oriented; at 
such time most of Europe will be asleep. Granted there might be a few 
out-of-Europe visitors at that time; but as my site is quite niche with limited 
visitors, I consider this acceptable downtime.


Nick


From: Paul Norman 
Sent: 13 November 2018 10:12:58
To: talk@openstreetmap.org
Subject: Re: [OSM-talk] osm2pgsql diff application with filtered OSM data

On 2018-11-11 7:53 AM, Nick Whitelegg wrote:


After thinking about this, I realised that I don't really want to update _all_ 
the data that often. The only thing I need to update on a weekly basis is the 
footpaths (I'm not so bothered if say the roads, or the pubs are a year out of 
date - as long as newly mapped footpaths appear quickly). So what I'm now doing 
is just doing an osmosis extract of paths weekly, deleting all data in the DB 
which I class as a 'path' and repopulating in amend mode.

This will leave your site down between the delete and import of new data.

It's also going to be fragile, because using append mode with a file that isn't 
a diff isn't supported, and if the area has a lot of footpaths, it'll be 
slower, since append has to do more work.

If you match the SQL delete and osmosis filtering carefully you shouldn't get 
too many errors, but you've probably got some to do with updates and changing 
object types.

As long as you're aware of these problems and it works for your needs, I'd say 
to go for it.
___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] osm2pgsql diff application with filtered OSM data

[Paul Norman]

On 2018-11-11 7:53 AM, Nick Whitelegg wrote:



After thinking about this, I realised that I don't really want to 
update _all_ the data that often. The only thing I need to update on a 
weekly basis is the footpaths (I'm not so bothered if say the roads, 
or the pubs are a year out of date - as long as newly mapped footpaths 
appear quickly). So what I'm now doing is just doing an osmosis 
extract of paths weekly, deleting all data in the DB which I class as 
a 'path' and repopulating in amend mode.




This will leave your site down between the delete and import of new data.

It's also going to be fragile, because using append mode with a file 
that isn't a diff isn't supported, and if the area has a lot of 
footpaths, it'll be slower, since append has to do more work.


If you match the SQL delete and osmosis filtering carefully you 
shouldn't get too many errors, but you've probably got some to do with 
updates and changing object types.


As long as you're aware of these problems and it works for your needs, 
I'd say to go for it.
___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] osm2pgsql diff application with filtered OSM data

[Nick Whitelegg]

Thanks for all the replies.


After thinking about this, I realised that I don't really want to update _all_ 
the data that often. The only thing I need to update on a weekly basis is the 
footpaths (I'm not so bothered if say the roads, or the pubs are a year out of 
date - as long as newly mapped footpaths appear quickly). So what I'm now doing 
is just doing an osmosis extract of paths weekly, deleting all data in the DB 
which I class as a 'path' and repopulating in amend mode.


Thanks,

Nick



From: Paul Norman 
Sent: 08 November 2018 20:10:14
To: talk@openstreetmap.org
Subject: Re: [OSM-talk] osm2pgsql diff application with filtered OSM data

On 2018-11-08 6:34 AM, Nick Whitelegg wrote:

At the moment I download full planet extracts about every 6 months. However, 
due to the limitations of my server, I filter out (with osmosis) a lot of stuff 
I don't need so that I am basically left with roads, footpaths, natural 
features, water features and selected POIs.


I'd like to move towards a system which applies diffs from geofabrik instead, 
and applies them regularly (daily or weekly) with osm2pgsql.


My question is this; given that not everything in the diff will be in my 
database (as I filter out what I don't need during the import process), will 
osm2pgsql apply the diff successfully or will it complain that not all features 
in the diff are in my database?

I can think of four ways to do this, all which have a different balance of 
correctness, performance, and ease of use.

There are two "right" ways to do this. The first one is to re-import every 
week. Because imports without slim tables (either --slim --drop or no --slim) 
are faster, this is a good option and needs less space than a database able to 
consume diffs.

The second right way involves keeping two files, one with the current full 
data, and one with the filtered data. Call these "planet.pbf" and 
"planet-filtered.pbf". Then when updating create "planet-new.pbf", filter it to 
get "planet-filtered-new.pbf", create a diff for the differences between 
"planet-filtered-new.pbf" and "planet-filtered.pbf", and apply that diff to the 
database. Then replace the old files with the new ones. This will keep the 
database correct.

A "wrong" way to do it is to import the filtered data, apply updates directly, 
and periodically delete data from the DB. The problem with this is that if 
someone adds one of the selected POI tags to a building that you have filtered 
out, osm2pgsql won't have the node data to create a geometry. This might be 
acceptable, depending on use case.

A less wrong way would be to modify your filtering so no nodes are filtered 
out. There are still potential errors with relations, but these are less 
common. If you're doing the planet or a large extract and using flat nodes 
there's no storage penalty for having all the nodes.
___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] osm2pgsql diff application with filtered OSM data

[Nick Whitelegg]

... append mode!





From: Nick Whitelegg
Sent: 11 November 2018 15:53:18
To: talk@openstreetmap.org
Subject: Re: [OSM-talk] osm2pgsql diff application with filtered OSM data



Thanks for all the replies.


After thinking about this, I realised that I don't really want to update _all_ 
the data that often. The only thing I need to update on a weekly basis is the 
footpaths (I'm not so bothered if say the roads, or the pubs are a year out of 
date - as long as newly mapped footpaths appear quickly). So what I'm now doing 
is just doing an osmosis extract of paths weekly, deleting all data in the DB 
which I class as a 'path' and repopulating in amend mode.


Thanks,

Nick



From: Paul Norman 
Sent: 08 November 2018 20:10:14
To: talk@openstreetmap.org
Subject: Re: [OSM-talk] osm2pgsql diff application with filtered OSM data

On 2018-11-08 6:34 AM, Nick Whitelegg wrote:

At the moment I download full planet extracts about every 6 months. However, 
due to the limitations of my server, I filter out (with osmosis) a lot of stuff 
I don't need so that I am basically left with roads, footpaths, natural 
features, water features and selected POIs.


I'd like to move towards a system which applies diffs from geofabrik instead, 
and applies them regularly (daily or weekly) with osm2pgsql.


My question is this; given that not everything in the diff will be in my 
database (as I filter out what I don't need during the import process), will 
osm2pgsql apply the diff successfully or will it complain that not all features 
in the diff are in my database?

I can think of four ways to do this, all which have a different balance of 
correctness, performance, and ease of use.

There are two "right" ways to do this. The first one is to re-import every 
week. Because imports without slim tables (either --slim --drop or no --slim) 
are faster, this is a good option and needs less space than a database able to 
consume diffs.

The second right way involves keeping two files, one with the current full 
data, and one with the filtered data. Call these "planet.pbf" and 
"planet-filtered.pbf". Then when updating create "planet-new.pbf", filter it to 
get "planet-filtered-new.pbf", create a diff for the differences between 
"planet-filtered-new.pbf" and "planet-filtered.pbf", and apply that diff to the 
database. Then replace the old files with the new ones. This will keep the 
database correct.

A "wrong" way to do it is to import the filtered data, apply updates directly, 
and periodically delete data from the DB. The problem with this is that if 
someone adds one of the selected POI tags to a building that you have filtered 
out, osm2pgsql won't have the node data to create a geometry. This might be 
acceptable, depending on use case.

A less wrong way would be to modify your filtering so no nodes are filtered 
out. There are still potential errors with relations, but these are less 
common. If you're doing the planet or a large extract and using flat nodes 
there's no storage penalty for having all the nodes.
___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] osm2pgsql diff application with filtered OSM data

[Paul Norman]

On 2018-11-08 6:34 AM, Nick Whitelegg wrote:


At the moment I download full planet extracts about every 6 months. 
However, due to the limitations of my server, I filter out (with 
osmosis) a lot of stuff I don't need so that I am basically left with 
roads, footpaths, natural features, water features and selected POIs.



I'd like to move towards a system which applies diffs from geofabrik 
instead, and applies them regularly (daily or weekly) with osm2pgsql.



My question is this; given that not everything in the diff will be in 
my database (as I filter out what I don't need during the import 
process), will osm2pgsql apply the diff successfully or will it 
complain that not all features in the diff are in my database?




I can think of four ways to do this, all which have a different balance 
of correctness, performance, and ease of use.


There are two "right" ways to do this. The first one is to re-import 
every week. Because imports without slim tables (either --slim --drop or 
no --slim) are faster, this is a good option and needs less space than a 
database able to consume diffs.


The second right way involves keeping two files, one with the current 
full data, and one with the filtered data. Call these "planet.pbf" and 
"planet-filtered.pbf". Then when updating create "planet-new.pbf", 
filter it to get "planet-filtered-new.pbf", create a diff for the 
differences between "planet-filtered-new.pbf" and "planet-filtered.pbf", 
and apply that diff to the database. Then replace the old files with the 
new ones. This will keep the database correct.


A "wrong" way to do it is to import the filtered data, apply updates 
directly, and periodically delete data from the DB. The problem with 
this is that if someone adds one of the selected POI tags to a building 
that you have filtered out, osm2pgsql won't have the node data to create 
a geometry. This might be acceptable, depending on use case.


A less wrong way would be to modify your filtering so no nodes are 
filtered out. There are still potential errors with relations, but these 
are less common. If you're doing the planet or a large extract and using 
flat nodes there's no storage penalty for having all the nodes.
___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] osm2pgsql diff application with filtered OSM data

[Andy Townsend]

On 08/11/2018 16:16, Darafei "Komяpa" Praliaskouski wrote:


Usually people also clip minutely osc, as per day database grows by a 
small country otherwise.


There's a worked example of that (in a slightly different context) at 
https://wiki.openstreetmap.org/wiki/User:SomeoneElse/Ubuntu_1804_tileserver_load#Updating_your_database_as_people_edit_OpenStreetMap 
.  That uses https://github.com/zverik/regional - I'm guessing you'll be 
able to "borrow" everything you need from there.


Best Regards,

Andy


___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] osm2pgsql diff application with filtered OSM data

[Nick Whitelegg]

Hello Darafei and Frederik,


OK - thanks for that.


What I will do therefore, to avoid unwanted data, is to generate a filtered 
planet extract and import that into the DB, and then generate another filtered 
extract and find the diff between the two. Advantage of that is that I'll only 
need to  do the first filtering run on the server, and subsequent runs on my 
local machine (assuming the resulting diff is small enough to upload from my 
local machine to the server).


Thanks,

Nick



From: Darafei "Komяpa" Praliaskouski 
Sent: 08 November 2018 16:16:48
To: Nick Whitelegg
Cc: osm-talk
Subject: Re: [OSM-talk] osm2pgsql diff application with filtered OSM data

Hi Nick,

osm2pgsql is tolerant to features absent in database. You can in theory even 
start with empty set of tables and just insert new diff data.

Usually people also clip minutely osc, as per day database grows by a small 
country otherwise.

чт, 8 нояб. 2018 г. в 17:37, Nick Whitelegg 
mailto:nick.whitel...@solent.ac.uk>>:


... sorry, when I say "full planet extracts" I mean only England, Wales, 
Scotland, Ireland (all) and Greece - not the entire planet.

Thanks,

Nick


From: Nick Whitelegg
Sent: 08 November 2018 14:34:17
To: osm-talk
Subject: osm2pgsql diff application with filtered OSM data



Hi,


Looking towards overhauling the import system I use for my Freemap site 
(free-map.org.uk<http://free-map.org.uk>) which is itself going to go through 
an overhaul in the near future by moving to Tangram and hopefully applying hill 
shading.


At the moment I download full planet extracts about every 6 months. However, 
due to the limitations of my server, I filter out (with osmosis) a lot of stuff 
I don't need so that I am basically left with roads, footpaths, natural 
features, water features and selected POIs.


I'd like to move towards a system which applies diffs from geofabrik instead, 
and applies them regularly (daily or weekly) with osm2pgsql.


My question is this; given that not everything in the diff will be in my 
database (as I filter out what I don't need during the import process), will 
osm2pgsql apply the diff successfully or will it complain that not all features 
in the diff are in my database?


Thanks,

Nick

___
talk mailing list
talk@openstreetmap.org<mailto:talk@openstreetmap.org>
https://lists.openstreetmap.org/listinfo/talk
--
Darafei Praliaskouski
Support me: http://patreon.com/komzpa
___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] osm2pgsql diff application with filtered OSM data

[Frederik Ramm]

Hi,

On 08.11.2018 15:34, Nick Whitelegg wrote:
> My question is this; given that not everything in the diff will be in my
> database (as I filter out what I don't need during the import process),
> will osm2pgsql apply the diff successfully or will it complain that not
> all features in the diff are in my database?

It will apply the diff as far as possible, but it will also add a lot of
unwanted/unnecessary stuff from the diff to the database which you would
then have to filter out again.

There's also a danger that you will miss some things. For example,
consider a rural garage (shop=car_repair) mapped as a way with four
nodes; you filter that out because you're not interested. Half a year
later the shop closes, and a tourism=hostel opens instead. Assuming you
are interested in this kind of POI, the diff that you process contains
the way with the new tags, but not the nodes since these have not
changed. The hostel will not appear in your osm2pgsql database because
the geometry cannot be built due to lack of nodes.

One way to avoid this downloading the full files, filtering them, and
then generating a diff by comparing the filtered file to the last
filtered file, and importing *that* diff.

Bye
Frederik

-- 
Frederik Ramm  ##  eMail frede...@remote.org  ##  N49°00'09" E008°23'33"

___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] osm2pgsql diff application with filtered OSM data

[Komяpa]

Hi Nick,

osm2pgsql is tolerant to features absent in database. You can in theory
even start with empty set of tables and just insert new diff data.

Usually people also clip minutely osc, as per day database grows by a small
country otherwise.

чт, 8 нояб. 2018 г. в 17:37, Nick Whitelegg :

>
> ... sorry, when I say "full planet extracts" I mean only England, Wales,
> Scotland, Ireland (all) and Greece - not the entire planet.
>
>
> Thanks,
>
> Nick
>
> --
> *From:* Nick Whitelegg
> *Sent:* 08 November 2018 14:34:17
> *To:* osm-talk
> *Subject:* osm2pgsql diff application with filtered OSM data
>
>
>
> Hi,
>
>
> Looking towards overhauling the import system I use for my Freemap site (
> free-map.org.uk) which is itself going to go through an overhaul in the
> near future by moving to Tangram and hopefully applying hill shading.
>
>
> At the moment I download full planet extracts about every 6 months.
> However, due to the limitations of my server, I filter out (with osmosis) a
> lot of stuff I don't need so that I am basically left with roads,
> footpaths, natural features, water features and selected POIs.
>
>
> I'd like to move towards a system which applies diffs from geofabrik
> instead, and applies them regularly (daily or weekly) with osm2pgsql.
>
>
> My question is this; given that not everything in the diff will be in my
> database (as I filter out what I don't need during the import process),
> will osm2pgsql apply the diff successfully or will it complain that not all
> features in the diff are in my database?
>
>
> Thanks,
>
> Nick
>
> ___
> talk mailing list
> talk@openstreetmap.org
> https://lists.openstreetmap.org/listinfo/talk
>
-- 
Darafei Praliaskouski
Support me: http://patreon.com/komzpa
___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] osm2pgsql diff application with filtered OSM data

[Nick Whitelegg]

... sorry, when I say "full planet extracts" I mean only England, Wales, 
Scotland, Ireland (all) and Greece - not the entire planet.

Thanks,

Nick


From: Nick Whitelegg
Sent: 08 November 2018 14:34:17
To: osm-talk
Subject: osm2pgsql diff application with filtered OSM data



Hi,


Looking towards overhauling the import system I use for my Freemap site 
(free-map.org.uk) which is itself going to go through an overhaul in the near 
future by moving to Tangram and hopefully applying hill shading.


At the moment I download full planet extracts about every 6 months. However, 
due to the limitations of my server, I filter out (with osmosis) a lot of stuff 
I don't need so that I am basically left with roads, footpaths, natural 
features, water features and selected POIs.


I'd like to move towards a system which applies diffs from geofabrik instead, 
and applies them regularly (daily or weekly) with osm2pgsql.


My question is this; given that not everything in the diff will be in my 
database (as I filter out what I don't need during the import process), will 
osm2pgsql apply the diff successfully or will it complain that not all features 
in the diff are in my database?


Thanks,

Nick

___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

[OSM-talk] osm2pgsql diff application with filtered OSM data

[Nick Whitelegg]

Hi,


Looking towards overhauling the import system I use for my Freemap site 
(free-map.org.uk) which is itself going to go through an overhaul in the near 
future by moving to Tangram and hopefully applying hill shading.


At the moment I download full planet extracts about every 6 months. However, 
due to the limitations of my server, I filter out (with osmosis) a lot of stuff 
I don't need so that I am basically left with roads, footpaths, natural 
features, water features and selected POIs.


I'd like to move towards a system which applies diffs from geofabrik instead, 
and applies them regularly (daily or weekly) with osm2pgsql.


My question is this; given that not everything in the diff will be in my 
database (as I filter out what I don't need during the import process), will 
osm2pgsql apply the diff successfully or will it complain that not all features 
in the diff are in my database?


Thanks,

Nick

___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] osm2pgsql 0.90.0 release

[Jeff McKenna]

Thank you for the update Paul, I am excited to try the improved MSVC 
compiler support soon.  (I am currently upgrading to Visual Studio 2015, 
from an old 2008)


-jeff


--
Jeff McKenna
MapServer Consulting and Training Services
http://www.gatewaygeomatics.com/




On 2016-03-02 10:27 AM, Paul Norman wrote:

Osm2pgsql 0.90.0 has been released. The major changes since 0.88.0 are

- Windows support is much improved, and osm2pgsql builds natively with MSVC

- Osm2pgsql now requires a C++11 compiler

- Memory overhead for very small extracts is significantly reduced

- Multi-threading support is enabled for Windows

- The build system has switched from autotools to cmake

- Parsing is now done by libosmium and protobuf-c-compiler is not required

- Multi-backend improvements and bug fixes

- PBF support is no longer optional

- Legacy code from 32-bit node IDs has been removed

- Builds now generate no GCC or Clang warnings or errors

Upgrading is strongly recommended for anyone using the multi backend or
Windows.

This will be the last version that supports PostGIS 1.5 or PostgreSQL
9.0. Both of these have been end of life for some time.

Future version will also use EPSG 3857 by default, not 900913.

A full list of commits can be found at
https://github.com/openstreetmap/osm2pgsql/compare/0.88.0...0.90.0





___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

[OSM-talk] osm2pgsql 0.90.0 release

[Paul Norman]

Osm2pgsql 0.90.0 has been released. The major changes since 0.88.0 are

- Windows support is much improved, and osm2pgsql builds natively with MSVC

- Osm2pgsql now requires a C++11 compiler

- Memory overhead for very small extracts is significantly reduced

- Multi-threading support is enabled for Windows

- The build system has switched from autotools to cmake

- Parsing is now done by libosmium and protobuf-c-compiler is not required

- Multi-backend improvements and bug fixes

- PBF support is no longer optional

- Legacy code from 32-bit node IDs has been removed

- Builds now generate no GCC or Clang warnings or errors

Upgrading is strongly recommended for anyone using the multi backend or 
Windows.


This will be the last version that supports PostGIS 1.5 or PostgreSQL 
9.0. Both of these have been end of life for some time.


Future version will also use EPSG 3857 by default, not 900913.

A full list of commits can be found at 
https://github.com/openstreetmap/osm2pgsql/compare/0.88.0...0.90.0
___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

[OSM-talk] osm2pgsql and JOSM osm files

[ri0ter]

Hi.
I've been trying to export osm file created by JOSM to my pgsql databse but
I'm having a problem with multipolygons. Everything seems to be fine in
windows build, but in linux version relations are parsed (as I see in slim
mode relations are added) but they are not added to planet_osm_polygon
table. Any ideas why's that happening or what am I doing wrong?
I'm using latest build from GIT but I also tried few older ones.



--
View this message in context: 
http://gis.19327.n5.nabble.com/osm2pgsql-and-JOSM-osm-files-tp5818686.html
Sent from the General Discussion mailing list archive at Nabble.com.

___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] osm2pgsql multipolygon parsing

[Stefan Keller]

2013/9/23 Kai Krueger kakrue...@gmail.com wrote:
 Indirectly it is a question of tagging schemas.

To me this is actually indirectly a question of a proper area type!
See e.g. Towards an Area Datatype for OSM from Jochen at SOTM
http://lanyrd.com/2013/sotm/scpkrr/

--Stefan


2013/9/23 Kai Krueger kakrue...@gmail.com

 quot;Petr Morávek [Xificurk]quot;-2 wrote
  Anyway, this thread was not started to debate tagging schemes, the
  question I (and others) wanted to discuss here is this:
  Given the data that are currently in the database, how should osm2pgsql
  handle the import to get as much as possible multipolygons right?

 Indirectly it is a question of tagging schemas. With osm2pgsql being the
 tool used in the default map rendering on osm.org and the prevalence of
 tagging for the renderer decisions on how it handles multipolygons will
 (and imho to a limited degree should) influence how people tag and what
 they
 perceive as correct tagging. Therefore it is important that there is a
 consensus of what the correct tagging schema is and make sure that is
 correctly supported by osm2pgsql. That is also why I think having this
 discussion on talk, rather than on github or the dev list is appropriate.

 We need to come to a consensus between all of the main tools (at least iD,
 P2, JOSM, osm2pgsql, osrm, ...) and the mappers to what the preferred,
 encouraged and supported standard for tagging multi-polygons is and make
 sure that all documentation is in line with this.





 --
 View this message in context:
 http://gis.19327.n5.nabble.com/osm2pgsql-multipolygon-parsing-tp5778300p5778654.html
 Sent from the General Discussion mailing list archive at Nabble.com.

 ___
 talk mailing list
 talk@openstreetmap.org
 https://lists.openstreetmap.org/listinfo/talk

___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] osm2pgsql multipolygon parsing

[Peter Wendorff]

Am 24.09.2013 10:12, schrieb Stefan Keller:
 2013/9/23 Kai Krueger kakrue...@gmail.com wrote:
 Indirectly it is a question of tagging schemas.
 
 To me this is actually indirectly a question of a proper area type!
 See e.g. Towards an Area Datatype for OSM from Jochen at SOTM
 http://lanyrd.com/2013/sotm/scpkrr/
+1
But the transition necessary is the same:
IF we could come to a stable, but backwards compatible solution by
step-by-step hardening the current multipolygons it would be easier to
create a new area datatype later converting multipolygons on the fly.

While it is NOT possible overall to convert ways to polygons later (as
you would have to do that tag-based, and have to make sure to duplicate
it if there are line-tags and area-tags on the same way and so on), it
might be possible to auto-convert multipolygons to area; but to do that
multipolygons have to be clearly defined and possible to handle.

The discussion of this thread tackles the problem left for any such
conversion in future: What is the area described by any given
multipolygon with the given tags on inner/outer/relation?

If we could come to a clear consensus about that, and if we could slowly
enforce fixing any problems according to that in the data, then a
conversion to a new area data type could be done with less errors and
problems than it could be done now.

But thanks for the area datatype argument, as it's an additional
argument to strenghten the Multipolygon definition and -interpretation
for the current (or very near future) osm.

regards
Peter

 
 --Stefan
 
 
 2013/9/23 Kai Krueger kakrue...@gmail.com
 
 quot;Petr Morávek [Xificurk]quot;-2 wrote
 Anyway, this thread was not started to debate tagging schemes, the
 question I (and others) wanted to discuss here is this:
 Given the data that are currently in the database, how should osm2pgsql
 handle the import to get as much as possible multipolygons right?

 Indirectly it is a question of tagging schemas. With osm2pgsql being the
 tool used in the default map rendering on osm.org and the prevalence of
 tagging for the renderer decisions on how it handles multipolygons will
 (and imho to a limited degree should) influence how people tag and what
 they
 perceive as correct tagging. Therefore it is important that there is a
 consensus of what the correct tagging schema is and make sure that is
 correctly supported by osm2pgsql. That is also why I think having this
 discussion on talk, rather than on github or the dev list is appropriate.

 We need to come to a consensus between all of the main tools (at least iD,
 P2, JOSM, osm2pgsql, osrm, ...) and the mappers to what the preferred,
 encouraged and supported standard for tagging multi-polygons is and make
 sure that all documentation is in line with this.





 --
 View this message in context:
 http://gis.19327.n5.nabble.com/osm2pgsql-multipolygon-parsing-tp5778300p5778654.html
 Sent from the General Discussion mailing list archive at Nabble.com.

 ___
 talk mailing list
 talk@openstreetmap.org
 https://lists.openstreetmap.org/listinfo/talk

 
 
 
 ___
 talk mailing list
 talk@openstreetmap.org
 https://lists.openstreetmap.org/listinfo/talk
 


___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] osm2pgsql multipolygon parsing

[Peter Wendorff]

Am 23.09.2013 18:48, schrieb Yves:
 Sorry, I meant osm2pgsql is not used for the slippy map ONLY.
 Thanks for all the feedback :)
Sure, but changing the DEFAULT behaviour to a more strict one while
keeping the old behaviour with a flag should enable anybody to keep the
old behaviour on demand; and whoever updates a software like this to a
new version should read the release-notes to get this IMHO (or be
patient if something changes).

regards
Peter

___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] osm2pgsql multipolygon parsing

[Pieren]

On Sun, Sep 22, 2013 at 12:29 PM, Peter Wendorff
wendo...@uni-paderborn.de wrote:
 The suggestion is to discourage this in all cases and encourage always
 tagging the relation (this is also straightforward and much easier as you
 can do A or B).
 +1

-1
Check cities with tens of thousands buildings. You will have sometime
the building tag on ways, sometimes on relations. Having the tag
always on the surrounding way is more consistent and easier to catch
for everybody, including newcomers.

Pieren

___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] osm2pgsql multipolygon parsing

[Janko Mihelić]

2013/9/23 Pieren pier...@gmail.com


 -1
 Check cities with tens of thousands buildings. You will have sometime
 the building tag on ways, sometimes on relations. Having the tag
 always on the surrounding way is more consistent and easier to catch
 for everybody, including newcomers.


Not if they use iD. In iD multipolygons and areas are selected in the same
way, by clicking in the building.

Janko
___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] osm2pgsql multipolygon parsing

[Martin Koppenhoefer]

 Am 23/set/2013 um 11:03 schrieb Pieren pier...@gmail.com:
 
 Check cities with tens of thousands buildings. You will have sometime
 the building tag on ways, sometimes on relations. Having the tag
 always on the surrounding way is more consistent and easier to catch
 for everybody, including newcomers.


it has a different meaning. tags on a closed way are for the whole area inside 
the way, tags on a mp relation are for the area of the outer minus the inner 
ways.

Where to put which tag depends on what you want to express.

cheers,
Martin
___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] osm2pgsql multipolygon parsing

[Paul Norman]

 From: Martin Koppenhoefer [mailto:dieterdre...@gmail.com]
 Subject: Re: [OSM-talk] osm2pgsql multipolygon parsing
 
 it has a different meaning. tags on a closed way are for the whole area
 inside the way, tags on a mp relation are for the area of the outer
 minus the inner ways.

Unless the closed way is a member of a multipolygon relation with no other
tags on the relation - then you'll have a resulting area with a hole. This
is a very well established means of tagging areas with holes (~22% of
type=multipolygon relations have no other tags).

The issues raised originally in the ticket are best explored through
examples

The first case is a way with natural=water and a MP relation with no other
tags. Both old osm2pgsql (0.82) and latest master version from git create an
area with a hole with natural=water on the area. There are no suggestions of
changing this.

The second is a way with natural=water and a MP relation with name=foo (with
the way as a member). Old osm2pgsql created an area with a hole with
natural=water, name=foo, latest master does too.

The third is the second with the same as the 2nd, but a name:de tag added.
In old osm2pgsql this created an area without a hole, in latest master it
creates one with a hole.

The fourth is a way with natural=water and a MP relation with foo=bar. In
old osm2pgsql this created an area without a hole, in latest master it
depends on the .style file used for import.

To make #3 consistent with #2 without breaking backwards compatibility
latest osm2pgsql doesn't use a pre-defined list of tags to differentiate
between old-style and new-style MPs, it uses its list of tags that
denote an area.

The specific issue that brought this up is actually fixed by only bringing
tags onto the MP area where all outer members have the tag, but there's
still a more general question.


___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] osm2pgsql multipolygon parsing

[Peter Wendorff]

Am 23.09.2013 11:59, schrieb Paul Norman:
 From: Martin Koppenhoefer [mailto:dieterdre...@gmail.com]
 Subject: Re: [OSM-talk] osm2pgsql multipolygon parsing

 it has a different meaning. tags on a closed way are for the whole area
 inside the way, tags on a mp relation are for the area of the outer
 minus the inner ways.
 
 Unless the closed way is a member of a multipolygon relation with no other
 tags on the relation - then you'll have a resulting area with a hole. This
 is a very well established means of tagging areas with holes (~22% of
 type=multipolygon relations have no other tags).
It's well established, but it's less correct from a theoretical point of
view, and establishment is produced here by software that supports wrong
tagging - in the past.
You propose to leave it as it is, which means software should estimate
the meaning because often there are errors in the tagging, while I
proposed (and as far as I understand at least Martin supports that) not
to ignore these errors with the goal to teach mappers about what's
correct instead of teaching them what's correct enough for application X
(be it the default mapnik style and -pipeline).

 The issues raised originally in the ticket are best explored through
 examples
 
 The first case is a way with natural=water and a MP relation with no other
 tags. Both old osm2pgsql (0.82) and latest master version from git create an
 area with a hole with natural=water on the area. There are no suggestions of
 changing this.
Problem:
Let's say the multipolygon relation was tagged before as depth=shallow
(and someone wanted to mark parts of the water as not deep - e.g. not
deep enough for boats). Let's say another mapper came along and found
this strange tags and - removing them - accidently left a multipolygon
without tags. This changes the meaning of stuff never touched again, as
you would now count the natural=water only for the area covered by the
multipolygon, excluding the hole.

 The second is a way with natural=water and a MP relation with name=foo (with
 the way as a member). Old osm2pgsql created an area with a hole with
 natural=water, name=foo, latest master does too.
What if the inner part does NOT have that name, but only the outer ring?
What if that is what somebody wanted to say?

regards
Peter

___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] osm2pgsql multipolygon parsing

[Petr Morávek [Xificurk]]

Dne 23.9.2013 11:59, Paul Norman napsal(a):
 Unless the closed way is a member of a multipolygon relation with no other
 tags on the relation - then you'll have a resulting area with a hole. This
 is a very well established means of tagging areas with holes (~22% of
 type=multipolygon relations have no other tags).

Yes, but if the relation has any[*] additional tags, you can't reliably
decide what was the intended purpose of this tagging. Imho the only
logical thing is to treat the relation and ways separately.
[*] Maybe some internal keys could be ignored, e.g. odbl, fixme, ...

 The issues raised originally in the ticket are best explored through
 examples
 
 The first case is a way with natural=water and a MP relation with no other
 tags. Both old osm2pgsql (0.82) and latest master version from git create an
 area with a hole with natural=water on the area. There are no suggestions of
 changing this.

Agreed.

 The second is a way with natural=water and a MP relation with name=foo (with
 the way as a member). Old osm2pgsql created an area with a hole with
 natural=water, name=foo, latest master does too.

Is this really correct?
In case of the name=foo it is probably safe assumption, but how about
multipolygon relation with only access=* tag (or something similar)? How
do you decide, if it means the area with holes is
natural=water+access=*, or that the whole area (with no holes) is
natural=water and only some parts have access=*?

 The fourth is a way with natural=water and a MP relation with foo=bar. In
 old osm2pgsql this created an area without a hole, in latest master it
 depends on the .style file used for import.

The dependence on on the .style file bothers me, I think it's a mistake
and will ultimately come back to haunt us. If you don't care about some
features and delete the lines from your .style file, the multipolygon
parsing should not change.

--

I propose that:
1) By default the relation and ways are treated separately
 - relation creates polygon with tags from the relation and ways are
processed on their own.
2) If and only if the relation has only type=multipolygon tag go to
backward compatibility mode. Copy over tags from outer ways that are
present on all of them and create polygon. Go over all member ways and
if all their tags are present on the created polygon, then mark them as
done, otherwise process them separately.

--

I have looked at all the multipolygon relations that do not have any of
the well-known polygon tags (the ones defined in default.style), this is
less than 27% of all the multipolygon relations. 85% of these relations
have type=multipolygon tag only, so the proposed change in fact affects
less than 4% of all the multipolygons.

More detailed breakdown of tags on multipolygons without well-known
polygon tags:
percentage  keys
85.5%   type
3.7%ref,type,id_ob,adr_les
1.7%type,name
1.5%area,type,highway


Best regards,
Petr Morávek aka Xificurk


___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] osm2pgsql multipolygon parsing

[Peter Wendorff]

Am 23.09.2013 15:20, schrieb Petr Morávek [Xificurk]:
 
 I propose that:
 1) By default the relation and ways are treated separately
  - relation creates polygon with tags from the relation and ways are
 processed on their own.
 2) If and only if the relation has only type=multipolygon tag go to
 backward compatibility mode. Copy over tags from outer ways that are
 present on all of them and create polygon. Go over all member ways and
 if all their tags are present on the created polygon, then mark them as
 done, otherwise process them separately.

+0.75 ;)

Agree, but I would use (2) if and only if backward compatibility mode
is active.
Additionally I would not use the backward compatibility mode for the
mapnik default sheet (after some time with big announcements an probably
a tool detecting/reporting possible problems for fixing).

With this we could
- enforce more correct tagging in future
- engage the mappers community to fix where the bad-style tagging is
used yet
- gracefully allow any other application to fall back to the old style
as long as they want to.

This might (!) degrade the perceived quality of the default mapnik layer
for some time (!), but IMHO it's worth it as it simplifies multipolygon
interpretation and on the long term teaches mappers to use the tagging
that is correct and matches that simplified interpretation.

regards
Peter

___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] osm2pgsql multipolygon parsing

[Yves]

Sorry, I meant osm2pgsql is not used for the slippy map ONLY.
Thanks for all the feedback :)
Yves
-- 
Envoyé de mon téléphone Android avec K-9 Mail. Excusez la brièveté.___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] osm2pgsql multipolygon parsing

[Petr Morávek [Xificurk]]

Dne 23.9.2013 16:03, Peter Wendorff napsal(a):
 Am 23.09.2013 15:20, schrieb Petr Morávek [Xificurk]:

 I propose that:
 1) By default the relation and ways are treated separately
  - relation creates polygon with tags from the relation and ways are
 processed on their own.
 2) If and only if the relation has only type=multipolygon tag go to
 backward compatibility mode. Copy over tags from outer ways that are
 present on all of them and create polygon. Go over all member ways and
 if all their tags are present on the created polygon, then mark them as
 done, otherwise process them separately.
 
 +0.75 ;)
 
 Agree, but I would use (2) if and only if backward compatibility mode
 is active.
 Additionally I would not use the backward compatibility mode for the
 mapnik default sheet (after some time with big announcements an probably
 a tool detecting/reporting possible problems for fixing).
 
 With this we could
 - enforce more correct tagging in future
 - engage the mappers community to fix where the bad-style tagging is
 used yet
 - gracefully allow any other application to fall back to the old style
 as long as they want to.
 
 This might (!) degrade the perceived quality of the default mapnik layer
 for some time (!), but IMHO it's worth it as it simplifies multipolygon
 interpretation and on the long term teaches mappers to use the tagging
 that is correct and matches that simplified interpretation.
 
 regards
 Peter

This is probably a good idea in the long-run, but there are things that
need to be done before this cut-off and all of them need a certain
transitional period.
1) You need to edit wiki and make people aware that this tagging scheme
is considered deprecated.
2) You need to make sure, that all the editors produce correct
multipolygon relations.
3) Fix the deprecated tagging on most of the elements.

I really don't think it's a good idea to suddenly stop rendering quarter
of the multipolygons out there just to enforce some thought-out
policy... I don't see who would benefit from this.


Anyway, this thread was not started to debate tagging schemes, the
question I (and others) wanted to discuss here is this:
Given the data that are currently in the database, how should osm2pgsql
handle the import to get as much as possible multipolygons right?


Best regards,
Petr Morávek aka Xificurk

___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] osm2pgsql multipolygon parsing

[Kai Krueger]

quot;Petr Morávek [Xificurk]quot;-2 wrote
 Anyway, this thread was not started to debate tagging schemes, the
 question I (and others) wanted to discuss here is this:
 Given the data that are currently in the database, how should osm2pgsql
 handle the import to get as much as possible multipolygons right?

Indirectly it is a question of tagging schemas. With osm2pgsql being the
tool used in the default map rendering on osm.org and the prevalence of
tagging for the renderer decisions on how it handles multipolygons will
(and imho to a limited degree should) influence how people tag and what they
perceive as correct tagging. Therefore it is important that there is a
consensus of what the correct tagging schema is and make sure that is
correctly supported by osm2pgsql. That is also why I think having this
discussion on talk, rather than on github or the dev list is appropriate.

We need to come to a consensus between all of the main tools (at least iD,
P2, JOSM, osm2pgsql, osrm, ...) and the mappers to what the preferred,
encouraged and supported standard for tagging multi-polygons is and make
sure that all documentation is in line with this.

 



--
View this message in context: 
http://gis.19327.n5.nabble.com/osm2pgsql-multipolygon-parsing-tp5778300p5778654.html
Sent from the General Discussion mailing list archive at Nabble.com.

___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] osm2pgsql multipolygon parsing

[Martin Koppenhoefer]

 Am 22/set/2013 um 04:14 schrieb Eugene Alvin Villar sea...@gmail.com:
 
 It's most likely that these people are not familiar with relations and they 
 see an outer way with no building=yes tag and decided to helpfully tag it.
 
 Because of this, a more complicated interpretation of tags, such as 
 Frederik's, leads to less breakage (think rendering) and is more in line with 
 people's expectations.


it leads to more broken data in the end, because it will continue to look OK on 
the map and many mappers are checking this visual feedback in order to judge 
their mapping.

cheers,
Martin
___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] osm2pgsql multipolygon parsing

[Peter Wendorff]

Am 22.09.2013 04:14, schrieb Eugene Alvin Villar:
 
 I agree that this is a good way of tagging multipolygons.
 
 Unfortunately, many people don't tag multipolygons in this way. I've seen
 people add building=yes to an outer way of a building with holes even
 though there's a multipolygon relation with that tag already. It's most
 likely that these people are not familiar with relations and they see an
 outer way with no building=yes tag and decided to helpfully tag it.
 
 Because of this, a more complicated interpretation of tags, such as
 Frederik's, leads to less breakage (think rendering) and is more in line
 with people's expectations.
 
Because of this I agree with Frederiks approach for any consumer-map
to produce. If I would print a map for a customer, I would follow his
assumptions most probably, but I disagree for the default style(s) on
the osm.org website, as these in fact are driving factors to how people tag.
If people tag multipolygons right (tm), this should be honoured by a
correct display on the map. If it's wrong they should see that there's
an error, therefore IMHO the default stylesheet MUST NOT gracefully
forgive these errors as a map more dedicated to end users would probably do.

Following these assumptions on the main map which is often used to check
the correctness of the own edits means to hide errors and to tell
mappers the wrong thing: you have done all right - where that's not the
case.

regards
Peter

___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] osm2pgsql multipolygon parsing

[yvecai]

Osm2pgsql is not used for the default map on osm.org.
While the current behaviour in osm2pgsql is OK for consumers, could a 
'strict' mode to handle mutipolygons be used on osm.org default map ?


Of course, it should be accompagnied with a large campaign of 
multi-polygons fix.


Yves

___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] osm2pgsql multipolygon parsing

[Martin Koppenhoefer]

2013/9/22 yvecai yve...@gmail.com

 Of course, it should be accompagnied with a large campaign of
 multi-polygons fix.



I'd suggest to start modifying the recommendations in the wiki:
http://wiki.openstreetmap.org/wiki/Multipolygon
reads:
If you have one closed way making up the outer ring and it does not
describe something in its own right, you *may* also put these tags on the
outer ring and leave the relation untagged.

The suggestion is to discourage this in all cases and encourage always
tagging the relation (this is also straightforward and much easier as you
can do A or B).

cheers,
Martin
___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] osm2pgsql multipolygon parsing

[SomeoneElse]

On 22/09/2013 10:03, yvecai wrote:


Of course, it should be accompagnied with a large campaign of 
multi-polygons fix.


... and a patch to any editors that don't create multipolygons in this 
format.  For example, here are three attempts at multipolygons in iD, P2 
and JOSM:


http://api06.dev.openstreetmap.org/#map=15/53.2692/-0.2806layers=D

(you may need to turn the data layer on manually to see the them)

The left-hand one was me trying to create one in iD (I've not worked out 
how to do that yet).  The middle one is P2, and the right-hand one is JOSM.


You'll notice that the right-hand one doesn't render in P2 despite being 
the arguably the more logically-tagged.  I suspect that this is and has 
been for some time a patches welcome type of situation.


Cheers,

Andy


___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] osm2pgsql multipolygon parsing

[Peter Wendorff]

Am 22.09.2013 11:31, schrieb Martin Koppenhoefer:
 2013/9/22 yvecai yve...@gmail.com
 
 Of course, it should be accompagnied with a large campaign of
 multi-polygons fix.

 
 
 I'd suggest to start modifying the recommendations in the wiki:
 http://wiki.openstreetmap.org/wiki/Multipolygon
 reads:
 If you have one closed way making up the outer ring and it does not
 describe something in its own right, you *may* also put these tags on the
 outer ring and leave the relation untagged.
 
 The suggestion is to discourage this in all cases and encourage always
 tagging the relation (this is also straightforward and much easier as you
 can do A or B).

+1

regards
Peter

___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

[OSM-talk] osm2pgsql multipolygon parsing

[Petr Morávek [Xificurk]]

Hello,

I've run into some problems with osm2pgsql parsing of multipolygon
relations, so I've opened an issue on github [1]. It turned out that the
behavior was recently changed.

The remaining question is, what should be the correct behavior?

The question is not easy to answer, because there are many multipolygon
relations with deprecated tagging scheme - having the relevant tags on
member ways instead of directly on the relation. Thus, osm2pgsql needs
to do some kind of magic to determine what tags were meant for the
multipolygon and what tags were meant for the individual ways only.

There is probably no perfect solution, but we need to decide which of
the imperfect ones will be considered the correct one ;-) So, what is
your opinion on this?


Best regards,
Petr Morávek aka Xificurk

[1] https://github.com/openstreetmap/osm2pgsql/issues/80

___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] osm2pgsql multipolygon parsing

[Frederik Ramm]

Hi,

 The remaining question is, what should be the correct behavior?

My assumption until now was:

* If a multipolygon is untagged - where untagged means that it has no
tags except a small list (type, source, source:*, note) then it will
simply receive *all* tags from all (outer) member ways, but processing
will fail if these tags differ in values.

* If the multipolygon is tagged (with anything except type, source, note
etc) then those are the tags for the polygon and way tags will not be
looked at.

* If the tags of the inner way differ from the tags of the outer way or
the tags of the relation, except source, note etc., then the inner way
will become a separate polygon.

All this was without trying to make a judgement about which tags are
polygon tags and which aren't.

Bye
Frederik

-- 
Frederik Ramm  ##  eMail frede...@remote.org  ##  N49°00'09 E008°23'33

___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] osm2pgsql multipolygon parsing

[Martin Koppenhoefer]

2013/9/21 Frederik Ramm frede...@remote.org

 Hi,

  The remaining question is, what should be the correct behavior?

 My assumption until now was:

 * If a multipolygon is untagged - where untagged means that it has no
 tags except a small list (type, source, source:*, note) then it will
 simply receive *all* tags from all (outer) member ways, but processing
 will fail if these tags differ in values.



but this introduces a lot of complexity, especially in the case of MPs with
many outer members, when you only want to modify the tags of a single one
of these (outer) ways. Many editors (e.g. on smartphones) will not even
notify you that the way is part of a relation.



 * If the multipolygon is tagged (with anything except type, source, note
 etc) then those are the tags for the polygon and way tags will not be
 looked at.



do you also include name in this list?



 * If the tags of the inner way differ from the tags of the outer way or
 the tags of the relation, except source, note etc., then the inner way
 will become a separate polygon.



all tags, or only the relevant ones? And if yes, which ones aren't
relevant?
And also here there might be cases where e.g. inside a forest you will want
to have another forest because of some properties they don't share (be it
name but also note etc.)

cheers,
Martin
___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] osm2pgsql multipolygon parsing

[Peter Wendorff]

IMHO it's clear:
- a tag on a way describes that way, if it's a closed way and the tag is
describing an area, the tag matches the complete area inside that polygon.
- if a way is outer of a multipolygon and there are tags on the way,
these tags nevertheless describe the whole area, including all holes, as
it's still a tag on the (simple) polygon.
- if a way is inner of a multipolygon and there are tags on the way,
these tags describe the polygon described by this way.
- tags on the multipolygon relation describe the area represented by the
relation - that is all outer polygons minus the inner polygons.

If we follow this assumption, it's
1) not conflicting with double-tagging, which is in fact often useful.
Example: A big park with a lake in the middle of a big lawn.
- the whole area (including lawn and lake) is a park = the outer
polygon is tagged as a park
- only the small patch in the middle is a lake = the inner polygon is
tagged as a lake
- the whole area except the lake surface is a lawn = tag the
multipolygon as lawn.
Tags:
outer: {leisure=park}
inner: {natural=water}
mp:{landuse=grass}

2) it's unambiguous
Following Frederiks assumptions instead my tagging of the example above
would be interpreted as:
- area of the multipolygon (outer minus inner): leisure=park
- inner: natural=water
- whole area: no attributes (am I right?)

Thus the park is missing now (or Frederiks description is incomplete).

An even better example is one where the inner has a subset of the
multipolygon tags, let's say:
mp: {landuse=forest, wood=coniferous}
inner: {landuse=forest}

Frederiks assumption lead here to:
- mp: landuse=forest, wood=coniferous
- inner: untagged, as the tags are ignored because they don't differ
from the mp.

regards
Peter


Am 21.09.2013 17:00, schrieb Frederik Ramm:
 Hi,
 
 The remaining question is, what should be the correct behavior?
 
 My assumption until now was:
 
 * If a multipolygon is untagged - where untagged means that it has no
 tags except a small list (type, source, source:*, note) then it will
 simply receive *all* tags from all (outer) member ways, but processing
 will fail if these tags differ in values.
 
 * If the multipolygon is tagged (with anything except type, source, note
 etc) then those are the tags for the polygon and way tags will not be
 looked at.
 
 * If the tags of the inner way differ from the tags of the outer way or
 the tags of the relation, except source, note etc., then the inner way
 will become a separate polygon.
 
 All this was without trying to make a judgement about which tags are
 polygon tags and which aren't.
 
 Bye
 Frederik
 


___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] osm2pgsql multipolygon parsing

[Eugene Alvin Villar]

On Sun, Sep 22, 2013 at 4:51 AM, Peter Wendorff
wendo...@uni-paderborn.dewrote:

 IMHO it's clear:
 - a tag on a way describes that way, if it's a closed way and the tag is
 describing an area, the tag matches the complete area inside that polygon.
 - if a way is outer of a multipolygon and there are tags on the way,
 these tags nevertheless describe the whole area, including all holes, as
 it's still a tag on the (simple) polygon.
 - if a way is inner of a multipolygon and there are tags on the way,
 these tags describe the polygon described by this way.
 - tags on the multipolygon relation describe the area represented by the
 relation - that is all outer polygons minus the inner polygons.


I agree that this is a good way of tagging multipolygons.

Unfortunately, many people don't tag multipolygons in this way. I've seen
people add building=yes to an outer way of a building with holes even
though there's a multipolygon relation with that tag already. It's most
likely that these people are not familiar with relations and they see an
outer way with no building=yes tag and decided to helpfully tag it.

Because of this, a more complicated interpretation of tags, such as
Frederik's, leads to less breakage (think rendering) and is more in line
with people's expectations.
___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] osm2pgsql import failed, no more space on device

[Frederik Ramm]

Hi,

On 09/07/2013 08:48 AM, Torsten Mohr wrote:
 I think --flat-nodes needs a parameter, but i'm not sure what to give there. 

A file name.

 I can't imagine that 480 Gb is not enough, can anybody tell me what went 
 wrong?

Are you *sure* that your PostgreSQL table space is actually on the 480
GB disk? Have you monitored disk usage during the import?

Bye
Frederik


-- 
Frederik Ramm  ##  eMail frede...@remote.org  ##  N49°00'09 E008°23'33

___
talk mailing list
talk@openstreetmap.org
http://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] Osm2pgsql/TileMill - how to detect whether way is in a route?

[Phil! Gold]

* Steve Bennett stevag...@gmail.com [2013-03-19 10:52 +1100]:
 One way to fudge this is to render the way, then render the route
 relation over the top, obscuring it. But that doesn't work if you want
 to use line smoothing (eg, line-smooth:0.6). The route relation will
 form a smooth bezier curve the intersection of two ways, but the way
 rendering will form a sharp point.
 
 So I'm just wondering if there is a way around this - some setting in
 osm2pgsql, some SQL magic I could play with.

I do something very similar with my highway shield rendering, which is
driven by route relations.  I go through the planet_osm_rels table that's
created by osm2pgsql's slim mode, which is considered cheating in some
circles.

My rendering has code at https://launchpad.net/osm-shields , but the meat
of what you're interested in is the following two functions:

-- osm2pgsql stores tags in PostgreSQL arrays, where even-numbered array
-- members are tags and odd-numbered members are the tags' values.  This
-- is a convenience function to go through such an array and retrieve the
-- value for a given tag.
CREATE OR REPLACE FUNCTION osm_get_tag_value(tags TEXT[], tag TEXT)
RETURNS TEXT
IMMUTABLE
RETURNS NULL ON NULL INPUT
LANGUAGE plpgsql
AS $$
DECLARE
  tag_index INTEGER;
BEGIN
  SELECT i INTO tag_index
FROM (SELECT generate_subscripts(tags, 1) AS i) AS indices
WHERE i % 2 = 1 AND tags[i] = tag;
  RETURN tags[tag_index + 1];
END;
$$;

CREATE OR REPLACE FUNCTION 
shields_get_route_memberships(planet_osm_rels.id%TYPE)
RETURNS fullref[]
STABLE
RETURNS NULL ON NULL INPUT
COST 5000
LANGUAGE SQL
AS $$
  SELECT 
shields_replace_groups(array_agg(ROW(shields_sanitize_text(network), 
shields_sanitize_text(ref))::fullref))
FROM (SELECT osm_get_tag_value(tags, 'route') route,
 UPPER(osm_get_tag_value(tags, 'network')) network,
 UPPER(COALESCE(NULLIF(osm_get_tag_value(tags, 'ref'), ''), 
osm_get_tag_value(tags, 'name'))) AS ref
FROM planet_osm_rels
WHERE parts  ARRAY[$1]
  AND parts[way_off:array_upper(parts, 1)]  ARRAY[$1]) AS 
ref_inner
WHERE route = 'road' AND network IS NOT NULL AND ref IS NOT NULL;
$$;

There's a lot of shield-rendering-specific code in there.  A more succinct
query might be the following (where $1 is the OSM ID of the way for which
you want to determine route membership):

SELECT *
  FROM (SELECT osm_get_tag_value(tags, 'route') route,
   osm_get_tag_value(tags, 'network') network,
   osm_get_tag_value(tags, 'ref') ref,
   osm_get_tag_value(tags, 'name') name
  FROM planet_osm_rels
  WHERE parts  ARRAY[$1]
AND parts[way_off:array_upper(parts, 1)]  ARRAY[$1]) AS 
ref_inner
  WHERE route IS NOT NULL;

Obviously, you can add your own parameters in the statement.

___
talk mailing list
talk@openstreetmap.org
http://lists.openstreetmap.org/listinfo/talk

[OSM-talk] Osm2pgsql/TileMill - how to detect whether way is in a route?

[Steve Bennett]

Hi,
  I'm trying to render a mountain bike map with TileMill. Some trails
are in route relations, eg:

3xWay: highway=path, mtb=yes
1xRelation: type=route, route=mtb, name=My Cool Trail

Osm2pgsql converts those to 4 rows: the relation is the complete trail.

What I want to do is not render any way that is also part of a route
relation - in that case I just want to render the route.

One way to fudge this is to render the way, then render the route
relation over the top, obscuring it. But that doesn't work if you want
to use line smoothing (eg, line-smooth:0.6). The route relation will
form a smooth bezier curve the intersection of two ways, but the way
rendering will form a sharp point.

So I'm just wondering if there is a way around this - some setting in
osm2pgsql, some SQL magic I could play with. The ultimate goal would
be to be able to write rules like:


#trails[highway=path][mb=yes][in_mtb_route=no] {
...
}

Thanks,
Steve

___
talk mailing list
talk@openstreetmap.org
http://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] Osm2pgsql/TileMill - how to detect whether way is in a route?

[Frederik Ramm]

Hi,

On 19.03.2013 00:52, Steve Bennett wrote:

So I'm just wondering if there is a way around this - some setting in
osm2pgsql, some SQL magic I could play with.


It will be very slow magic, but two things come to mind:

One would use the line geometry table and ask for routes in the vicinity 
to be subtracted from the way geometry; if the resulting geometry is 
empty then your way was part of a route. In pseudo SQL something like


select way,attribute,attribute,attribute,st_isempty(st_difference(way, 
(select way from planet_osm_roads t2 where t2.osm_id0 and 
t2.has_mtb_route_tags and t2.way  way) as in_mtb_route from 
planet_osm_ways...


The other would be using the planet_osm_rels table that only exists ins 
slim mode and from there get a list of all relations the way in question 
is a member of, check those relations for mtb routes...


The only *efficient* way I can think of is extending osm2pgsql so that 
it sets an extra flag for ways that are part of a route relation.


Bye
Frederik

--
Frederik Ramm  ##  eMail frede...@remote.org  ##  N49°00'09 E008°23'33

___
talk mailing list
talk@openstreetmap.org
http://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] osm2pgsql hstore (was: Wind turbines no longer rendered on mapnik layer)

[Nathan Edgars II]

On 2/16/2012 6:00 PM, Jochen Topf wrote:

Generic key names can be confusing, especially when one OSM object is used for
multiple things. Say there is a way tagged as railway and at the same time this
way is part of an area tagged as a generating station. Does power_source mean
the type of generating station or the type of power used by the railway
(overhead vs. third rail vs. unelectrified)? I hope this example is
hypothetical, but people do strange things in OSM...


A better example is a highway=service service=alley with a railway=rail 
service=spur down the middle.


___
talk mailing list
talk@openstreetmap.org
http://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] osm2pgsql hstore (was: Wind turbines no longer rendered on mapnik layer)

[Jochen Topf]

On Fri, Feb 17, 2012 at 07:44:41AM +, Jukka Rahkonen wrote:
 Frederik Ramm frederik at remote.org writes:
 
  
  Hi,
  
  On 02/16/2012 07:25 PM, Graham Jones wrote:
   This reminded me of a question I have been meaning to ask for quite a
   while - is a database generated by osm2pgsql with an hstore expected to
   perform similarly to one without?
  
  I never ran one with hstore when I think of what this must mean for the 
  database engine, and storage space, then I shudder and would not be 
  surprised by the factor 5 you mentioned.
 
 How about doing it with relations? Let the importer program create four 
 tables 
 osm_point
 osm_line
 osm_polygon
 osm_relation
 
 These tables would each have two attributes: Geometry and osm_is. Geometries 
 in
 the osm_relation would be of type geometry collection, that is, a collection
 of whatever, and it could hold for example the tranport route relation with 
 the
 route and bus stops and everything in one PostGIS geometry object.
 
 Then there should be one or four tables for tags (everything in one table or
 split to suit the geometry tables). The three attributes would be osm_id, key,
 and value. Osm_id would be used as a foreing key for joining geometries and
 attributes. If osm_id can not be guaranteed to be unique then there should be
 point_id, line_id, polygon_id and relation_id added to suitable places.
 
 Every attribute would be indexed. For Mapnik use where all that is needed is 
 to
 do simple SQL selects.  I guess that this database would be faster to query 
 than
 hstore even it also contains all the tags. It might be faster for Mapnik than
 the current tables with extra wide attribute schema because now attributes are
 not indexed at all (or are they?). SQL queries according to tags and values or
 even with a part of the tag and value strings would be easier to do than from
 hstore. People could enhance attributes for some special uses by converting
 max_speeds and other values which are actually measures from strings to 
 integers
 or doubles etc.
 
 This option is so obvious that I believe that someone must have tried it
 already. It would be nice to hear about experiences.

Much too slow. Requiring joins on every query is not a good idea. And having
the right indexes is important. You can't just index everything and hope it
would do the right thing. (The most important index btw is the geometry index
not the attribute indexes, although having the right attribute indexes here
or there can help.)

Jochen
-- 
Jochen Topf  joc...@remote.org  http://www.remote.org/jochen/  +49-721-388298


___
talk mailing list
talk@openstreetmap.org
http://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] osm2pgsql hstore (was: Wind turbines no longer rendered on mapnik layer)

[Komяpa]

 - is a database generated by osm2pgsql with an hstore expected to perform
 similarly to one without?

It performs at the same speed for me.

Of course you're not supposed to use hstore for every tag, but just
for those on highest zoom levels, where spatial indexes are used
mostly, not indexes for other columns.

-- 
Darafei Komяpa Praliaskouski
OSM BY Team - http://openstreetmap.by/
xmpp:m...@komzpa.net mailto:m...@komzpa.net

___
talk mailing list
talk@openstreetmap.org
http://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] osm2pgsql hstore (was: Wind turbines no longer rendered on mapnik layer)

[Stephan Knauss]

Komяpa writes: 


- is a database generated by osm2pgsql with an hstore expected to perform
similarly to one without?


It performs at the same speed for me. 


Of course you're not supposed to use hstore for every tag, but just
for those on highest zoom levels, where spatial indexes are used
mostly, not indexes for other columns.


if you are using Postgres 9.1 you are advised to update to 9.1.2. There was 
a bug introduced that prevented index usage. For example testing for a tag 
like

SELECT * from table WHERE hstore ? tag
did not use an index and could make a query horribly slow. We experienced 
this on Toolserver. 



Stephan

___
talk mailing list
talk@openstreetmap.org
http://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] osm2pgsql hstore (was: Wind turbines no longer rendered on mapnik layer)

[Graham Jones]

Thanks - I will give it another try - I have got a computer running Ubuntu
11.10, which has Postgres 9.1.2.   I will create a second database with an
hstore and see how it compares.

Graham.

On 17 February 2012 11:36, Stephan Knauss o...@stephans-server.de wrote:

 Komяpa writes:

 - is a database generated by osm2pgsql with an hstore expected to perform
 similarly to one without?


 It performs at the same speed for me.
 Of course you're not supposed to use hstore for every tag, but just
 for those on highest zoom levels, where spatial indexes are used
 mostly, not indexes for other columns.


 if you are using Postgres 9.1 you are advised to update to 9.1.2. There
 was a bug introduced that prevented index usage. For example testing for a
 tag like
 SELECT * from table WHERE hstore ? tag
 did not use an index and could make a query horribly slow. We experienced
 this on Toolserver.

 Stephan




-- 
Graham Jones
Hartlepool, UK.
___
talk mailing list
talk@openstreetmap.org
http://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] osm2pgsql hstore (was: Wind turbines no longer rendered on mapnik layer)

[Jukka Rahkonen]

Jochen Topf jochen at remote.org writes:


 
 Much too slow. Requiring joins on every query is not a good idea. And having
 the right indexes is important. You can't just index everything and hope it
 would do the right thing. (The most important index btw is the geometry index
 not the attribute indexes, although having the right attribute indexes here
 or there can help.)
 
I sent by followup accidentally to osm-dev list. The main message was that
perhaps there is a need to do some tests before saying if hstore or joined
tables are faster.

Here is a Spatialite database for testing

http://latuviitta.org/documents/relation_test.zip

First trials suggest that joined tables with proper indexes are not at all
slow and compared with the standard tables created by osm2pgsql the query
times can be several times faster. This is not a fair comparison because
osm2pgsql tables are missing the attribute indexes but so they normally do
in the Mapnik chain. Query times vary, so try yourself. Data is osm_lines
from Berlin from some months ago.

By the way, recent development around GDAL suggests that Spatialite may be
faster with spatial and attribute queries than PostGIS. Improvements in GDAL
Spatialite driver made Mapserver about 20 times faster than it used to be and
now Spatialite is faster that PostGIS and even shapefiles.


query from osm_line from osm2pgsql
===
select geometry, highway from osm_line
where highway='primary'
0.531 seconds

query from joined tables
===
select * from line_geometry, tags
where tags.tag='highway' and tags.value='primary'
and tags.join_id=line_geometry.osm_id
0.059 seconds


-Jukka Rahkonen-


___
talk mailing list
talk@openstreetmap.org
http://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] osm2pgsql hstore (was: Wind turbines no longer rendered on mapnik layer)

[Jochen Topf]

On Fri, Feb 17, 2012 at 02:02:26PM +, Jukka Rahkonen wrote:
 Jochen Topf jochen at remote.org writes:
 
 
  
  Much too slow. Requiring joins on every query is not a good idea. And having
  the right indexes is important. You can't just index everything and hope it
  would do the right thing. (The most important index btw is the geometry 
  index
  not the attribute indexes, although having the right attribute indexes here
  or there can help.)
  
 I sent by followup accidentally to osm-dev list. The main message was that
 perhaps there is a need to do some tests before saying if hstore or joined
 tables are faster.
 
 Here is a Spatialite database for testing
 
 http://latuviitta.org/documents/relation_test.zip
 
 First trials suggest that joined tables with proper indexes are not at all
 slow and compared with the standard tables created by osm2pgsql the query
 times can be several times faster. This is not a fair comparison because
 osm2pgsql tables are missing the attribute indexes but so they normally do
 in the Mapnik chain. Query times vary, so try yourself. Data is osm_lines
 from Berlin from some months ago.

Sorry, but that kind of test doesn't really tell you anything about real
performance on a real system. Do this test with data for the whole planet with
one process updating the database and multiple readers accessing the db at the
same time. And results you get with sqlite do not tell you anything about
results for postgresql.

 By the way, recent development around GDAL suggests that Spatialite may be
 faster with spatial and attribute queries than PostGIS. Improvements in GDAL
 Spatialite driver made Mapserver about 20 times faster than it used to be and
 now Spatialite is faster that PostGIS and even shapefiles.
 
 
 query from osm_line from osm2pgsql
 ===
 select geometry, highway from osm_line
 where highway='primary'
 0.531 seconds
 
 query from joined tables
 ===
 select * from line_geometry, tags
 where tags.tag='highway' and tags.value='primary'
 and tags.join_id=line_geometry.osm_id
 0.059 seconds

These numbers are also irrelevant for our discussion here because Mapnik
doesn't do this kinds of queries. Mapnik always queries with a bounding box.

Jochen
-- 
Jochen Topf  joc...@remote.org  http://www.remote.org/jochen/  +49-721-388298


___
talk mailing list
talk@openstreetmap.org
http://lists.openstreetmap.org/listinfo/talk

[OSM-talk] osm2pgsql hstore (was: Wind turbines no longer rendered on mapnik layer)

[Graham Jones]

On 16 February 2012 16:25, Frederik Ramm frede...@remote.org wrote:

 Hi,


 On 02/16/2012 03:05 PM, Kay Drangmeister wrote:

 Isn't there a hstore in the rendering-db?


 No.

 This reminded me of a question I have been meaning to ask for quite a
while - is a database generated by osm2pgsql with an hstore expected to
perform similarly to one without?

The only time I tried to use hstore, the performance of the whole database
was dreadful (at least a factor of 5 slower, but I didn't do any proper
benchmarking - I just gave up and re-created it without hstore and the
problem went away).It may have been that something went wrong with the
import that I did not notice, and it was not an hstore issue.

Using hstore would be really good given the fashion for generating new keys
with colons in them, but not with that reduction in performance, so I keep
having to re-import my database to add new keys.

grumble Why create a key generator:power_source rather than just use
power_source.  power_source is much more generic so you could re-cycle it
for things like district heating, but generator:power_source is only ever
going to be used for generating stations, and needs a new column in the
database. /grumble.   I think I just prefer more generic, re-usable keys
rather than trying to invent a new one for each situation

Regards


Graham.

-- 
Graham Jones
Hartlepool, UK.
___
talk mailing list
talk@openstreetmap.org
http://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] osm2pgsql hstore (was: Wind turbines no longer rendered on mapnik layer)

[Frederik Ramm]

Hi,

On 02/16/2012 07:25 PM, Graham Jones wrote:

This reminded me of a question I have been meaning to ask for quite a
while - is a database generated by osm2pgsql with an hstore expected to
perform similarly to one without?


I never ran one with hstore when I think of what this must mean for the 
database engine, and storage space, then I shudder and would not be 
surprised by the factor 5 you mentioned.



I keep having to re-import my database to add new keys.


If you want to add a new key but avoid re-importing the full database, 
you can add the key to osm2pgsql's style file, somehow generate an .osm 
file that contains only the objects that have this tag, wrap this .osm 
file into a osmChangemodify.../modify/osmChange instead of 
osm.../osm and throw it at osm2pgsql in append mode.



power_source is much more generic so you could re-cycle
it for things like district heating, but generator:power_source is only
ever going to be used for generating stations, and needs a new column in
the database.


Yes, that's the plan I believe. People like keys that are understandable 
without context. Contrast this with a tag width=2 - where you have no 
chance in knowing what it means without looking at the whole thing.


Bye
Frederik

--
Frederik Ramm  ##  eMail frede...@remote.org  ##  N49°00'09 E008°23'33

___
talk mailing list
talk@openstreetmap.org
http://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] osm2pgsql hstore (was: Wind turbines no longer rendered on mapnik layer)

[Graham Jones]

On 16 February 2012 18:51, Frederik Ramm frede...@remote.org wrote:

 On 02/16/2012 07:25 PM, Graham Jones wrote:

 This reminded me of a question I have been meaning to ask for quite a
 while - is a database generated by osm2pgsql with an hstore expected to
 perform similarly to one without?


 I never ran one with hstore when I think of what this must mean for the
 database engine, and storage space, then I shudder and would not be
 surprised by the factor 5 you mentioned.


Thanks - I don't feel so bad about giving up so easily now!

If you want to add a new key but avoid re-importing the full database, you
 can add the key to osm2pgsql's style file, somehow generate an .osm file
 that contains only the objects that have this tag, wrap this .osm file into
 a osmChangemodify.../**modify/osmChange instead of
 osm.../osm and throw it at osm2pgsql in append mode.


That's a neat idea - will try that next time, thanks.


 People like keys that are understandable without context. Contrast this
 with a tag width=2 - where you have no chance in knowing what it means
 without looking at the whole thing.


Yes - I think it is just a style difference - I am quite happy to interpret
it from context for the sake of having a smaller number of unique keys to
think about.

Cheers

Graham

-- 
Graham Jones
Hartlepool, UK.
___
talk mailing list
talk@openstreetmap.org
http://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] osm2pgsql hstore (was: Wind turbines no longer rendered on mapnik layer)

[Lennard]

On 16-2-2012 19:25, Graham Jones wrote:


grumble Why create a key generator:power_source rather than just use
power_source.  power_source is much more generic so you could re-cycle
it for things like district heating, but generator:power_source is only
ever going to be used for generating stations, and needs a new column in
the database. /grumble.   I think I just prefer more generic,
re-usable keys rather than trying to invent a new one for each situation


This is a core problem with the public_transport=* scheme as well. This 
tag in and of itself is fine, but the whole additional shebang that is 
train=yes/no, bus=yes/no, subway=yes/no, etc etc is so dreadful (in the 
context of the rendering chain) that every time I read #2798 I feel the 
urge to run away screaming.


http://trac.openstreetmap.org/ticket/2798

--
Lennard

___
talk mailing list
talk@openstreetmap.org
http://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] osm2pgsql hstore (was: Wind turbines no longer rendered on mapnik layer)

[Jochen Topf]

On Thu, Feb 16, 2012 at 06:25:30PM +, Graham Jones wrote:
 grumble Why create a key generator:power_source rather than just use
 power_source.  power_source is much more generic so you could re-cycle it
 for things like district heating, but generator:power_source is only ever
 going to be used for generating stations, and needs a new column in the
 database. /grumble.   I think I just prefer more generic, re-usable keys
 rather than trying to invent a new one for each situation

Generic key names can be confusing, especially when one OSM object is used for
multiple things. Say there is a way tagged as railway and at the same time this
way is part of an area tagged as a generating station. Does power_source mean
the type of generating station or the type of power used by the railway
(overhead vs. third rail vs. unelectrified)? I hope this example is
hypothetical, but people do strange things in OSM...

Non-generic tags also make tools such as Taginfo more useful, because each tag
stands on its own. They make it immediately clear if the tagging is incomplete.
If the generator tag got lost, how do we know what the power_source tag
is supposed to mean?

So on the one side you have generic tags like type and class and id which
are very confusing. You never know what they belong to. On the other side you
have rather complex tagging schemas like in your generator:... example. Thats
sometimes hard to use, too. We still have to figure out where the sweet spot is,
but I tend to prefer the more explicit keys.

Jochen
-- 
Jochen Topf  joc...@remote.org  http://www.remote.org/jochen/  +49-721-388298


___
talk mailing list
talk@openstreetmap.org
http://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] osm2pgsql hstore (was: Wind turbines no longer rendered on mapnik layer)

[Martin Koppenhoefer]

2012/2/17 Jochen Topf joc...@remote.org:
 On Thu, Feb 16, 2012 at 06:25:30PM +, Graham Jones wrote:
 Generic key names can be confusing, especially when one OSM object is used for
 multiple things.


+1.
E.g. an object tagged barrier=fence, height=2, landuse=forest. In this
example you could also use 2 objects: a closed way for the fence and a
multipolygon for the area. Linear objects we can create with route
relations (instead of overlapping ways). We don't have currently a
relation for nodes and maybe we don't need this utterly (you could
place another node nearby), but it would solve some cases where you
want to state the topology precisely (e.g. 2 objects at the same
pole). This way we can have things (expressed by a relation) which
have their geometry only as an attribute, instead of a geometry that
_is_ something.


 Non-generic tags also make tools such as Taginfo more useful, because each tag
 stands on its own. They make it immediately clear if the tagging is 
 incomplete.
 If the generator tag got lost, how do we know what the power_source tag
 is supposed to mean?


+1 for taginfo, but I don't think we need the namespaces for the case
something gets lost (there is the history for this).

Cheers,
Martin

___
talk mailing list
talk@openstreetmap.org
http://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] osm2pgsql hstore (was: Wind turbines no longer rendered on mapnik layer)

[Jukka Rahkonen]

Frederik Ramm frederik at remote.org writes:

 
 Hi,
 
 On 02/16/2012 07:25 PM, Graham Jones wrote:
  This reminded me of a question I have been meaning to ask for quite a
  while - is a database generated by osm2pgsql with an hstore expected to
  perform similarly to one without?
 
 I never ran one with hstore when I think of what this must mean for the 
 database engine, and storage space, then I shudder and would not be 
 surprised by the factor 5 you mentioned.

How about doing it with relations? Let the importer program create four tables 
osm_point
osm_line
osm_polygon
osm_relation

These tables would each have two attributes: Geometry and osm_is. Geometries in
the osm_relation would be of type geometry collection, that is, a collection
of whatever, and it could hold for example the tranport route relation with the
route and bus stops and everything in one PostGIS geometry object.

Then there should be one or four tables for tags (everything in one table or
split to suit the geometry tables). The three attributes would be osm_id, key,
and value. Osm_id would be used as a foreing key for joining geometries and
attributes. If osm_id can not be guaranteed to be unique then there should be
point_id, line_id, polygon_id and relation_id added to suitable places.

Every attribute would be indexed. For Mapnik use where all that is needed is to
do simple SQL selects.  I guess that this database would be faster to query than
hstore even it also contains all the tags. It might be faster for Mapnik than
the current tables with extra wide attribute schema because now attributes are
not indexed at all (or are they?). SQL queries according to tags and values or
even with a part of the tag and value strings would be easier to do than from
hstore. People could enhance attributes for some special uses by converting
max_speeds and other values which are actually measures from strings to integers
or doubles etc.

This option is so obvious that I believe that someone must have tried it
already. It would be nice to hear about experiences.

-Jukka Rahkonen-


___
talk mailing list
talk@openstreetmap.org
http://lists.openstreetmap.org/listinfo/talk

[OSM-talk] osm2pgsql - some tables missing

[Kenneth Gonsalves]

hi,

on importing data with osm2pgsql, there is no error, but some tables are
missing:

gis=# \dt
   List of relations
 Schema |Name| Type  |  Owner  
++---+-
 public | geometry_columns   | table | gisuser
 public | planet_osm_line| table | gisuser
 public | planet_osm_point   | table | gisuser
 public | planet_osm_polygon | table | gisuser
 public | planet_osm_roads   | table | gisuser
 public | spatial_ref_sys| table | gisuser
(6 rows)

any idea as to what step I have missed?
-- 
regards
Kenneth Gonsalves


___
talk mailing list
talk@openstreetmap.org
http://lists.openstreetmap.org/listinfo/talk

[OSM-talk] osm2pgsql - some tables missing

[kenneth gonsalves]

hi,

on importing data with osm2pgsql, there is no error, but some tables are
missing:

gis=# \dt
   List of relations
 Schema |Name| Type  |  Owner  
++---+-
 public | geometry_columns   | table | gisuser
 public | planet_osm_line| table | gisuser
 public | planet_osm_point   | table | gisuser
 public | planet_osm_polygon | table | gisuser
 public | planet_osm_roads   | table | gisuser
 public | spatial_ref_sys| table | gisuser
(6 rows)

any idea as to what step I have missed?
-- 
regards
Kenneth Gonsalves



___
talk mailing list
talk@openstreetmap.org
http://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] osm2pgsql - some tables missing

[Frederik Ramm]

Hi,

On 08/10/2011 09:54 AM, kenneth gonsalves wrote:

on importing data with osm2pgsql, there is no error, but some tables are
missing:


This is the normal set of tables that you get when importing without 
--slim. It is fully sufficient for rendering. Only if you want to do 
incremental updates will you need the _nodes, _ways, and _rels tables.


Bye
Frederik

___
talk mailing list
talk@openstreetmap.org
http://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] osm2pgsql - some tables missing

[kenneth gonsalves]

On Wed, 2011-08-10 at 10:00 +0200, Frederik Ramm wrote:
 On 08/10/2011 09:54 AM, kenneth gonsalves wrote:
  on importing data with osm2pgsql, there is no error, but some tables
 are
  missing:
 
 This is the normal set of tables that you get when importing without 
 --slim. It is fully sufficient for rendering. Only if you want to do 
 incremental updates will you need the _nodes, _ways, and _rels
 tables. 

thanks - got my tables.


___
talk mailing list
talk@openstreetmap.org
http://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] osm2pgsql for 64-bit

[Stephan Knauss]

On 09.06.2011 16:10, Saphy Mo wrote:

For windows 64 there is need of
  osm2pgsql for 64-bit.
is there any version of that available for users?


a while ago I tried a cross compile and failed on the needed include 
library from postgres 8.3. I was not able to create a 64bit lib of it.


With postgres 9 it should be possible as this is also available as 64 
bit on windows.


Meanwhile you can patch the 32 bit executable to be large address aware. 
This gives you 4G to use.


Stephan

___
talk mailing list
talk@openstreetmap.org
http://lists.openstreetmap.org/listinfo/talk

[OSM-talk] osm2pgsql for 64-bit

[Saphy Mo]

For windows 64 there is need of 

osm2pgsql for 64-bit.is there any version of that available for users?

regards,
Saphy___
talk mailing list
talk@openstreetmap.org
http://lists.openstreetmap.org/listinfo/talk

[OSM-talk] osm2pgsql error in Oxford/Cotswolds .osm data 19/12/09

[Nick Whitelegg]

Hello everyone,

I'm having problems loading in a slab of OSM data for the Oxford/Cotswolds area 
for the UK extract for 19/12/09 downloaded from geofabrik.de.

osm2pgsql stops with the error Error allocating ways (no other info is 
produced)

The relevant file is 
http://nick.dev.openstreetmap.org/downloads/planet/oxford-cotswolds-191209.osm.bz2.

Has anyone else seen problems loading data from the Oxford/Cotswolds area from 
recent planets?

Thanks,
Nick
___
talk mailing list
talk@openstreetmap.org
http://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] osm2pgsql error in Oxford/Cotswolds .osm data 19/12/09

[Jon Burgess]

On Tue, 2009-12-22 at 17:21 +, Nick Whitelegg wrote:
 Hello everyone,
 
 I'm having problems loading in a slab of OSM data for the Oxford/Cotswolds 
 area for the UK extract for 19/12/09 downloaded from geofabrik.de.
 
 osm2pgsql stops with the error Error allocating ways (no other info is 
 produced)
 
 The relevant file is 
 http://nick.dev.openstreetmap.org/downloads/planet/oxford-cotswolds-191209.osm.bz2.
 
 Has anyone else seen problems loading data from the Oxford/Cotswolds area 
 from recent planets?

The error means it ran out of memory. Try running in the --slim mode
instead.

Jon



___
talk mailing list
talk@openstreetmap.org
http://lists.openstreetmap.org/listinfo/talk

[OSM-talk] osm2pgsql usage

[Nakor]

   Hello,

I asked that question on the mapnik mailing list but was told that the OSM
mailing list would have more knowledgeable people on this issue:

I was in the process of setting up Mapnik and as a start I was using an OSM
file from around my area to test with small data.

I get stuck with an error message in osm2pgsql:

osm2pgsql SVN version 0.67-18179

Using projection SRS 900913 (Spherical Mercator)
Setting up table: planet_osm_point
Setting up table: planet_osm_line
Setting up table: planet_osm_polygon
Setting up table: planet_osm_roads
Mid: Ram, scale=100

Reading in file: ../maps/Detroit_091010.osm
Processing: Node(434k) Way(50k) Relation(0k)osm2pgsql: osm2pgsql.c:262:
StartElement: Assertion `xrole' failed.

I searched the web a bit but did not find much, except from some memories
issues. I monitored the memory during the process and had at least 1Gb free,
so I guess this is not the issue.

The data file comes from XAPI (
http://www.informationfreeway.org/api/0.6/map?bbox=-83.56,42.17,-83.01,42.71)
and works fine for other purposes like generating a map for my GPS
using
mkgmap.

Any help would be appreciated.

  Thanks,

N.
___
talk mailing list
talk@openstreetmap.org
http://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] osm2pgsql usage

[Peter Körner]

 The data file comes from XAPI ( 
 http://www.informationfreeway.org/api/0.6/map?bbox=-83.56,42.17,-83.01,42.71 
 ) and works fine for other purposes like generating a map for my GPS 
 using mkgmap.

Have you tried loading a even smaller part from api? This could sort out 
problems with the xapi extensions..

Peter

___
talk mailing list
talk@openstreetmap.org
http://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] osm2pgsql

[Jon Burgess]

On Fri, 2009-06-26 at 18:28 -0400, James McManus wrote:
 Hi - I'm trying to use osm2pgsql to extract a subset area of OSM, but
 the --bbox option does not appear to be working.  I downloaded
 planet-090617.osm.bz2 and then issued the following command:
 
 osm2pgsql --bbox -0.5,51.25,0.5,51.75 -m -d gis planet-090617.osm.bz2
 
 
 At first it appears to be working producing the following output:
 
 planet-090617.osm.bz2
 osm2pgsql SVN version 0.66-16084
 
 Using projection SRS 900913 (Spherical Mercator)
 Applying Bounding box: -0.50,51.25 to 0.50,51.75
 
 Setting up table: planet_osm_point
 Setting up table: planet_osm_line
 Setting up table: planet_osm_polygon
 Setting up table: planet_osm_roads
 Mid: Ram, scale=100
 
 Reading in file: planet-090617.osm.bz2
 
 Processing: Node(1400k) Way(0k) Relation(0k)
 
 But it runs for hours and uses up all of my RAM.  I eventually have to kill 
 it. 
 How long should it take to subset a small area such as this?

How much RAM and swap do you have?

It take at least 2 hours to process the full planet, this is time taken
just to do the bzip2 decompression and XML parsing without even writing
anything out to the database. There is a lot of data in the full planet
file.

The algorithms in the code are optimised for the full planet import and
are not as efficient as they could be for very small areas.

Using the --slim option should fix the memory issue.

Alternatively you could start with a UK or England only extract and save
yourself a lot of bandwidth and import time.

Jon



___
talk mailing list
talk@openstreetmap.org
http://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] osm2pgsql

[Jon Burgess]

On Fri, 2009-06-26 at 23:44 +0100, Jon Burgess wrote:
 On Fri, 2009-06-26 at 18:28 -0400, James McManus wrote:
  
  But it runs for hours and uses up all of my RAM.  I eventually have
 to kill it. 
  How long should it take to subset a small area such as this?
 
 How much RAM and swap do you have?
 
 It take at least 2 hours to process the full planet, this is time
 taken
 just to do the bzip2 decompression and XML parsing without even
 writing
 anything out to the database. There is a lot of data in the full
 planet
 file.
 
 The algorithms in the code are optimised for the full planet import
 and
 are not as efficient as they could be for very small areas.
 
 Using the --slim option should fix the memory issue.
 
 Alternatively you could start with a UK or England only extract and
 save
 yourself a lot of bandwidth and import time.

To give you some firm numbers, the full planet import on the main tile
server with 12GB of RAM takes around 12 hours. On my home machine I
normally import a UK only planet file and it takes about 15 minutes and
about 2GB of RAM.

The --slim option reduces the RAM usage but the import time will
increase if you have less RAM to cache the DB in memory during the
import.

Jon


___
talk mailing list
talk@openstreetmap.org
http://lists.openstreetmap.org/listinfo/talk

[OSM-talk] osm2pgsql and proper/legacy mercator

[Francois Van Der Biest]

Hi list,

osm2pgsql --help says:
-m|--merc: Store data in proper spherical mercator (default)
-M|--oldmerc: Store data in the legacy OSM mercator format

I'm wondering what's the difference between those two srs.
Which one is epsg:900913 (aka epsg:3785
http://www.spatialreference.org/ref/epsg/3785/) ?

My experience (importing an osm dump into postgis, then exporting to
shapefiles) would let me think that the legacy OSM mercator format
is epsg:3785. So, what's the other one ?

Thank's,
F.

___
talk mailing list
talk@openstreetmap.org
http://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] osm2pgsql and proper/legacy mercator

[Andy Allan]

On Fri, May 1, 2009 at 4:52 PM, Francois Van Der Biest
francois.vanderbi...@camptocamp.com wrote:
 Hi list,

 osm2pgsql --help says:
 -m|--merc: Store data in proper spherical mercator (default)
 -M|--oldmerc: Store data in the legacy OSM mercator format

 I'm wondering what's the difference between those two srs.
 Which one is epsg:900913 (aka epsg:3785
 http://www.spatialreference.org/ref/epsg/3785/) ?

 My experience (importing an osm dump into postgis, then exporting to
 shapefiles) would let me think that the legacy OSM mercator format
 is epsg:3785. So, what's the other one ?

I'm moderately sure that --oldmerc was when someone in OSM got the
maths slightly wrong. Use -m for (as it says) proper spherical
mercator aka epsg:900913

Cheers,
Andy

___
talk mailing list
talk@openstreetmap.org
http://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] osm2pgsql and proper/legacy mercator

[Jon Burgess]

On Fri, 2009-05-01 at 17:52 +0200, Francois Van Der Biest wrote:
 Hi list,
 
 osm2pgsql --help says:
 -m|--merc: Store data in proper spherical mercator (default)
 -M|--oldmerc: Store data in the legacy OSM mercator format
 
 I'm wondering what's the difference between those two srs.
 Which one is epsg:900913 (aka epsg:3785
 http://www.spatialreference.org/ref/epsg/3785/) ?
 
 My experience (importing an osm dump into postgis, then exporting to
 shapefiles) would let me think that the legacy OSM mercator format
 is epsg:3785. So, what's the other one ?

The other one should be epsg:3395

$ ./osm2pgsql -vh
...
Supported projections:
Latlong (-l) SRS:  4326 (none)
WGS84 Mercator  (  ) SRS:  3395 +proj=merc +datum=WGS84  +k=1.0 +units=m 
+over +no_defs
Spherical Mercator  (-m) SRS:900913 +proj=merc +a=6378137 +b=6378137 
+lat_ts=0.0 +lon_0=0.0 +x_0=0.0 +y_0=0 +k=1.0 +units=m +nadgri...@null +no_defs 
+over

The seconds line of the help should show (-M) for the legacy projection.
I've just fixed it in SVN.

   Jon



___
talk mailing list
talk@openstreetmap.org
http://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] osm2pgsql and proper/legacy mercator

[Juan Lucas Dominguez Rubio]

Hi, according to the source code, the legacy option -M is using this projection:
 
  [PROJ_MERC]= { 
 descr: WGS84 Mercator, 
 proj4text: +proj=merc +datum=WGS84  +k=1.0 +units=m +over +no_defs, 
 srs:3395, 
 option:  }
 
Unlike EPSG:900913, that projection is not assuming that the Earth is a perfect 
sphere, therefore:
 
- it seems to be cartographically more correct
- it probably needs far more computation
- the result probably differs from that of EPSG:900913 by a few meters
 
Regards
Juan Lucas
 
 



De: talk-boun...@openstreetmap.org en nombre de Francois Van 
Der Biest
Enviado el: vie 01/05/2009 17:52
Para: talk@openstreetmap.org
Asunto: [OSM-talk] osm2pgsql and proper/legacy mercator



Hi list,

osm2pgsql --help says:
-m|--merc: Store data in proper spherical mercator (default)
-M|--oldmerc: Store data in the legacy OSM mercator format

I'm wondering what's the difference between those two srs.
Which one is epsg:900913 (aka epsg:3785
http://www.spatialreference.org/ref/epsg/3785/) ?

My experience (importing an osm dump into postgis, then 
exporting to
shapefiles) would let me think that the legacy OSM mercator 
format
is epsg:3785. So, what's the other one ?

Thank's,
F.

___
talk mailing list
talk@openstreetmap.org
http://lists.openstreetmap.org/listinfo/talk


___
talk mailing list
talk@openstreetmap.org
http://lists.openstreetmap.org/listinfo/talk

[OSM-talk] Osm2pgsql and column names with two underscores

[Jukka Rahkonen]

Hi,

An osm2pgsql user writes on the forum about importing special tags
into PostGIS.
He has defined for example these tags:
node   openGeoDB:telephone_area_code   text 
node   openGeoDB:license_plate_code  text

Import fails and error message is:
CREATE TABLE planet_osm_point ( osm_id int4,access text,
admin_level text, 

---SNIP---
openGeoDB:postal_codes text,openGeoDB:telephone_are a_code,
openGeoDB:type text,openGeoDB:license_plate _code,z_order
int4 );

failed: ERROR:  type a_code does not exist
LINE 1: ...eoDB:postal_codes text,
openGeoDB:telephone_are a_code,op...
 ^

Error occurred, cleaning up

Look at these:
openGeoDB:telephone_are a_code,
openGeoDB:license_plate _code

It looks like column name gets truncated at the second 
underline character. Could it be some bug in osm2pgsql,
or is it some other place where the error occurs?

-Jukka Rahkonen-



___
talk mailing list
talk@openstreetmap.org
http://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] Osm2pgsql and column names with two underscores

[Martijn van Oosterhout]

On Mon, Mar 30, 2009 at 2:08 PM, Jukka Rahkonen
jukka.rahko...@mmmtike.fi wrote:
 Hi,

 An osm2pgsql user writes on the forum about importing special tags
 into PostGIS.
 He has defined for example these tags:
 node       openGeoDB:telephone_area_code       text
 node       openGeoDB:license_plate_code          text

 Import fails and error message is:
 CREATE TABLE planet_osm_point ( osm_id int4,access text,
 admin_level text,

Ok, it's got nothing to do with the underscores and more to do with
the fact that the field names are limited to 23 characters by
osm2pgsql. I don't really know where this restriction comes from, does
anyone else have ideas?

It should be easy to fix in output-pgsql.c. Fix the read_style_file()
function to increase the length of the buffers.

Have a nice day,
-- 
Martijn van Oosterhout klep...@gmail.com http://svana.org/kleptog/

___
talk mailing list
talk@openstreetmap.org
http://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] Osm2pgsql and column names with two underscores

[sly (sylvain letuffe)]

On Monday 30 March 2009 14:08, Jukka Rahkonen wrote:
 Hi,

Hi,

( better transfert this to dev ?)


 An osm2pgsql user writes on the forum about importing special tags
 into PostGIS.
 He has defined for example these tags:
 node   openGeoDB:telephone_area_code   text 
 node   openGeoDB:license_plate_code  text

I've read his problem, and proposed converting the osm input file with smaller 
tag name. The other solution might be to patch osm2pgsql.

 Look at these:
 openGeoDB:telephone_are a_code,
 openGeoDB:license_plate _code
 
 It looks like column name gets truncated at the second 
 underline character. 

Coincidence ;-)

Both tag name are also truncated at 24 chars exactly, let's see...

 Could it be some bug in osm2pgsql, 
 or is it some other place where the error occurs?

Looks like I went lucky :
# grep 24 output-pgsql.c
char buffer[1024];
  char buffer[1024];
char osmtype[24];
char tag[24];
char datatype[24];

Recompiling osm2pgsql might help, if that limit of 24 is not present elsewhere


-- 
sly 
Sylvain Letuffe li...@letuffe.org
qui suis-je : http://slyserv.dyndns.org



___
talk mailing list
talk@openstreetmap.org
http://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] Osm2pgsql and column names with two underscores

[Cartinus]

On Monday 30 March 2009 14:08:11 Jukka Rahkonen wrote:
 Look at these:
 openGeoDB:telephone_are a_code,
 openGeoDB:license_plate _code

 It looks like column name gets truncated at the second
 underline character. Could it be some bug in osm2pgsql,
 or is it some other place where the error occurs?

It doesn't get truncated at the second underscore as there is a character 
before the second underscore at the first case. Looks like it gets truncated 
after the 23th character.


-- 
m.v.g.,
Cartinus

___
talk mailing list
talk@openstreetmap.org
http://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] Osm2pgsql fails with Finland.osm.bz2

[Frederik Ramm]

Hi,

Jukka Rahkonen wrote:
 Processing: Node(2835k) Way(194k) Relation(0k)terminate called after throwing 
 an
  instance of 'geos::util::TopologyException'
   what():  TopologyException: found non-noded intersection between 
 2.78294e+006
 9.41427e+006, 2.78315e+006 9.41433e+006 and 2.78326e+006 9.41433e+006, 
 2.78269e+
 006 9.41433e+006 2.78315e+006 9.41433e+006

Reverse-projecting that into lat/lon gives 24.999;64.252. If you check 
the area in OSM Inspector:

http://tools.geofabrik.de/osmi/?view=geometrylon=24.94860lat=64.24186zoom=11opacity=1.00

you will see that there are a number of geometry errors and quite 
possibly self-intersecting ways near the town of Oulainen. One of these 
is very likely responsible for the error you are seeing, however I 
cannot tell you why osm2pgsql chokes on this when it happily processes 
the myriad other geometry errors that OSM Inspector spots.

You can select an object and then use the icons on the right to load it 
in an (already running) JOSM or launch Potlatch for that object.

Bye
Frederik


___
talk mailing list
talk@openstreetmap.org
http://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] Osm2pgsql fails with Finland.osm.bz2

[Jon Burgess]

On Wed, 2009-02-04 at 15:27 +, Jukka Rahkonen wrote:
 Hi,
 
 I have been importing Finland.osm.bz2 dataset from Geofabrik into Postgis 
 every
 day with osm2pgsql.exe (on Windows) but now it fails. The error looks like 
 this:
 Reading in file: finland.osm.bz2
 Processing: Node(2835k) Way(37k) Relation(0k)Error allocating ways
 Error occurred, cleaning up
 
 I tried next to use Scandinavian dataset from hypercube.telascience.org.  
 Result
 is about the same:
 Reading in file: planet-scand-090203.osm.gz
 Processing: Node(6670k) Way(0k) Relation(0k)Error allocating nodes
 Error occurred, cleaning up

These first two look like an out of memory condition causing malloc() to
fail. 

 I made a third trial by using another version of osm2pgsql.exe with Finnish 
 data
 but again with no luck. A bit different error message though:
 Reading in file: finland.osm.bz2
 Processing: Node(2835k) Way(194k) Relation(0k)terminate called after throwing 
 an
  instance of 'geos::util::TopologyException'
   what():  TopologyException: found non-noded intersection between 
 2.78294e+006
 9.41427e+006, 2.78315e+006 9.41433e+006 and 2.78326e+006 9.41433e+006, 
 2.78269e+
 006 9.41433e+006 2.78315e+006 9.41433e+006

Normally the code catches exceptions from geos and ignores them. In some
builds of osm2pgsql.exe this mechanism is broken and a geometry
exception will instead cause the program to abort. It looks like this is
one of these broken versions.

 Does anybody has an idea about what is going on, and if it is just a data 
 error,
 how to find and correct it?

This version of osm2pgsql.exe it should not have the exception problem:
http://tile.openstreetmap.org/direct/osm2pgsql.zip

Run the utility with the --slim parameter. This should fix any out of
memory problems but it will be a bit slower and require more disk space.

Jon



___
talk mailing list
talk@openstreetmap.org
http://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] Osm2pgsql fails with Finland.osm.bz2

[Rahkonen Jukka]

Jon Burgess jburgess777 at googlemail.com writes:

 
 On Wed, 2009-02-04 at 15:27 +, Jukka Rahkonen wrote:
  Hi,
  
  I have been importing Finland.osm.bz2 dataset from Geofabrik into Postgis 
  every
  day with osm2pgsql.exe (on Windows) but now it fails. The error looks like 
  this:
  Reading in file: finland.osm.bz2
  Processing: Node(2835k) Way(37k) Relation(0k)Error allocating ways
  Error occurred, cleaning up

 
 These first two look like an out of memory condition causing malloc() to
 fail. 

Right, another computer with more memory imports the same data with same program
versions.
 
 
  I made a third trial by using another version of osm2pgsql.exe with Finnish 
  data
  but again with no luck. A bit different error message though:
  Reading in file: finland.osm.bz2
  Processing: Node(2835k) Way(194k) Relation(0k)terminate called after 
  throwing an
   instance of 'geos::util::TopologyException'

...
 
 Normally the code catches exceptions from geos and ignores them. In some
 builds of osm2pgsql.exe this mechanism is broken and a geometry
 exception will instead cause the program to abort. It looks like this is
 one of these broken versions.
...
 This version of osm2pgsql.exe it should not have the exception problem:
 http://tile.openstreetmap.org/direct/osm2pgsql.zip

Right again, this one does not stop to topology errors. So no problem at all,
the amount of data had just reached the limits of the first computer I used.

___
talk mailing list
talk@openstreetmap.org
http://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] osm2pgsql planet: frustrations, cutoffs, and idempotence

[Jochen Topf]

On Sun, Oct 26, 2008 at 06:11:04PM -0700, Michal Migurski wrote:
 What is the difference between osmosis and osm2pgsql, with regards to  
 postGIS?

osm2pgsql creates the structure needed for Mapnik. Osmosis creates a
structure more simliar to the one in the OSM central database.

 If I've been maintaining a dataset based on osm2pgsql with the  
 provided default.style, would a dataset based on osmosis result in a  
 substantially different table structure?

Yes.

Jochen
-- 
Jochen Topf  [EMAIL PROTECTED]  http://www.remote.org/jochen/  +49-721-388298


___
talk mailing list
talk@openstreetmap.org
http://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] osm2pgsql planet: frustrations, cutoffs, and idempotence

[Tom Hughes]

Shaun McDonald wrote:
 On 27 Oct 2008, at 00:50, Michal Migurski wrote:
 
 Planet dumps are not snapshots - they do not represent a consistent
 view at any particular point in time because they take a number of
 hours to generate, during which time new changes are constantly
 being made to the contents of the database.
 
 Shouldn't it be possible to ignore any changes that happen after the
 cutoff, though?
 
 At the moment we don't look at the time stamps when dumping the planet  
 file.

It's not as simple as that - you also have to switch to reading the 
history tables rather than the current tables or you won't be able to 
see what the state of the object used to be if it has changed since the 
snapshot time.

Which means you're reading much more data, and either having to track 
the state of each object (in order to find the most recent valid change) 
or you have to index scan so that you're seeing things in timestamp order.

Tom

-- 
Tom Hughes ([EMAIL PROTECTED])
http://www.compton.nu/

___
talk mailing list
talk@openstreetmap.org
http://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] osm2pgsql planet: frustrations, cutoffs, and idempotence

[Jochen Topf]

On Mon, Oct 27, 2008 at 08:22:32AM +, Tom Hughes wrote:
 Shaun McDonald wrote:
  On 27 Oct 2008, at 00:50, Michal Migurski wrote:
  
  Planet dumps are not snapshots - they do not represent a consistent
  view at any particular point in time because they take a number of
  hours to generate, during which time new changes are constantly
  being made to the contents of the database.
  
  Shouldn't it be possible to ignore any changes that happen after the
  cutoff, though?
  
  At the moment we don't look at the time stamps when dumping the planet  
  file.
 
 It's not as simple as that - you also have to switch to reading the 
 history tables rather than the current tables or you won't be able to 
 see what the state of the object used to be if it has changed since the 
 snapshot time.
 
 Which means you're reading much more data, and either having to track 
 the state of each object (in order to find the most recent valid change) 
 or you have to index scan so that you're seeing things in timestamp order.

If the planet dump plus the diff from the same day is what everybody
wants anyway, why not do this on the server side and hold the planet
back after the first diff is available, run this over the planet and
then publish that as the planet?

Jochen
-- 
Jochen Topf  [EMAIL PROTECTED]  http://www.remote.org/jochen/  +49-721-388298


___
talk mailing list
talk@openstreetmap.org
http://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] osm2pgsql planet: frustrations, cutoffs, and idempotence

[Brett Henderson]

Others have already commented on most of your points but I'll add my 
thoughts in case there's some gaps.

Michal Migurski wrote:
 Hi,

 I've been trying to keep up to date with the dumps and diffs from 
 http://planet.openstreetmap.org/ 
 , and I'm running into a number of bugs related to cutoff dates.

 In keeping my Bay Area tiles 
 (http://mike.teczno.com/notes/cascadenik-openstreetmap.html 
 ) up to date, I've been grabbing complete planet.osm dumps about once  
 per month, and filling in the intervening time with daily diffs. I've  
 noticed some misalignments between the data in the dumps and the  
 osm2pgsql importer that leads to unavoidable holes in the data.

 It seems that they could be fixed in either osm2pgsql, the planet  
 files, or both.

 The final event in each weekly planet dump does not fall on an even  
 day boundary. In the case of the most recent Oct. 22nd planet.osm, it  
 was necessary to experiment with hourly diffs from that day to find  
 that the boundary was approx. 2:00pm. Hourlies up to and including  
 2008102213-2008102214.osc.gz failed, hourlies after that succeeded. I  
 could go more granular here, checking the minute diffs as well for a  
 more precise breakpoint, but it seems odd that the planet dump does  
 not break cleanly on a midnight boundary so that it's possible to pick  
 up the differences moving forward.
   
Yep, as others have commented there are two tables types in the osm 
database; current tables, and history tables.  The planet dumper just 
reads current tables which is the fastest approach.  Unfortunately the 
current tables change constantly during the planet generation process 
resulting in inconsistencies.  It is possible to produce a consistent 
snapshot reading history tables and osmosis has the ability to do just 
that but it is significantly slower.  It is also possible to produce a 
consistent snapshot by taking an inconsistent planet and applying 
changesets from a point in time prior to the planet dump beginning 
through to a point after completion, this effectively produces the same 
result at much reduced load on the main database.
 osm2pgsql itself notifies the user of inconsistencies by failing. I  
 can see that effort has been put into making it more resilient (e.g. 
 http://trac.openstreetmap.org/changeset/10464) 
 . Does osm2pgsql have something like a `--force` switch? I haven't  
 been able to find one. In looking at the diff files, it seems that it  
 should be possible to ignore possible conflicts by simply overwriting  
 whatever's in the DB with whatever's in the .osc file.
   
Yes, that's true.  I can't comment on osm2pgsql but when osmosis 
processes changeset files it does exactly that.
 Finally, the boundaries between the hourlies and dailies seem  
 misaligned.
   
This shouldn't be the case.
 After running the remaining hourlies for the 22nd, I attempted to pick  
 up on the 23rd with a daily. The final hourly I used was  
 2008102223-2008102300.osc.gz. It's my expectation that I should be  
 able to immediately follow that with 20081023-20081024.osc.gz, but  
 this led to duplicate key violation suggesting that there's an overlap  
 between the two files. Continuing with hourlies *works*, but is  
 tedious and I suspect slower than the dailies.
   
You should have been able to do what you've suggested.  If you are 
finding problems, please provide me with some example data which is 
misaligned between the two types of changesets.  I've gone to a fair bit 
of trouble to ensure that timestamp management is correct.  For example, 
all changesets and file names are using UTC even though the database 
itself is using BST.  If I've made a mistake somewhere I'd like to know 
about it.  Given that daily, hourly and minute changesets are using 
*identical* code, I find it hard to believe they're inconsistent with 
each other.
 My sense from reading other people's experiences has been that it's a  
 common pattern to rely solely on the weekly planet dumps, incurring  
 the substantial overhead of parsing and importing the full 5GB dump  
 once every week, and then re-rendering the complete set of tiles.
   
For a long time weekly planet dumps were the only bulk data available.  
Osmosis changesets have been on the scene for some time now though and 
are gradually being utilised by more and more clients.  As the planet 
grows, this will become more critical.  Who knows, if the kinks 
gradually get ironed out of the osm2pgsql program we may even begin to 
see the main mapnik tile generator move to using changesets.
 My hope has been to proceed in a more incremental fashion, since this  
 makes it possible to track what specific tiles need to be re-rendered  
 on a near-constant schedule, based on actual content or activity, vs.  
 simple cache expiration. Right now I'm doing this daily, I'd like to  
 do it as often as hourly.
   
Yep, that was one of my original aims.
 I can see a few possible solutions.

 The cutoff times for 

Re: [OSM-talk] osm2pgsql planet: frustrations, cutoffs, and idempotence

[Brett Henderson]

Jochen Topf wrote:
 If the planet dump plus the diff from the same day is what everybody
 wants anyway, why not do this on the server side and hold the planet
 back after the first diff is available, run this over the planet and
 then publish that as the planet?
   
It would add delay to the planet creation process.  I don't know how 
much of an issue that would be.

How many people still download the full planet on a regular basis?  I 
would hope that people would begin to use changesets even if they only 
require a complete xml file.  For bandwidth reasons alone the gains are 
well worthwhile, plus you can get far more regular updates than weekly.  
The script below automates keeping a snapshot file in sync:
http://svn.openstreetmap.org/applications/utils/osmosis/script/contrib/replicate_osm_file.sh


___
talk mailing list
talk@openstreetmap.org
http://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] osm2pgsql planet: frustrations, cutoffs, and idempotence

[Frederik Ramm]

Hi,

Brett Henderson wrote:
 Brett Henderson has offered to look into creating the dailies from 
 history as well, but I don't know about the status of that.
   
 Are you referring to the daily changesets? 
[...]
 Or did you mean planets instead of dailies? 

Mix-up on my part, sorry, yes I meant the planets.

Bye
Frederik

-- 
Frederik Ramm  ##  eMail [EMAIL PROTECTED]  ##  N49°00'09 E008°23'33

___
talk mailing list
talk@openstreetmap.org
http://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] osm2pgsql planet: frustrations, cutoffs, and idempotence

[Michal Migurski]

 Yep, as others have commented there are two tables types in the osm  
 database; current tables, and history tables.  The planet dumper  
 just reads current tables which is the fastest approach.   
 Unfortunately the current tables change constantly during the  
 planet generation process resulting in inconsistencies.  It is  
 possible to produce a consistent snapshot reading history tables  
 and osmosis has the ability to do just that but it is significantly  
 slower.  It is also possible to produce a consistent snapshot by  
 taking an inconsistent planet and applying changesets from a point  
 in time prior to the planet dump beginning through to a point after  
 completion, this effectively produces the same result at much  
 reduced load on the main database.


I'm liking Jochen Topf's suggestion here:

If the planet dump plus the diff from the same day is what everybody  
wants anyway, why not do this on the server side and hold the planet  
back after the first diff is available, run this over the planet and  
then publish that as the planet?


 Finally, the boundaries between the hourlies and dailies seem   
 misaligned.


 This shouldn't be the case.
 After running the remaining hourlies for the 22nd, I attempted to  
 pick  up on the 23rd with a daily. The final hourly I used was   
 2008102223-2008102300.osc.gz. It's my expectation that I should be   
 able to immediately follow that with 20081023-20081024.osc.gz, but   
 this led to duplicate key violation suggesting that there's an  
 overlap  between the two files. Continuing with hourlies *works*,  
 but is  tedious and I suspect slower than the dailies.


 You should have been able to do what you've suggested.  If you are  
 finding problems, please provide me with some example data which is  
 misaligned between the two types of changesets.

Try the two files mentioned above - that's where I saw this behavior,  
they're quite recent.

2008102223-2008102300.osc.gz
20081023-20081024.osc.gz


 My sense from reading other people's experiences has been that it's  
 a  common pattern to rely solely on the weekly planet dumps,  
 incurring  the substantial overhead of parsing and importing the  
 full 5GB dump  once every week, and then re-rendering the complete  
 set of tiles.


 For a long time weekly planet dumps were the only bulk data  
 available.  Osmosis changesets have been on the scene for some time  
 now though and are gradually being utilised by more and more  
 clients.  As the planet grows, this will become more critical.  Who  
 knows, if the kinks gradually get ironed out of the osm2pgsql  
 program we may even begin to see the main mapnik tile generator move  
 to using changesets.

I would love to rely on these exclusively, it's much more efficient.  
But, I was seeing a fair bit of information fall through the cracks so  
that's why I'm re-synching to planet every four weeks.



 I can see a few possible solutions.

 The cutoff times for files on planet.openstreetmap.org could  
 behave  more consistently. A weekly dump should end at 11:59pm so  
 that dailies  can immediately pick up user activity. Hourly and  
 daily dumps should  be synchronized. This seems more difficult.


 You only need a single consistent snapshot to get started.  You can  
 download a planet, then download the two daily changesets either  
 side of the planet generation window, then use osmosis to patch the  
 planet.  This will give you a consistent snapshot.  Once you've  
 imported that into your target database you can then start using  
 daily changesets to keep up to date (or hourly or minute as  
 appropriate).

 While it would be nice to have planet dumps already in consistent  
 form, it does add a significant overhead to the whole process.  It's  
 not terribly hard to fix on the client side.

Probably what I need to do is get a fresh update of osm2pgsql. I can  
see now that the revision I'm using is older than #10464, where some  
inconsistency resilience was added.


-mike.




michal migurski- [EMAIL PROTECTED]
  415.558.1610




___
talk mailing list
talk@openstreetmap.org
http://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] osm2pgsql planet: frustrations, cutoffs, and idempotence

[Martijn van Oosterhout]

On Mon, Oct 27, 2008 at 9:39 PM, Michal Migurski [EMAIL PROTECTED] wrote:
 I'm liking Jochen Topf's suggestion here:

If the planet dump plus the diff from the same day is what everybody
 wants anyway, why not do this on the server side and hold the planet
 back after the first diff is available, run this over the planet and
 then publish that as the planet?

1. Because there are plenty of uses for the planet dump that don't
need consistant snapshots.

2. Because such consistant snapshots have been available elsewhere for
quite a while now and people who need them can get them. There's no
particular reason why it has to be on the same site as the normal
planet dumps.
 Probably what I need to do is get a fresh update of osm2pgsql. I can
 see now that the revision I'm using is older than #10464, where some
 inconsistency resilience was added.

Umm, yeah. I was ofcourse assuming you were running the latest
version, otherwise anything is possible, The creates-as-modifies fix
was done two months ago.

Have a nice day,
-- 
Martijn van Oosterhout [EMAIL PROTECTED] http://svana.org/kleptog/

___
talk mailing list
talk@openstreetmap.org
http://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] osm2pgsql planet: frustrations, cutoffs, and idempotence

[Martijn van Oosterhout]

On Mon, Oct 27, 2008 at 9:40 PM, Michal Migurski [EMAIL PROTECTED] wrote:
 Now that I think about it though, I think what I did was take one of
 the planet dumps from http://hypercube.telascience.org/planet/ (which
 *are* consistant snapshots), and run the dailies from there.

 Is there any reason to not use those? They seem to be more frequent
 than the planet.openstreetmap.org ones - is there some disadvantage?
 How are they created?

Umm, they are created by taking the planet dumps and applying the
daily diffs every day. They are used to produce consistant snapshots
of for example, NL and by the coastline checker (which really likes
having consistant snapshots to work with).

Have a nice day,
-- 
Martijn van Oosterhout [EMAIL PROTECTED] http://svana.org/kleptog/

___
talk mailing list
talk@openstreetmap.org
http://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] osm2pgsql planet: frustrations, cutoffs, and idempotence

[Brett Henderson]

On Tue, Oct 28, 2008 at 7:39 AM, Michal Migurski [EMAIL PROTECTED] wrote:

  Finally, the boundaries between the hourlies and dailies seem
  misaligned.
 
 
  This shouldn't be the case.
  After running the remaining hourlies for the 22nd, I attempted to
  pick  up on the 23rd with a daily. The final hourly I used was
  2008102223-2008102300.osc.gz. It's my expectation that I should be
  able to immediately follow that with 20081023-20081024.osc.gz, but
  this led to duplicate key violation suggesting that there's an
  overlap  between the two files. Continuing with hourlies *works*,
  but is  tedious and I suspect slower than the dailies.
 
 
  You should have been able to do what you've suggested.  If you are
  finding problems, please provide me with some example data which is
  misaligned between the two types of changesets.

 Try the two files mentioned above - that's where I saw this behavior,
 they're quite recent.

2008102223-2008102300.osc.gz
20081023-20081024.osc.gz


I need you to provide some specific examples of broken data.  If you can say
that way 27123456 is created in both of the above files even though they
are for different time periods then I can take a look at why this may have
occurred.  Just saying that there is misalignment between those two files
doesn't help me at all.  Presumably you ran into a specific problem and
received a specific error message, this is the kind of information I need.
I only do this project in my spare time and can't go looking for problems
that I'm not sure even exist, I have enough known problems to look into
already :-)





  My sense from reading other people's experiences has been that it's
  a  common pattern to rely solely on the weekly planet dumps,
  incurring  the substantial overhead of parsing and importing the
  full 5GB dump  once every week, and then re-rendering the complete
  set of tiles.
 
 
  For a long time weekly planet dumps were the only bulk data
  available.  Osmosis changesets have been on the scene for some time
  now though and are gradually being utilised by more and more
  clients.  As the planet grows, this will become more critical.  Who
  knows, if the kinks gradually get ironed out of the osm2pgsql
  program we may even begin to see the main mapnik tile generator move
  to using changesets.

 I would love to rely on these exclusively, it's much more efficient.
 But, I was seeing a fair bit of information fall through the cracks so
 that's why I'm re-synching to planet every four weeks.


Again, please provide some specific examples.  If data is being missed I'd
like to know about it.  Osmosis provides some tools that may be useful
here.  You can download a planet, apply changesets for a week, then compare
against the next planet and see what the differences are.  Obviously both
planets would need appropriate changesets applied to make them consistent
before performing a comparison to eliminate noise.

I probably should do some of these comparisons myself, but again just
haven't found time yet and nobody else has complained about missing data.
The minute changesets run 5 minutes behind the API so could potentially miss
data if a lock is held for several minutes.  The daily and hourly changesets
run at least 20 minutes behind API (forget off the top of my head) and
should be extremely unlikely to miss data.

Brett
___
talk mailing list
talk@openstreetmap.org
http://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] osm2pgsql planet: frustrations, cutoffs, and idempotence

[Michal Migurski]

 On Mon, Oct 27, 2008 at 9:39 PM, Michal Migurski [EMAIL PROTECTED]  
 wrote:
 I'm liking Jochen Topf's suggestion here:

   If the planet dump plus the diff from the same day is what  
 everybody
 wants anyway, why not do this on the server side and hold the planet
 back after the first diff is available, run this over the planet and
 then publish that as the planet?

 1. Because there are plenty of uses for the planet dump that don't
 need consistant snapshots.

Those uses would not be impacted by consistent snapshots.


 2. Because such consistant snapshots have been available elsewhere for
 quite a while now and people who need them can get them. There's no
 particular reason why it has to be on the same site as the normal
 planet dumps.

Yet there is no link to these places from planet.openstreetmap.org  
that indicates that the files available there differ in some important  
or useful way. The telascience.org source you suggested is described  
as extracts of NL, Scandinavia and Taiwan at 
http://wiki.openstreetmap.org/index.php/Planet.osm 
, rather than a complete dump of Planet with different datetime  
boundaries.

I'm happy to keep bellying up to the trial  error bar here, but as I  
mention in a previous mail, the volume of data involved means that  
individual attempts at the data (successful or not) have multiple-day  
costs associated with them.


 Umm, yeah. I was ofcourse assuming you were running the latest
 version, otherwise anything is possible, The creates-as-modifies fix
 was done two months ago.


I'll recompile and replace the two-month-old version of osm2pgsql I've  
been using.

-mike.


michal migurski- [EMAIL PROTECTED]
  415.558.1610




___
talk mailing list
talk@openstreetmap.org
http://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] osm2pgsql planet: frustrations, cutoffs, and idempotence

[Tom Hughes]

Michal Migurski wrote:

 The final event in each weekly planet dump does not fall on an even  
 day boundary. In the case of the most recent Oct. 22nd planet.osm, it  
 was necessary to experiment with hourly diffs from that day to find  
 that the boundary was approx. 2:00pm. Hourlies up to and including  
 2008102213-2008102214.osc.gz failed, hourlies after that succeeded. I  
 could go more granular here, checking the minute diffs as well for a  
 more precise breakpoint, but it seems odd that the planet dump does  
 not break cleanly on a midnight boundary so that it's possible to pick  
 up the differences moving forward.

Planet dumps are not snapshots - they do not represent a consistent view 
at any particular point in time because they take a number of hours to 
generate, during which time new changes are constantly being made to the 
contents of the database.

I believe that it is supposed to be safe to apply diffs which overlap 
with the planet dump in order to bring it to a consistent state however.

 The cutoff times for files on planet.openstreetmap.org could behave  
 more consistently. A weekly dump should end at 11:59pm so that dailies  
 can immediately pick up user activity. Hourly and daily dumps should  
 be synchronized. This seems more difficult.

As explained above, there is no cutoff time as such, and it isn't 
possible to implement one as things stand. It may be possible once we 
have working transactions, though it's not clear that a transaction that 
lasts many hours would be sensible or workable.

BTW I'm not sure why you CCed the OSMF board on this... I don't think it 
needs their input at all.

Tom

-- 
Tom Hughes ([EMAIL PROTECTED])
http://www.compton.nu/

___
talk mailing list
talk@openstreetmap.org
http://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] osm2pgsql planet: frustrations, cutoffs, and idempotence

[Frederik Ramm]

Hi,

Michal Migurski wrote:
 I've noticed some misalignments between the data in the dumps and the  
 osm2pgsql importer that leads to unavoidable holes in the data.

As TomH has already said, this is not a bug, it stems from the fact that 
the full planet export reads the current tables and as such is subject 
to changes that occur during the export process. (There may even be 
inconsistencies when something like this happens: Exporter dumps nodes, 
exporter starts dumping ways, user adds new node into way, new way 
version is dumped referring to new node that is not in the dump.)

The daily, hourly, and minutely diffs have a clean cutoff date because 
they are taken from the history tables.

Brett Henderson has offered to look into creating the dailies from 
history as well, but I don't know about the status of that.

If you use osmosis, it is safe (and in fact recommended) that, after 
loading the database with a planet file initially, you should load that 
same day's diff file as the first diff, creating a clean cutoff point. 
It is possible that the same is not working with osm2pgsql, I have no 
experience there.

Bye
Frederik

-- 
Frederik Ramm  ##  eMail [EMAIL PROTECTED]  ##  N49°00'09 E008°23'33

___
talk mailing list
talk@openstreetmap.org
http://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] osm2pgsql planet: frustrations, cutoffs, and idempotence

[Michal Migurski]

 The final event in each weekly planet dump does not fall on an  
 even  day boundary. In the case of the most recent Oct. 22nd  
 planet.osm, it  was necessary to experiment with hourly diffs from  
 that day to find  that the boundary was approx. 2:00pm. Hourlies up  
 to and including  2008102213-2008102214.osc.gz failed, hourlies  
 after that succeeded. I  could go more granular here, checking the  
 minute diffs as well for a  more precise breakpoint, but it seems  
 odd that the planet dump does  not break cleanly on a midnight  
 boundary so that it's possible to pick  up the differences moving  
 forward.

 Planet dumps are not snapshots - they do not represent a consistent  
 view at any particular point in time because they take a number of  
 hours to generate, during which time new changes are constantly  
 being made to the contents of the database.

Shouldn't it be possible to ignore any changes that happen after the  
cutoff, though? I may not understand the structure of the OSM  
database, but it seems like if it supports rollbacks, then in theory  
it ought to be possible to only include things before a given  
timestamp when creating the dump file. That, or make it clear what the  
actual cutoff time is in the dumpfile.

I understand that in practice, practice is different from theory. =)


 I believe that it is supposed to be safe to apply diffs which  
 overlap with the planet dump in order to bring it to a consistent  
 state however.

This is what I would have hoped, however osm2pgsql does not appear to  
allow it. It feels like the easiest solution would be to give  
osm2pgsql a --force option, and add some explanation of timing and  
cutoffs to http://planet.openstreetmap.org/README.


 BTW I'm not sure why you CCed the OSMF board on this... I don't  
 think it needs their input at all.

Mikel Maron suggested that I cc: team@, when I spoke to him about this  
a few days ago, because it's connected to a *.openstreetmap.org service.

Thanks for your reply!



-mike.


michal migurski- [EMAIL PROTECTED]
  415.558.1610




___
talk mailing list
talk@openstreetmap.org
http://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] osm2pgsql planet: frustrations, cutoffs, and idempotence

[Michal Migurski]

On Oct 26, 2008, at 5:50 PM, Frederik Ramm wrote:

 Brett Henderson has offered to look into creating the dailies from  
 history as well, but I don't know about the status of that.

 If you use osmosis, it is safe (and in fact recommended) that, after  
 loading the database with a planet file initially, you should load  
 that same day's diff file as the first diff, creating a clean cutoff  
 point. It is possible that the same is not working with osm2pgsql, I  
 have no experience there.


What is the difference between osmosis and osm2pgsql, with regards to  
postGIS?

If I've been maintaining a dataset based on osm2pgsql with the  
provided default.style, would a dataset based on osmosis result in a  
substantially different table structure?

-mike.


michal migurski- [EMAIL PROTECTED]
  415.558.1610




___
talk mailing list
talk@openstreetmap.org
http://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] osm2pgsql planet: frustrations, cutoffs, and idempotence

[Shaun McDonald]

On 27 Oct 2008, at 00:50, Michal Migurski wrote:

 The final event in each weekly planet dump does not fall on an
 even  day boundary. In the case of the most recent Oct. 22nd
 planet.osm, it  was necessary to experiment with hourly diffs from
 that day to find  that the boundary was approx. 2:00pm. Hourlies up
 to and including  2008102213-2008102214.osc.gz failed, hourlies
 after that succeeded. I  could go more granular here, checking the
 minute diffs as well for a  more precise breakpoint, but it seems
 odd that the planet dump does  not break cleanly on a midnight
 boundary so that it's possible to pick  up the differences moving
 forward.

 Planet dumps are not snapshots - they do not represent a consistent
 view at any particular point in time because they take a number of
 hours to generate, during which time new changes are constantly
 being made to the contents of the database.

 Shouldn't it be possible to ignore any changes that happen after the
 cutoff, though?

At the moment we don't look at the time stamps when dumping the planet  
file.

 I may not understand the structure of the OSM
 database, but it seems like if it supports rollbacks, then in theory
 it ought to be possible to only include things before a given
 timestamp when creating the dump file. That, or make it clear what the
 actual cutoff time is in the dumpfile.

We currently don't support rollbacks. It would require a rewrite of  
the dump script, and more time and processing to be able to produce a  
consistent planet dump.



 I understand that in practice, practice is different from theory. =)

Have you got the rails port running?




 I believe that it is supposed to be safe to apply diffs which
 overlap with the planet dump in order to bring it to a consistent
 state however.

 This is what I would have hoped, however osm2pgsql does not appear to
 allow it. It feels like the easiest solution would be to give
 osm2pgsql a --force option, and add some explanation of timing and
 cutoffs to http://planet.openstreetmap.org/README.


The initial import that you do with osm2pgsql, must be using a special  
mode to allow diff imports. Could it be that you need to update to the  
latest version of osm2pgsql? You should be able to happily apply the  
diffs to an inconsistent planet dump, to get a consistent planet dump.

This will become easier when the version numbers are exposed in the  
0.6 API. The diff mechanism would then be able to look at the version  
numbers of the nodes/ways/relations and be able to deal with them  
appropriately.

Shaun


___
talk mailing list
talk@openstreetmap.org
http://lists.openstreetmap.org/listinfo/talk

[OSM-talk] osm2pgsql password for non-interactive use

[Michal Migurski]

Hi,

It appears that osm2pgsql (SVN version 0.55-20080901) does not have a  
way to specify the db password on the command line. Is this true?

It'd be nice to be able to run it without needing to babysit the  
password prompt at the beginning. Can this be added a flag?

-mike.


michal migurski- [EMAIL PROTECTED]
  415.558.1610




___
talk mailing list
talk@openstreetmap.org
http://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] osm2pgsql - invalid input syntax for integer : Breërivier

[Ricardo Peironcely]

Yes, downloading the last version fron SVN the error was resolved.

Thanks.
Rpr


2008/4/16, Martijn van Oosterhout [EMAIL PROTECTED]:

 On Wed, Apr 16, 2008 at 7:05 PM, Ricardo Peironcely
 [EMAIL PROTECTED] wrote:
 
  I've a problem with osm2pgsql and the last version of planet file.
 
  When I try to execute the process, always receive the same error:
 invalid
  input syntax for integer: Breërivier
 
  Any one knows something about this?

 Do you have the latest version of osm2pgsql? I beleive this was fixed
 a few days ago...

 Have a nice day,
 --
 Martijn van Oosterhout [EMAIL PROTECTED] http://svana.org/kleptog/

___
talk mailing list
talk@openstreetmap.org
http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/talk

[OSM-talk] osm2pgsql - invalid input syntax for integer : Breërivier

[Ricardo Peironcely]

I've a problem with osm2pgsql and the last version of planet file.

When I try to execute the process, always receive the same error: invalid
input syntax for integer: Breërivier

Any one knows something about this?

Thanks in advance.
Rpr

./osm2pgsql -u -U postgres -d map planet-080409.osm.bz2
osm2pgsql SVN version 0.53-20080413 $Rev: 7345 $

Password:
Using projection SRS 3395 (WGS84 Mercator)
Setting up table: planet_osm_point
Setting up table: planet_osm_line
Setting up table: planet_osm_polygon
Setting up table: planet_osm_roads

Reading in file: /Users/rpr/Desktop/planet-080409.osm.bz2
Processing: Node(231808k) Way(18398k) Relation(0k)exit
Processing: Node(231808k) Way(18781k) Relation(6k)
Node stats: total(231808103), max(256523410)
Way stats: total(18781596), max(23689770)
Relation stats: total(6967), max(9763)

Writing way(1249k)
COPY_END for planet_osm_point failed: ERROR:  invalid input syntax for
integer: Breërivier
CONTEXT:  COPY planet_osm_point, line 919015, column admin_level:
Breërivier

Error occurred, cleaning up



-- 
Un saludo.

Ricardo Peironcely
___
talk mailing list
talk@openstreetmap.org
http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/talk