Re: [OSM-talk] osm2pgsql 1.2.0

[Sarah Hoffmann]

Hi,

On Tue, Nov 05, 2019 at 12:16:30AM +0100, wambac...@posteo.de wrote:
> i'm using osm2pgsql-0.96 right now. Is it possible (and meaningful) to
> switch to osm2pgsql-1.2.0?
> 
> Must i change anything in my database? Runing diff updates using a flatfile.

You can switch from 0.96 to 1.2 any time. The database layout
is compatible between the two versions. The only difference is
that 1.x no longer supports old-style multipolygons. That should
not be an issue with the curent planet anyway.

I'd always recommend using the latest release to get the latest fixes.
However, the main performance improvements for 1.2 are noticable during
import. If you prefer a packaged 0.96 over a self-compiled 1.2, then
it's not too much loss to remain with the packages version.

Sarah


___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

[OSM-talk] osm2pgsql 1.2.0

[wambacher]

Hi,

i'm using osm2pgsql-0.96 right now. Is it possible (and meaningful) to 
switch to osm2pgsql-1.2.0?


Must i change anything in my database? Runing diff updates using a flatfile.

Regards
walter

--
My projects:

Admin Boundaries of the World 
Missing Boundaries 


Emergency Map 
Postal Code Map (Germany only) 
Fools (QA for zipcodes in Germany) 
Postcode Boundaries of Germany 
OSM Software Watchlist 

___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] osm2pgsql diff application with filtered OSM data

[Nick Whitelegg]

>This will leave your site down between the delete and import of new data.

>It's also going to be fragile, because using append mode with a file that 
>isn't a diff isn't supported, and if the area has a lot of footpaths, it'll be 
>slower, since >append has to do more work.

>If you match the SQL delete and osmosis filtering carefully you shouldn't get 
>too many errors, but you've probably got some to do with updates and >changing 
>object types.

>As long as you're aware of these problems and it works for your needs, I'd say 
>to go for it.


Thanks. I have actually used append frequently over several years and never had 
a problem with it; main issue is that memory is expensive on servers so I have 
to do take various less-than-ideal steps to work round low memory.


The intention is to bring the site down over the (say) 2-3 hours the update is 
taking place; this is what I used to do when I last performed weekly updates. 
This would be scheduled for (probably) Wednesday 0100-0400 UK time, a time that 
the server is likely to see little traffic as it's UK and Ireland oriented; at 
such time most of Europe will be asleep. Granted there might be a few 
out-of-Europe visitors at that time; but as my site is quite niche with limited 
visitors, I consider this acceptable downtime.


Nick


From: Paul Norman 
Sent: 13 November 2018 10:12:58
To: talk@openstreetmap.org
Subject: Re: [OSM-talk] osm2pgsql diff application with filtered OSM data

On 2018-11-11 7:53 AM, Nick Whitelegg wrote:


After thinking about this, I realised that I don't really want to update _all_ 
the data that often. The only thing I need to update on a weekly basis is the 
footpaths (I'm not so bothered if say the roads, or the pubs are a year out of 
date - as long as newly mapped footpaths appear quickly). So what I'm now doing 
is just doing an osmosis extract of paths weekly, deleting all data in the DB 
which I class as a 'path' and repopulating in amend mode.

This will leave your site down between the delete and import of new data.

It's also going to be fragile, because using append mode with a file that isn't 
a diff isn't supported, and if the area has a lot of footpaths, it'll be 
slower, since append has to do more work.

If you match the SQL delete and osmosis filtering carefully you shouldn't get 
too many errors, but you've probably got some to do with updates and changing 
object types.

As long as you're aware of these problems and it works for your needs, I'd say 
to go for it.
___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] osm2pgsql diff application with filtered OSM data

[Paul Norman]

On 2018-11-11 7:53 AM, Nick Whitelegg wrote:



After thinking about this, I realised that I don't really want to 
update _all_ the data that often. The only thing I need to update on a 
weekly basis is the footpaths (I'm not so bothered if say the roads, 
or the pubs are a year out of date - as long as newly mapped footpaths 
appear quickly). So what I'm now doing is just doing an osmosis 
extract of paths weekly, deleting all data in the DB which I class as 
a 'path' and repopulating in amend mode.




This will leave your site down between the delete and import of new data.

It's also going to be fragile, because using append mode with a file 
that isn't a diff isn't supported, and if the area has a lot of 
footpaths, it'll be slower, since append has to do more work.


If you match the SQL delete and osmosis filtering carefully you 
shouldn't get too many errors, but you've probably got some to do with 
updates and changing object types.


As long as you're aware of these problems and it works for your needs, 
I'd say to go for it.
___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] osm2pgsql diff application with filtered OSM data

[Nick Whitelegg]

Thanks for all the replies.


After thinking about this, I realised that I don't really want to update _all_ 
the data that often. The only thing I need to update on a weekly basis is the 
footpaths (I'm not so bothered if say the roads, or the pubs are a year out of 
date - as long as newly mapped footpaths appear quickly). So what I'm now doing 
is just doing an osmosis extract of paths weekly, deleting all data in the DB 
which I class as a 'path' and repopulating in amend mode.


Thanks,

Nick



From: Paul Norman 
Sent: 08 November 2018 20:10:14
To: talk@openstreetmap.org
Subject: Re: [OSM-talk] osm2pgsql diff application with filtered OSM data

On 2018-11-08 6:34 AM, Nick Whitelegg wrote:

At the moment I download full planet extracts about every 6 months. However, 
due to the limitations of my server, I filter out (with osmosis) a lot of stuff 
I don't need so that I am basically left with roads, footpaths, natural 
features, water features and selected POIs.


I'd like to move towards a system which applies diffs from geofabrik instead, 
and applies them regularly (daily or weekly) with osm2pgsql.


My question is this; given that not everything in the diff will be in my 
database (as I filter out what I don't need during the import process), will 
osm2pgsql apply the diff successfully or will it complain that not all features 
in the diff are in my database?

I can think of four ways to do this, all which have a different balance of 
correctness, performance, and ease of use.

There are two "right" ways to do this. The first one is to re-import every 
week. Because imports without slim tables (either --slim --drop or no --slim) 
are faster, this is a good option and needs less space than a database able to 
consume diffs.

The second right way involves keeping two files, one with the current full 
data, and one with the filtered data. Call these "planet.pbf" and 
"planet-filtered.pbf". Then when updating create "planet-new.pbf", filter it to 
get "planet-filtered-new.pbf", create a diff for the differences between 
"planet-filtered-new.pbf" and "planet-filtered.pbf", and apply that diff to the 
database. Then replace the old files with the new ones. This will keep the 
database correct.

A "wrong" way to do it is to import the filtered data, apply updates directly, 
and periodically delete data from the DB. The problem with this is that if 
someone adds one of the selected POI tags to a building that you have filtered 
out, osm2pgsql won't have the node data to create a geometry. This might be 
acceptable, depending on use case.

A less wrong way would be to modify your filtering so no nodes are filtered 
out. There are still potential errors with relations, but these are less 
common. If you're doing the planet or a large extract and using flat nodes 
there's no storage penalty for having all the nodes.
___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] osm2pgsql diff application with filtered OSM data

[Nick Whitelegg]

... append mode!





From: Nick Whitelegg
Sent: 11 November 2018 15:53:18
To: talk@openstreetmap.org
Subject: Re: [OSM-talk] osm2pgsql diff application with filtered OSM data



Thanks for all the replies.


After thinking about this, I realised that I don't really want to update _all_ 
the data that often. The only thing I need to update on a weekly basis is the 
footpaths (I'm not so bothered if say the roads, or the pubs are a year out of 
date - as long as newly mapped footpaths appear quickly). So what I'm now doing 
is just doing an osmosis extract of paths weekly, deleting all data in the DB 
which I class as a 'path' and repopulating in amend mode.


Thanks,

Nick



From: Paul Norman 
Sent: 08 November 2018 20:10:14
To: talk@openstreetmap.org
Subject: Re: [OSM-talk] osm2pgsql diff application with filtered OSM data

On 2018-11-08 6:34 AM, Nick Whitelegg wrote:

At the moment I download full planet extracts about every 6 months. However, 
due to the limitations of my server, I filter out (with osmosis) a lot of stuff 
I don't need so that I am basically left with roads, footpaths, natural 
features, water features and selected POIs.


I'd like to move towards a system which applies diffs from geofabrik instead, 
and applies them regularly (daily or weekly) with osm2pgsql.


My question is this; given that not everything in the diff will be in my 
database (as I filter out what I don't need during the import process), will 
osm2pgsql apply the diff successfully or will it complain that not all features 
in the diff are in my database?

I can think of four ways to do this, all which have a different balance of 
correctness, performance, and ease of use.

There are two "right" ways to do this. The first one is to re-import every 
week. Because imports without slim tables (either --slim --drop or no --slim) 
are faster, this is a good option and needs less space than a database able to 
consume diffs.

The second right way involves keeping two files, one with the current full 
data, and one with the filtered data. Call these "planet.pbf" and 
"planet-filtered.pbf". Then when updating create "planet-new.pbf", filter it to 
get "planet-filtered-new.pbf", create a diff for the differences between 
"planet-filtered-new.pbf" and "planet-filtered.pbf", and apply that diff to the 
database. Then replace the old files with the new ones. This will keep the 
database correct.

A "wrong" way to do it is to import the filtered data, apply updates directly, 
and periodically delete data from the DB. The problem with this is that if 
someone adds one of the selected POI tags to a building that you have filtered 
out, osm2pgsql won't have the node data to create a geometry. This might be 
acceptable, depending on use case.

A less wrong way would be to modify your filtering so no nodes are filtered 
out. There are still potential errors with relations, but these are less 
common. If you're doing the planet or a large extract and using flat nodes 
there's no storage penalty for having all the nodes.
___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] osm2pgsql diff application with filtered OSM data

[Paul Norman]

On 2018-11-08 6:34 AM, Nick Whitelegg wrote:


At the moment I download full planet extracts about every 6 months. 
However, due to the limitations of my server, I filter out (with 
osmosis) a lot of stuff I don't need so that I am basically left with 
roads, footpaths, natural features, water features and selected POIs.



I'd like to move towards a system which applies diffs from geofabrik 
instead, and applies them regularly (daily or weekly) with osm2pgsql.



My question is this; given that not everything in the diff will be in 
my database (as I filter out what I don't need during the import 
process), will osm2pgsql apply the diff successfully or will it 
complain that not all features in the diff are in my database?




I can think of four ways to do this, all which have a different balance 
of correctness, performance, and ease of use.


There are two "right" ways to do this. The first one is to re-import 
every week. Because imports without slim tables (either --slim --drop or 
no --slim) are faster, this is a good option and needs less space than a 
database able to consume diffs.


The second right way involves keeping two files, one with the current 
full data, and one with the filtered data. Call these "planet.pbf" and 
"planet-filtered.pbf". Then when updating create "planet-new.pbf", 
filter it to get "planet-filtered-new.pbf", create a diff for the 
differences between "planet-filtered-new.pbf" and "planet-filtered.pbf", 
and apply that diff to the database. Then replace the old files with the 
new ones. This will keep the database correct.


A "wrong" way to do it is to import the filtered data, apply updates 
directly, and periodically delete data from the DB. The problem with 
this is that if someone adds one of the selected POI tags to a building 
that you have filtered out, osm2pgsql won't have the node data to create 
a geometry. This might be acceptable, depending on use case.


A less wrong way would be to modify your filtering so no nodes are 
filtered out. There are still potential errors with relations, but these 
are less common. If you're doing the planet or a large extract and using 
flat nodes there's no storage penalty for having all the nodes.
___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] osm2pgsql diff application with filtered OSM data

[Andy Townsend]

On 08/11/2018 16:16, Darafei "Komяpa" Praliaskouski wrote:


Usually people also clip minutely osc, as per day database grows by a 
small country otherwise.


There's a worked example of that (in a slightly different context) at 
https://wiki.openstreetmap.org/wiki/User:SomeoneElse/Ubuntu_1804_tileserver_load#Updating_your_database_as_people_edit_OpenStreetMap 
.  That uses https://github.com/zverik/regional - I'm guessing you'll be 
able to "borrow" everything you need from there.


Best Regards,

Andy


___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] osm2pgsql diff application with filtered OSM data

[Nick Whitelegg]

Hello Darafei and Frederik,


OK - thanks for that.


What I will do therefore, to avoid unwanted data, is to generate a filtered 
planet extract and import that into the DB, and then generate another filtered 
extract and find the diff between the two. Advantage of that is that I'll only 
need to  do the first filtering run on the server, and subsequent runs on my 
local machine (assuming the resulting diff is small enough to upload from my 
local machine to the server).


Thanks,

Nick



From: Darafei "Komяpa" Praliaskouski 
Sent: 08 November 2018 16:16:48
To: Nick Whitelegg
Cc: osm-talk
Subject: Re: [OSM-talk] osm2pgsql diff application with filtered OSM data

Hi Nick,

osm2pgsql is tolerant to features absent in database. You can in theory even 
start with empty set of tables and just insert new diff data.

Usually people also clip minutely osc, as per day database grows by a small 
country otherwise.

чт, 8 нояб. 2018 г. в 17:37, Nick Whitelegg 
mailto:nick.whitel...@solent.ac.uk>>:


... sorry, when I say "full planet extracts" I mean only England, Wales, 
Scotland, Ireland (all) and Greece - not the entire planet.

Thanks,

Nick


From: Nick Whitelegg
Sent: 08 November 2018 14:34:17
To: osm-talk
Subject: osm2pgsql diff application with filtered OSM data



Hi,


Looking towards overhauling the import system I use for my Freemap site 
(free-map.org.uk<http://free-map.org.uk>) which is itself going to go through 
an overhaul in the near future by moving to Tangram and hopefully applying hill 
shading.


At the moment I download full planet extracts about every 6 months. However, 
due to the limitations of my server, I filter out (with osmosis) a lot of stuff 
I don't need so that I am basically left with roads, footpaths, natural 
features, water features and selected POIs.


I'd like to move towards a system which applies diffs from geofabrik instead, 
and applies them regularly (daily or weekly) with osm2pgsql.


My question is this; given that not everything in the diff will be in my 
database (as I filter out what I don't need during the import process), will 
osm2pgsql apply the diff successfully or will it complain that not all features 
in the diff are in my database?


Thanks,

Nick

___
talk mailing list
talk@openstreetmap.org<mailto:talk@openstreetmap.org>
https://lists.openstreetmap.org/listinfo/talk
--
Darafei Praliaskouski
Support me: http://patreon.com/komzpa
___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] osm2pgsql diff application with filtered OSM data

[Frederik Ramm]

Hi,

On 08.11.2018 15:34, Nick Whitelegg wrote:
> My question is this; given that not everything in the diff will be in my
> database (as I filter out what I don't need during the import process),
> will osm2pgsql apply the diff successfully or will it complain that not
> all features in the diff are in my database?

It will apply the diff as far as possible, but it will also add a lot of
unwanted/unnecessary stuff from the diff to the database which you would
then have to filter out again.

There's also a danger that you will miss some things. For example,
consider a rural garage (shop=car_repair) mapped as a way with four
nodes; you filter that out because you're not interested. Half a year
later the shop closes, and a tourism=hostel opens instead. Assuming you
are interested in this kind of POI, the diff that you process contains
the way with the new tags, but not the nodes since these have not
changed. The hostel will not appear in your osm2pgsql database because
the geometry cannot be built due to lack of nodes.

One way to avoid this downloading the full files, filtering them, and
then generating a diff by comparing the filtered file to the last
filtered file, and importing *that* diff.

Bye
Frederik

-- 
Frederik Ramm  ##  eMail frede...@remote.org  ##  N49°00'09" E008°23'33"

___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] osm2pgsql diff application with filtered OSM data

[Komяpa]

Hi Nick,

osm2pgsql is tolerant to features absent in database. You can in theory
even start with empty set of tables and just insert new diff data.

Usually people also clip minutely osc, as per day database grows by a small
country otherwise.

чт, 8 нояб. 2018 г. в 17:37, Nick Whitelegg :

>
> ... sorry, when I say "full planet extracts" I mean only England, Wales,
> Scotland, Ireland (all) and Greece - not the entire planet.
>
>
> Thanks,
>
> Nick
>
> --
> *From:* Nick Whitelegg
> *Sent:* 08 November 2018 14:34:17
> *To:* osm-talk
> *Subject:* osm2pgsql diff application with filtered OSM data
>
>
>
> Hi,
>
>
> Looking towards overhauling the import system I use for my Freemap site (
> free-map.org.uk) which is itself going to go through an overhaul in the
> near future by moving to Tangram and hopefully applying hill shading.
>
>
> At the moment I download full planet extracts about every 6 months.
> However, due to the limitations of my server, I filter out (with osmosis) a
> lot of stuff I don't need so that I am basically left with roads,
> footpaths, natural features, water features and selected POIs.
>
>
> I'd like to move towards a system which applies diffs from geofabrik
> instead, and applies them regularly (daily or weekly) with osm2pgsql.
>
>
> My question is this; given that not everything in the diff will be in my
> database (as I filter out what I don't need during the import process),
> will osm2pgsql apply the diff successfully or will it complain that not all
> features in the diff are in my database?
>
>
> Thanks,
>
> Nick
>
> ___
> talk mailing list
> talk@openstreetmap.org
> https://lists.openstreetmap.org/listinfo/talk
>
-- 
Darafei Praliaskouski
Support me: http://patreon.com/komzpa
___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] osm2pgsql diff application with filtered OSM data

[Nick Whitelegg]

... sorry, when I say "full planet extracts" I mean only England, Wales, 
Scotland, Ireland (all) and Greece - not the entire planet.

Thanks,

Nick


From: Nick Whitelegg
Sent: 08 November 2018 14:34:17
To: osm-talk
Subject: osm2pgsql diff application with filtered OSM data



Hi,


Looking towards overhauling the import system I use for my Freemap site 
(free-map.org.uk) which is itself going to go through an overhaul in the near 
future by moving to Tangram and hopefully applying hill shading.


At the moment I download full planet extracts about every 6 months. However, 
due to the limitations of my server, I filter out (with osmosis) a lot of stuff 
I don't need so that I am basically left with roads, footpaths, natural 
features, water features and selected POIs.


I'd like to move towards a system which applies diffs from geofabrik instead, 
and applies them regularly (daily or weekly) with osm2pgsql.


My question is this; given that not everything in the diff will be in my 
database (as I filter out what I don't need during the import process), will 
osm2pgsql apply the diff successfully or will it complain that not all features 
in the diff are in my database?


Thanks,

Nick

___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

[OSM-talk] osm2pgsql diff application with filtered OSM data

[Nick Whitelegg]

Hi,


Looking towards overhauling the import system I use for my Freemap site 
(free-map.org.uk) which is itself going to go through an overhaul in the near 
future by moving to Tangram and hopefully applying hill shading.


At the moment I download full planet extracts about every 6 months. However, 
due to the limitations of my server, I filter out (with osmosis) a lot of stuff 
I don't need so that I am basically left with roads, footpaths, natural 
features, water features and selected POIs.


I'd like to move towards a system which applies diffs from geofabrik instead, 
and applies them regularly (daily or weekly) with osm2pgsql.


My question is this; given that not everything in the diff will be in my 
database (as I filter out what I don't need during the import process), will 
osm2pgsql apply the diff successfully or will it complain that not all features 
in the diff are in my database?


Thanks,

Nick

___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] osm2pgsql 0.90.0 release

[Jeff McKenna]

Thank you for the update Paul, I am excited to try the improved MSVC 
compiler support soon.  (I am currently upgrading to Visual Studio 2015, 
from an old 2008)


-jeff


--
Jeff McKenna
MapServer Consulting and Training Services
http://www.gatewaygeomatics.com/




On 2016-03-02 10:27 AM, Paul Norman wrote:

Osm2pgsql 0.90.0 has been released. The major changes since 0.88.0 are

- Windows support is much improved, and osm2pgsql builds natively with MSVC

- Osm2pgsql now requires a C++11 compiler

- Memory overhead for very small extracts is significantly reduced

- Multi-threading support is enabled for Windows

- The build system has switched from autotools to cmake

- Parsing is now done by libosmium and protobuf-c-compiler is not required

- Multi-backend improvements and bug fixes

- PBF support is no longer optional

- Legacy code from 32-bit node IDs has been removed

- Builds now generate no GCC or Clang warnings or errors

Upgrading is strongly recommended for anyone using the multi backend or
Windows.

This will be the last version that supports PostGIS 1.5 or PostgreSQL
9.0. Both of these have been end of life for some time.

Future version will also use EPSG 3857 by default, not 900913.

A full list of commits can be found at
https://github.com/openstreetmap/osm2pgsql/compare/0.88.0...0.90.0





___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

[OSM-talk] osm2pgsql 0.90.0 release

[Paul Norman]

Osm2pgsql 0.90.0 has been released. The major changes since 0.88.0 are

- Windows support is much improved, and osm2pgsql builds natively with MSVC

- Osm2pgsql now requires a C++11 compiler

- Memory overhead for very small extracts is significantly reduced

- Multi-threading support is enabled for Windows

- The build system has switched from autotools to cmake

- Parsing is now done by libosmium and protobuf-c-compiler is not required

- Multi-backend improvements and bug fixes

- PBF support is no longer optional

- Legacy code from 32-bit node IDs has been removed

- Builds now generate no GCC or Clang warnings or errors

Upgrading is strongly recommended for anyone using the multi backend or 
Windows.


This will be the last version that supports PostGIS 1.5 or PostgreSQL 
9.0. Both of these have been end of life for some time.


Future version will also use EPSG 3857 by default, not 900913.

A full list of commits can be found at 
https://github.com/openstreetmap/osm2pgsql/compare/0.88.0...0.90.0
___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

[OSM-talk] osm2pgsql and JOSM osm files

[ri0ter]

Hi.
I've been trying to export osm file created by JOSM to my pgsql databse but
I'm having a problem with multipolygons. Everything seems to be fine in
windows build, but in linux version relations are parsed (as I see in slim
mode relations are added) but they are not added to planet_osm_polygon
table. Any ideas why's that happening or what am I doing wrong?
I'm using latest build from GIT but I also tried few older ones.



--
View this message in context: 
http://gis.19327.n5.nabble.com/osm2pgsql-and-JOSM-osm-files-tp5818686.html
Sent from the General Discussion mailing list archive at Nabble.com.

___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] osm2pgsql multipolygon parsing

[Peter Wendorff]

Am 23.09.2013 18:48, schrieb Yves:
> Sorry, I meant "osm2pgsql is not used for the slippy map ONLY".
> Thanks for all the feedback :)
Sure, but changing the DEFAULT behaviour to a more strict one while
keeping the old behaviour with a flag should enable anybody to keep the
old behaviour on demand; and whoever updates a software like this to a
new version should read the release-notes to get this IMHO (or be
patient if something changes).

regards
Peter

___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] osm2pgsql multipolygon parsing

[Peter Wendorff]

Am 24.09.2013 10:12, schrieb Stefan Keller:
> 2013/9/23 Kai Krueger  wrote:
>> Indirectly it is a question of tagging schemas.
> 
> To me this is actually indirectly a question of a proper area type!
> See e.g. "Towards an Area Datatype for OSM" from Jochen at SOTM
> http://lanyrd.com/2013/sotm/scpkrr/
+1
But the transition necessary is the same:
IF we could come to a stable, but backwards compatible solution by
step-by-step hardening the current multipolygons it would be easier to
create a new area datatype later converting multipolygons on the fly.

While it is NOT possible overall to convert ways to polygons later (as
you would have to do that tag-based, and have to make sure to duplicate
it if there are line-tags and area-tags on the same way and so on), it
might be possible to auto-convert multipolygons to area; but to do that
multipolygons have to be clearly defined and possible to handle.

The discussion of this thread tackles the problem left for any such
conversion in future: What is the area described by any given
multipolygon with the given tags on inner/outer/relation?

If we could come to a clear consensus about that, and if we could slowly
enforce fixing any problems according to that in the data, then a
conversion to a new area data type could be done with less errors and
problems than it could be done now.

But thanks for the area datatype argument, as it's an additional
argument to strenghten the Multipolygon definition and -interpretation
for the current (or very near future) osm.

regards
Peter

> 
> --Stefan
> 
> 
> 2013/9/23 Kai Krueger 
> 
>> "Petr Morávek [Xificurk]"-2 wrote
>>> Anyway, this thread was not started to debate tagging schemes, the
>>> question I (and others) wanted to discuss here is this:
>>> Given the data that are currently in the database, how should osm2pgsql
>>> handle the import to get as much as possible multipolygons right?
>>
>> Indirectly it is a question of tagging schemas. With osm2pgsql being the
>> tool used in the default map rendering on osm.org and the prevalence of
>> "tagging for the renderer" decisions on how it handles multipolygons will
>> (and imho to a limited degree should) influence how people tag and what
>> they
>> perceive as correct tagging. Therefore it is important that there is a
>> consensus of what the correct tagging schema is and make sure that is
>> correctly supported by osm2pgsql. That is also why I think having this
>> discussion on talk, rather than on github or the dev list is appropriate.
>>
>> We need to come to a consensus between all of the main tools (at least iD,
>> P2, JOSM, osm2pgsql, osrm, ...) and the mappers to what the preferred,
>> encouraged and supported standard for tagging multi-polygons is and make
>> sure that all documentation is in line with this.
>>
>>
>>
>>
>>
>> --
>> View this message in context:
>> http://gis.19327.n5.nabble.com/osm2pgsql-multipolygon-parsing-tp5778300p5778654.html
>> Sent from the General Discussion mailing list archive at Nabble.com.
>>
>> ___
>> talk mailing list
>> talk@openstreetmap.org
>> https://lists.openstreetmap.org/listinfo/talk
>>
> 
> 
> 
> ___
> talk mailing list
> talk@openstreetmap.org
> https://lists.openstreetmap.org/listinfo/talk
> 


___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] osm2pgsql multipolygon parsing

[Stefan Keller]

2013/9/23 Kai Krueger  wrote:
> Indirectly it is a question of tagging schemas.

To me this is actually indirectly a question of a proper area type!
See e.g. "Towards an Area Datatype for OSM" from Jochen at SOTM
http://lanyrd.com/2013/sotm/scpkrr/

--Stefan


2013/9/23 Kai Krueger 

> "Petr Morávek [Xificurk]"-2 wrote
> > Anyway, this thread was not started to debate tagging schemes, the
> > question I (and others) wanted to discuss here is this:
> > Given the data that are currently in the database, how should osm2pgsql
> > handle the import to get as much as possible multipolygons right?
>
> Indirectly it is a question of tagging schemas. With osm2pgsql being the
> tool used in the default map rendering on osm.org and the prevalence of
> "tagging for the renderer" decisions on how it handles multipolygons will
> (and imho to a limited degree should) influence how people tag and what
> they
> perceive as correct tagging. Therefore it is important that there is a
> consensus of what the correct tagging schema is and make sure that is
> correctly supported by osm2pgsql. That is also why I think having this
> discussion on talk, rather than on github or the dev list is appropriate.
>
> We need to come to a consensus between all of the main tools (at least iD,
> P2, JOSM, osm2pgsql, osrm, ...) and the mappers to what the preferred,
> encouraged and supported standard for tagging multi-polygons is and make
> sure that all documentation is in line with this.
>
>
>
>
>
> --
> View this message in context:
> http://gis.19327.n5.nabble.com/osm2pgsql-multipolygon-parsing-tp5778300p5778654.html
> Sent from the General Discussion mailing list archive at Nabble.com.
>
> ___
> talk mailing list
> talk@openstreetmap.org
> https://lists.openstreetmap.org/listinfo/talk
>
___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] osm2pgsql multipolygon parsing

[Kai Krueger]

"Petr Morávek [Xificurk]"-2 wrote
> Anyway, this thread was not started to debate tagging schemes, the
> question I (and others) wanted to discuss here is this:
> Given the data that are currently in the database, how should osm2pgsql
> handle the import to get as much as possible multipolygons right?

Indirectly it is a question of tagging schemas. With osm2pgsql being the
tool used in the default map rendering on osm.org and the prevalence of
"tagging for the renderer" decisions on how it handles multipolygons will
(and imho to a limited degree should) influence how people tag and what they
perceive as correct tagging. Therefore it is important that there is a
consensus of what the correct tagging schema is and make sure that is
correctly supported by osm2pgsql. That is also why I think having this
discussion on talk, rather than on github or the dev list is appropriate.

We need to come to a consensus between all of the main tools (at least iD,
P2, JOSM, osm2pgsql, osrm, ...) and the mappers to what the preferred,
encouraged and supported standard for tagging multi-polygons is and make
sure that all documentation is in line with this.

 



--
View this message in context: 
http://gis.19327.n5.nabble.com/osm2pgsql-multipolygon-parsing-tp5778300p5778654.html
Sent from the General Discussion mailing list archive at Nabble.com.

___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] osm2pgsql multipolygon parsing

[Petr Morávek [Xificurk]]

Dne 23.9.2013 16:03, Peter Wendorff napsal(a):
> Am 23.09.2013 15:20, schrieb "Petr Morávek [Xificurk]":
>>
>> I propose that:
>> 1) By default the relation and ways are treated separately
>>  - relation creates polygon with tags from the relation and ways are
>> processed on their own.
>> 2) If and only if the relation has only type=multipolygon tag go to
>> "backward compatibility mode". Copy over tags from outer ways that are
>> present on all of them and create polygon. Go over all member ways and
>> if all their tags are present on the created polygon, then mark them as
>> done, otherwise process them separately.
> 
> +0.75 ;)
> 
> Agree, but I would use (2) if and only if "backward compatibility mode"
> is active.
> Additionally I would not use the backward compatibility mode for the
> mapnik default sheet (after some time with big announcements an probably
> a tool detecting/reporting possible problems for fixing).
> 
> With this we could
> - "enforce" more correct tagging in future
> - engage the mappers community to fix where the bad-style tagging is
> used yet
> - gracefully allow any other application to fall back to the old style
> as long as they want to.
> 
> This might (!) degrade the perceived quality of the default mapnik layer
> for some time (!), but IMHO it's worth it as it simplifies multipolygon
> interpretation and on the long term teaches mappers to use the tagging
> that is correct and matches that simplified interpretation.
> 
> regards
> Peter

This is probably a good idea in the long-run, but there are things that
need to be done before this cut-off and all of them need a certain
transitional period.
1) You need to edit wiki and make people aware that this tagging scheme
is considered deprecated.
2) You need to make sure, that all the editors produce correct
multipolygon relations.
3) Fix the deprecated tagging on most of the elements.

I really don't think it's a good idea to suddenly stop rendering quarter
of the multipolygons out there just to enforce some thought-out
policy... I don't see who would benefit from this.


Anyway, this thread was not started to debate tagging schemes, the
question I (and others) wanted to discuss here is this:
Given the data that are currently in the database, how should osm2pgsql
handle the import to get as much as possible multipolygons right?


Best regards,
Petr Morávek aka Xificurk

___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] osm2pgsql multipolygon parsing

[Yves]

Sorry, I meant "osm2pgsql is not used for the slippy map ONLY".
Thanks for all the feedback :)
Yves
-- 
Envoyé de mon téléphone Android avec K-9 Mail. Excusez la brièveté.___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] osm2pgsql multipolygon parsing

[Peter Wendorff]

Am 23.09.2013 15:20, schrieb "Petr Morávek [Xificurk]":
> 
> I propose that:
> 1) By default the relation and ways are treated separately
>  - relation creates polygon with tags from the relation and ways are
> processed on their own.
> 2) If and only if the relation has only type=multipolygon tag go to
> "backward compatibility mode". Copy over tags from outer ways that are
> present on all of them and create polygon. Go over all member ways and
> if all their tags are present on the created polygon, then mark them as
> done, otherwise process them separately.

+0.75 ;)

Agree, but I would use (2) if and only if "backward compatibility mode"
is active.
Additionally I would not use the backward compatibility mode for the
mapnik default sheet (after some time with big announcements an probably
a tool detecting/reporting possible problems for fixing).

With this we could
- "enforce" more correct tagging in future
- engage the mappers community to fix where the bad-style tagging is
used yet
- gracefully allow any other application to fall back to the old style
as long as they want to.

This might (!) degrade the perceived quality of the default mapnik layer
for some time (!), but IMHO it's worth it as it simplifies multipolygon
interpretation and on the long term teaches mappers to use the tagging
that is correct and matches that simplified interpretation.

regards
Peter

___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] osm2pgsql multipolygon parsing

[Petr Morávek [Xificurk]]

Dne 23.9.2013 11:59, Paul Norman napsal(a):
> Unless the closed way is a member of a multipolygon relation with no other
> tags on the relation - then you'll have a resulting area with a hole. This
> is a very well established means of tagging areas with holes (~22% of
> type=multipolygon relations have no other tags).

Yes, but if the relation has any[*] additional tags, you can't reliably
decide what was the intended purpose of this tagging. Imho the only
logical thing is to treat the relation and ways separately.
[*] Maybe some internal keys could be ignored, e.g. odbl, fixme, ...

> The issues raised originally in the ticket are best explored through
> examples
> 
> The first case is a way with natural=water and a MP relation with no other
> tags. Both old osm2pgsql (0.82) and latest master version from git create an
> area with a hole with natural=water on the area. There are no suggestions of
> changing this.

Agreed.

> The second is a way with natural=water and a MP relation with name=foo (with
> the way as a member). Old osm2pgsql created an area with a hole with
> natural=water, name=foo, latest master does too.

Is this really correct?
In case of the name=foo it is probably safe assumption, but how about
multipolygon relation with only access=* tag (or something similar)? How
do you decide, if it means the area with holes is
natural=water+access=*, or that the whole area (with no holes) is
natural=water and only some parts have access=*?

> The fourth is a way with natural=water and a MP relation with foo=bar. In
> old osm2pgsql this created an area without a hole, in latest master it
> depends on the .style file used for import.

The dependence on on the .style file bothers me, I think it's a mistake
and will ultimately come back to haunt us. If you don't care about some
features and delete the lines from your .style file, the multipolygon
parsing should not change.

--

I propose that:
1) By default the relation and ways are treated separately
 - relation creates polygon with tags from the relation and ways are
processed on their own.
2) If and only if the relation has only type=multipolygon tag go to
"backward compatibility mode". Copy over tags from outer ways that are
present on all of them and create polygon. Go over all member ways and
if all their tags are present on the created polygon, then mark them as
done, otherwise process them separately.

--

I have looked at all the multipolygon relations that do not have any of
the well-known polygon tags (the ones defined in default.style), this is
less than 27% of all the multipolygon relations. 85% of these relations
have type=multipolygon tag only, so the proposed change in fact affects
less than 4% of all the multipolygons.

More detailed breakdown of tags on multipolygons without well-known
polygon tags:
percentage  keys
85.5%   type
3.7%ref,type,id_ob,adr_les
1.7%type,name
1.5%area,type,highway


Best regards,
Petr Morávek aka Xificurk


___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] osm2pgsql multipolygon parsing

[Peter Wendorff]

Am 23.09.2013 11:59, schrieb Paul Norman:
>> From: Martin Koppenhoefer [mailto:dieterdre...@gmail.com]
>> Subject: Re: [OSM-talk] osm2pgsql multipolygon parsing
>>
>> it has a different meaning. tags on a closed way are for the whole area
>> inside the way, tags on a mp relation are for the area of the outer
>> minus the inner ways.
> 
> Unless the closed way is a member of a multipolygon relation with no other
> tags on the relation - then you'll have a resulting area with a hole. This
> is a very well established means of tagging areas with holes (~22% of
> type=multipolygon relations have no other tags).
It's well established, but it's less correct from a theoretical point of
view, and establishment is produced here by software that supports wrong
tagging - in the past.
You propose to leave it as it is, which means software should estimate
the meaning because often there are errors in the tagging, while I
proposed (and as far as I understand at least Martin supports that) not
to ignore these errors with the goal to teach mappers about what's
correct instead of teaching them what's correct enough for application X
(be it the default mapnik style and -pipeline).

> The issues raised originally in the ticket are best explored through
> examples
> 
> The first case is a way with natural=water and a MP relation with no other
> tags. Both old osm2pgsql (0.82) and latest master version from git create an
> area with a hole with natural=water on the area. There are no suggestions of
> changing this.
Problem:
Let's say the multipolygon relation was tagged before as depth=shallow
(and someone wanted to mark parts of the water as not deep - e.g. not
deep enough for boats). Let's say another mapper came along and found
this strange tags and - removing them - accidently left a multipolygon
without tags. This changes the meaning of stuff never touched again, as
you would now count the natural=water only for the area covered by the
multipolygon, excluding the hole.

> The second is a way with natural=water and a MP relation with name=foo (with
> the way as a member). Old osm2pgsql created an area with a hole with
> natural=water, name=foo, latest master does too.
What if the inner part does NOT have that name, but only the outer ring?
What if that is what somebody wanted to say?

regards
Peter

___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] osm2pgsql multipolygon parsing

[Paul Norman]

> From: Martin Koppenhoefer [mailto:dieterdre...@gmail.com]
> Subject: Re: [OSM-talk] osm2pgsql multipolygon parsing
> 
> it has a different meaning. tags on a closed way are for the whole area
> inside the way, tags on a mp relation are for the area of the outer
> minus the inner ways.

Unless the closed way is a member of a multipolygon relation with no other
tags on the relation - then you'll have a resulting area with a hole. This
is a very well established means of tagging areas with holes (~22% of
type=multipolygon relations have no other tags).

The issues raised originally in the ticket are best explored through
examples

The first case is a way with natural=water and a MP relation with no other
tags. Both old osm2pgsql (0.82) and latest master version from git create an
area with a hole with natural=water on the area. There are no suggestions of
changing this.

The second is a way with natural=water and a MP relation with name=foo (with
the way as a member). Old osm2pgsql created an area with a hole with
natural=water, name=foo, latest master does too.

The third is the second with the same as the 2nd, but a name:de tag added.
In old osm2pgsql this created an area without a hole, in latest master it
creates one with a hole.

The fourth is a way with natural=water and a MP relation with foo=bar. In
old osm2pgsql this created an area without a hole, in latest master it
depends on the .style file used for import.

To make #3 consistent with #2 without breaking backwards compatibility
latest osm2pgsql doesn't use a pre-defined list of tags to differentiate
between "old-style" and "new-style" MPs, it uses its list of tags that
denote an area.

The specific issue that brought this up is actually fixed by only bringing
tags onto the MP area where all outer members have the tag, but there's
still a more general question.


___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] osm2pgsql multipolygon parsing

[Martin Koppenhoefer]

> Am 23/set/2013 um 11:03 schrieb Pieren :
> 
> Check cities with tens of thousands buildings. You will have sometime
> the building tag on ways, sometimes on relations. Having the tag
> always on the surrounding way is more consistent and easier to catch
> for everybody, including newcomers.


it has a different meaning. tags on a closed way are for the whole area inside 
the way, tags on a mp relation are for the area of the outer minus the inner 
ways.

Where to put which tag depends on what you want to express.

cheers,
Martin
___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] osm2pgsql multipolygon parsing

[Janko Mihelić]

2013/9/23 Pieren 

>
> -1
> Check cities with tens of thousands buildings. You will have sometime
> the building tag on ways, sometimes on relations. Having the tag
> always on the surrounding way is more consistent and easier to catch
> for everybody, including newcomers.
>
>
Not if they use iD. In iD multipolygons and areas are selected in the same
way, by clicking in the building.

Janko
___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] osm2pgsql multipolygon parsing

[Pieren]

On Sun, Sep 22, 2013 at 12:29 PM, Peter Wendorff
 wrote:
>> The suggestion is to discourage this in all cases and encourage always
>> tagging the relation (this is also straightforward and much easier as "you
>> can do A or B").
> +1

-1
Check cities with tens of thousands buildings. You will have sometime
the building tag on ways, sometimes on relations. Having the tag
always on the surrounding way is more consistent and easier to catch
for everybody, including newcomers.

Pieren

___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] osm2pgsql multipolygon parsing

[Peter Wendorff]

Am 22.09.2013 11:31, schrieb Martin Koppenhoefer:
> 2013/9/22 yvecai 
> 
>> Of course, it should be accompagnied with a large campaign of
>> multi-polygons fix.
>>
> 
> 
> I'd suggest to start modifying the recommendations in the wiki:
> http://wiki.openstreetmap.org/wiki/Multipolygon
> reads:
> "If you have one closed way making up the outer ring and it does not
> describe something in its own right, you *may* also put these tags on the
> outer ring and leave the relation untagged."
> 
> The suggestion is to discourage this in all cases and encourage always
> tagging the relation (this is also straightforward and much easier as "you
> can do A or B").

+1

regards
Peter

___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] osm2pgsql multipolygon parsing

[SomeoneElse]

On 22/09/2013 10:03, yvecai wrote:


Of course, it should be accompagnied with a large campaign of 
multi-polygons fix.


... and a patch to any editors that don't create multipolygons in this 
format.  For example, here are three attempts at multipolygons in iD, P2 
and JOSM:


http://api06.dev.openstreetmap.org/#map=15/53.2692/-0.2806&layers=D

(you may need to turn the data layer on manually to see the them)

The left-hand one was me trying to create one in iD (I've not worked out 
how to do that yet).  The middle one is P2, and the right-hand one is JOSM.


You'll notice that the right-hand one doesn't render in P2 despite being 
the arguably the more logically-tagged.  I suspect that this is and has 
been for some time a "patches welcome" type of situation.


Cheers,

Andy


___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] osm2pgsql multipolygon parsing

[Martin Koppenhoefer]

2013/9/22 yvecai 

> Of course, it should be accompagnied with a large campaign of
> multi-polygons fix.
>


I'd suggest to start modifying the recommendations in the wiki:
http://wiki.openstreetmap.org/wiki/Multipolygon
reads:
"If you have one closed way making up the outer ring and it does not
describe something in its own right, you *may* also put these tags on the
outer ring and leave the relation untagged."

The suggestion is to discourage this in all cases and encourage always
tagging the relation (this is also straightforward and much easier as "you
can do A or B").

cheers,
Martin
___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] osm2pgsql multipolygon parsing

[yvecai]

Osm2pgsql is not used for the default map on osm.org.
While the current behaviour in osm2pgsql is OK for consumers, could a 
'strict' mode to handle mutipolygons be used on osm.org default map ?


Of course, it should be accompagnied with a large campaign of 
multi-polygons fix.


Yves

___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] osm2pgsql multipolygon parsing

[Peter Wendorff]

Am 22.09.2013 04:14, schrieb Eugene Alvin Villar:
> 
> I agree that this is a good way of tagging multipolygons.
> 
> Unfortunately, many people don't tag multipolygons in this way. I've seen
> people add building=yes to an outer way of a building with holes even
> though there's a multipolygon relation with that tag already. It's most
> likely that these people are not familiar with relations and they see an
> outer way with no building=yes tag and decided to "helpfully" tag it.
> 
> Because of this, a more complicated interpretation of tags, such as
> Frederik's, leads to less breakage (think rendering) and is more in line
> with people's expectations.
> 
Because of this I agree with Frederiks approach for any "consumer"-map
to produce. If I would print a map for a customer, I would follow his
assumptions most probably, but I disagree for the "default" style(s) on
the osm.org website, as these in fact are driving factors to how people tag.
If people tag multipolygons right (tm), this should be honoured by a
correct display on the map. If it's wrong they should see that there's
an error, therefore IMHO the default stylesheet MUST NOT gracefully
forgive these errors as a map more dedicated to end users would probably do.

Following these assumptions on the main map which is often used to check
the correctness of the own edits means to hide errors and to tell
mappers the wrong thing: you have done all right - where that's not the
case.

regards
Peter

___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] osm2pgsql multipolygon parsing

[Martin Koppenhoefer]

> Am 22/set/2013 um 04:14 schrieb Eugene Alvin Villar :
> 
> It's most likely that these people are not familiar with relations and they 
> see an outer way with no building=yes tag and decided to "helpfully" tag it.
> 
> Because of this, a more complicated interpretation of tags, such as 
> Frederik's, leads to less breakage (think rendering) and is more in line with 
> people's expectations.


it leads to more broken data in the end, because it will continue to look OK on 
the map and many mappers are checking this visual feedback in order to judge 
their mapping.

cheers,
Martin
___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] osm2pgsql multipolygon parsing

[Eugene Alvin Villar]

On Sun, Sep 22, 2013 at 4:51 AM, Peter Wendorff
wrote:

> IMHO it's clear:
> - a tag on a way describes that way, if it's a closed way and the tag is
> describing an area, the tag matches the complete area inside that polygon.
> - if a way is outer of a multipolygon and there are tags on the way,
> these tags nevertheless describe the whole area, including all holes, as
> it's still a tag on the (simple) polygon.
> - if a way is inner of a multipolygon and there are tags on the way,
> these tags describe the polygon described by this way.
> - tags on the multipolygon relation describe the area represented by the
> relation - that is all outer polygons minus the inner polygons.
>

I agree that this is a good way of tagging multipolygons.

Unfortunately, many people don't tag multipolygons in this way. I've seen
people add building=yes to an outer way of a building with holes even
though there's a multipolygon relation with that tag already. It's most
likely that these people are not familiar with relations and they see an
outer way with no building=yes tag and decided to "helpfully" tag it.

Because of this, a more complicated interpretation of tags, such as
Frederik's, leads to less breakage (think rendering) and is more in line
with people's expectations.
___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] osm2pgsql multipolygon parsing

[Peter Wendorff]

IMHO it's clear:
- a tag on a way describes that way, if it's a closed way and the tag is
describing an area, the tag matches the complete area inside that polygon.
- if a way is outer of a multipolygon and there are tags on the way,
these tags nevertheless describe the whole area, including all holes, as
it's still a tag on the (simple) polygon.
- if a way is inner of a multipolygon and there are tags on the way,
these tags describe the polygon described by this way.
- tags on the multipolygon relation describe the area represented by the
relation - that is all outer polygons minus the inner polygons.

If we follow this assumption, it's
1) not conflicting with double-tagging, which is in fact often useful.
Example: A big park with a lake in the middle of a big lawn.
- the whole area (including lawn and lake) is a park => the outer
polygon is tagged as a park
- only the small patch in the middle is a lake => the inner polygon is
tagged as a lake
- the whole area except the lake surface is a lawn => tag the
multipolygon as lawn.
Tags:
outer: {leisure=park}
inner: {natural=water}
mp:{landuse=grass}

2) it's unambiguous
Following Frederiks assumptions instead my tagging of the example above
would be interpreted as:
- area of the multipolygon (outer minus inner): leisure=park
- inner: natural=water
- whole area: no attributes (am I right?)

Thus the park is missing now (or Frederiks description is incomplete).

An even better example is one where the inner has a subset of the
multipolygon tags, let's say:
mp: {landuse=forest, wood=coniferous}
inner: {landuse=forest}

Frederiks assumption lead here to:
- mp: landuse=forest, wood=coniferous
- inner: untagged, as the tags are ignored because they don't differ
from the mp.

regards
Peter


Am 21.09.2013 17:00, schrieb Frederik Ramm:
> Hi,
> 
>> The remaining question is, what should be the correct behavior?
> 
> My assumption until now was:
> 
> * If a multipolygon is untagged - where "untagged" means that it has no
> tags except a small list (type, source, source:*, note) then it will
> simply receive *all* tags from all (outer) member ways, but processing
> will fail if these tags differ in values.
> 
> * If the multipolygon is tagged (with anything except type, source, note
> etc) then those are the tags for the polygon and way tags will not be
> looked at.
> 
> * If the tags of the inner way differ from the tags of the outer way or
> the tags of the relation, except source, note etc., then the inner way
> will become a separate polygon.
> 
> All this was without trying to make a judgement about which tags are
> "polygon tags" and which aren't.
> 
> Bye
> Frederik
> 


___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] osm2pgsql multipolygon parsing

[Martin Koppenhoefer]

2013/9/21 Frederik Ramm 

> Hi,
>
> > The remaining question is, what should be the correct behavior?
>
> My assumption until now was:
>
> * If a multipolygon is untagged - where "untagged" means that it has no
> tags except a small list (type, source, source:*, note) then it will
> simply receive *all* tags from all (outer) member ways, but processing
> will fail if these tags differ in values.
>


but this introduces a lot of complexity, especially in the case of MPs with
many outer members, when you only want to modify the tags of a single one
of these (outer) ways. Many editors (e.g. on smartphones) will not even
notify you that the way is part of a relation.



> * If the multipolygon is tagged (with anything except type, source, note
> etc) then those are the tags for the polygon and way tags will not be
> looked at.
>


do you also include "name" in this list?



> * If the tags of the inner way differ from the tags of the outer way or
> the tags of the relation, except source, note etc., then the inner way
> will become a separate polygon.
>


all tags, or only "the relevant" ones? And if yes, which ones aren't
relevant?
And also here there might be cases where e.g. inside a forest you will want
to have another forest because of some properties they don't share (be it
name but also note etc.)

cheers,
Martin
___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] osm2pgsql multipolygon parsing

[Frederik Ramm]

Hi,

> The remaining question is, what should be the correct behavior?

My assumption until now was:

* If a multipolygon is untagged - where "untagged" means that it has no
tags except a small list (type, source, source:*, note) then it will
simply receive *all* tags from all (outer) member ways, but processing
will fail if these tags differ in values.

* If the multipolygon is tagged (with anything except type, source, note
etc) then those are the tags for the polygon and way tags will not be
looked at.

* If the tags of the inner way differ from the tags of the outer way or
the tags of the relation, except source, note etc., then the inner way
will become a separate polygon.

All this was without trying to make a judgement about which tags are
"polygon tags" and which aren't.

Bye
Frederik

-- 
Frederik Ramm  ##  eMail frede...@remote.org  ##  N49°00'09" E008°23'33"

___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

[OSM-talk] osm2pgsql multipolygon parsing

[Petr Morávek [Xificurk]]

Hello,

I've run into some problems with osm2pgsql parsing of multipolygon
relations, so I've opened an issue on github [1]. It turned out that the
behavior was recently changed.

The remaining question is, what should be the correct behavior?

The question is not easy to answer, because there are many multipolygon
relations with deprecated tagging scheme - having the relevant tags on
member ways instead of directly on the relation. Thus, osm2pgsql needs
to do some kind of magic to determine what tags were meant for the
multipolygon and what tags were meant for the individual ways only.

There is probably no perfect solution, but we need to decide which of
the imperfect ones will be considered the "correct" one ;-) So, what is
your opinion on this?


Best regards,
Petr Morávek aka Xificurk

[1] https://github.com/openstreetmap/osm2pgsql/issues/80

___
talk mailing list
talk@openstreetmap.org
https://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] osm2pgsql import failed, no more space on device

[Frederik Ramm]

Hi,

On 09/07/2013 08:48 AM, Torsten Mohr wrote:
> I think --flat-nodes needs a parameter, but i'm not sure what to give there. 

A file name.

> I can't imagine that 480 Gb is not enough, can anybody tell me what went 
> wrong?

Are you *sure* that your PostgreSQL table space is actually on the 480
GB disk? Have you monitored disk usage during the import?

Bye
Frederik


-- 
Frederik Ramm  ##  eMail frede...@remote.org  ##  N49°00'09" E008°23'33"

___
talk mailing list
talk@openstreetmap.org
http://lists.openstreetmap.org/listinfo/talk

[OSM-talk] osm2pgsql import failed, no more space on device

[Torsten Mohr]

Hello,

i have set up PostgresQL on a 480 Gb SSD, so there should be heaps of space 
for the database and the import.

The commented log of the import is at the end of this mail.

The import was running for less than a day and stopped sometime in the night.

I think --flat-nodes needs a parameter, but i'm not sure what to give there.  
But also, in another mail it was said it saves ~ 20Gb.


I can't imagine that 480 Gb is not enough, can anybody tell me what went 
wrong?

I think the import stopped quite early, maybe this is a hint on what went 
wrong?

I don't know how to further analyse the problem, maybe somebody has a hint?
 

Thanks for any hints
Torsten


postgres@schleim:/local/ftp/osm3> osm2pgsql -c -d postgis --flat-nodes 
planet-130904.osm.bz2
osm2pgsql SVN version 0.83.0 (64bit id space)

Usage error. For further information see:
osm2pgsql -h|--help

=> --flat-nodes does not seem available though it is mentioned
in the examples of osm2pgsql -h

postgres@schleim:/local/ftp/osm3> osm2pgsql -c -d postgis -s 
planet-130904.osm.bz2
osm2pgsql SVN version 0.83.0 (64bit id space)

Using projection SRS 900913 (Spherical Mercator)
Setting up table: planet_osm_point
HINWEIS:  Tabelle »planet_osm_point« existiert nicht, wird übersprungen
HINWEIS:  Tabelle »planet_osm_point_tmp« existiert nicht, wird übersprungen
Setting up table: planet_osm_line
HINWEIS:  Tabelle »planet_osm_line« existiert nicht, wird übersprungen
HINWEIS:  Tabelle »planet_osm_line_tmp« existiert nicht, wird übersprungen
Setting up table: planet_osm_polygon
HINWEIS:  Tabelle »planet_osm_polygon« existiert nicht, wird übersprungen
HINWEIS:  Tabelle »planet_osm_polygon_tmp« existiert nicht, wird übersprungen
Setting up table: planet_osm_roads
HINWEIS:  Tabelle »planet_osm_roads« existiert nicht, wird übersprungen
HINWEIS:  Tabelle »planet_osm_roads_tmp« existiert nicht, wird übersprungen
Using built-in tag processing pipeline
Allocating memory for dense node cache
Allocating dense node cache in one big chunk
Allocating memory for sparse node cache
Sharing dense sparse
Node-cache: cache=800MB, maxblocks=102401*8192, allocation method=11
Mid: pgsql, scale=100 cache=800
Setting up table: planet_osm_nodes
HINWEIS:  Tabelle »planet_osm_nodes« existiert nicht, wird übersprungen
HINWEIS:  CREATE TABLE / PRIMARY KEY erstellt implizit einen Index 
»planet_osm_nodes_pkey« für Tabelle »planet_osm_nodes«
Setting up table: planet_osm_ways
HINWEIS:  Tabelle »planet_osm_ways« existiert nicht, wird übersprungen
HINWEIS:  CREATE TABLE / PRIMARY KEY erstellt implizit einen Index 
»planet_osm_ways_pkey« für Tabelle »planet_osm_ways«
Setting up table: planet_osm_rels
HINWEIS:  Tabelle »planet_osm_rels« existiert nicht, wird übersprungen
HINWEIS:  CREATE TABLE / PRIMARY KEY erstellt implizit einen Index 
»planet_osm_rels_pkey« für Tabelle »planet_osm_rels«

Reading in file: planet-130904.osm.bz2
Processing: Node(2013050k 165.2k/s) Way(0k 0.00k/s) Relation(0 0.00/s)COPY_END 
for COPY planet_osm_nodes FROM STDIN;
 failed: FEHLER:  konnte Datei »base/16384/19767.5« nicht erweitern: Auf dem 
Gerät ist kein Speicherplatz mehr verfügbar
HINT:  Prüfen Sie den freien Festplattenplatz.
CONTEXT:  COPY planet_osm_nodes, Zeile 123372446: »166865452465558107   
-931871821  \N«

Error occurred, cleaning up
postgres@schleim:/local/ftp/osm3> 


___
talk mailing list
talk@openstreetmap.org
http://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] Osm2pgsql/TileMill - how to detect whether way is in a route?

[Phil! Gold]

* Steve Bennett  [2013-03-19 10:52 +1100]:
> One way to fudge this is to render the way, then render the route
> relation over the top, obscuring it. But that doesn't work if you want
> to use line smoothing (eg, line-smooth:0.6). The route relation will
> form a smooth bezier curve the intersection of two ways, but the way
> rendering will form a sharp point.
> 
> So I'm just wondering if there is a way around this - some setting in
> osm2pgsql, some SQL magic I could play with.

I do something very similar with my highway shield rendering, which is
driven by route relations.  I go through the planet_osm_rels table that's
created by osm2pgsql's slim mode, which is considered cheating in some
circles.

My rendering has code at https://launchpad.net/osm-shields , but the meat
of what you're interested in is the following two functions:

-- osm2pgsql stores tags in PostgreSQL arrays, where even-numbered array
-- members are tags and odd-numbered members are the tags' values.  This
-- is a convenience function to go through such an array and retrieve the
-- value for a given tag.
CREATE OR REPLACE FUNCTION osm_get_tag_value(tags TEXT[], tag TEXT)
RETURNS TEXT
IMMUTABLE
RETURNS NULL ON NULL INPUT
LANGUAGE plpgsql
AS $$
DECLARE
  tag_index INTEGER;
BEGIN
  SELECT i INTO tag_index
FROM (SELECT generate_subscripts(tags, 1) AS i) AS indices
WHERE i % 2 = 1 AND tags[i] = tag;
  RETURN tags[tag_index + 1];
END;
$$;

CREATE OR REPLACE FUNCTION 
shields_get_route_memberships(planet_osm_rels.id%TYPE)
RETURNS fullref[]
STABLE
RETURNS NULL ON NULL INPUT
COST 5000
LANGUAGE SQL
AS $$
  SELECT 
shields_replace_groups(array_agg(ROW(shields_sanitize_text(network), 
shields_sanitize_text(ref))::fullref))
FROM (SELECT osm_get_tag_value(tags, 'route') route,
 UPPER(osm_get_tag_value(tags, 'network')) network,
 UPPER(COALESCE(NULLIF(osm_get_tag_value(tags, 'ref'), ''), 
osm_get_tag_value(tags, 'name'))) AS ref
FROM planet_osm_rels
WHERE parts && ARRAY[$1]
  AND parts[way_off:array_upper(parts, 1)] && ARRAY[$1]) AS 
ref_inner
WHERE route = 'road' AND network IS NOT NULL AND ref IS NOT NULL;
$$;

There's a lot of shield-rendering-specific code in there.  A more succinct
query might be the following (where $1 is the OSM ID of the way for which
you want to determine route membership):

SELECT *
  FROM (SELECT osm_get_tag_value(tags, 'route') route,
   osm_get_tag_value(tags, 'network') network,
   osm_get_tag_value(tags, 'ref') ref,
   osm_get_tag_value(tags, 'name') name
  FROM planet_osm_rels
  WHERE parts && ARRAY[$1]
AND parts[way_off:array_upper(parts, 1)] && ARRAY[$1]) AS 
ref_inner
  WHERE route IS NOT NULL;

Obviously, you can add your own parameters in the statement.

___
talk mailing list
talk@openstreetmap.org
http://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] Osm2pgsql/TileMill - how to detect whether way is in a route?

[Frederik Ramm]

Hi,

On 19.03.2013 00:52, Steve Bennett wrote:

So I'm just wondering if there is a way around this - some setting in
osm2pgsql, some SQL magic I could play with.


It will be very slow magic, but two things come to mind:

One would use the line geometry table and ask for routes in the vicinity 
to be subtracted from the way geometry; if the resulting geometry is 
empty then your way was part of a route. In pseudo SQL something like


select way,attribute,attribute,attribute,st_isempty(st_difference(way, 
(select way from planet_osm_roads t2 where t2.osm_id<0 and 
t2.has_mtb_route_tags and t2.way && way) as in_mtb_route from 
planet_osm_ways...


The other would be using the planet_osm_rels table that only exists ins 
slim mode and from there get a list of all relations the way in question 
is a member of, check those relations for mtb routes...


The only *efficient* way I can think of is extending osm2pgsql so that 
it sets an extra flag for ways that are part of a route relation.


Bye
Frederik

--
Frederik Ramm  ##  eMail frede...@remote.org  ##  N49°00'09" E008°23'33"

___
talk mailing list
talk@openstreetmap.org
http://lists.openstreetmap.org/listinfo/talk

[OSM-talk] Osm2pgsql/TileMill - how to detect whether way is in a route?

[Steve Bennett]

Hi,
  I'm trying to render a mountain bike map with TileMill. Some trails
are in route relations, eg:

3xWay: highway=path, mtb=yes
1xRelation: type=route, route=mtb, name="My Cool Trail"

Osm2pgsql converts those to 4 rows: the relation is the complete trail.

What I want to do is not render any way that is also part of a route
relation - in that case I just want to render the route.

One way to fudge this is to render the way, then render the route
relation over the top, obscuring it. But that doesn't work if you want
to use line smoothing (eg, line-smooth:0.6). The route relation will
form a smooth bezier curve the intersection of two ways, but the way
rendering will form a sharp point.

So I'm just wondering if there is a way around this - some setting in
osm2pgsql, some SQL magic I could play with. The ultimate goal would
be to be able to write rules like:


#trails[highway="path"][mb="yes"][in_mtb_route="no"] {
...
}

Thanks,
Steve

___
talk mailing list
talk@openstreetmap.org
http://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] osm2pgsql hstore (was: Wind turbines no longer rendered on mapnik layer)

[Nathan Edgars II]

On 2/16/2012 6:00 PM, Jochen Topf wrote:

Generic key names can be confusing, especially when one OSM object is used for
multiple things. Say there is a way tagged as railway and at the same time this
way is part of an area tagged as a generating station. Does "power_source" mean
the type of generating station or the type of power used by the railway
(overhead vs. third rail vs. unelectrified)? I hope this example is
hypothetical, but people do strange things in OSM...


A better example is a highway=service service=alley with a railway=rail 
service=spur down the middle.


___
talk mailing list
talk@openstreetmap.org
http://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] osm2pgsql hstore (was: Wind turbines no longer rendered on mapnik layer)

[Jochen Topf]

On Fri, Feb 17, 2012 at 02:02:26PM +, Jukka Rahkonen wrote:
> Jochen Topf  remote.org> writes:
> 
> 
> > 
> > Much too slow. Requiring joins on every query is not a good idea. And having
> > the right indexes is important. You can't just index everything and hope it
> > would do the right thing. (The most important index btw is the geometry 
> > index
> > not the attribute indexes, although having the right attribute indexes here
> > or there can help.)
>  
> I sent by followup accidentally to osm-dev list. The main message was that
> perhaps there is a need to do some tests before saying if hstore or joined
> tables are faster.
> 
> Here is a Spatialite database for testing
> 
> http://latuviitta.org/documents/relation_test.zip
> 
> First trials suggest that joined tables with proper indexes are not at all
> slow and compared with the standard tables created by osm2pgsql the query
> times can be several times faster. This is not a fair comparison because
> osm2pgsql tables are missing the attribute indexes but so they normally do
> in the Mapnik chain. Query times vary, so try yourself. Data is osm_lines
> from Berlin from some months ago.

Sorry, but that kind of test doesn't really tell you anything about real
performance on a real system. Do this test with data for the whole planet with
one process updating the database and multiple readers accessing the db at the
same time. And results you get with sqlite do not tell you anything about
results for postgresql.

> By the way, recent development around GDAL suggests that Spatialite may be
> faster with spatial and attribute queries than PostGIS. Improvements in GDAL
> Spatialite driver made Mapserver about 20 times faster than it used to be and
> now Spatialite is faster that PostGIS and even shapefiles.
> 
> 
> query from osm_line from osm2pgsql
> ===
> select geometry, highway from osm_line
> where highway='primary'
> 0.531 seconds
> 
> query from joined tables
> ===
> select * from line_geometry, tags
> where tags.tag='highway' and tags.value='primary'
> and tags.join_id=line_geometry.osm_id
> 0.059 seconds

These numbers are also irrelevant for our discussion here because Mapnik
doesn't do this kinds of queries. Mapnik always queries with a bounding box.

Jochen
-- 
Jochen Topf  joc...@remote.org  http://www.remote.org/jochen/  +49-721-388298


___
talk mailing list
talk@openstreetmap.org
http://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] osm2pgsql hstore (was: Wind turbines no longer rendered on mapnik layer)

[Jukka Rahkonen]

Jochen Topf  remote.org> writes:


> 
> Much too slow. Requiring joins on every query is not a good idea. And having
> the right indexes is important. You can't just index everything and hope it
> would do the right thing. (The most important index btw is the geometry index
> not the attribute indexes, although having the right attribute indexes here
> or there can help.)
 
I sent by followup accidentally to osm-dev list. The main message was that
perhaps there is a need to do some tests before saying if hstore or joined
tables are faster.

Here is a Spatialite database for testing

http://latuviitta.org/documents/relation_test.zip

First trials suggest that joined tables with proper indexes are not at all
slow and compared with the standard tables created by osm2pgsql the query
times can be several times faster. This is not a fair comparison because
osm2pgsql tables are missing the attribute indexes but so they normally do
in the Mapnik chain. Query times vary, so try yourself. Data is osm_lines
from Berlin from some months ago.

By the way, recent development around GDAL suggests that Spatialite may be
faster with spatial and attribute queries than PostGIS. Improvements in GDAL
Spatialite driver made Mapserver about 20 times faster than it used to be and
now Spatialite is faster that PostGIS and even shapefiles.


query from osm_line from osm2pgsql
===
select geometry, highway from osm_line
where highway='primary'
0.531 seconds

query from joined tables
===
select * from line_geometry, tags
where tags.tag='highway' and tags.value='primary'
and tags.join_id=line_geometry.osm_id
0.059 seconds


-Jukka Rahkonen-


___
talk mailing list
talk@openstreetmap.org
http://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] osm2pgsql hstore (was: Wind turbines no longer rendered on mapnik layer)

[Graham Jones]

Thanks - I will give it another try - I have got a computer running Ubuntu
11.10, which has Postgres 9.1.2.   I will create a second database with an
hstore and see how it compares.

Graham.

On 17 February 2012 11:36, Stephan Knauss  wrote:

> Komяpa writes:
>
>> - is a database generated by osm2pgsql with an hstore expected to perform
>>> similarly to one without?
>>>
>>
>> It performs at the same speed for me.
>> Of course you're not supposed to use hstore for every tag, but just
>> for those on highest zoom levels, where spatial indexes are used
>> mostly, not indexes for other columns.
>>
>
> if you are using Postgres 9.1 you are advised to update to 9.1.2. There
> was a bug introduced that prevented index usage. For example testing for a
> tag like
> SELECT * from table WHERE hstore ? tag
> did not use an index and could make a query horribly slow. We experienced
> this on Toolserver.
>
> Stephan
>



-- 
Graham Jones
Hartlepool, UK.
___
talk mailing list
talk@openstreetmap.org
http://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] osm2pgsql hstore (was: Wind turbines no longer rendered on mapnik layer)

[Stephan Knauss]

Komяpa writes: 


- is a database generated by osm2pgsql with an hstore expected to perform
similarly to one without?


It performs at the same speed for me. 


Of course you're not supposed to use hstore for every tag, but just
for those on highest zoom levels, where spatial indexes are used
mostly, not indexes for other columns.


if you are using Postgres 9.1 you are advised to update to 9.1.2. There was 
a bug introduced that prevented index usage. For example testing for a tag 
like

SELECT * from table WHERE hstore ? tag
did not use an index and could make a query horribly slow. We experienced 
this on Toolserver. 



Stephan

___
talk mailing list
talk@openstreetmap.org
http://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] osm2pgsql hstore (was: Wind turbines no longer rendered on mapnik layer)

[Komяpa]

> - is a database generated by osm2pgsql with an hstore expected to perform
> similarly to one without?

It performs at the same speed for me.

Of course you're not supposed to use hstore for every tag, but just
for those on highest zoom levels, where spatial indexes are used
mostly, not indexes for other columns.

-- 
Darafei "Komяpa" Praliaskouski
OSM BY Team - http://openstreetmap.by/
xmpp:m...@komzpa.net mailto:m...@komzpa.net

___
talk mailing list
talk@openstreetmap.org
http://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] osm2pgsql hstore (was: Wind turbines no longer rendered on mapnik layer)

[Jochen Topf]

On Fri, Feb 17, 2012 at 07:44:41AM +, Jukka Rahkonen wrote:
> Frederik Ramm  remote.org> writes:
> 
> > 
> > Hi,
> > 
> > On 02/16/2012 07:25 PM, Graham Jones wrote:
> > > This reminded me of a question I have been meaning to ask for quite a
> > > while - is a database generated by osm2pgsql with an hstore expected to
> > > perform similarly to one without?
> > 
> > I never ran one with hstore when I think of what this must mean for the 
> > database engine, and storage space, then I shudder and would not be 
> > surprised by the factor 5 you mentioned.
> 
> How about doing it with relations? Let the importer program create four 
> tables 
> osm_point
> osm_line
> osm_polygon
> osm_relation
> 
> These tables would each have two attributes: Geometry and osm_is. Geometries 
> in
> the osm_relation would be of type "geometry collection", that is, a collection
> of whatever, and it could hold for example the tranport route relation with 
> the
> route and bus stops and everything in one PostGIS geometry object.
> 
> Then there should be one or four tables for tags (everything in one table or
> split to suit the geometry tables). The three attributes would be osm_id, key,
> and value. Osm_id would be used as a foreing key for joining geometries and
> attributes. If osm_id can not be guaranteed to be unique then there should be
> point_id, line_id, polygon_id and relation_id added to suitable places.
> 
> Every attribute would be indexed. For Mapnik use where all that is needed is 
> to
> do simple SQL selects.  I guess that this database would be faster to query 
> than
> hstore even it also contains all the tags. It might be faster for Mapnik than
> the current tables with extra wide attribute schema because now attributes are
> not indexed at all (or are they?). SQL queries according to tags and values or
> even with a part of the tag and value strings would be easier to do than from
> hstore. People could enhance attributes for some special uses by converting
> max_speeds and other values which are actually measures from strings to 
> integers
> or doubles etc.
> 
> This option is so obvious that I believe that someone must have tried it
> already. It would be nice to hear about experiences.

Much too slow. Requiring joins on every query is not a good idea. And having
the right indexes is important. You can't just index everything and hope it
would do the right thing. (The most important index btw is the geometry index
not the attribute indexes, although having the right attribute indexes here
or there can help.)

Jochen
-- 
Jochen Topf  joc...@remote.org  http://www.remote.org/jochen/  +49-721-388298


___
talk mailing list
talk@openstreetmap.org
http://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] osm2pgsql hstore (was: Wind turbines no longer rendered on mapnik layer)

[Jukka Rahkonen]

Frederik Ramm  remote.org> writes:

> 
> Hi,
> 
> On 02/16/2012 07:25 PM, Graham Jones wrote:
> > This reminded me of a question I have been meaning to ask for quite a
> > while - is a database generated by osm2pgsql with an hstore expected to
> > perform similarly to one without?
> 
> I never ran one with hstore when I think of what this must mean for the 
> database engine, and storage space, then I shudder and would not be 
> surprised by the factor 5 you mentioned.

How about doing it with relations? Let the importer program create four tables 
osm_point
osm_line
osm_polygon
osm_relation

These tables would each have two attributes: Geometry and osm_is. Geometries in
the osm_relation would be of type "geometry collection", that is, a collection
of whatever, and it could hold for example the tranport route relation with the
route and bus stops and everything in one PostGIS geometry object.

Then there should be one or four tables for tags (everything in one table or
split to suit the geometry tables). The three attributes would be osm_id, key,
and value. Osm_id would be used as a foreing key for joining geometries and
attributes. If osm_id can not be guaranteed to be unique then there should be
point_id, line_id, polygon_id and relation_id added to suitable places.

Every attribute would be indexed. For Mapnik use where all that is needed is to
do simple SQL selects.  I guess that this database would be faster to query than
hstore even it also contains all the tags. It might be faster for Mapnik than
the current tables with extra wide attribute schema because now attributes are
not indexed at all (or are they?). SQL queries according to tags and values or
even with a part of the tag and value strings would be easier to do than from
hstore. People could enhance attributes for some special uses by converting
max_speeds and other values which are actually measures from strings to integers
or doubles etc.

This option is so obvious that I believe that someone must have tried it
already. It would be nice to hear about experiences.

-Jukka Rahkonen-


___
talk mailing list
talk@openstreetmap.org
http://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] osm2pgsql hstore (was: Wind turbines no longer rendered on mapnik layer)

[Martin Koppenhoefer]

2012/2/17 Jochen Topf :
> On Thu, Feb 16, 2012 at 06:25:30PM +, Graham Jones wrote:
> Generic key names can be confusing, especially when one OSM object is used for
> multiple things.


+1.
E.g. an object tagged barrier=fence, height=2, landuse=forest. In this
example you could also use 2 objects: a closed way for the fence and a
multipolygon for the area. Linear objects we can create with route
relations (instead of overlapping ways). We don't have currently a
relation for nodes and maybe we don't need this utterly (you could
place another node "nearby"), but it would solve some cases where you
want to state the topology precisely (e.g. 2 objects at the same
pole). This way we can have "things" (expressed by a relation) which
have their geometry only as an attribute, instead of a geometry that
_is_ something.


> Non-generic tags also make tools such as Taginfo more useful, because each tag
> stands on its own. They make it immediately clear if the tagging is 
> incomplete.
> If the "generator" tag got lost, how do we know what the "power_source" tag
> is supposed to mean?


+1 for taginfo, but I don't think we need the namespaces for the case
something gets "lost" (there is the history for this).

Cheers,
Martin

___
talk mailing list
talk@openstreetmap.org
http://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] osm2pgsql hstore (was: Wind turbines no longer rendered on mapnik layer)

[Jochen Topf]

On Thu, Feb 16, 2012 at 06:25:30PM +, Graham Jones wrote:
>  Why create a key generator:power_source rather than just use
> power_source.  power_source is much more generic so you could re-cycle it
> for things like district heating, but generator:power_source is only ever
> going to be used for generating stations, and needs a new column in the
> database. .   I think I just prefer more generic, re-usable keys
> rather than trying to invent a new one for each situation

Generic key names can be confusing, especially when one OSM object is used for
multiple things. Say there is a way tagged as railway and at the same time this
way is part of an area tagged as a generating station. Does "power_source" mean
the type of generating station or the type of power used by the railway
(overhead vs. third rail vs. unelectrified)? I hope this example is
hypothetical, but people do strange things in OSM...

Non-generic tags also make tools such as Taginfo more useful, because each tag
stands on its own. They make it immediately clear if the tagging is incomplete.
If the "generator" tag got lost, how do we know what the "power_source" tag
is supposed to mean?

So on the one side you have generic tags like "type" and "class" and "id" which
are very confusing. You never know what they belong to. On the other side you
have rather complex tagging schemas like in your generator:... example. Thats
sometimes hard to use, too. We still have to figure out where the sweet spot is,
but I tend to prefer the more explicit keys.

Jochen
-- 
Jochen Topf  joc...@remote.org  http://www.remote.org/jochen/  +49-721-388298


___
talk mailing list
talk@openstreetmap.org
http://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] osm2pgsql hstore (was: Wind turbines no longer rendered on mapnik layer)

[Lennard]

On 16-2-2012 19:25, Graham Jones wrote:


 Why create a key generator:power_source rather than just use
power_source.  power_source is much more generic so you could re-cycle
it for things like district heating, but generator:power_source is only
ever going to be used for generating stations, and needs a new column in
the database. .   I think I just prefer more generic,
re-usable keys rather than trying to invent a new one for each situation


This is a core problem with the public_transport=* scheme as well. This 
tag in and of itself is fine, but the whole additional shebang that is 
train=yes/no, bus=yes/no, subway=yes/no, etc etc is so dreadful (in the 
context of the rendering chain) that every time I read #2798 I feel the 
urge to run away screaming.


http://trac.openstreetmap.org/ticket/2798

--
Lennard

___
talk mailing list
talk@openstreetmap.org
http://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] osm2pgsql hstore (was: Wind turbines no longer rendered on mapnik layer)

[Graham Jones]

On 16 February 2012 18:51, Frederik Ramm  wrote:

> On 02/16/2012 07:25 PM, Graham Jones wrote:
>
>> This reminded me of a question I have been meaning to ask for quite a
>> while - is a database generated by osm2pgsql with an hstore expected to
>> perform similarly to one without?
>>
>
> I never ran one with hstore when I think of what this must mean for the
> database engine, and storage space, then I shudder and would not be
> surprised by the factor 5 you mentioned.


Thanks - I don't feel so bad about giving up so easily now!

If you want to add a new key but avoid re-importing the full database, you
> can add the key to osm2pgsql's style file, somehow generate an .osm file
> that contains only the objects that have this tag, wrap this .osm file into
> a "..." instead of
> "..." and throw it at osm2pgsql in append mode.


That's a neat idea - will try that next time, thanks.


> People like keys that are understandable without context. Contrast this
> with a tag "width=2" - where you have no chance in knowing what it means
> without looking at the whole thing.


Yes - I think it is just a style difference - I am quite happy to interpret
it from context for the sake of having a smaller number of unique keys to
think about.

Cheers

Graham

-- 
Graham Jones
Hartlepool, UK.
___
talk mailing list
talk@openstreetmap.org
http://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] osm2pgsql hstore (was: Wind turbines no longer rendered on mapnik layer)

[Frederik Ramm]

Hi,

On 02/16/2012 07:25 PM, Graham Jones wrote:

This reminded me of a question I have been meaning to ask for quite a
while - is a database generated by osm2pgsql with an hstore expected to
perform similarly to one without?


I never ran one with hstore when I think of what this must mean for the 
database engine, and storage space, then I shudder and would not be 
surprised by the factor 5 you mentioned.



I keep having to re-import my database to add new keys.


If you want to add a new key but avoid re-importing the full database, 
you can add the key to osm2pgsql's style file, somehow generate an .osm 
file that contains only the objects that have this tag, wrap this .osm 
file into a "..." instead of 
"..." and throw it at osm2pgsql in append mode.



power_source is much more generic so you could re-cycle
it for things like district heating, but generator:power_source is only
ever going to be used for generating stations, and needs a new column in
the database.


Yes, that's the plan I believe. People like keys that are understandable 
without context. Contrast this with a tag "width=2" - where you have no 
chance in knowing what it means without looking at the whole thing.


Bye
Frederik

--
Frederik Ramm  ##  eMail frede...@remote.org  ##  N49°00'09" E008°23'33"

___
talk mailing list
talk@openstreetmap.org
http://lists.openstreetmap.org/listinfo/talk

[OSM-talk] osm2pgsql hstore (was: Wind turbines no longer rendered on mapnik layer)

[Graham Jones]

On 16 February 2012 16:25, Frederik Ramm  wrote:

> Hi,
>
>
> On 02/16/2012 03:05 PM, Kay Drangmeister wrote:
>
>> Isn't there a hstore in the rendering-db?
>>
>
> No.
>
> This reminded me of a question I have been meaning to ask for quite a
while - is a database generated by osm2pgsql with an hstore expected to
perform similarly to one without?

The only time I tried to use hstore, the performance of the whole database
was dreadful (at least a factor of 5 slower, but I didn't do any proper
benchmarking - I just gave up and re-created it without hstore and the
problem went away).It may have been that something went wrong with the
import that I did not notice, and it was not an hstore issue.

Using hstore would be really good given the fashion for generating new keys
with colons in them, but not with that reduction in performance, so I keep
having to re-import my database to add new keys.

 Why create a key generator:power_source rather than just use
power_source.  power_source is much more generic so you could re-cycle it
for things like district heating, but generator:power_source is only ever
going to be used for generating stations, and needs a new column in the
database. .   I think I just prefer more generic, re-usable keys
rather than trying to invent a new one for each situation

Regards


Graham.

-- 
Graham Jones
Hartlepool, UK.
___
talk mailing list
talk@openstreetmap.org
http://lists.openstreetmap.org/listinfo/talk

[OSM-talk] osm2pgsql import united_states data: alaska, hawaii seems missing

[Min Wang]

Hi

   (1)  I downloaded the united_stats data from
http://downloads.cloudmade.com/americas/northern_america/united_states

   then inported data to postgres/postgis:

 ./osm2pgsql  --slim   -S default.style -d gis -C 2048 
--number-processes=1 --cache-strategy=dense  united_states.osm.bz2


  The map can show US-48 states, but no alaska, hawaii.

  and seems united_state.osm do contain the hawaii data.

 grep Honolulu united_states.osm












  What could be wrong?

(2) osm2pgsql : how to append extraction to existing data?

I have imported alaska into postgres:
 ./osm2pgsql  --slim   -S default.style -d gis -C 2048 
--number-processes=1 --cache-strategy=dense  alaska.osm.bz2


   and the data can show on the map.

   then I tried to append the hawaii data to the existing data.

acorrding to : 
http://lists.openstreetmap.org/pipermail/talk/2011-October/060436.html


seems it can use -a without --slim to append the data,

 but I tried times to append data without success.

./osm2pgsql  -a -S default.style -d gis -C 2048 --number-processes=1 
--cache-strategy=dense  hawaii.osm.bz2



any suggestion?


Thanks


Min Wang






___
talk mailing list
talk@openstreetmap.org
http://lists.openstreetmap.org/listinfo/talk

[OSM-talk] osm2pgsql - some tables missing

[Kenneth Gonsalves]

hi,

on importing data with osm2pgsql, there is no error, but some tables are
missing:

gis=# \dt
   List of relations
 Schema |Name| Type  |  Owner  
++---+-
 public | geometry_columns   | table | gisuser
 public | planet_osm_line| table | gisuser
 public | planet_osm_point   | table | gisuser
 public | planet_osm_polygon | table | gisuser
 public | planet_osm_roads   | table | gisuser
 public | spatial_ref_sys| table | gisuser
(6 rows)

any idea as to what step I have missed?
-- 
regards
Kenneth Gonsalves


___
talk mailing list
talk@openstreetmap.org
http://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] osm2pgsql - some tables missing

[kenneth gonsalves]

On Wed, 2011-08-10 at 10:00 +0200, Frederik Ramm wrote:
> On 08/10/2011 09:54 AM, kenneth gonsalves wrote:
> > on importing data with osm2pgsql, there is no error, but some tables
> are
> > missing:
> 
> This is the normal set of tables that you get when importing without 
> --slim. It is fully sufficient for rendering. Only if you want to do 
> incremental updates will you need the _nodes, _ways, and _rels
> tables. 

thanks - got my tables.


___
talk mailing list
talk@openstreetmap.org
http://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] osm2pgsql - some tables missing

[Frederik Ramm]

Hi,

On 08/10/2011 09:54 AM, kenneth gonsalves wrote:

on importing data with osm2pgsql, there is no error, but some tables are
missing:


This is the normal set of tables that you get when importing without 
--slim. It is fully sufficient for rendering. Only if you want to do 
incremental updates will you need the _nodes, _ways, and _rels tables.


Bye
Frederik

___
talk mailing list
talk@openstreetmap.org
http://lists.openstreetmap.org/listinfo/talk

[OSM-talk] osm2pgsql - some tables missing

[kenneth gonsalves]

hi,

on importing data with osm2pgsql, there is no error, but some tables are
missing:

gis=# \dt
   List of relations
 Schema |Name| Type  |  Owner  
++---+-
 public | geometry_columns   | table | gisuser
 public | planet_osm_line| table | gisuser
 public | planet_osm_point   | table | gisuser
 public | planet_osm_polygon | table | gisuser
 public | planet_osm_roads   | table | gisuser
 public | spatial_ref_sys| table | gisuser
(6 rows)

any idea as to what step I have missed?
-- 
regards
Kenneth Gonsalves



___
talk mailing list
talk@openstreetmap.org
http://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] osm2pgsql for 64-bit

[Stephan Knauss]

On 09.06.2011 16:10, Saphy Mo wrote:

For windows 64 there is need of
  osm2pgsql for 64-bit.
is there any version of that available for users?


a while ago I tried a cross compile and failed on the needed include 
library from postgres 8.3. I was not able to create a 64bit lib of it.


With postgres 9 it should be possible as this is also available as 64 
bit on windows.


Meanwhile you can patch the 32 bit executable to be large address aware. 
This gives you 4G to use.


Stephan

___
talk mailing list
talk@openstreetmap.org
http://lists.openstreetmap.org/listinfo/talk

[OSM-talk] osm2pgsql for 64-bit

[Saphy Mo]

For windows 64 there is need of 

osm2pgsql for 64-bit.is there any version of that available for users?

regards,
Saphy___
talk mailing list
talk@openstreetmap.org
http://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] osm2pgsql error in Oxford/Cotswolds .osm data 19/12/09

[Jon Burgess]

On Tue, 2009-12-22 at 17:21 +, Nick Whitelegg wrote:
> Hello everyone,
> 
> I'm having problems loading in a slab of OSM data for the Oxford/Cotswolds 
> area for the UK extract for 19/12/09 downloaded from geofabrik.de.
> 
> osm2pgsql stops with the error "Error allocating ways" (no other info is 
> produced)
> 
> The relevant file is 
> http://nick.dev.openstreetmap.org/downloads/planet/oxford-cotswolds-191209.osm.bz2.
> 
> Has anyone else seen problems loading data from the Oxford/Cotswolds area 
> from recent planets?

The error means it ran out of memory. Try running in the --slim mode
instead.

Jon



___
talk mailing list
talk@openstreetmap.org
http://lists.openstreetmap.org/listinfo/talk

[OSM-talk] osm2pgsql error in Oxford/Cotswolds .osm data 19/12/09

[Nick Whitelegg]

Hello everyone,

I'm having problems loading in a slab of OSM data for the Oxford/Cotswolds area 
for the UK extract for 19/12/09 downloaded from geofabrik.de.

osm2pgsql stops with the error "Error allocating ways" (no other info is 
produced)

The relevant file is 
http://nick.dev.openstreetmap.org/downloads/planet/oxford-cotswolds-191209.osm.bz2.

Has anyone else seen problems loading data from the Oxford/Cotswolds area from 
recent planets?

Thanks,
Nick
___
talk mailing list
talk@openstreetmap.org
http://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] osm2pgsql usage

[Peter Körner]

> The data file comes from XAPI ( 
> http://www.informationfreeway.org/api/0.6/map?bbox=-83.56,42.17,-83.01,42.71 
> ) and works fine for other purposes like generating a map for my GPS 
> using mkgmap.

Have you tried loading a even smaller part from api? This could sort out 
problems with the xapi extensions..

Peter

___
talk mailing list
talk@openstreetmap.org
http://lists.openstreetmap.org/listinfo/talk

[OSM-talk] osm2pgsql usage

[Nakor]

   Hello,

I asked that question on the mapnik mailing list but was told that the OSM
mailing list would have more knowledgeable people on this issue:

I was in the process of setting up Mapnik and as a start I was using an OSM
file from around my area to test with "small" data.

I get stuck with an error message in osm2pgsql:

osm2pgsql SVN version 0.67-18179

Using projection SRS 900913 (Spherical Mercator)
Setting up table: planet_osm_point
Setting up table: planet_osm_line
Setting up table: planet_osm_polygon
Setting up table: planet_osm_roads
Mid: Ram, scale=100

Reading in file: ../maps/Detroit_091010.osm
Processing: Node(434k) Way(50k) Relation(0k)osm2pgsql: osm2pgsql.c:262:
StartElement: Assertion `xrole' failed.

I searched the web a bit but did not find much, except from some memories
issues. I monitored the memory during the process and had at least 1Gb free,
so I guess this is not the issue.

The data file comes from XAPI (
http://www.informationfreeway.org/api/0.6/map?bbox=-83.56,42.17,-83.01,42.71)
and works fine for other purposes like generating a map for my GPS
using
mkgmap.

Any help would be appreciated.

  Thanks,

N.
___
talk mailing list
talk@openstreetmap.org
http://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] osm2pgsql

[Jon Burgess]

On Fri, 2009-06-26 at 23:44 +0100, Jon Burgess wrote:
> On Fri, 2009-06-26 at 18:28 -0400, James McManus wrote:
> > 
> > But it runs for hours and uses up all of my RAM.  I eventually have
> to kill it. 
> > How long should it take to subset a small area such as this?
> 
> How much RAM and swap do you have?
> 
> It take at least 2 hours to process the full planet, this is time
> taken
> just to do the bzip2 decompression and XML parsing without even
> writing
> anything out to the database. There is a lot of data in the full
> planet
> file.
> 
> The algorithms in the code are optimised for the full planet import
> and
> are not as efficient as they could be for very small areas.
> 
> Using the --slim option should fix the memory issue.
> 
> Alternatively you could start with a UK or England only extract and
> save
> yourself a lot of bandwidth and import time.

To give you some firm numbers, the full planet import on the main tile
server with 12GB of RAM takes around 12 hours. On my home machine I
normally import a UK only planet file and it takes about 15 minutes and
about 2GB of RAM.

The --slim option reduces the RAM usage but the import time will
increase if you have less RAM to cache the DB in memory during the
import.

Jon


___
talk mailing list
talk@openstreetmap.org
http://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] osm2pgsql

[Jon Burgess]

On Fri, 2009-06-26 at 18:28 -0400, James McManus wrote:
> Hi - I'm trying to use osm2pgsql to extract a subset area of OSM, but
> the --bbox option does not appear to be working.  I downloaded
> planet-090617.osm.bz2 and then issued the following command:
> 
> osm2pgsql --bbox -0.5,51.25,0.5,51.75 -m -d gis planet-090617.osm.bz2
> 
> 
> At first it appears to be working producing the following output:
> 
> planet-090617.osm.bz2
> osm2pgsql SVN version 0.66-16084
> 
> Using projection SRS 900913 (Spherical Mercator)
> Applying Bounding box: -0.50,51.25 to 0.50,51.75
> 
> Setting up table: planet_osm_point
> Setting up table: planet_osm_line
> Setting up table: planet_osm_polygon
> Setting up table: planet_osm_roads
> Mid: Ram, scale=100
> 
> Reading in file: planet-090617.osm.bz2
> 
> Processing: Node(1400k) Way(0k) Relation(0k)
> 
> But it runs for hours and uses up all of my RAM.  I eventually have to kill 
> it. 
> How long should it take to subset a small area such as this?

How much RAM and swap do you have?

It take at least 2 hours to process the full planet, this is time taken
just to do the bzip2 decompression and XML parsing without even writing
anything out to the database. There is a lot of data in the full planet
file.

The algorithms in the code are optimised for the full planet import and
are not as efficient as they could be for very small areas.

Using the --slim option should fix the memory issue.

Alternatively you could start with a UK or England only extract and save
yourself a lot of bandwidth and import time.

Jon



___
talk mailing list
talk@openstreetmap.org
http://lists.openstreetmap.org/listinfo/talk

[OSM-talk] osm2pgsql

[James McManus]

Hi - I'm trying to use osm2pgsql to extract a subset area of OSM, but the
--bbox option does not appear to be working.  I downloaded
planet-090617.osm.bz2 and then issued the following command:

osm2pgsql --bbox -0.5,51.25,0.5,51.75 -m -d gis planet-090617.osm.bz2

At first it appears to be working producing the following output:

planet-090617.osm.bz2
osm2pgsql SVN version 0.66-16084

Using projection SRS 900913 (Spherical Mercator)
Applying Bounding box: -0.50,51.25 to 0.50,51.75
Setting up table: planet_osm_point
Setting up table: planet_osm_line
Setting up table: planet_osm_polygon
Setting up table: planet_osm_roads
Mid: Ram, scale=100

Reading in file: planet-090617.osm.bz2
Processing: Node(1400k) Way(0k) Relation(0k)

But it runs for hours and uses up all of my RAM.  I eventually have to kill it.
How long should it take to subset a small area such as this?

Thanks
Jim
___
talk mailing list
talk@openstreetmap.org
http://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] osm2pgsql and proper/legacy mercator

[Juan Lucas Dominguez Rubio]

Hi, according to the source code, the legacy option -M is using this projection:
 
  [PROJ_MERC]= { 
 descr: "WGS84 Mercator", 
 proj4text: "+proj=merc +datum=WGS84  +k=1.0 +units=m +over +no_defs", 
 srs:3395, 
 option: "" }
 
Unlike EPSG:900913, that projection is not assuming that the Earth is a perfect 
sphere, therefore:
 
- it seems to be cartographically more correct
- it probably needs far more computation
- the result probably differs from that of EPSG:900913 by a few meters
 
Regards
Juan Lucas
 
 



De: talk-boun...@openstreetmap.org en nombre de Francois Van 
Der Biest
Enviado el: vie 01/05/2009 17:52
Para: talk@openstreetmap.org
        Asunto: [OSM-talk] osm2pgsql and proper/legacy mercator



Hi list,

osm2pgsql --help says:
-m|--merc: Store data in proper spherical mercator (default)
-M|--oldmerc: Store data in the legacy OSM mercator format

I'm wondering what's the difference between those two srs.
Which one is epsg:900913 (aka epsg:3785
http://www.spatialreference.org/ref/epsg/3785/) ?

My experience (importing an osm dump into postgis, then 
exporting to
shapefiles) would let me think that the "legacy OSM mercator 
format"
is epsg:3785. So, what's the other one ?

Thank's,
F.

___
talk mailing list
talk@openstreetmap.org
http://lists.openstreetmap.org/listinfo/talk


___
talk mailing list
talk@openstreetmap.org
http://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] osm2pgsql and proper/legacy mercator

[Jon Burgess]

On Fri, 2009-05-01 at 17:52 +0200, Francois Van Der Biest wrote:
> Hi list,
> 
> osm2pgsql --help says:
> -m|--merc: Store data in proper spherical mercator (default)
> -M|--oldmerc: Store data in the legacy OSM mercator format
> 
> I'm wondering what's the difference between those two srs.
> Which one is epsg:900913 (aka epsg:3785
> http://www.spatialreference.org/ref/epsg/3785/) ?
> 
> My experience (importing an osm dump into postgis, then exporting to
> shapefiles) would let me think that the "legacy OSM mercator format"
> is epsg:3785. So, what's the other one ?

The other one should be epsg:3395

$ ./osm2pgsql -vh
...
Supported projections:
Latlong (-l) SRS:  4326 (none)
WGS84 Mercator  (  ) SRS:  3395 +proj=merc +datum=WGS84  +k=1.0 +units=m 
+over +no_defs
Spherical Mercator  (-m) SRS:900913 +proj=merc +a=6378137 +b=6378137 
+lat_ts=0.0 +lon_0=0.0 +x_0=0.0 +y_0=0 +k=1.0 +units=m +nadgri...@null +no_defs 
+over

The seconds line of the help should show (-M) for the legacy projection.
I've just fixed it in SVN.

   Jon



___
talk mailing list
talk@openstreetmap.org
http://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] osm2pgsql and proper/legacy mercator

[Andy Allan]

On Fri, May 1, 2009 at 4:52 PM, Francois Van Der Biest
 wrote:
> Hi list,
>
> osm2pgsql --help says:
> -m|--merc: Store data in proper spherical mercator (default)
> -M|--oldmerc: Store data in the legacy OSM mercator format
>
> I'm wondering what's the difference between those two srs.
> Which one is epsg:900913 (aka epsg:3785
> http://www.spatialreference.org/ref/epsg/3785/) ?
>
> My experience (importing an osm dump into postgis, then exporting to
> shapefiles) would let me think that the "legacy OSM mercator format"
> is epsg:3785. So, what's the other one ?

I'm moderately sure that --oldmerc was when someone in OSM got the
maths slightly wrong. Use -m for (as it says) proper spherical
mercator aka epsg:900913

Cheers,
Andy

___
talk mailing list
talk@openstreetmap.org
http://lists.openstreetmap.org/listinfo/talk

[OSM-talk] osm2pgsql and proper/legacy mercator

[Francois Van Der Biest]

Hi list,

osm2pgsql --help says:
-m|--merc: Store data in proper spherical mercator (default)
-M|--oldmerc: Store data in the legacy OSM mercator format

I'm wondering what's the difference between those two srs.
Which one is epsg:900913 (aka epsg:3785
http://www.spatialreference.org/ref/epsg/3785/) ?

My experience (importing an osm dump into postgis, then exporting to
shapefiles) would let me think that the "legacy OSM mercator format"
is epsg:3785. So, what's the other one ?

Thank's,
F.

___
talk mailing list
talk@openstreetmap.org
http://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] Osm2pgsql and column names with two underscores

[Cartinus]

On Monday 30 March 2009 14:08:11 Jukka Rahkonen wrote:
> Look at these:
> "openGeoDB:telephone_are" a_code,
> "openGeoDB:license_plate" _code
>
> It looks like column name gets truncated at the second
> underline character. Could it be some bug in osm2pgsql,
> or is it some other place where the error occurs?

It doesn't get truncated at the second underscore as there is a character 
before the second underscore at the first case. Looks like it gets truncated 
after the 23th character.


-- 
m.v.g.,
Cartinus

___
talk mailing list
talk@openstreetmap.org
http://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] Osm2pgsql and column names with two underscores

[sly (sylvain letuffe)]

On Monday 30 March 2009 14:08, Jukka Rahkonen wrote:
> Hi,

Hi,

( better transfert this to dev ?)


> An osm2pgsql user writes on the forum about importing special tags
> into PostGIS.
> He has defined for example these tags:
> node   openGeoDB:telephone_area_code   text 
> node   openGeoDB:license_plate_code  text

I've read his problem, and proposed converting the osm input file with smaller 
tag name. The other solution might be to patch osm2pgsql.

> Look at these:
> "openGeoDB:telephone_are" a_code,
> "openGeoDB:license_plate" _code
> 
> It looks like column name gets truncated at the second 
> underline character. 

Coincidence ;-)

Both tag name are also truncated at 24 chars exactly, let's see...

> Could it be some bug in osm2pgsql, 
> or is it some other place where the error occurs?

Looks like I went lucky :
# grep 24 output-pgsql.c
char buffer[1024];
  char buffer[1024];
char osmtype[24];
char tag[24];
char datatype[24];

Recompiling osm2pgsql might help, if that limit of 24 is not present elsewhere


-- 
sly 
Sylvain Letuffe li...@letuffe.org
qui suis-je : http://slyserv.dyndns.org



___
talk mailing list
talk@openstreetmap.org
http://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] Osm2pgsql and column names with two underscores

[Martijn van Oosterhout]

On Mon, Mar 30, 2009 at 2:08 PM, Jukka Rahkonen
 wrote:
> Hi,
>
> An osm2pgsql user writes on the forum about importing special tags
> into PostGIS.
> He has defined for example these tags:
> node       openGeoDB:telephone_area_code       text
> node       openGeoDB:license_plate_code          text
>
> Import fails and error message is:
> CREATE TABLE planet_osm_point ( osm_id int4,"access" text,
> "admin_level" text,

Ok, it's got nothing to do with the underscores and more to do with
the fact that the field names are limited to 23 characters by
osm2pgsql. I don't really know where this restriction comes from, does
anyone else have ideas?

It should be easy to fix in output-pgsql.c. Fix the read_style_file()
function to increase the length of the buffers.

Have a nice day,
-- 
Martijn van Oosterhout  http://svana.org/kleptog/

___
talk mailing list
talk@openstreetmap.org
http://lists.openstreetmap.org/listinfo/talk

[OSM-talk] Osm2pgsql and column names with two underscores

[Jukka Rahkonen]

Hi,

An osm2pgsql user writes on the forum about importing special tags
into PostGIS.
He has defined for example these tags:
node   openGeoDB:telephone_area_code   text 
node   openGeoDB:license_plate_code  text

Import fails and error message is:
CREATE TABLE planet_osm_point ( osm_id int4,"access" text,
"admin_level" text, 

---SNIP---
"openGeoDB:postal_codes" text,"openGeoDB:telephone_are" a_code,
"openGeoDB:type" text,"openGeoDB:license_plate" _code,"z_order"
int4 );

failed: ERROR:  type "a_code" does not exist
LINE 1: ...eoDB:postal_codes" text,
"openGeoDB:telephone_are" a_code,"op...
 ^

Error occurred, cleaning up

Look at these:
"openGeoDB:telephone_are" a_code,
"openGeoDB:license_plate" _code

It looks like column name gets truncated at the second 
underline character. Could it be some bug in osm2pgsql,
or is it some other place where the error occurs?

-Jukka Rahkonen-



___
talk mailing list
talk@openstreetmap.org
http://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] Osm2pgsql fails with Finland.osm.bz2

[Rahkonen Jukka]

Jon Burgess  googlemail.com> writes:

> 
> On Wed, 2009-02-04 at 15:27 +, Jukka Rahkonen wrote:
> > Hi,
> > 
> > I have been importing Finland.osm.bz2 dataset from Geofabrik into Postgis 
> > every
> > day with osm2pgsql.exe (on Windows) but now it fails. The error looks like 
> > this:
> > Reading in file: finland.osm.bz2
> > Processing: Node(2835k) Way(37k) Relation(0k)Error allocating ways
> > Error occurred, cleaning up

> 
> These first two look like an out of memory condition causing malloc() to
> fail. 

Right, another computer with more memory imports the same data with same program
versions.
 
> 
> > I made a third trial by using another version of osm2pgsql.exe with Finnish 
> > data
> > but again with no luck. A bit different error message though:
> > Reading in file: finland.osm.bz2
> > Processing: Node(2835k) Way(194k) Relation(0k)terminate called after 
> > throwing an
> >  instance of 'geos::util::TopologyException'

...
> 
> Normally the code catches exceptions from geos and ignores them. In some
> builds of osm2pgsql.exe this mechanism is broken and a geometry
> exception will instead cause the program to abort. It looks like this is
> one of these broken versions.
...
> This version of osm2pgsql.exe it should not have the exception problem:
> http://tile.openstreetmap.org/direct/osm2pgsql.zip

Right again, this one does not stop to topology errors. So no problem at all,
the amount of data had just reached the limits of the first computer I used.

___
talk mailing list
talk@openstreetmap.org
http://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] Osm2pgsql fails with Finland.osm.bz2

[Jon Burgess]

On Wed, 2009-02-04 at 15:27 +, Jukka Rahkonen wrote:
> Hi,
> 
> I have been importing Finland.osm.bz2 dataset from Geofabrik into Postgis 
> every
> day with osm2pgsql.exe (on Windows) but now it fails. The error looks like 
> this:
> Reading in file: finland.osm.bz2
> Processing: Node(2835k) Way(37k) Relation(0k)Error allocating ways
> Error occurred, cleaning up
> 
> I tried next to use Scandinavian dataset from hypercube.telascience.org.  
> Result
> is about the same:
> Reading in file: planet-scand-090203.osm.gz
> Processing: Node(6670k) Way(0k) Relation(0k)Error allocating nodes
> Error occurred, cleaning up

These first two look like an out of memory condition causing malloc() to
fail. 

> I made a third trial by using another version of osm2pgsql.exe with Finnish 
> data
> but again with no luck. A bit different error message though:
> Reading in file: finland.osm.bz2
> Processing: Node(2835k) Way(194k) Relation(0k)terminate called after throwing 
> an
>  instance of 'geos::util::TopologyException'
>   what():  TopologyException: found non-noded intersection between 
> 2.78294e+006
> 9.41427e+006, 2.78315e+006 9.41433e+006 and 2.78326e+006 9.41433e+006, 
> 2.78269e+
> 006 9.41433e+006 2.78315e+006 9.41433e+006

Normally the code catches exceptions from geos and ignores them. In some
builds of osm2pgsql.exe this mechanism is broken and a geometry
exception will instead cause the program to abort. It looks like this is
one of these broken versions.

> Does anybody has an idea about what is going on, and if it is just a data 
> error,
> how to find and correct it?

This version of osm2pgsql.exe it should not have the exception problem:
http://tile.openstreetmap.org/direct/osm2pgsql.zip

Run the utility with the "--slim" parameter. This should fix any out of
memory problems but it will be a bit slower and require more disk space.

Jon



___
talk mailing list
talk@openstreetmap.org
http://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] Osm2pgsql fails with Finland.osm.bz2

[Frederik Ramm]

Hi,

Jukka Rahkonen wrote:
> Processing: Node(2835k) Way(194k) Relation(0k)terminate called after throwing 
> an
>  instance of 'geos::util::TopologyException'
>   what():  TopologyException: found non-noded intersection between 
> 2.78294e+006
> 9.41427e+006, 2.78315e+006 9.41433e+006 and 2.78326e+006 9.41433e+006, 
> 2.78269e+
> 006 9.41433e+006 2.78315e+006 9.41433e+006

Reverse-projecting that into lat/lon gives 24.999;64.252. If you check 
the area in OSM Inspector:

http://tools.geofabrik.de/osmi/?view=geometry&lon=24.94860&lat=64.24186&zoom=11&opacity=1.00

you will see that there are a number of geometry errors and quite 
possibly self-intersecting ways near the town of Oulainen. One of these 
is very likely responsible for the error you are seeing, however I 
cannot tell you why osm2pgsql chokes on this when it happily processes 
the myriad other geometry errors that OSM Inspector spots.

You can select an object and then use the icons on the right to load it 
in an (already running) JOSM or launch Potlatch for that object.

Bye
Frederik


___
talk mailing list
talk@openstreetmap.org
http://lists.openstreetmap.org/listinfo/talk

[OSM-talk] Osm2pgsql fails with Finland.osm.bz2

[Jukka Rahkonen]

Hi,

I have been importing Finland.osm.bz2 dataset from Geofabrik into Postgis every
day with osm2pgsql.exe (on Windows) but now it fails. The error looks like this:
Reading in file: finland.osm.bz2
Processing: Node(2835k) Way(37k) Relation(0k)Error allocating ways
Error occurred, cleaning up

I tried next to use Scandinavian dataset from hypercube.telascience.org.  Result
is about the same:
Reading in file: planet-scand-090203.osm.gz
Processing: Node(6670k) Way(0k) Relation(0k)Error allocating nodes
Error occurred, cleaning up

I made a third trial by using another version of osm2pgsql.exe with Finnish data
but again with no luck. A bit different error message though:
Reading in file: finland.osm.bz2
Processing: Node(2835k) Way(194k) Relation(0k)terminate called after throwing an
 instance of 'geos::util::TopologyException'
  what():  TopologyException: found non-noded intersection between 2.78294e+006
9.41427e+006, 2.78315e+006 9.41433e+006 and 2.78326e+006 9.41433e+006, 2.78269e+
006 9.41433e+006 2.78315e+006 9.41433e+006

Does anybody has an idea about what is going on, and if it is just a data error,
how to find and correct it?



___
talk mailing list
talk@openstreetmap.org
http://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] osm2pgsql & planet: frustrations, cutoffs, and idempotence

[Michal Migurski]

> On Mon, Oct 27, 2008 at 9:39 PM, Michal Migurski <[EMAIL PROTECTED]>  
> wrote:
>> I'm liking Jochen Topf's suggestion here:
>>
>>   "If the planet dump plus the diff from the same day is what  
>> everybody
>> wants anyway, why not do this on the server side and hold the planet
>> back after the first diff is available, run this over the planet and
>> then publish that as the planet?"
>
> 1. Because there are plenty of uses for the planet dump that don't
> need consistant snapshots.

Those uses would not be impacted by consistent snapshots.


> 2. Because such consistant snapshots have been available elsewhere for
> quite a while now and people who need them can get them. There's no
> particular reason why it has to be on the same site as the normal
> planet dumps.

Yet there is no link to these places from planet.openstreetmap.org  
that indicates that the files available there differ in some important  
or useful way. The telascience.org source you suggested is described  
as "extracts of NL, Scandinavia and Taiwan" at 
http://wiki.openstreetmap.org/index.php/Planet.osm 
, rather than a complete dump of Planet with different datetime  
boundaries.

I'm happy to keep bellying up to the trial & error bar here, but as I  
mention in a previous mail, the volume of data involved means that  
individual attempts at the data (successful or not) have multiple-day  
costs associated with them.


> Umm, yeah. I was ofcourse assuming you were running the latest
> version, otherwise anything is possible, The creates-as-modifies fix
> was done two months ago.


I'll recompile and replace the two-month-old version of osm2pgsql I've  
been using.

-mike.


michal migurski- [EMAIL PROTECTED]
  415.558.1610




___
talk mailing list
talk@openstreetmap.org
http://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] osm2pgsql & planet: frustrations, cutoffs, and idempotence

[Brett Henderson]

On Tue, Oct 28, 2008 at 7:39 AM, Michal Migurski <[EMAIL PROTECTED]> wrote:

> >> Finally, the boundaries between the hourlies and dailies seem
> >> misaligned.
> >>
> >
> > This shouldn't be the case.
> >> After running the remaining hourlies for the 22nd, I attempted to
> >> pick  up on the 23rd with a daily. The final hourly I used was
> >> 2008102223-2008102300.osc.gz. It's my expectation that I should be
> >> able to immediately follow that with 20081023-20081024.osc.gz, but
> >> this led to duplicate key violation suggesting that there's an
> >> overlap  between the two files. Continuing with hourlies *works*,
> >> but is  tedious and I suspect slower than the dailies.
> >>
> >
> > You should have been able to do what you've suggested.  If you are
> > finding problems, please provide me with some example data which is
> > misaligned between the two types of changesets.
>
> Try the two files mentioned above - that's where I saw this behavior,
> they're quite recent.
>
>2008102223-2008102300.osc.gz
>20081023-20081024.osc.gz


I need you to provide some specific examples of broken data.  If you can say
that "way 27123456 is created in both of the above files even though they
are for different time periods" then I can take a look at why this may have
occurred.  Just saying that there is misalignment between those two files
doesn't help me at all.  Presumably you ran into a specific problem and
received a specific error message, this is the kind of information I need.
I only do this project in my spare time and can't go looking for problems
that I'm not sure even exist, I have enough known problems to look into
already :-)


>
>
>
> >> My sense from reading other people's experiences has been that it's
> >> a  common pattern to rely solely on the weekly planet dumps,
> >> incurring  the substantial overhead of parsing and importing the
> >> full 5GB dump  once every week, and then re-rendering the complete
> >> set of tiles.
> >>
> >
> > For a long time weekly planet dumps were the only bulk data
> > available.  Osmosis changesets have been on the scene for some time
> > now though and are gradually being utilised by more and more
> > clients.  As the planet grows, this will become more critical.  Who
> > knows, if the kinks gradually get ironed out of the osm2pgsql
> > program we may even begin to see the main mapnik tile generator move
> > to using changesets.
>
> I would love to rely on these exclusively, it's much more efficient.
> But, I was seeing a fair bit of information fall through the cracks so
> that's why I'm re-synching to planet every four weeks.


Again, please provide some specific examples.  If data is being missed I'd
like to know about it.  Osmosis provides some tools that may be useful
here.  You can download a planet, apply changesets for a week, then compare
against the next planet and see what the differences are.  Obviously both
planets would need appropriate changesets applied to make them consistent
before performing a comparison to eliminate noise.

I probably should do some of these comparisons myself, but again just
haven't found time yet and nobody else has complained about missing data.
The minute changesets run 5 minutes behind the API so could potentially miss
data if a lock is held for several minutes.  The daily and hourly changesets
run at least 20 minutes behind API (forget off the top of my head) and
should be extremely unlikely to miss data.

Brett
___
talk mailing list
talk@openstreetmap.org
http://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] osm2pgsql & planet: frustrations, cutoffs, and idempotence

[Martijn van Oosterhout]

On Mon, Oct 27, 2008 at 9:40 PM, Michal Migurski <[EMAIL PROTECTED]> wrote:
>> Now that I think about it though, I think what I did was take one of
>> the planet dumps from http://hypercube.telascience.org/planet/ (which
>> *are* consistant snapshots), and run the dailies from there.
>
> Is there any reason to not use those? They seem to be more frequent
> than the planet.openstreetmap.org ones - is there some disadvantage?
> How are they created?

Umm, they are created by taking the planet dumps and applying the
daily diffs every day. They are used to produce consistant snapshots
of for example, NL and by the coastline checker (which really likes
having consistant snapshots to work with).

Have a nice day,
-- 
Martijn van Oosterhout <[EMAIL PROTECTED]> http://svana.org/kleptog/

___
talk mailing list
talk@openstreetmap.org
http://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] osm2pgsql & planet: frustrations, cutoffs, and idempotence

[Martijn van Oosterhout]

On Mon, Oct 27, 2008 at 9:39 PM, Michal Migurski <[EMAIL PROTECTED]> wrote:
> I'm liking Jochen Topf's suggestion here:
>
>"If the planet dump plus the diff from the same day is what everybody
> wants anyway, why not do this on the server side and hold the planet
> back after the first diff is available, run this over the planet and
> then publish that as the planet?"

1. Because there are plenty of uses for the planet dump that don't
need consistant snapshots.

2. Because such consistant snapshots have been available elsewhere for
quite a while now and people who need them can get them. There's no
particular reason why it has to be on the same site as the normal
planet dumps.
> Probably what I need to do is get a fresh update of osm2pgsql. I can
> see now that the revision I'm using is older than #10464, where some
> inconsistency resilience was added.

Umm, yeah. I was ofcourse assuming you were running the latest
version, otherwise anything is possible, The creates-as-modifies fix
was done two months ago.

Have a nice day,
-- 
Martijn van Oosterhout <[EMAIL PROTECTED]> http://svana.org/kleptog/

___
talk mailing list
talk@openstreetmap.org
http://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] osm2pgsql & planet: frustrations, cutoffs, and idempotence

[Michal Migurski]

On Oct 27, 2008, at 12:59 AM, Martijn van Oosterhout wrote:

> On Mon, Oct 27, 2008 at 1:10 AM, Michal Migurski <[EMAIL PROTECTED]>  
> wrote:
>> The final event in each weekly planet dump does not fall on an even
>> day boundary. In the case of the most recent Oct. 22nd planet.osm, it
>> was necessary to experiment with hourly diffs from that day to find
>> that the boundary was approx. 2:00pm. Hourlies up to and including
>> 2008102213-2008102214.osc.gz failed, hourlies after that succeeded. I
>> could go more granular here, checking the minute diffs as well for a
>> more precise breakpoint, but it seems odd that the planet dump does
>> not break cleanly on a midnight boundary so that it's possible to  
>> pick
>> up the differences moving forward.
>
> As I recall, osm2pgsql did support this kind of operation (or at least
> it did last time I tried, it was discussed on the list). All creates
> in diffs are treated as delete+insert. You don't actually say what the
> error was you ran into though so I can't be sure if you're talking
> about the same problem.
>
> Now that I think about it though, I think what I did was take one of
> the planet dumps from http://hypercube.telascience.org/planet/ (which
> *are* consistant snapshots), and run the dailies from there.

Is there any reason to not use those? They seem to be more frequent  
than the planet.openstreetmap.org ones - is there some disadvantage?  
How are they created?

-mike.


michal migurski- [EMAIL PROTECTED]
  415.558.1610




___
talk mailing list
talk@openstreetmap.org
http://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] osm2pgsql & planet: frustrations, cutoffs, and idempotence

[Michal Migurski]

>> Yep, as others have commented there are two tables types in the osm  
>> database; current tables, and history tables.  The planet dumper  
>> just reads current tables which is the fastest approach.   
>> Unfortunately the current tables change constantly during the  
>> planet generation process resulting in inconsistencies.  It is  
>> possible to produce a consistent snapshot reading history tables  
>> and osmosis has the ability to do just that but it is significantly  
>> slower.  It is also possible to produce a consistent snapshot by  
>> taking an inconsistent planet and applying changesets from a point  
>> in time prior to the planet dump beginning through to a point after  
>> completion, this effectively produces the same result at much  
>> reduced load on the main database.
>>

I'm liking Jochen Topf's suggestion here:

"If the planet dump plus the diff from the same day is what everybody  
wants anyway, why not do this on the server side and hold the planet  
back after the first diff is available, run this over the planet and  
then publish that as the planet?"


>> Finally, the boundaries between the hourlies and dailies seem   
>> misaligned.
>>
>
> This shouldn't be the case.
>> After running the remaining hourlies for the 22nd, I attempted to  
>> pick  up on the 23rd with a daily. The final hourly I used was   
>> 2008102223-2008102300.osc.gz. It's my expectation that I should be   
>> able to immediately follow that with 20081023-20081024.osc.gz, but   
>> this led to duplicate key violation suggesting that there's an  
>> overlap  between the two files. Continuing with hourlies *works*,  
>> but is  tedious and I suspect slower than the dailies.
>>
>
> You should have been able to do what you've suggested.  If you are  
> finding problems, please provide me with some example data which is  
> misaligned between the two types of changesets.

Try the two files mentioned above - that's where I saw this behavior,  
they're quite recent.

2008102223-2008102300.osc.gz
20081023-20081024.osc.gz


>> My sense from reading other people's experiences has been that it's  
>> a  common pattern to rely solely on the weekly planet dumps,  
>> incurring  the substantial overhead of parsing and importing the  
>> full 5GB dump  once every week, and then re-rendering the complete  
>> set of tiles.
>>
>
> For a long time weekly planet dumps were the only bulk data  
> available.  Osmosis changesets have been on the scene for some time  
> now though and are gradually being utilised by more and more  
> clients.  As the planet grows, this will become more critical.  Who  
> knows, if the kinks gradually get ironed out of the osm2pgsql  
> program we may even begin to see the main mapnik tile generator move  
> to using changesets.

I would love to rely on these exclusively, it's much more efficient.  
But, I was seeing a fair bit of information fall through the cracks so  
that's why I'm re-synching to planet every four weeks.



>> I can see a few possible solutions.
>>
>> The cutoff times for files on planet.openstreetmap.org could  
>> behave  more consistently. A weekly dump should end at 11:59pm so  
>> that dailies  can immediately pick up user activity. Hourly and  
>> daily dumps should  be synchronized. This seems more difficult.
>>
>
> You only need a single consistent snapshot to get started.  You can  
> download a planet, then download the two daily changesets either  
> side of the planet generation window, then use osmosis to patch the  
> planet.  This will give you a consistent snapshot.  Once you've  
> imported that into your target database you can then start using  
> daily changesets to keep up to date (or hourly or minute as  
> appropriate).
>
> While it would be nice to have planet dumps already in consistent  
> form, it does add a significant overhead to the whole process.  It's  
> not terribly hard to fix on the client side.

Probably what I need to do is get a fresh update of osm2pgsql. I can  
see now that the revision I'm using is older than #10464, where some  
inconsistency resilience was added.


-mike.

>


michal migurski- [EMAIL PROTECTED]
  415.558.1610




___
talk mailing list
talk@openstreetmap.org
http://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] osm2pgsql & planet: frustrations, cutoffs, and idempotence

[Frederik Ramm]

Hi,

Brett Henderson wrote:
>> Brett Henderson has offered to look into creating the dailies from 
>> history as well, but I don't know about the status of that.
>>   
> Are you referring to the daily changesets? 
[...]
> Or did you mean planets instead of dailies? 

Mix-up on my part, sorry, yes I meant the planets.

Bye
Frederik

-- 
Frederik Ramm  ##  eMail [EMAIL PROTECTED]  ##  N49°00'09" E008°23'33"

___
talk mailing list
talk@openstreetmap.org
http://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] osm2pgsql & planet: frustrations, cutoffs, and idempotence

[Brett Henderson]

Jochen Topf wrote:
> If the planet dump plus the diff from the same day is what everybody
> wants anyway, why not do this on the server side and hold the planet
> back after the first diff is available, run this over the planet and
> then publish that as the planet?
>   
It would add delay to the planet creation process.  I don't know how 
much of an issue that would be.

How many people still download the full planet on a regular basis?  I 
would hope that people would begin to use changesets even if they only 
require a complete xml file.  For bandwidth reasons alone the gains are 
well worthwhile, plus you can get far more regular updates than weekly.  
The script below automates keeping a snapshot file in sync:
http://svn.openstreetmap.org/applications/utils/osmosis/script/contrib/replicate_osm_file.sh


___
talk mailing list
talk@openstreetmap.org
http://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] osm2pgsql & planet: frustrations, cutoffs, and idempotence

[Brett Henderson]

Frederik Ramm wrote:
> Hi,
>
> Michal Migurski wrote:
>   
>> I've noticed some misalignments between the data in the dumps and the  
>> osm2pgsql importer that leads to unavoidable holes in the data.
>> 
>
> As TomH has already said, this is not a bug, it stems from the fact that 
> the full planet export reads the "current" tables and as such is subject 
> to changes that occur during the export process. (There may even be 
> inconsistencies when something like this happens: Exporter dumps nodes, 
> exporter starts dumping ways, user adds new node into way, new way 
> version is dumped referring to new node that is not in the dump.)
>
> The daily, hourly, and minutely diffs have a clean cutoff date because 
> they are taken from the history tables.
>
> Brett Henderson has offered to look into creating the dailies from 
> history as well, but I don't know about the status of that.
>   
Are you referring to the daily changesets?  The daily, hourly and minute 
changesets are all using the same osmosis-extract-mysql application and 
the only difference is the interval being used.  For a while it was 
using a shell script for dailies that spaetz initially created and I 
extended, but it was unreliable in the face of database outages and is 
no longer being used.  That was the reason for the switch from bzip to 
gzip compression.

Or did you mean planets instead of dailies?  I have a working 
implementation but it is slower than the existing planet dump process so 
I've never tried to introduce it.  It would be faster to automate 
patching of a current table planet with some changesets.

Brett


___
talk mailing list
talk@openstreetmap.org
http://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] osm2pgsql & planet: frustrations, cutoffs, and idempotence

[Brett Henderson]

Others have already commented on most of your points but I'll add my 
thoughts in case there's some gaps.

Michal Migurski wrote:
> Hi,
>
> I've been trying to keep up to date with the dumps and diffs from 
> http://planet.openstreetmap.org/ 
> , and I'm running into a number of bugs related to cutoff dates.
>
> In keeping my Bay Area tiles 
> (http://mike.teczno.com/notes/cascadenik-openstreetmap.html 
> ) up to date, I've been grabbing complete planet.osm dumps about once  
> per month, and filling in the intervening time with daily diffs. I've  
> noticed some misalignments between the data in the dumps and the  
> osm2pgsql importer that leads to unavoidable holes in the data.
>
> It seems that they could be fixed in either osm2pgsql, the planet  
> files, or both.
>
> The final event in each weekly planet dump does not fall on an even  
> day boundary. In the case of the most recent Oct. 22nd planet.osm, it  
> was necessary to experiment with hourly diffs from that day to find  
> that the boundary was approx. 2:00pm. Hourlies up to and including  
> 2008102213-2008102214.osc.gz failed, hourlies after that succeeded. I  
> could go more granular here, checking the minute diffs as well for a  
> more precise breakpoint, but it seems odd that the planet dump does  
> not break cleanly on a midnight boundary so that it's possible to pick  
> up the differences moving forward.
>   
Yep, as others have commented there are two tables types in the osm 
database; current tables, and history tables.  The planet dumper just 
reads current tables which is the fastest approach.  Unfortunately the 
current tables change constantly during the planet generation process 
resulting in inconsistencies.  It is possible to produce a consistent 
snapshot reading history tables and osmosis has the ability to do just 
that but it is significantly slower.  It is also possible to produce a 
consistent snapshot by taking an inconsistent planet and applying 
changesets from a point in time prior to the planet dump beginning 
through to a point after completion, this effectively produces the same 
result at much reduced load on the main database.
> osm2pgsql itself notifies the user of inconsistencies by failing. I  
> can see that effort has been put into making it more resilient (e.g. 
> http://trac.openstreetmap.org/changeset/10464) 
> . Does osm2pgsql have something like a `--force` switch? I haven't  
> been able to find one. In looking at the diff files, it seems that it  
> should be possible to ignore possible conflicts by simply overwriting  
> whatever's in the DB with whatever's in the .osc file.
>   
Yes, that's true.  I can't comment on osm2pgsql but when osmosis 
processes changeset files it does exactly that.
> Finally, the boundaries between the hourlies and dailies seem  
> misaligned.
>   
This shouldn't be the case.
> After running the remaining hourlies for the 22nd, I attempted to pick  
> up on the 23rd with a daily. The final hourly I used was  
> 2008102223-2008102300.osc.gz. It's my expectation that I should be  
> able to immediately follow that with 20081023-20081024.osc.gz, but  
> this led to duplicate key violation suggesting that there's an overlap  
> between the two files. Continuing with hourlies *works*, but is  
> tedious and I suspect slower than the dailies.
>   
You should have been able to do what you've suggested.  If you are 
finding problems, please provide me with some example data which is 
misaligned between the two types of changesets.  I've gone to a fair bit 
of trouble to ensure that timestamp management is correct.  For example, 
all changesets and file names are using UTC even though the database 
itself is using BST.  If I've made a mistake somewhere I'd like to know 
about it.  Given that daily, hourly and minute changesets are using 
*identical* code, I find it hard to believe they're inconsistent with 
each other.
> My sense from reading other people's experiences has been that it's a  
> common pattern to rely solely on the weekly planet dumps, incurring  
> the substantial overhead of parsing and importing the full 5GB dump  
> once every week, and then re-rendering the complete set of tiles.
>   
For a long time weekly planet dumps were the only bulk data available.  
Osmosis changesets have been on the scene for some time now though and 
are gradually being utilised by more and more clients.  As the planet 
grows, this will become more critical.  Who knows, if the kinks 
gradually get ironed out of the osm2pgsql program we may even begin to 
see the main mapnik tile generator move to using changesets.
> My hope has been to proceed in a more incremental fashion, since this  
> makes it possible to track what specific tiles need to be re-rendered  
> on a near-constant schedule, based on actual content or activity, vs.  
> simple cache expiration. Right now I'm doing this daily, I'd like to  
> do it as often as hourly.
>   
Yep, that was one of my original aims.
> I can

Re: [OSM-talk] osm2pgsql & planet: frustrations, cutoffs, and idempotence

[Jochen Topf]

On Mon, Oct 27, 2008 at 08:22:32AM +, Tom Hughes wrote:
> Shaun McDonald wrote:
> > On 27 Oct 2008, at 00:50, Michal Migurski wrote:
> > 
> >>> Planet dumps are not snapshots - they do not represent a consistent
> >>> view at any particular point in time because they take a number of
> >>> hours to generate, during which time new changes are constantly
> >>> being made to the contents of the database.
>  >>
> >> Shouldn't it be possible to ignore any changes that happen after the
> >> cutoff, though?
> > 
> > At the moment we don't look at the time stamps when dumping the planet  
> > file.
> 
> It's not as simple as that - you also have to switch to reading the 
> history tables rather than the current tables or you won't be able to 
> see what the state of the object used to be if it has changed since the 
> snapshot time.
> 
> Which means you're reading much more data, and either having to track 
> the state of each object (in order to find the most recent valid change) 
> or you have to index scan so that you're seeing things in timestamp order.

If the planet dump plus the diff from the same day is what everybody
wants anyway, why not do this on the server side and hold the planet
back after the first diff is available, run this over the planet and
then publish that as the planet?

Jochen
-- 
Jochen Topf  [EMAIL PROTECTED]  http://www.remote.org/jochen/  +49-721-388298


___
talk mailing list
talk@openstreetmap.org
http://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] osm2pgsql & planet: frustrations, cutoffs, and idempotence

[Tom Hughes]

Shaun McDonald wrote:
> On 27 Oct 2008, at 00:50, Michal Migurski wrote:
> 
>>> Planet dumps are not snapshots - they do not represent a consistent
>>> view at any particular point in time because they take a number of
>>> hours to generate, during which time new changes are constantly
>>> being made to the contents of the database.
 >>
>> Shouldn't it be possible to ignore any changes that happen after the
>> cutoff, though?
> 
> At the moment we don't look at the time stamps when dumping the planet  
> file.

It's not as simple as that - you also have to switch to reading the 
history tables rather than the current tables or you won't be able to 
see what the state of the object used to be if it has changed since the 
snapshot time.

Which means you're reading much more data, and either having to track 
the state of each object (in order to find the most recent valid change) 
or you have to index scan so that you're seeing things in timestamp order.

Tom

-- 
Tom Hughes ([EMAIL PROTECTED])
http://www.compton.nu/

___
talk mailing list
talk@openstreetmap.org
http://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] osm2pgsql & planet: frustrations, cutoffs, and idempotence

[Martijn van Oosterhout]

On Mon, Oct 27, 2008 at 1:10 AM, Michal Migurski <[EMAIL PROTECTED]> wrote:
> The final event in each weekly planet dump does not fall on an even
> day boundary. In the case of the most recent Oct. 22nd planet.osm, it
> was necessary to experiment with hourly diffs from that day to find
> that the boundary was approx. 2:00pm. Hourlies up to and including
> 2008102213-2008102214.osc.gz failed, hourlies after that succeeded. I
> could go more granular here, checking the minute diffs as well for a
> more precise breakpoint, but it seems odd that the planet dump does
> not break cleanly on a midnight boundary so that it's possible to pick
> up the differences moving forward.

As I recall, osm2pgsql did support this kind of operation (or at least
it did last time I tried, it was discussed on the list). All creates
in diffs are treated as delete+insert. You don't actually say what the
error was you ran into though so I can't be sure if you're talking
about the same problem.

Now that I think about it though, I think what I did was take one of
the planet dumps from http://hypercube.telascience.org/planet/ (which
*are* consistant snapshots), and run the dailies from there.

Have a nice day,
-- 
Martijn van Oosterhout <[EMAIL PROTECTED]> http://svana.org/kleptog/

___
talk mailing list
talk@openstreetmap.org
http://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] osm2pgsql & planet: frustrations, cutoffs, and idempotence

[Jochen Topf]

On Sun, Oct 26, 2008 at 06:11:04PM -0700, Michal Migurski wrote:
> What is the difference between osmosis and osm2pgsql, with regards to  
> postGIS?

osm2pgsql creates the structure needed for Mapnik. Osmosis creates a
structure more simliar to the one in the OSM central database.

> If I've been maintaining a dataset based on osm2pgsql with the  
> provided default.style, would a dataset based on osmosis result in a  
> substantially different table structure?

Yes.

Jochen
-- 
Jochen Topf  [EMAIL PROTECTED]  http://www.remote.org/jochen/  +49-721-388298


___
talk mailing list
talk@openstreetmap.org
http://lists.openstreetmap.org/listinfo/talk

Re: [OSM-talk] osm2pgsql & planet: frustrations, cutoffs, and idempotence

[Shaun McDonald]

On 27 Oct 2008, at 00:50, Michal Migurski wrote:

>>> The final event in each weekly planet dump does not fall on an
>>> even  day boundary. In the case of the most recent Oct. 22nd
>>> planet.osm, it  was necessary to experiment with hourly diffs from
>>> that day to find  that the boundary was approx. 2:00pm. Hourlies up
>>> to and including  2008102213-2008102214.osc.gz failed, hourlies
>>> after that succeeded. I  could go more granular here, checking the
>>> minute diffs as well for a  more precise breakpoint, but it seems
>>> odd that the planet dump does  not break cleanly on a midnight
>>> boundary so that it's possible to pick  up the differences moving
>>> forward.
>>
>> Planet dumps are not snapshots - they do not represent a consistent
>> view at any particular point in time because they take a number of
>> hours to generate, during which time new changes are constantly
>> being made to the contents of the database.
>
> Shouldn't it be possible to ignore any changes that happen after the
> cutoff, though?

At the moment we don't look at the time stamps when dumping the planet  
file.

> I may not understand the structure of the OSM
> database, but it seems like if it supports rollbacks, then in theory
> it ought to be possible to only include things before a given
> timestamp when creating the dump file. That, or make it clear what the
> actual cutoff time is in the dumpfile.

We currently don't support rollbacks. It would require a rewrite of  
the dump script, and more time and processing to be able to produce a  
consistent planet dump.

>
>
> I understand that in practice, practice is different from theory. =)

Have you got the rails port running?

>
>
>
>> I believe that it is supposed to be safe to apply diffs which
>> overlap with the planet dump in order to bring it to a consistent
>> state however.
>
> This is what I would have hoped, however osm2pgsql does not appear to
> allow it. It feels like the easiest solution would be to give
> osm2pgsql a --force option, and add some explanation of timing and
> cutoffs to http://planet.openstreetmap.org/README.
>

The initial import that you do with osm2pgsql, must be using a special  
mode to allow diff imports. Could it be that you need to update to the  
latest version of osm2pgsql? You should be able to happily apply the  
diffs to an inconsistent planet dump, to get a consistent planet dump.

This will become easier when the version numbers are exposed in the  
0.6 API. The diff mechanism would then be able to look at the version  
numbers of the nodes/ways/relations and be able to deal with them  
appropriately.

Shaun


___
talk mailing list
talk@openstreetmap.org
http://lists.openstreetmap.org/listinfo/talk