Re: [osmosis-dev] keeping an AOI-API-DB up to date
Hi Marco, I'm not sure if you've received an answer elsewhere, but here goes. There are a number of ways your data can end up with referential integrity problems. The most likely is simply that by cutting an area out using the bounding polygon task you'll end up with missing nodes. Some ways will be included because they have nodes within the bounding polygon, but nodes that are part of the way but lie outside the polygon will not be included. You have a couple of choices: - Set clipIncompleteEntities=true on your --bp task. This will modify the ways and relations to fix referential integrity, but be aware that the entities no longer match the API database so they *MUST NOT* be re-uploaded. - Simply drop the offending constraint from the database. This is what I've done in the past. I can't guarantee it will work with rails, but it certainly used to in the past. Brett On Tue, Feb 9, 2010 at 2:50 AM, Marco Lechner - FOSSGIS e.V. < marco.lech...@fossgis.de> wrote: > Hi I tried to use the described way to keep a AOI (area of interest) > OSM-file up to date. The following errors or unexpected behaviour occured: > > 1. downloaded planetfile (planet-100120.osm.bz2) > 2. osmosis (0.32) with --bp completeWays=yes and completeRelations=yes > to cut out myAOI > 3. downloaded daily-diffs (20100120-20100121.osc, ...) > 4. merged the diffs using osmosis => 20100120-20100120.osc > 5. applied merged diffs using: >./bin/osmosis --rxc 20100120-20100126.osc --rx aoi-100120.osm.bz2 > --ac --bp file=aoi.pff completeWays=yes completeRelations=yes --wx > file=aoi-100126.osm.bz2 > 6. tried to import both aoi-osm.bz2 into separate PostgreSQL-DBs > using API-Scheme. >Results: >No Errors: - bzcat ./aoi-100120.osm.bz2 | ./bin/osmosis --rx file=- > --wd database=osmAOI_api_1 host=localhost:5432 user=osmuser password="pass" >Error: - bzcat ./aoi-100126.osm.bz2 | ./bin/osmosis --rx file=- --wd > database=osmAOI_api_2 host=localhost:5432 user=osmuser password="pass" > Error message: > Unable to load current way nodes. >at > > org.openstreetmap.osmosis.core.apidb.v0_6.ApidbWriter.complete(ApidbWriter.java:980) >at > org.openstreetmap.osmosis.core.xml.v0_6.XmlReader.run(XmlReader.java:110) >at java.lang.Thread.run(Thread.java:619) > Caused by: org.postgresql.util.PSQLException: ERROR: Insert or Update in > table »current_way_nodes« verletzt foreignkey-Constraint > »current_way_nodes_node_id_fkey« > Detail: key (node_id)=(581918002) does not exist in table »current_nodes«. > > The error message says it clear. My workflow to keep aoi.osm.bz2 up to > date seems to produce an "erroneous" aoi.osm.bz2 or at least a osm-file > that differs from a freshly cutted one. I expected that cutting an AOI > from a planetfile should produce the same results than keeping my AOI up > to date by applying planet*.osc and cutting these back to myAOI. > > Before trying to: > > bzcat ./aoi-100120.osm.bz2 | ./bin/osmosis --rx file=- --wd > database=osmAOI_api_1 host=localhost:5432 user=osmuser password="pass" > > ./bin/osmosis --rxc 20100120-20100126.osc --rd database=osmAOI_api_1 > host=localhost:5432 user=osmuser password="pass" --ac --bp file=aoi.pff > completeWays=yes completeRelations=yes --wd database=osmAOI_api_1 > host=localhost:5432 user=osmuser password="pass" > > will this keep my apiDB of an AOI up to date? > > Marco > > P.S. I'm not sure if this is more or less a user-question and therefore > using the wrong list, but I guess it moves quite next to the > feature-wish section. > > Lennard schrieb: > > Marco Lechner - FOSSGIS e.V. wrote: > > > > > >> Otheriwse I have to apply the diff onto a planetfile and cut it everyday > >> or apply the planet diff to ma AOI.osm and cut it afterwards. > >> Is this a proper workflow? > >> Or what ist the best task to keep an AOI.osm (cutted by osmosis using > >> --bp) up-to-date? > >> > > > > Apply the diffs first, then pass it through the --bp task before writing > > it out again: > > > > osmosis --rxc 20100127-20100128.osc.gz --rx AIO-20100127.osm --ac --bp > > file=/path/to/your/polygon/file --wx AIO-20100128.osm > > > > > > > ___ > osmosis-dev mailing list > osmosis-dev@openstreetmap.org > http://lists.openstreetmap.org/listinfo/osmosis-dev > ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] keeping an AOI-API-DB up to date
Hi Brett, thanx for your answer. I thought using completeWays=yes and completeRelations=yes takes care of nodes outside my bounding polygon. 1. If a node is created new it will be inside my merged osc and therefore be inside my patched AOI.osm and (if part of a way or relation) put in my recently cutted AOI.osm 2. If a node is deleted it will be deleted by patching the merged osc 3. the only problem seems to exist if an existing node outside the polygon is changed and integrated into a way or relation inside my AOI. Because the node doesn't exist in my AOI (also not in the patched AOI before cutting it again), because it isn't created freshly. Am I right? Or are there situations I miss? I'm very interessted in creating a consistent regionally bordered OSM-DB. Marco Brett Henderson schrieb: > Hi Marco, > > I'm not sure if you've received an answer elsewhere, but here goes. > > There are a number of ways your data can end up with referential > integrity problems. The most likely is simply that by cutting an area > out using the bounding polygon task you'll end up with missing nodes. > Some ways will be included because they have nodes within the bounding > polygon, but nodes that are part of the way but lie outside the > polygon will not be included. > > You have a couple of choices: > > * Set clipIncompleteEntities=true on your --bp task. This will > modify the ways and relations to fix referential integrity, but > be aware that the entities no longer match the API database so > they *MUST NOT* be re-uploaded. > * Simply drop the offending constraint from the database. This is > what I've done in the past. I can't guarantee it will work with > rails, but it certainly used to in the past. > > Brett > > On Tue, Feb 9, 2010 at 2:50 AM, Marco Lechner - FOSSGIS e.V. > mailto:marco.lech...@fossgis.de>> wrote: > > Hi I tried to use the described way to keep a AOI (area of interest) > OSM-file up to date. The following errors or unexpected behaviour > occured: > > 1. downloaded planetfile (planet-100120.osm.bz2) > 2. osmosis (0.32) with --bp completeWays=yes and completeRelations=yes > to cut out myAOI > 3. downloaded daily-diffs (20100120-20100121.osc, ...) > 4. merged the diffs using osmosis => 20100120-20100120.osc > 5. applied merged diffs using: >./bin/osmosis --rxc 20100120-20100126.osc --rx aoi-100120.osm.bz2 > --ac --bp file=aoi.pff completeWays=yes completeRelations=yes --wx > file=aoi-100126.osm.bz2 > 6. tried to import both aoi-osm.bz2 into separate PostgreSQL-DBs > using API-Scheme. >Results: >No Errors: - bzcat ./aoi-100120.osm.bz2 | ./bin/osmosis --rx file=- > --wd database=osmAOI_api_1 host=localhost:5432 user=osmuser > password="pass" >Error: - bzcat ./aoi-100126.osm.bz2 | ./bin/osmosis --rx file=- > --wd > database=osmAOI_api_2 host=localhost:5432 user=osmuser password="pass" > Error message: > Unable to load current way nodes. >at > > org.openstreetmap.osmosis.core.apidb.v0_6.ApidbWriter.complete(ApidbWriter.java:980) >at > org.openstreetmap.osmosis.core.xml.v0_6.XmlReader.run(XmlReader.java:110) >at java.lang.Thread.run(Thread.java:619) > Caused by: org.postgresql.util.PSQLException: ERROR: Insert or > Update in > table »current_way_nodes« verletzt foreignkey-Constraint > »current_way_nodes_node_id_fkey« > Detail: key (node_id)=(581918002) does not exist in table > »current_nodes«. > > The error message says it clear. My workflow to keep aoi.osm.bz2 up to > date seems to produce an "erroneous" aoi.osm.bz2 or at least a > osm-file > that differs from a freshly cutted one. I expected that cutting an AOI > from a planetfile should produce the same results than keeping my > AOI up > to date by applying planet*.osc and cutting these back to myAOI. > > Before trying to: > > bzcat ./aoi-100120.osm.bz2 | ./bin/osmosis --rx file=- --wd > database=osmAOI_api_1 host=localhost:5432 user=osmuser password="pass" > > ./bin/osmosis --rxc 20100120-20100126.osc --rd database=osmAOI_api_1 > host=localhost:5432 user=osmuser password="pass" --ac --bp > file=aoi.pff > completeWays=yes completeRelations=yes --wd database=osmAOI_api_1 > host=localhost:5432 user=osmuser password="pass" > > will this keep my apiDB of an AOI up to date? > > Marco > > P.S. I'm not sure if this is more or less a user-question and > therefore > using the wrong list, but I guess it moves quite next to the > feature-wish section. > > Lennard schrieb: > > Marco Lechner - FOSSGIS e.V. wrote: > > > > > >> Otheriwse I have to apply the diff onto a planetfile and cut it > everyday > >> or apply the planet diff to ma AOI.osm and cut it afterwards. > >> Is this a proper workflow? > >> Or what ist the best task to keep a
Re: [osmosis-dev] keeping an AOI-API-DB up to date
Hi Marco, On Tue, Feb 23, 2010 at 12:08 AM, Marco Lechner - FOSSGIS e.V. < marco.lech...@fossgis.de> wrote: > Hi Brett, > > thanx for your answer. > I thought using completeWays=yes and completeRelations=yes takes care of > nodes outside my bounding polygon. > Those options will fix the problem, but they are horrendously slow. If they work for you then that's fine, but most users avoid those options if at all possible. > > 1. If a node is created new it will be inside my merged osc and > therefore be inside my patched AOI.osm and (if part of a way or > relation) put in my recently cutted AOI.osm > Correct. > 2. If a node is deleted it will be deleted by patching the merged osc > Also correct. > 3. the only problem seems to exist if an existing node outside the > polygon is changed and integrated into a way or relation inside my AOI. > Because the node doesn't exist in my AOI (also not in the patched AOI > before cutting it again), because it isn't created freshly. Am I right? > Or are there situations I miss? > Yes, also correct :-) Well kind of anyway. If the node is modified as well as the newly containing way or relation then it will be available for inclusion (if you're using completeWays or completeRelations anyway). But if the node is not modified but it is added to a new or existing way or relation then it cannot be included. However if you're not using the completeWays and completeRelations options then it will never be included regardless of scenario. > > I'm very interessted in creating a consistent regionally bordered OSM-DB. > Creating a perfect bordered extract is not simple. But you do have a couple of options: 1. Load the complete planet (or at least an area somewhat larger than the one you require) into the Osmosis "simple" schema via the --write-pgsql-dump task, then use the --read-pgsql and --dataset-bounding-box task to extract a perfect extract including all required nodes. Keep the database up to date with minute changesets and the --write-pgsql-change task. 2. Create the extract in two parts. First part is to extract a region from the overall planet that is larger than your required area, but small enough to keep the result file small. Do this with the default --bounding-box task. Then run the --bounding-box task over the smaller result file with completeWays and completeRelations both enabled. I haven't tried this myself, but I think it should work. It's not a simple problem unfortunately, and unless you have a really really good reason to make the extract correct, it's much simpler to make some compromises. Brett ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
