Re: [osmosis-dev] keeping an AOI-API-DB up to date
Hi Brett, thanx for your answer. I thought using completeWays=yes and completeRelations=yes takes care of nodes outside my bounding polygon. 1. If a node is created new it will be inside my merged osc and therefore be inside my patched AOI.osm and (if part of a way or relation) put in my recently cutted AOI.osm 2. If a node is deleted it will be deleted by patching the merged osc 3. the only problem seems to exist if an existing node outside the polygon is changed and integrated into a way or relation inside my AOI. Because the node doesn't exist in my AOI (also not in the patched AOI before cutting it again), because it isn't created freshly. Am I right? Or are there situations I miss? I'm very interessted in creating a consistent regionally bordered OSM-DB. Marco Brett Henderson schrieb: > Hi Marco, > > I'm not sure if you've received an answer elsewhere, but here goes. > > There are a number of ways your data can end up with referential > integrity problems. The most likely is simply that by cutting an area > out using the bounding polygon task you'll end up with missing nodes. > Some ways will be included because they have nodes within the bounding > polygon, but nodes that are part of the way but lie outside the > polygon will not be included. > > You have a couple of choices: > > * Set clipIncompleteEntities=true on your --bp task. This will > modify the ways and relations to fix referential integrity, but > be aware that the entities no longer match the API database so > they *MUST NOT* be re-uploaded. > * Simply drop the offending constraint from the database. This is > what I've done in the past. I can't guarantee it will work with > rails, but it certainly used to in the past. > > Brett > > On Tue, Feb 9, 2010 at 2:50 AM, Marco Lechner - FOSSGIS e.V. > mailto:marco.lech...@fossgis.de>> wrote: > > Hi I tried to use the described way to keep a AOI (area of interest) > OSM-file up to date. The following errors or unexpected behaviour > occured: > > 1. downloaded planetfile (planet-100120.osm.bz2) > 2. osmosis (0.32) with --bp completeWays=yes and completeRelations=yes > to cut out myAOI > 3. downloaded daily-diffs (20100120-20100121.osc, ...) > 4. merged the diffs using osmosis => 20100120-20100120.osc > 5. applied merged diffs using: >./bin/osmosis --rxc 20100120-20100126.osc --rx aoi-100120.osm.bz2 > --ac --bp file=aoi.pff completeWays=yes completeRelations=yes --wx > file=aoi-100126.osm.bz2 > 6. tried to import both aoi-osm.bz2 into separate PostgreSQL-DBs > using API-Scheme. >Results: >No Errors: - bzcat ./aoi-100120.osm.bz2 | ./bin/osmosis --rx file=- > --wd database=osmAOI_api_1 host=localhost:5432 user=osmuser > password="pass" >Error: - bzcat ./aoi-100126.osm.bz2 | ./bin/osmosis --rx file=- > --wd > database=osmAOI_api_2 host=localhost:5432 user=osmuser password="pass" > Error message: > Unable to load current way nodes. >at > > org.openstreetmap.osmosis.core.apidb.v0_6.ApidbWriter.complete(ApidbWriter.java:980) >at > org.openstreetmap.osmosis.core.xml.v0_6.XmlReader.run(XmlReader.java:110) >at java.lang.Thread.run(Thread.java:619) > Caused by: org.postgresql.util.PSQLException: ERROR: Insert or > Update in > table »current_way_nodes« verletzt foreignkey-Constraint > »current_way_nodes_node_id_fkey« > Detail: key (node_id)=(581918002) does not exist in table > »current_nodes«. > > The error message says it clear. My workflow to keep aoi.osm.bz2 up to > date seems to produce an "erroneous" aoi.osm.bz2 or at least a > osm-file > that differs from a freshly cutted one. I expected that cutting an AOI > from a planetfile should produce the same results than keeping my > AOI up > to date by applying planet*.osc and cutting these back to myAOI. > > Before trying to: > > bzcat ./aoi-100120.osm.bz2 | ./bin/osmosis --rx file=- --wd > database=osmAOI_api_1 host=localhost:5432 user=osmuser password="pass" > > ./bin/osmosis --rxc 20100120-20100126.osc --rd database=osmAOI_api_1 > host=localhost:5432 user=osmuser password="pass" --ac --bp > file=aoi.pff > completeWays=yes completeRelations=yes --wd database=osmAOI_api_1 > host=localhost:5432 user=osmuser password="pass" > > will this keep my apiDB of an AOI up to date? > > Marco > > P.S. I'm not sure if this is more or less a user-question and > therefore > using the wrong list, but I guess it moves quite next to the > feature-wish section. > > Lennard schrieb: > > Marco Lechner - FOSSGIS e.V. wrote: > > > > > >> Otheriwse I have to apply the diff onto a planetfile and cut it > everyday > >> or apply the planet diff to ma AOI.osm and cut it afterwards. > >> Is this a proper workflow? > >> Or what ist the best task to keep a
Re: [osmosis-dev] keeping an AOI-API-DB up to date
Hi Marco, I'm not sure if you've received an answer elsewhere, but here goes. There are a number of ways your data can end up with referential integrity problems. The most likely is simply that by cutting an area out using the bounding polygon task you'll end up with missing nodes. Some ways will be included because they have nodes within the bounding polygon, but nodes that are part of the way but lie outside the polygon will not be included. You have a couple of choices: - Set clipIncompleteEntities=true on your --bp task. This will modify the ways and relations to fix referential integrity, but be aware that the entities no longer match the API database so they *MUST NOT* be re-uploaded. - Simply drop the offending constraint from the database. This is what I've done in the past. I can't guarantee it will work with rails, but it certainly used to in the past. Brett On Tue, Feb 9, 2010 at 2:50 AM, Marco Lechner - FOSSGIS e.V. < marco.lech...@fossgis.de> wrote: > Hi I tried to use the described way to keep a AOI (area of interest) > OSM-file up to date. The following errors or unexpected behaviour occured: > > 1. downloaded planetfile (planet-100120.osm.bz2) > 2. osmosis (0.32) with --bp completeWays=yes and completeRelations=yes > to cut out myAOI > 3. downloaded daily-diffs (20100120-20100121.osc, ...) > 4. merged the diffs using osmosis => 20100120-20100120.osc > 5. applied merged diffs using: >./bin/osmosis --rxc 20100120-20100126.osc --rx aoi-100120.osm.bz2 > --ac --bp file=aoi.pff completeWays=yes completeRelations=yes --wx > file=aoi-100126.osm.bz2 > 6. tried to import both aoi-osm.bz2 into separate PostgreSQL-DBs > using API-Scheme. >Results: >No Errors: - bzcat ./aoi-100120.osm.bz2 | ./bin/osmosis --rx file=- > --wd database=osmAOI_api_1 host=localhost:5432 user=osmuser password="pass" >Error: - bzcat ./aoi-100126.osm.bz2 | ./bin/osmosis --rx file=- --wd > database=osmAOI_api_2 host=localhost:5432 user=osmuser password="pass" > Error message: > Unable to load current way nodes. >at > > org.openstreetmap.osmosis.core.apidb.v0_6.ApidbWriter.complete(ApidbWriter.java:980) >at > org.openstreetmap.osmosis.core.xml.v0_6.XmlReader.run(XmlReader.java:110) >at java.lang.Thread.run(Thread.java:619) > Caused by: org.postgresql.util.PSQLException: ERROR: Insert or Update in > table »current_way_nodes« verletzt foreignkey-Constraint > »current_way_nodes_node_id_fkey« > Detail: key (node_id)=(581918002) does not exist in table »current_nodes«. > > The error message says it clear. My workflow to keep aoi.osm.bz2 up to > date seems to produce an "erroneous" aoi.osm.bz2 or at least a osm-file > that differs from a freshly cutted one. I expected that cutting an AOI > from a planetfile should produce the same results than keeping my AOI up > to date by applying planet*.osc and cutting these back to myAOI. > > Before trying to: > > bzcat ./aoi-100120.osm.bz2 | ./bin/osmosis --rx file=- --wd > database=osmAOI_api_1 host=localhost:5432 user=osmuser password="pass" > > ./bin/osmosis --rxc 20100120-20100126.osc --rd database=osmAOI_api_1 > host=localhost:5432 user=osmuser password="pass" --ac --bp file=aoi.pff > completeWays=yes completeRelations=yes --wd database=osmAOI_api_1 > host=localhost:5432 user=osmuser password="pass" > > will this keep my apiDB of an AOI up to date? > > Marco > > P.S. I'm not sure if this is more or less a user-question and therefore > using the wrong list, but I guess it moves quite next to the > feature-wish section. > > Lennard schrieb: > > Marco Lechner - FOSSGIS e.V. wrote: > > > > > >> Otheriwse I have to apply the diff onto a planetfile and cut it everyday > >> or apply the planet diff to ma AOI.osm and cut it afterwards. > >> Is this a proper workflow? > >> Or what ist the best task to keep an AOI.osm (cutted by osmosis using > >> --bp) up-to-date? > >> > > > > Apply the diffs first, then pass it through the --bp task before writing > > it out again: > > > > osmosis --rxc 20100127-20100128.osc.gz --rx AIO-20100127.osm --ac --bp > > file=/path/to/your/polygon/file --wx AIO-20100128.osm > > > > > > > ___ > osmosis-dev mailing list > osmosis-dev@openstreetmap.org > http://lists.openstreetmap.org/listinfo/osmosis-dev > ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] Slow to import into a postgres api schema
On Mon, Feb 22, 2010 at 12:31 AM, Kai Krueger wrote: > On 02/21/2010 11:43 AM, Brett Henderson wrote: > >> >> > Having a --write-api0.6-dump command might not be a bad idea. Although the > --wd task is simpler from a users point of view if it can be made to be > equally fast. Sorry, I wasn't clear there. I wasn't suggesting we create a --write-apidb-dump task. I was just suggesting that there may be some re-usable code in the --write-pgsql-dump task because it already formats a number of PostgreSQL data types into the COPY format. Although I haven't looked at the new JDBC COPY support so I'm not sure if that even makes sense. > > > >> >>As a proof of concept, I added statements into disableIndexes to >>manually drop each index and then recreate them in enableIndexes. >>Together with using the Copy command (supported in the postgres 8.4 >> JDBC >>driver), my initial experiments show a speedup of 3 - 4 times on the >>initial population of the tables (i.e. without populating the current >>tables, but I suspect that this step can be similarly sped up). These >>numbers were obtained using small country extracts (e.g. 1 - 20 Mb in >>bz2 size), but I would guess that they hold up with the full planet >>imports too. >> >>The main benefit comes from disabling the indecies, and the copy >> command >>seems less important. >> >> >>The patch I have is quite ugly (and untested for correctness), as it >>breaks the levels of abstraction and has to hard code all the available >>indecies. So my question is, what would be the best way to do this in a >>clean way? Looking at the speedups obtained and the time involved in >>imports, it seems like it might be worth it. >> >> >> If it's truly 3-4 times faster then it's worth a lot of effort. I don't >> have a lot of time to get involved in this myself though, so if you have >> some time to write a maintainable patch, then I'd be very grateful. My >> only ask is that you stick around to get it working and provide support >> until it is proven stable. >> > > I have very little time myself at the moment. But perhaps I will be able to > come up with something. Perhaps not too soon though. At least not a > maintainable patch that integrates nicely with moduler structure of osmosis. Okay. See how you go, and post whatever you come up with back to the list. > > * I'd prefer to make this a new task within Osmosis. The current >> >> one is called --write-apidb I think. You could create a new one >> called something like --write-apidb-fast. If you can get it >> stable and it works well, then we can point --write-apidb at your >> task and delete the current one. >> > > Oh, that would mean I would actually have to understand this wonderful > extensible structure of osmosis rather than just hack some random bits and > pieces into it... ;-) In case you don't know already, creating a new task is a matter of modifying TaskRegistrar to register your task factory, creating a new task factory (use those called from TaskRegistrar as an example), then create a new Task implementing the Sink interface (use the existing DB tasks as an example). How much of the rest of Osmosis you choose to use is largely up to you, although more re-use is definitely a good thing from a maintenance perspective. Cheers, Brett ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
