[osmosis-dev] Crash on processing nodes?
Osmosis ran yesterday for CycleStreets, but broke today. I think it is because this node was changed yesterday evening: http://www.openstreetmap.org/browse/node/620823 I don't know why it has crashed. The osmosis command and debug output follow. Simon osmosis -v 100 --read-xml data/osm/downloads/great_britain.osm enableDateParsing=no --bounding-polygon file="import/fz_cambridge.txt" --read-xml data/osm/downloads/ireland.osm enableDateParsing=no --bounding-polygon file="import/fz_cambridge.txt" --merge --write-apidb dbType=mysql host=localhost database=britainOSM user=import password=xxx validateSchemaVersion=no 09-Jan-2010 11:17:11 org.openstreetmap.osmosis.core.Osmosis run INFO: Osmosis Version 0.32 09-Jan-2010 11:17:13 org.openstreetmap.osmosis.core.TaskRegistrar loadJPFPlugins FINE: Searching for JPF plugins. 09-Jan-2010 11:17:13 org.openstreetmap.osmosis.core.TaskRegistrar gatherJpfPlugins FINER: Loading plugins in /websites/www/content/plugins 09-Jan-2010 11:17:13 org.openstreetmap.osmosis.core.TaskRegistrar gatherJpfPlugins FINER: Loading plugins in /root/.openstreetmap/osmosis/plugins 09-Jan-2010 11:17:13 org.openstreetmap.osmosis.core.TaskRegistrar gatherJpfPlugins FINER: Loading plugins in /websites/www/content/null/openstreetmap/osmosis/plugins 09-Jan-2010 11:17:13 org.openstreetmap.osmosis.core.TaskRegistrar loadJPFPlugins FINE: Registering the core plugin. 09-Jan-2010 11:17:13 org.openstreetmap.osmosis.core.TaskRegistrar registerCorePlugin FINEST: Plugin URL: jar:file:/usr/local/osmosis/osmosis-0.32/osmosis.jar!/org/openstreetmap/osmosis/core/plugin/plugin.xml 09-Jan-2010 11:17:15 org.openstreetmap.osmosis.core.TaskRegistrar loadJPFPlugins FINE: Registering the extension plugins. 09-Jan-2010 11:17:15 org.openstreetmap.osmosis.core.Osmosis run INFO: Preparing pipeline. 09-Jan-2010 11:17:15 org.openstreetmap.osmosis.core.pipeline.common.Pipeline prepare FINE: Building tasks. 09-Jan-2010 11:17:15 org.openstreetmap.osmosis.core.pipeline.common.Pipeline buildTasks FINE: Created task "1-read-xml" 09-Jan-2010 11:17:15 org.openstreetmap.osmosis.core.pipeline.common.Pipeline buildTasks FINE: Created task "2-bounding-polygon" 09-Jan-2010 11:17:15 org.openstreetmap.osmosis.core.pipeline.common.Pipeline buildTasks FINE: Created task "3-read-xml" 09-Jan-2010 11:17:15 org.openstreetmap.osmosis.core.pipeline.common.Pipeline buildTasks FINE: Created task "4-bounding-polygon" 09-Jan-2010 11:17:15 org.openstreetmap.osmosis.core.pipeline.common.Pipeline buildTasks FINE: Created task "5-merge" 09-Jan-2010 11:17:15 org.openstreetmap.osmosis.core.pipeline.common.Pipeline buildTasks FINE: Created task "6-write-apidb" 09-Jan-2010 11:17:15 org.openstreetmap.osmosis.core.pipeline.common.Pipeline prepare FINE: Connecting tasks. 09-Jan-2010 11:17:15 org.openstreetmap.osmosis.core.pipeline.common.PipeTasks putTask FINE: Task "1-read-xml" produced unnamed pipe stored at level 1 in the default pipe stack. 09-Jan-2010 11:17:15 org.openstreetmap.osmosis.core.pipeline.common.Pipeline connectTasks FINE: Connected task "1-read-xml" 09-Jan-2010 11:17:15 org.openstreetmap.osmosis.core.pipeline.common.PipeTasks retrieveTask FINE: Task "2-bounding-polygon" consumed unnamed pipe stored at level 1 in the default pipe stack. 09-Jan-2010 11:17:15 org.openstreetmap.osmosis.core.pipeline.common.PipeTasks putTask FINE: Task "2-bounding-polygon" produced unnamed pipe stored at level 1 in the default pipe stack. 09-Jan-2010 11:17:15 org.openstreetmap.osmosis.core.pipeline.common.Pipeline connectTasks FINE: Connected task "2-bounding-polygon" 09-Jan-2010 11:17:15 org.openstreetmap.osmosis.core.pipeline.common.PipeTasks putTask FINE: Task "3-read-xml" produced unnamed pipe stored at level 2 in the default pipe stack. 09-Jan-2010 11:17:15 org.openstreetmap.osmosis.core.pipeline.common.Pipeline connectTasks FINE: Connected task "3-read-xml" 09-Jan-2010 11:17:15 org.openstreetmap.osmosis.core.pipeline.common.PipeTasks retrieveTask FINE: Task "4-bounding-polygon" consumed unnamed pipe stored at level 2 in the default pipe stack. 09-Jan-2010 11:17:15 org.openstreetmap.osmosis.core.pipeline.common.PipeTasks putTask FINE: Task "4-bounding-polygon" produced unnamed pipe stored at level 2 in the default pipe stack. 09-Jan-2010 11:17:15 org.openstreetmap.osmosis.core.pipeline.common.Pipeline connectTasks FINE: Connected task "4-bounding-polygon" 09-Jan-2010 11:17:15 org.openstreetmap.osmosis.core.pipeline.common.PipeTasks retrieveTask FINE: Task "5-merge" consumed unnamed pipe stored at level 2 in the default pipe stack. 09-Jan-2010 11:17:15 org.openstreetmap.osmosis.core.pipeline.common.PipeTasks retrieveTask FINE: Task "5-merge" consumed unnamed pipe stored at level 1 in the default pipe stack. 09-Jan-2010 11:17:15 org.openstreetmap.osmosis.core.pipeline.common.PipeTasks putTask FINE: Task "5-merge" produced unnamed pipe stored at level 1 in the default pipe stack. 09-Jan-2010 11:17:15 org.openstreetmap.osmosis.core.pipeline.common.Pipeline con
Re: [osmosis-dev] Crash on processing nodes?
2010/1/9 Simon Nuttall : > Osmosis ran yesterday for CycleStreets, but broke today. > > I think it is because this node was changed yesterday evening: > > http://www.openstreetmap.org/browse/node/620823 > > I don't know why it has crashed. What more I have found out... 1. Today's great_britain.osm was generated by osmosis 0.31.1 , whereas yesterdays by osmosis 0.31 2. The offending node appears twice in the .osm: but only once in yesterday's .osm: 3. There are no tags for the node. Simon ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] Crash on processing nodes?
It was suggested that I try to sort the nodes first, so I tried that using: osmosis -v 100 --read-xml data/osm/downloads/great_britain.osm enableDateParsing=no --bounding-polygon file="import/fz_cambridge.txt" --sort-0.6 --write-xml data/osm/downloads/great_britain.sorted.osm but it made no difference to the order of the appearance of these two nodes: which seem to be causing the problem, with the same error report previously reported: SEVERE: Thread for task 1-read-xml failed org.openstreetmap.osmosis.core.OsmosisRuntimeException: Pipeline entities are not sorted, previous entity type=Node, id=620823, version=9 current entity type=Node, id=620823, version=10. at org.openstreetmap.osmosis.core.merge.v0_6.impl.SortedEntityPipeValidator.process(SortedEntityPipeValidator.java:47) Simon ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] Crash on processing nodes?
Hi Simon, The duplicate node entry could be caused by a couple of reasons: 1. There were bugs in some builds of Osmosis 0.31.1. Is there a reason you're using 0.31.1? The latest stable version is 0.32. You can get it here: http://wiki.openstreetmap.org/index.php/Osmosis#Latest_Stable_Version 2. Are you using replication diffs instead of daily diffs? These diffs can contain multiple version of the same entity because they are full history diffs. The delta style daily diffs are more appropriate for patching osm files. Brett On Sun, Jan 10, 2010 at 7:54 PM, Simon Nuttall wrote: > It was suggested that I try to sort the nodes first, so I tried that using: > > osmosis -v 100 --read-xml data/osm/downloads/great_britain.osm > enableDateParsing=no --bounding-polygon file="import/fz_cambridge.txt" > --sort-0.6 --write-xml data/osm/downloads/great_britain.sorted.osm > > but it made no difference to the order of the appearance of these two > nodes: > > uid="215789" user="smncrsk" changeset="3560454" lat="52.1920125" > lon="0.1884885"/> >uid="215789" user="smncrsk" changeset="3564864" lat="52.191996" > lon="0.1884762"/> > > which seem to be causing the problem, with the same error report > previously reported: > > SEVERE: Thread for task 1-read-xml failed > org.openstreetmap.osmosis.core.OsmosisRuntimeException: Pipeline > entities are not sorted, previous entity type=Node, id=620823, > version=9 current entity type=Node, id=620823, version=10. >at > org.openstreetmap.osmosis.core.merge.v0_6.impl.SortedEntityPipeValidator.process(SortedEntityPipeValidator.java:47) > > Simon > ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] Crash on processing nodes?
2010/1/10 Brett Henderson : > Hi Simon, > > The duplicate node entry could be caused by a couple of reasons: > > 1. There were bugs in some builds of Osmosis 0.31.1. Is there a reason > you're using 0.31.1? The latest stable version is 0.32. You can get it > here: > http://wiki.openstreetmap.org/index.php/Osmosis#Latest_Stable_Version > > 2. Are you using replication diffs instead of daily diffs? These diffs can > contain multiple version of the same entity because they are full history > diffs. The delta style daily diffs are more appropriate for patching osm > files. I think both these points are more a message for geofabrik.de than me because I'm using the diff's downloaded from their site. I am already using 0.32 for CycleSteets processing. thanks Simon ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] Crash on processing nodes?
2010/1/10 Simon Nuttall : > I think both these points are more a message for geofabrik.de than me > because I'm using the diff's downloaded from their site. Sorry, I wasn't clear, I meant to say that CycleStreets uses the ireland and great_britain extracts that we download from geofabrik.de, and their system seems to be using 0.31.1 now. Simon ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] Crash on processing nodes?
Hi, Simon Nuttall wrote: > 2010/1/10 Simon Nuttall : >> I think both these points are more a message for geofabrik.de than me >> because I'm using the diff's downloaded from their site. > > Sorry, I wasn't clear, I meant to say that CycleStreets uses the > ireland and great_britain extracts that we download from geofabrik.de, > and their system seems to be using 0.31.1 now. The problem was that in switching from --rci to --rri I didn't add the --simc filter, which led to duplicate objects in the extracts. It should be fixed now. I don't think that it was related to the Osmosis version, but switched to 0.32 nonetheless. Bye Frederik ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] Crash on processing nodes?
On Mon, Jan 11, 2010 at 6:40 PM, Frederik Ramm wrote: > Hi, > > Simon Nuttall wrote: > > 2010/1/10 Simon Nuttall : > >> I think both these points are more a message for geofabrik.de than me > >> because I'm using the diff's downloaded from their site. > > > > Sorry, I wasn't clear, I meant to say that CycleStreets uses the > > ireland and great_britain extracts that we download from geofabrik.de, > > and their system seems to be using 0.31.1 now. > > The problem was that in switching from --rci to --rri I didn't add the > --simc filter, which led to duplicate objects in the extracts. It should > be fixed now. I don't think that it was related to the Osmosis version, > but switched to 0.32 nonetheless. > I probably should fix the --apply-change so that it either fails or handles history diffs properly ... it's fairly non-obvious. ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] Crash on processing nodes?
On Mon, Jan 11, 2010 at 8:38 PM, Brett Henderson wrote: > On Mon, Jan 11, 2010 at 6:40 PM, Frederik Ramm wrote: > >> >> The problem was that in switching from --rci to --rri I didn't add the >> --simc filter, which led to duplicate objects in the extracts. It should >> be fixed now. I don't think that it was related to the Osmosis version, >> but switched to 0.32 nonetheless. >> > > I probably should fix the --apply-change so that it either fails or handles > history diffs properly ... it's fairly non-obvious. > > I've checked in a change to detect this scenario and fail now. It's only in svn for now. Brett ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] Crash on processing nodes?
On one of the CycleStreets machines it takes 17.3 hours to extract and merge data from the Britain and Ireland GeoFabrik planets using: osmosis -v 100 --read-xml data/osm/downloads/great_britain.osm enableDateParsing=no --sort-0.6 type="TypeThenId" --read-xml data/osm/downloads/ireland.osm enableDateParsing=no --sort-0.6 type="TypeThenId" --merge --write-apidb dbType=mysql host=localhost database=britainOSM user=import password=xxx validateSchemaVersion=no (On our main machine the same only takes about 5 hours.) During that time it builds an 11 GB innodb file in /var/lib/mysql/ibdata1 Osmosis fills the britainOSM database, which I then treat as 'read only' to build the CycleStreets routing tables in a separate database. As I'm only filling a database and then treating it as read-only do I really need all the transaction processing that is provided by innodb? Is there a way of turning those features off to speed up all the processing and use less disk space? Simon ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
Re: [osmosis-dev] Crash on processing nodes?
On Wed, Jan 20, 2010 at 11:24 PM, Simon Nuttall wrote: > On one of the CycleStreets machines it takes 17.3 hours to extract and > merge data from the Britain and Ireland GeoFabrik planets using: > > osmosis -v 100 --read-xml data/osm/downloads/great_britain.osm > enableDateParsing=no --sort-0.6 type="TypeThenId" --read-xml > data/osm/downloads/ireland.osm enableDateParsing=no --sort-0.6 > type="TypeThenId" --merge --write-apidb dbType=mysql host=localhost > database=britainOSM user=import password=xxx validateSchemaVersion=no > > (On our main machine the same only takes about 5 hours.) > > During that time it builds an 11 GB innodb file in /var/lib/mysql/ibdata1 > > > Osmosis fills the britainOSM database, which I then treat as 'read > only' to build the CycleStreets routing tables in a separate database. > > As I'm only filling a database and then treating it as read-only do I > really need all the transaction processing that is provided by innodb? > Is there a way of turning those features off to speed up all the > processing and use less disk space? > You could possibly add populateCurrentTables=no to the --write-apidb task and use the history tables for your queries. After a planet import they contain identical data to the current tables. Are the history tables still MyISAM or are they InnoDB? I haven't used MySQL for a while. If the history tables are MyISAM and you disable current table population you should get very large performance speedups. Brett ___ osmosis-dev mailing list osmosis-dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/osmosis-dev
