On 27 Oct 2008, at 00:50, Michal Migurski wrote: >>> The final event in each weekly planet dump does not fall on an >>> even day boundary. In the case of the most recent Oct. 22nd >>> planet.osm, it was necessary to experiment with hourly diffs from >>> that day to find that the boundary was approx. 2:00pm. Hourlies up >>> to and including 2008102213-2008102214.osc.gz failed, hourlies >>> after that succeeded. I could go more granular here, checking the >>> minute diffs as well for a more precise breakpoint, but it seems >>> odd that the planet dump does not break cleanly on a midnight >>> boundary so that it's possible to pick up the differences moving >>> forward. >> >> Planet dumps are not snapshots - they do not represent a consistent >> view at any particular point in time because they take a number of >> hours to generate, during which time new changes are constantly >> being made to the contents of the database. > > Shouldn't it be possible to ignore any changes that happen after the > cutoff, though?
At the moment we don't look at the time stamps when dumping the planet file. > I may not understand the structure of the OSM > database, but it seems like if it supports rollbacks, then in theory > it ought to be possible to only include things before a given > timestamp when creating the dump file. That, or make it clear what the > actual cutoff time is in the dumpfile. We currently don't support rollbacks. It would require a rewrite of the dump script, and more time and processing to be able to produce a consistent planet dump. > > > I understand that in practice, practice is different from theory. =) Have you got the rails port running? > > > >> I believe that it is supposed to be safe to apply diffs which >> overlap with the planet dump in order to bring it to a consistent >> state however. > > This is what I would have hoped, however osm2pgsql does not appear to > allow it. It feels like the easiest solution would be to give > osm2pgsql a --force option, and add some explanation of timing and > cutoffs to http://planet.openstreetmap.org/README. > The initial import that you do with osm2pgsql, must be using a special mode to allow diff imports. Could it be that you need to update to the latest version of osm2pgsql? You should be able to happily apply the diffs to an inconsistent planet dump, to get a consistent planet dump. This will become easier when the version numbers are exposed in the 0.6 API. The diff mechanism would then be able to look at the version numbers of the nodes/ways/relations and be able to deal with them appropriately. Shaun _______________________________________________ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
