[OSM-dev] Diff issues: minutely overwritten by later data/incompl. daily diffs

[mmd]

Hi there,

I'm currently struggling with two different issues when processing daily
and minutely diff files.

Originally I planned to set up a full Overpass API instance using daily
diffs. During processing, some ways/nodes were found to be missing,
which shouldn't have happened in the first place. Now trying again with
minutely diffs, I noticed some minutely diff files got overwritten by
much later data on the server. The two separate issues in more detail:


1/ First 234 OdBl minutely diffs overwritten by 2016 data
-

This currently seems to affect the files
http://planet.openstreetmap.org/replication/minute/000/000/001.state.txt
http://planet.openstreetmap.org/replication/minute/000/000/001.osc.gz
to
http://planet.openstreetmap.org/replication/minute/000/000/234.state.txt
http://planet.openstreetmap.org/replication/minute/000/000/234.osc.gz

For some reason those minutely diffs now include data from May 2016 (!)

Where did the original minutely diffs from September 2012 go? Can they
be restored from backup?


2/ Some daily diffs incomplete
--

Way 326324100, version 1 was somehow not included in a daily diff,
although the respective minutely diff includes that version.



Respective files I checked:

http://planet.openstreetmap.org/replication/minute/001/252/157.osc.gz
http://planet.openstreetmap.org/replication/day/000/000/876.osc.gz

I thought daily diffs should always include all minutely diffs for that
day. Was this caused by some osmosis hiccup? Maybe this affects other
daily diffs as well.

Processing daily diffs is in fact a very convenient to load 4 years
worth of OSM data (needed for full history), rather than downloading
more than 2 Mio. minutely diffs.

Thanks for looking into this.

regards,


___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev

Re: [OSM-dev] Diff issues: minutely overwritten by later data/incompl. daily diffs

[Paul Norman]

On 8/14/2016 2:56 AM, mmd wrote:

Processing daily diffs is in fact a very convenient to load 4 years
worth of OSM data (needed for full history), rather than downloading
more than 2 Mio. minutely diffs.


I don't know about the data issues, but I suggest you use the full 
history. Over half the data in OSM is covered by those diffs. The full 
history is available as a faster to parse PBF format, and an initial 
import is generally faster than consuming diffs.


___
dev mailing list
dev@openstreetmap.org
https://lists.openstreetmap.org/listinfo/dev