Re: [OSM-talk] osm2pgsql & planet: frustrations, cutoffs, and idempotence

2008-10-27 Thread Michal Migurski
> On Mon, Oct 27, 2008 at 9:39 PM, Michal Migurski <[EMAIL PROTECTED]> > wrote: >> I'm liking Jochen Topf's suggestion here: >> >> "If the planet dump plus the diff from the same day is what >> everybody >> wants anyway, why not do this on the server side and hold the planet >> back after

Re: [OSM-talk] osm2pgsql & planet: frustrations, cutoffs, and idempotence

2008-10-27 Thread Brett Henderson
On Tue, Oct 28, 2008 at 7:39 AM, Michal Migurski <[EMAIL PROTECTED]> wrote: > >> Finally, the boundaries between the hourlies and dailies seem > >> misaligned. > >> > > > > This shouldn't be the case. > >> After running the remaining hourlies for the 22nd, I attempted to > >> pick up on the 23rd

Re: [OSM-talk] osm2pgsql & planet: frustrations, cutoffs, and idempotence

2008-10-27 Thread Martijn van Oosterhout
On Mon, Oct 27, 2008 at 9:40 PM, Michal Migurski <[EMAIL PROTECTED]> wrote: >> Now that I think about it though, I think what I did was take one of >> the planet dumps from http://hypercube.telascience.org/planet/ (which >> *are* consistant snapshots), and run the dailies from there. > > Is there a

Re: [OSM-talk] osm2pgsql & planet: frustrations, cutoffs, and idempotence

2008-10-27 Thread Martijn van Oosterhout
On Mon, Oct 27, 2008 at 9:39 PM, Michal Migurski <[EMAIL PROTECTED]> wrote: > I'm liking Jochen Topf's suggestion here: > >"If the planet dump plus the diff from the same day is what everybody > wants anyway, why not do this on the server side and hold the planet > back after the first diff

Re: [OSM-talk] osm2pgsql & planet: frustrations, cutoffs, and idempotence

2008-10-27 Thread Michal Migurski
On Oct 27, 2008, at 12:59 AM, Martijn van Oosterhout wrote: > On Mon, Oct 27, 2008 at 1:10 AM, Michal Migurski <[EMAIL PROTECTED]> > wrote: >> The final event in each weekly planet dump does not fall on an even >> day boundary. In the case of the most recent Oct. 22nd planet.osm, it >> was neces

Re: [OSM-talk] osm2pgsql & planet: frustrations, cutoffs, and idempotence

2008-10-27 Thread Michal Migurski
>> Yep, as others have commented there are two tables types in the osm >> database; current tables, and history tables. The planet dumper >> just reads current tables which is the fastest approach. >> Unfortunately the current tables change constantly during the >> planet generation proce

Re: [OSM-talk] osm2pgsql & planet: frustrations, cutoffs, and idempotence

2008-10-27 Thread Frederik Ramm
Hi, Brett Henderson wrote: >> Brett Henderson has offered to look into creating the dailies from >> history as well, but I don't know about the status of that. >> > Are you referring to the daily changesets? [...] > Or did you mean planets instead of dailies? Mix-up on my part, sorry, yes I

Re: [OSM-talk] osm2pgsql & planet: frustrations, cutoffs, and idempotence

2008-10-27 Thread Brett Henderson
Jochen Topf wrote: > If the planet dump plus the diff from the same day is what everybody > wants anyway, why not do this on the server side and hold the planet > back after the first diff is available, run this over the planet and > then publish that as the planet? > It would add delay to the p

Re: [OSM-talk] osm2pgsql & planet: frustrations, cutoffs, and idempotence

2008-10-27 Thread Brett Henderson
Frederik Ramm wrote: > Hi, > > Michal Migurski wrote: > >> I've noticed some misalignments between the data in the dumps and the >> osm2pgsql importer that leads to unavoidable holes in the data. >> > > As TomH has already said, this is not a bug, it stems from the fact that > the full p

Re: [OSM-talk] osm2pgsql & planet: frustrations, cutoffs, and idempotence

2008-10-27 Thread Brett Henderson
Others have already commented on most of your points but I'll add my thoughts in case there's some gaps. Michal Migurski wrote: > Hi, > > I've been trying to keep up to date with the dumps and diffs from > http://planet.openstreetmap.org/ > , and I'm running into a number of bugs related to cut

Re: [OSM-talk] osm2pgsql & planet: frustrations, cutoffs, and idempotence

2008-10-27 Thread Jochen Topf
On Mon, Oct 27, 2008 at 08:22:32AM +, Tom Hughes wrote: > Shaun McDonald wrote: > > On 27 Oct 2008, at 00:50, Michal Migurski wrote: > > > >>> Planet dumps are not snapshots - they do not represent a consistent > >>> view at any particular point in time because they take a number of > >>> hour

Re: [OSM-talk] osm2pgsql & planet: frustrations, cutoffs, and idempotence

2008-10-27 Thread Tom Hughes
Shaun McDonald wrote: > On 27 Oct 2008, at 00:50, Michal Migurski wrote: > >>> Planet dumps are not snapshots - they do not represent a consistent >>> view at any particular point in time because they take a number of >>> hours to generate, during which time new changes are constantly >>> being ma

Re: [OSM-talk] osm2pgsql & planet: frustrations, cutoffs, and idempotence

2008-10-27 Thread Martijn van Oosterhout
On Mon, Oct 27, 2008 at 1:10 AM, Michal Migurski <[EMAIL PROTECTED]> wrote: > The final event in each weekly planet dump does not fall on an even > day boundary. In the case of the most recent Oct. 22nd planet.osm, it > was necessary to experiment with hourly diffs from that day to find > that the

Re: [OSM-talk] osm2pgsql & planet: frustrations, cutoffs, and idempotence

2008-10-27 Thread Jochen Topf
On Sun, Oct 26, 2008 at 06:11:04PM -0700, Michal Migurski wrote: > What is the difference between osmosis and osm2pgsql, with regards to > postGIS? osm2pgsql creates the structure needed for Mapnik. Osmosis creates a structure more simliar to the one in the OSM central database. > If I've been

Re: [OSM-talk] osm2pgsql & planet: frustrations, cutoffs, and idempotence

2008-10-26 Thread Shaun McDonald
On 27 Oct 2008, at 00:50, Michal Migurski wrote: >>> The final event in each weekly planet dump does not fall on an >>> even day boundary. In the case of the most recent Oct. 22nd >>> planet.osm, it was necessary to experiment with hourly diffs from >>> that day to find that the boundary was a

Re: [OSM-talk] osm2pgsql & planet: frustrations, cutoffs, and idempotence

2008-10-26 Thread Michal Migurski
On Oct 26, 2008, at 5:50 PM, Frederik Ramm wrote: > Brett Henderson has offered to look into creating the dailies from > history as well, but I don't know about the status of that. > > If you use osmosis, it is safe (and in fact recommended) that, after > loading the database with a planet fil

Re: [OSM-talk] osm2pgsql & planet: frustrations, cutoffs, and idempotence

2008-10-26 Thread Michal Migurski
>> The final event in each weekly planet dump does not fall on an >> even day boundary. In the case of the most recent Oct. 22nd >> planet.osm, it was necessary to experiment with hourly diffs from >> that day to find that the boundary was approx. 2:00pm. Hourlies up >> to and including

Re: [OSM-talk] osm2pgsql & planet: frustrations, cutoffs, and idempotence

2008-10-26 Thread Frederik Ramm
Hi, Michal Migurski wrote: > I've noticed some misalignments between the data in the dumps and the > osm2pgsql importer that leads to unavoidable holes in the data. As TomH has already said, this is not a bug, it stems from the fact that the full planet export reads the "current" tables and as

Re: [OSM-talk] osm2pgsql & planet: frustrations, cutoffs, and idempotence

2008-10-26 Thread Tom Hughes
Michal Migurski wrote: > The final event in each weekly planet dump does not fall on an even > day boundary. In the case of the most recent Oct. 22nd planet.osm, it > was necessary to experiment with hourly diffs from that day to find > that the boundary was approx. 2:00pm. Hourlies up to an

[OSM-talk] osm2pgsql & planet: frustrations, cutoffs, and idempotence

2008-10-26 Thread Michal Migurski
Hi, I've been trying to keep up to date with the dumps and diffs from http://planet.openstreetmap.org/ , and I'm running into a number of bugs related to cutoff dates. In keeping my Bay Area tiles (http://mike.teczno.com/notes/cascadenik-openstreetmap.html ) up to date, I've been grabbing com