Re: [OSM-dev] osm2pgsql fails on pbf file

2011-07-21 Thread Parveen Arora
On Fri, Jul 22, 2011 at 5:02 AM, Stephan Knauss wrote: > Hi, > > I tried osm2pgsql svn head revision. > > compiles fine. When trying to process asia.osm.pbf from geofabrik it fails: > > Reading in file: ../asia.osm.pbf > error parsing member id of DenseNodes > error parsing member dense of Primiti

[OSM-dev] osm2pgsql fails on pbf file

2011-07-21 Thread Stephan Knauss
Hi, I tried osm2pgsql svn head revision. compiles fine. When trying to process asia.osm.pbf from geofabrik it fails: Reading in file: ../asia.osm.pbf error parsing member id of DenseNodes error parsing member dense of PrimitiveGroup error parsing member stringtable of PrimitiveBlock Error unpac

Re: [OSM-dev] Updating Planet and Reliability

2011-07-21 Thread Stephan Knauss
Frederik Ramm writes: One other thing comes to mind. The most frequent problem with my minutely Osmosis updates is that about once every 2 months it simply hangs in some sort of timeout. I submitted a fix for this a while ago. If you build your own osmosis from SVN than you already have the

Re: [OSM-dev] Updating Planet and Reliability

2011-07-21 Thread Andrew Ayre
On 7/21/2011 12:48 PM, John Smith wrote: > On 21 July 2011 21:41, Andrew Ayre wrote: >> Yes, changes are applied sequentially, i.e. one set every 24 hours in my >> case as that is how frequently I run Osmosis to perform the change merge. > > So what is it you want to do exactly, the state file sa

Re: [OSM-dev] Updating Planet and Reliability

2011-07-21 Thread John Smith
On 21 July 2011 21:41, Andrew Ayre wrote: > Yes, changes are applied sequentially, i.e. one set every 24 hours in my > case as that is how frequently I run Osmosis to perform the change merge. So what is it you want to do exactly, the state file saved to the hdd is the last one successfully appli

Re: [OSM-dev] Updating Planet and Reliability

2011-07-21 Thread Andrew Ayre
On 7/21/2011 12:36 PM, John Smith wrote: > On 21 July 2011 21:33, Andrew Ayre wrote: >> From the date/time of the last successfully applied changes. > > It already does that, but that's not what you're expecting, you're > expecting that changes are applied sequentially, not as a group. Yes, chan

Re: [OSM-dev] Updating Planet and Reliability

2011-07-21 Thread John Smith
On 21 July 2011 21:33, Andrew Ayre wrote: > From the date/time of the last successfully applied changes. It already does that, but that's not what you're expecting, you're expecting that changes are applied sequentially, not as a group. ___ dev mailing

Re: [OSM-dev] Updating Planet and Reliability

2011-07-21 Thread Frederik Ramm
Hi, On 07/21/11 10:07, Andrew Ayre wrote: What protection is there in Osmosis to recover from this without missing any changes? If none, how are people solving this in their scripts? One other thing comes to mind. The most frequent problem with my minutely Osmosis updates is that about once e

Re: [OSM-dev] Updating Planet and Reliability

2011-07-21 Thread Andrew Ayre
On 7/21/2011 12:29 PM, John Smith wrote: > On 21 July 2011 21:26, Andrew Ayre wrote: >> I don't care about resuming partially completed change merges if that is >> what you mean. What I do care about is detecting an interrupted merge >> and starting it again from the beginning. I don't expect rebo

Re: [OSM-dev] Updating Planet and Reliability

2011-07-21 Thread John Smith
On 21 July 2011 21:26, Andrew Ayre wrote: > I don't care about resuming partially completed change merges if that is > what you mean. What I do care about is detecting an interrupted merge > and starting it again from the beginning. I don't expect reboots to > happen very often, so taking an extra

Re: [OSM-dev] Updating Planet and Reliability

2011-07-21 Thread Andrew Ayre
On 7/21/2011 12:17 PM, John Smith wrote: > You misunderstood me, I didn't say anything took 5 minutes, nor that > the memory state file would be any where near 85 bytes, Osmosis does a > lot of the merging in memory to be able to come back to a point close > in time you would need to dump the memor

Re: [OSM-dev] Updating Planet and Reliability

2011-07-21 Thread John Smith
On 21 July 2011 21:08, Andrew Ayre wrote: > Merging changes into the 14G planet file takes my server about an hour, > so I don't think writing an 85 byte state file at the end of it will > impact performance much. > > I'm interested to know what commands you use to merge changes into a > planet fi

Re: [OSM-dev] Updating Planet and Reliability

2011-07-21 Thread Andrew Ayre
On 7/21/2011 12:02 PM, John Smith wrote: > On 21 July 2011 20:45, Andrew Ayre wrote: >> Is there any possibility of changing Osmosis so arbitrary post merge >> commands could be executed before finalizing the state file? E.g.: > > The problem is if osmosis posted a memory state file every say 5 >

Re: [OSM-dev] Updating Planet and Reliability

2011-07-21 Thread John Smith
On 21 July 2011 20:45, Andrew Ayre wrote: > Is there any possibility of changing Osmosis so arbitrary post merge > commands could be executed before finalizing the state file? E.g.: The problem is if osmosis posted a memory state file every say 5 minutes, this may seriously impact on performance,

Re: [OSM-dev] Updating Planet and Reliability

2011-07-21 Thread Andrew Ayre
On 7/21/2011 11:28 AM, Brett Henderson wrote: > You will however need a wrapper shell script to rename the new planet > file to the original file name and this leaves a very small window for > something to go wrong between the state.txt being updated and the file > being renamed. > > Brett Thanks

Re: [OSM-dev] Updating Planet and Reliability

2011-07-21 Thread Brett Henderson
On Thu, Jul 21, 2011 at 6:18 PM, Frederik Ramm wrote: > Hi, > > > On 07/21/2011 10:07 AM, Andrew Ayre wrote: > >> What protection is there in Osmosis to recover from this without missing >> any changes? If none, how are people solving this in their scripts? >> > > I usually do this: > > * get lat

Re: [OSM-dev] Updating Planet and Reliability

2011-07-21 Thread Frederik Ramm
Hi, On 07/21/2011 10:07 AM, Andrew Ayre wrote: What protection is there in Osmosis to recover from this without missing any changes? If none, how are people solving this in their scripts? I usually do this: * get latest changes, save in latest.osc * if a file named changes-to-apply.osc exists

[OSM-dev] Updating Planet and Reliability

2011-07-21 Thread Andrew Ayre
Keeping my copy of the planet up to date is a two-step process with Osmosis. Get the latest changes and applying them. This takes about an hour on my server which is enough time for some other user to reboot the server without realizing/knowing. What protection is there in Osmosis to recover from