Hi, Thanks for that, loads of good info in there!
>Playing with indexes, especially partial indexes on the ways columns, >reflecting the where condition of the most expensive queries may help. I'm told that the easiest way to start on this is with pgAdmin, which raises a very basic question - what's the username and password I can connect to the gis database with? I presume it's listed somewhere in the setup scripts, but I've not had the chance to go through them. Cheers, Joseph On 19 October 2011 14:57, Kai Krueger <kakrue...@gmail.com> wrote: > Hi, > > On 10/19/2011 05:01 AM, Joseph Reeves wrote: >> Hi Kai, >> >>> The default Ram cache size is 800Mb. You can increase it with the -C >>> parameter of osm2pgsql. But given that your netbook probably doesn't >>> have that much ram, I am not sure increasing that option is such a good >>> idea. >> >> -C only has an effect in slim mode, unless I'm wrong? > > That also changed with the commit a few days ago. As again for smaller > extracts, who have a sparse distribution of node IDs, the cache was very > inefficient, I reused the improved node cache from the --slim mode. > Perhaps unfortunately, that also incorporated the limit and effect of > the -C parameter. On 64 bit machines at least, you can simply set the -C > parameter very high as it only reserves virtual memory. Only the amount > actually used will result in physical ram allocation. It is perhaps a > little bit more problematic on 32 bit Operating Systems though, as > virtual memory is also fairly limited. > > Setting a very high -C parameter and --cache-strategy chunked basically > gets you back to the old behavior though. > > As I was hoping >> to append data to my db, I'm running osm2pgsql in full fat mode. >> Thanks for the --cache-strategy tip - I got the import working with >> the sparse option. It seems to be working surprisingly quickly in >> fact. > > Fat mode definitely has its advantage in speed, perhaps especially on > slow disk systems like a netbook. This is perhaps why it was a bit > unfortunate that non-slim mode previously was (and for ways and > relations it still is) very wasteful with ram for extracts. > >> >> Of course, as you said, getting the data into the db is one thing, but >> actually using it is another matter. The netbook is now rendering >> tiles for a large strip across Europe - this often takes a while to >> get tiles created, but I think it should work for what I need it for. >> If the load gets too high I can always empty the db and add extracts >> as I need them. Before that, however, I think I'll try and find any >> database optimisations that might exist. > > Playing with indexes, especially partial indexes on the ways columns, > reflecting the where condition of the most expensive queries may help. > > If you do come up with good optimizations, it may be good to collect > them somewhere to build up a knowledge base for optimization tips. > > Kai > >> >> Thanks again for all this, >> >> Cheers, Joseph >> >> >> >> On 18 October 2011 18:34, Kai Krueger <kakrue...@gmail.com> wrote: >>> On 10/18/2011 10:48 AM, Joseph Reeves wrote: >>>>> It is possible that one could catch the errors in slim mode and then only >>>>> do the expensive diff processing for those node / ways that are >>>>> >duplicate in the extracts. >>>> >>>> Interesting, although I think this is beyond the limits of my OSM >>>> skills. >>> >>> Yes, that comment was more directed at my self that I should look into >>> that, or if any other dev of osm2pgsql gets to it first... ;-) >>> >>> Unfortunately Austria seems to be beyond the capabilities of >>>> my netbook; an import without --slim gives the error: >>>> >>>> Node cache size is too small to fit all nodes. Please increase cache size >>>> >>>> Presumably a slim import would help, but this would then fail because >>>> of overlapping ways... I can't Google up anyone suffering that error >>>> message before; I guess nobody else is trying to get a number of >>>> European countries into a db on their netbook... >>> >>> You will likely not find that error message on google yet, as if I am >>> not mistaken, I only commited that error message two days ago. >>> >>> The default Ram cache size is 800Mb. You can increase it with the -C >>> parameter of osm2pgsql. But given that your netbook probably doesn't >>> have that much ram, I am not sure increasing that option is such a good >>> idea. >>> >>> Given that you can't fit Austria into 800Mb, I suspect you are using the >>> 32bit version of osm2pgsql. It defaults to the old (for extracts >>> inefficient) cache allocation strategy. You can try using the >>> "--cache-strategy" option and set it to either optimized or sparse. That >>> should be more efficient for extracts than the default method. In the >>> optimized option, you might run out of virtual memory on 32 bit though. >>> >>> But you might have troubles with Austria on a netbook anyway, as it >>> might still be too resource intense. >>> >>> Kai >>> >>>> >>>> Thanks again, >>>> >>>> Joseph >>>> >>>> >>>> >>>> >>>> On 18 October 2011 16:59, Kai Krueger <kakrue...@gmail.com> wrote: >>>>> On 10/18/11 9:31 AM, Joseph Reeves wrote: >>>>>> >>>>>> Hi Kai, >>>>>> >>>>>>> The pre-rendered tiles are stored in /var/lib/mod_tile/default. You can >>>>>>> simply delete those files and they will automatically get rerendered the >>>>>>> next time you view them. >>>>>> >>>>>> Great, thanks, that's working great. >>>>>> >>>>>>> I have seen that you appear to need to restart renderd (sudo >>>>>>> /etc/init.d/renderd restart) after a new import, as it otherwise appears >>>>>>> to >>>>>>> still use old data (It is kind of odd, so I might have the wrong >>>>>>> impression >>>>>>> here). >>>>>> >>>>>> Sorry, I should've said in my previous email - I was assuming this to >>>>>> be the case. Out of interest, is there any log output from renderd? >>>>> >>>>> sudo tail -f /var/log/syslog >>>>> >>>>> should give you some output of what renderd is doing, including which >>>>> tiles >>>>> it is rendering and when it has completed them >>>>>> >>>>>> I'm running this on a netbook so tiles take a while to be produced; >>>>> >>>>> I'd imagine a netbook might struggle a little with rendering tiles on the >>>>> fly... ;-) But eventually they should be done. >>>>>> >>>>>> it >>>>>> would be interesting to see some logs so that I know it's all working >>>>>> as I expect. >>>>>> >>>>>>> However, what you are trying to do is as far as I know not supported by >>>>>>> osm2pgsql. Although it seems to be a much requested feature, I don't >>>>>>> think >>>>>>> osm2pgsql currently handles importing of multiple extracts. The --append >>>>>>> option doesn't really do what you would think it does. >>>>>> >>>>>> I tried the append flag and got an error about an already existing way >>>>>> - it would be good if osm2pgsql would simply ignore any ways that >>>>>> already exist in the database. I re-ran osm2pgsql without the --slim >>>>>> option, however, and the import was successful. I currently have >>>>>> Bulgaria and Romania working on my netbook :) >>>>> >>>>> Interesting that the non-slim mode works with appending multiple extracts. >>>>> >>>>> It is possible that one could catch the errors in slim mode and then only >>>>> do >>>>> the expensive diff processing for those node / ways that are duplicate in >>>>> the extracts. >>>>>> >>>>>> Am trying to re-import Turkey now, then onwards with bits of Europe! >>>>>> If it all works out do you mind if I do a bit of wiki fiddling on your >>>>>> instructions? >>>>> >>>>> No go ahead and improve the instructions >>>>>> >>>>>> Thanks again, >>>>>> >>>>>> Joseph >>>>>> >>>>> >>>>> >>> >>> > > _______________________________________________ talk mailing list talk@openstreetmap.org http://lists.openstreetmap.org/listinfo/talk
