Re: [mkgmap-dev] New, faster splitter

2010-06-10 Thread Ralf Kleineisel
On 06/10/2010 01:21 AM, Chris Miller wrote: pass required. Additionally, if you do specify a --cache parameter but only one pass ends up being required, no cache will be generated anyway since it would only slow things down. I don't think this is a good idea. I often split the Europe

Re: [mkgmap-dev] New, faster splitter

2010-06-10 Thread Chris Miller
Hello Ralf, I don't think this is a good idea. I often split the Europe excerpt, and only after running mkgmap I find out that one of the tiles is too big so I have to split again with a modified areas.list. Creating a cache during the first pass helps a lot there. Hmmm I was wondering

Re: [mkgmap-dev] New, faster splitter

2010-06-10 Thread Marko Mäkelä
Hi Chris, thanks for the quick response. On Wed, Jun 09, 2010 at 11:21:21PM +, Chris Miller wrote: I've just checked in r112 which should address this. You no longer need to specify a --cache parameter with stdin and --split-file if there's only one pass required. I tried that, but

Re: [mkgmap-dev] New, faster splitter

2010-06-10 Thread Chris Miller
Hi Marko, I tried that, but splitter wrote almost empty *.osm.gz files (150ish bytes, just containing the osm ...bounds ...//osm) and terminated before consuming all input: Hmm, I encountered that during my testing but thought I'd fixed it. Guess I must have missed something, I'll take a

Re: [mkgmap-dev] New, faster splitter

2010-06-10 Thread Chris Miller
Hello Marko, I tried that, but splitter wrote almost empty *.osm.gz files (150ish bytes, just containing the osm ...bounds ...//osm) and terminated before consuming all input: This should now be fixed in r113. Chris ___ mkgmap-dev mailing list

Re: [mkgmap-dev] New, faster splitter

2010-06-09 Thread Marko Mäkelä
Hello Chris, I've just checked in r111 which enables osm data to be read from stdin. For it to work, the following three conditions must be met: 1) there must be no osm files specified as parameters 2) a valid --cache parameter must be supplied 3) there must not be an existing cache from

Re: [mkgmap-dev] New, faster splitter

2010-06-09 Thread Chris Miller
Hello Marko, Thanks, I am trying it out now. This warning seems redundant for this use case: * WARNING: No valid existing cache found but caching was requested. * Ahh... good catch. I hadn't considered the case where you supply an areas.list file. In that situation, if (and only if) a

Re: [mkgmap-dev] New, faster splitter

2010-06-08 Thread Lambertus
Just to say: thanks WanMil and Chris! (now I need to upgrade to a quadcore to benefit from all these goodies...) ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

Re: [mkgmap-dev] New, faster splitter

2010-06-08 Thread Chris Miller
Hello Marko, I would appreciate that, because I have to do some preprocessing (moving end nodes of the coastline to the tile boundary, deleting auto-generated bad multipolygons) after decompressing the osm file. The sed or perl filter could sit between bzip2 and splitter, and all three

Re: [mkgmap-dev] New, faster splitter

2010-06-07 Thread Chris Miller
WanMil's patch has inspired me to add some additional threading to the splitter. In r110 the loading of osm files is now performed in a separate thread from the parsing. This provides an especially big benefit when reading in a .bz2 compressed osm file since the decompression happens in

Re: [mkgmap-dev] New, faster splitter

2010-06-07 Thread Felix Hartmann
bz2 compressed file, no cache: No threading: 238s r109 threading: 193s r110 threading: 136s bz2 compressed file, cache: No threading: 170s r109 threading: 123s r110 threading: 92s Uncompressed osm file, no cache: No threading: 132s r109 threading: 80s r110 threading: 76s

Re: [mkgmap-dev] New, faster splitter

2010-06-05 Thread Felix Hartmann
On 05.06.2010 15:32, Chris Miller wrote: WanMil sent through a very nice patch for the splitter that writes out the results using multiple threads, giving a pretty big speedup during the 2nd half of the split if you have a multicore PC. For me, the time it takes to split the UK is less than

Re: [mkgmap-dev] New, faster splitter

2010-06-05 Thread WanMil
On 05.06.2010 15:32, Chris Miller wrote: WanMil sent through a very nice patch for the splitter that writes out the results using multiple threads, giving a pretty big speedup during the 2nd half of the split if you have a multicore PC. For me, the time it takes to split the UK is less than

Re: [mkgmap-dev] New, faster splitter

2010-06-05 Thread Chris Miller
Hello Felix, What about max memory needs. Is there a big increase, or doesn't it matter, because the first phase of the split is the important part? Is this also working without --cache, or only when using --cache?? It is the second stage that requires the most memory, however with this patch