Re: [mkgmap-dev] New, faster splitter

2010-06-10 Thread Ralf Kleineisel
On 06/10/2010 01:21 AM, Chris Miller wrote: > pass required. Additionally, if you do specify a --cache parameter but only > one pass ends up being required, no cache will be generated anyway since > it would only slow things down. I don't think this is a good idea. I often split the Europe exce

Re: [mkgmap-dev] New, faster splitter

2010-06-10 Thread Chris Miller
Hello Ralf, > I don't think this is a good idea. I often split the Europe excerpt, > and only after running mkgmap I find out that one of the tiles is too > big so I have to split again with a modified areas.list. Creating a > cache during the first pass helps a lot there. Hmmm I was wondering wh

Re: [mkgmap-dev] New, faster splitter

2010-06-10 Thread Marko Mäkelä
Hi Chris, thanks for the quick response. On Wed, Jun 09, 2010 at 11:21:21PM +, Chris Miller wrote: >I've just checked in r112 which should address this. You no longer need >to specify a --cache parameter with stdin and --split-file if there's >only one pass required. I tried that, but spli

Re: [mkgmap-dev] New, faster splitter

2010-06-10 Thread Chris Miller
Hi Marko, > I tried that, but splitter wrote almost empty *.osm.gz files (150ish > bytes, just containing the ) and > terminated before consuming all input: Hmm, I encountered that during my testing but thought I'd fixed it. Guess I must have missed something, I'll take a look tonight. Sorry for

Re: [mkgmap-dev] Increasing the performance of whole-planet splits.

2010-06-10 Thread Scott Crosby
> It'll be a few days before I have time to look at this in much detail and > give you my thoughts but from what I can see it sounds promising. We > actually > use protobufs at my job for shunting around vast amounts of data so I'm > pretty > familiar with the pros and cons of them. One concern is

[mkgmap-dev] Increasing the performance of whole-planet splits.

2010-06-10 Thread Scott Crosby
Hello! I would like to submit the first of a set of performance optimizations to the splitter that improve the performance and reduce memory usage. The end result is that a core 2 duo can split the entire planet in 3.5 hours with -Xmx1700m and without using any temporary space. (67 minutes on a 8gb

[mkgmap-dev] Getting rid of the 'Node in too many areas' warning.

2010-06-10 Thread Scott Crosby
This patch to the splitter that makes this warning MUCH more rare. With this patch, I can split the entire planet into 17219 areas with max-nodes=5 resolution=14 without a single 'too many areas' warning among the the first 25 of 68 passes. Without this patch, a max-nodes=50 split has dozen

Re: [mkgmap-dev] New, faster splitter

2010-06-10 Thread Chris Miller
Hello Marko, > I tried that, but splitter wrote almost empty *.osm.gz files (150ish > bytes, just containing the ) and > terminated before consuming all input: This should now be fixed in r113. Chris ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgm

Re: [mkgmap-dev] Increasing the performance of whole-planet splits.

2010-06-10 Thread Chris Miller
Hello Scott, > Hello! I would like to submit the first of a set of performance > optimizations to the > splitter that improve the performance and reduce memory usage. Fantastic! > Perhaps the biggest change is that I've made the splitter accept my > binary OSM format. It'll be a few days before

Re: [mkgmap-dev] Increasing the performance of whole-planet splits.

2010-06-10 Thread Chris Miller
Hello Scott, > I envisioned it as an alternate format that could exist in parallel > with the osm, but also be smaller and faster. I still need to clean up > the osmosis patches and turn them into a plugin. OK sounds like we're both thinking along the same lines then. Note that the current cache

Re: [mkgmap-dev] Increasing the performance of whole-planet splits.

2010-06-10 Thread Scott Crosby
> > The reason for this was the costs of initializing were showing up in > > the profile. > > Do you mean the cost of initializing the Element instances? If so then yes, > that's something that I'd caught previously in profiling too, hence why > they > were being reused and reset (prior to r109 tha