Hi Gerd Certainly no cache. Maybe reduce the chunk size, but this might increase copying.
It could be improved by doing a linear chunk split/sort then a multi -way merge. This would avoid lots of copying assuming the following: Using the original array to store sorted chunks demands that another array of the same full size is needed for the final merge. If each sorted key chunk is converted to a object chunk and these merged, then although the same total size is needed, it is made of number of smaller arrays. The most space efficient solution might be have Mdr11 "implements Comparable" and generate pairs of sortkeys on the fly and let the java sort take care of all the details. The other use of LargeListSorter is Mdr7. I get a higher hit-rate (~50%) for the first/partialSorter (1046096 allstreets). However, for the repeated fullNameSorter on the partial results, most of the lists are just 1 long, with very few more than 10. I guess this depends on --road-name-config/split-name and use of shields etc, but LargeListSorted seems overkill. Ticker On Tue, 2021-05-11 at 19:15 +0000, Gerd Petermann wrote: > Hi Ticker, > > I think the cache is not meant to improve run time, it is used to > deduplicate and thus reduce memory. Maybe it would be better to use a > smaller chunk size and no cache. > No idea why I didn't use > merged = new ArrayList<>(len); > > Gerd > > ________________________________________ > Von: mkgmap-dev <mkgmap-dev-boun...@lists.mkgmap.org.uk> im Auftrag > von Ticker Berkin <rwb-mkg...@jagit.co.uk> > Gesendet: Dienstag, 11. Mai 2021 19:19 > An: Development list for mkgmap > Betreff: Re: [mkgmap-dev] MDR building out-of-memory > > Hi Gerd > > I've looked at this, and if Mdr5 space becomes a problem again, I'll > consider converting to it. > > A couple of comments: > > My map had 2909735 poi so the sort chunk size was ~727433. The cache > sizes after each chunk were 563095, 603595, 597239 & 605718. Is the > cache worth-while for this low hit-rate? Just running the Gmapsupp > combiner on existing tiles (without --route, so no streets), I got a > run time of 1 min 44 secs with the cache and 1 min 30 without! > However > most of this time is copying the tiles into gmapsupp.img so not an > accurate statistic. > > You could pre-allocate List<> "merged" with the correct size. > > Ticker > > On Tue, 2021-05-11 at 14:40 +0000, Gerd Petermann wrote: > > Hi Ticker, > > > > I've committed the patch as is. I've not seen big changes in > > performance, but I've used a different (already existing) set of > > files which was created with my own style. For me, > > Mdr11.preWriteImpl() is the most problematic part reg. OOM errors. > > > > Maybe look at the code which uses LargeListSorter. > > > > Gerd > > > > ________________________________________ > > Von: mkgmap-dev <mkgmap-dev-boun...@lists.mkgmap.org.uk> im Auftrag > > von Ticker Berkin <rwb-mkg...@jagit.co.uk> > > Gesendet: Dienstag, 11. Mai 2021 13:27 > > An: Development list for mkgmap > > Betreff: Re: [mkgmap-dev] MDR building out-of-memory > > > > Hi Gerd > > > > Here is updated version of patch. > > > > Changes from last: > > > > Uses your cache code for region and country (in 2 places). For > > British > > Isles, there are 190 regions and 7 countries, so I don't think the > > extra memory will be a problem and there should be some performance > > benefit. > > > > Delays allocating cities until it can use sortKeys.size() for > > initial > > allocation. For above map this is 0.07% too big, so I don't think > > trimToSize() is worthwhile. > > > > Shares the Sort object between the 4 methods. > > > > Ticker > > _______________________________________________ > > mkgmap-dev mailing list > > mkgmap-dev@lists.mkgmap.org.uk > > https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > _______________________________________________ > mkgmap-dev mailing list > mkgmap-dev@lists.mkgmap.org.uk > https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > _______________________________________________ > mkgmap-dev mailing list > mkgmap-dev@lists.mkgmap.org.uk > https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev _______________________________________________ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev