Re: [mkgmap-dev] max-jobs patch
Hi tested on my linux box, looks good Heap is 4.4 GB with 8 threads. With the unpatched version i'm using -Xmx8G for the heap Bernd Am Donnerstag, 8. Februar 2018, 15:34:23 CET schrieb Gerd Petermann: > binary is here: > http://files.mkgmap.org.uk/download/415/mkgmap.jar > > For those users who already used --max-jobs with or without a paramter > before there should be no difference. signature.asc Description: This is a digitally signed message part. ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Re: [mkgmap-dev] max-jobs patch
Hi Bernd, okay, I think that confirms my estimate of ~500M per thread. See also my previous answer to Henning. Gerd Von: mkgmap-dev im Auftrag von Bernd Weigelt Gesendet: Samstag, 10. Februar 2018 09:43 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] max-jobs patch Hi tested on my linux box, looks good Heap is 4.4 GB with 8 threads. With the unpatched version i'm using -Xmx8G for the heap Bernd Am Donnerstag, 8. Februar 2018, 15:34:23 CET schrieb Gerd Petermann: > binary is here: > http://files.mkgmap.org.uk/download/415/mkgmap.jar > > For those users who already used --max-jobs with or without a paramter > before there should be no difference. ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
[mkgmap-dev] WG: AW: max-jobs patch
Sorry, answered to Henning instead of the list, see below... Gerd Von: Gerd Petermann Gesendet: Samstag, 10. Februar 2018 09:52 An: Henning Scholland Betreff: AW: AW: [mkgmap-dev] max-jobs patch Hi Henning, splitter uses multiple threads by default, but up to now only during the last phase that writes the tiles it can use more than 2. Not a big problem, most of the work is I/O bound. The workload in splitter is completely different to mkgmap. Peak memory depends on the --keep-complete option and the max-areas option, not so much on the number of parallel threads. In some cases I see slightly better throughput when I use --max-threads=6 on my 4 core machine, but only a few percent. The last time when I measured it I was quite happy with the defaults. Gerd Von: Henning Scholland Gesendet: Samstag, 10. Februar 2018 08:46 An: Gerd Petermann; Development list for mkgmap Betreff: Re: AW: [mkgmap-dev] max-jobs patch Hi Gerd, maybe splitter can be modified similar. Henning ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Re: [mkgmap-dev] max-jobs patch
Hi Gerd the unpatched version uses up to 8.7 GB with the options for mkgmap I'm using maxnodes=512000 to create an area.list and use this file for every build. At the moment i see no problem with the patched version Bernd Am Samstag, 10. Februar 2018, 09:56:41 CET schrieb Gerd Petermann: > Hi Bernd, > > okay, I think that confirms my estimate of ~500M per thread. > See also my previous answer to Henning. > > Gerd > > signature.asc Description: This is a digitally signed message part. ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Re: [mkgmap-dev] max-jobs patch
Hi Bernd, not sure what you want to express here. When you use --max-jobs or max-jobs=8 with the unpatched version you should see exactly the same numbers on an 8 core machine, the patch only changes the default for max-jobs. Gerd Von: mkgmap-dev im Auftrag von Bernd Weigelt Gesendet: Samstag, 10. Februar 2018 10:19:09 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] max-jobs patch Hi Gerd the unpatched version uses up to 8.7 GB with the options for mkgmap I'm using maxnodes=512000 to create an area.list and use this file for every build. At the moment i see no problem with the patched version Bernd Am Samstag, 10. Februar 2018, 09:56:41 CET schrieb Gerd Petermann: > Hi Bernd, > > okay, I think that confirms my estimate of ~500M per thread. > See also my previous answer to Henning. > > Gerd > > ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Re: [mkgmap-dev] max-jobs patch
Hi Gerd, I have put together the attached patch that should result in a reasonable value for max-jobs if it is defaulted. This version attempts to calculate a suitable value by processing one task, seeing how much memory it used, then dividing the available memory by the memory used to determine the value before running the remaining tasks. I have tried a variety of values of -Xmx on my PC, and it seems to work for both a county and the whole of the UK. I haven't tried it on a continent, as I don't have suitable data. I can't guarantee that every combination of memory and CPU will work, but if it does fail, you will still get a sensible error message telling you what to change. At the end of the run, if insufficient memory is allocated to the Java heap to allow all the CPU cores to be used, it will suggest a value for -Xmx. For my maps, it seems to suggest about 400Mb per thread. My build of the UK without specifying --max-jobs has reduced from about 3 hours to 45 minutes. I have also made a few further minor improvements to the documentation to make the form more consistent, and changed the logged execution time from milliseconds to a more useful hours, minutes and seconds (or milliseconds when less than 1 second). Regards, Mike -Original Message- From: Gerd Petermann [mailto:gpetermann_muenc...@hotmail.com] Sent: 07 February 2018 19:06 To: Development list for mkgmap Subject: Re: [mkgmap-dev] max-jobs patch Hi Mike, I did not yet try it. My understanding is that the Garbage Collection will try hard to avoid an out of heap exception. In other words: A user might see better throughput with fewer parallel jobs. What are your experiences? Did you try with large maps, say > 40 large tiles ? Gerd Von: mkgmap-dev im Auftrag von Mike Baggaley Gesendet: Mittwoch, 7. Februar 2018 01:58 An: 'Development list for mkgmap' Betreff: Re: [mkgmap-dev] max-jobs patch Hi Gerd, please find attached v2 of the max-jobs patch. This version additionally catches out of heap memory exceptions and displays a message suggesting either the use of the Java -Xmx option to increase the available heap memory or the mkgmap --max-jobs option with a smaller value to reduce the memory requirement (the latter is suggested only if using more than one thread). How does that seem? Regards, Mike -Original Message- From: Gerd Petermann [mailto:gpetermann_muenc...@hotmail.com] Sent: 05 February 2018 07:28 To: Mike Baggaley ; 'Development list for mkgmap' Subject: Re: [mkgmap-dev] max-jobs patch Hi Mike, thought about this again. Maybe this change is too simple. With multiple jobs one also needs more heap (-Xmx JRE option). If you create rather large tiles with splitter (max-nodes=160) you need 0.5 - 1 GB for each job. Not sure what happens when a user creates a map with 10 tiles on an 8-core machine without any -Xmx option. I fear this will result in OutOfMemory exception, so better check the available heap as well. Gerd Von: mkgmap-dev im Auftrag von Mike Baggaley Gesendet: Sonntag, 4. Februar 2018 15:14 An: 'Development list for mkgmap' Betreff: [mkgmap-dev] max-jobs patch Hi Gerd, Please find attached a patch that amends the default behaviour if the --max-jobs option is not specified, to using a value equal to the number of CPU cores, as suggested in a previous post. The documentation is also amended to reflect the change. This halves the execution time of mkgmap for building a map of Staffordshire on my 8-core PC when --max-jobs is not specified (I didn't know about this option previously and was unaware the performance could have been improved). Cheers, Mike maxjobs-v3.patch Description: Binary data ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Re: [mkgmap-dev] max-jobs patch
Hi Gerd, these are my runtimes of several compilations of my Turkey map, splitted with 1.500.000 nodes, compiled witout index, gmapi and gmappsupp, but including DEM and routing etc. RAM in GB Threads Time1 in ms Time2 in ms 12 8 328106 323605 8 8 323135 341242 4 8 378460 408714 4 7 364123 361501 4 6 349048 343866 4 4 371786 366581 4 1 845778 797376 2 4 527267 509698 2 1 804117 853805 It seems a lack of RAM slows more down if there are two many jobs. For 4GB Ram, 6 threads are faster than 8 threads, even using only 4 threads is faster than using 8. Henning On 10.02.2018 16:41, Gerd Petermann wrote: On 10.02.2018 16:41, Gerd Petermann wrote: > Hi Henning, > > thanks for the numbers. I assume that your JRE sets xmx=4G because you > probably have 16G. > My machine has 4 cores and 8G and the JRE sets xmx=2G when I don't specify a > value. > When you compile a large number of tiles peak memory is reached when building > the index. > > Maybe you can do some more tests with a smaller set of tiles? > I'd like to know if the run time changes much when you compile e.g. 32 tiles > with > a) -Xmx8GB and --max-jobs=8 > b) -Xmx4GB and --max-jobs=8 > c) -Xmx4GB and --max-jobs=7 > d) -Xmx4GB and --max-jobs=6 > > I suggest to do each test at least 2 times and start with one unmeasured test > to reduce > IO caching effects. You should also not use --index, --gmapi or --gmapsupp > for these tests. > > Gerd > > > > Von: Henning Scholland > Gesendet: Samstag, 10. Februar 2018 07:23 > An: Gerd Petermann; Development list for mkgmap > Betreff: Re: AW: [mkgmap-dev] max-jobs patch > > Hi Gerd, > Hi Mike, > > I gave it a try on all my maps (splitted with max-nodes=1.600.000). > Removed xmx and max jobs. mkgmap used all 8 logical cores and about 4GB > of memory. So it seems, Gerd was correct with the 500mb per job. So I > would suggest to limit max-jobs with 32bit java to 3 jobs and to > RAM/500mb on 64bit java. The used jobs should be minimum of both CPU and > RAM. > > Regarding to higher amount of RAM I was mentioning before, I was calling > mkgmap with xmx=12G, maybe java is not freeing the heap in that case. It > was during dem-branch. Maybe during that time mkgmap wasn't also that > efficient. > > Henning > ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Re: [mkgmap-dev] max-jobs patch
Hi Mike, I've compiled r4107 with your patch: http://files.mkgmap.org.uk/download/418/mkgmap.jar I did not yet try it because I am still fighting with the crash in MapSource. I am not sure if the usage of classes like com.sun.management.OperatingSystemMXBean are supported with openjdk? Will this also work on a Mac? Gerd -- Sent from: http://gis.19327.n8.nabble.com/Mkgmap-Development-f5324443.html ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev