Re: [mkgmap-dev] [PATCH v10] make maps in parallel
On Tue, May 19, 2009 at 10:33:01AM +0100, Mark Burton wrote: Further testing shows that you can get different output on subsequent runs on the same input (ignoring the time stamps). So far, all of the differences appear to be caused by data being output in a different order rather than the data itself being different. So, I believe that the resulting maps are still good. You should understand that the ordering of some of the map's data structures is not important as long as all the cross-references match up. Has anyone written a lint program for *.img files that would validate all the cross-references? Or a program to pretty-print the data structures in sorted format? The sorted pretty-print should be identical across runs. Marko ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Re: [mkgmap-dev] [PATCH v10] make maps in parallel
Hi Marko, Has anyone written a lint program for *.img files that would validate all the cross-references? Or a program to pretty-print the data structures in sorted format? The sorted pretty-print should be identical across runs. There are various programs around for printing out the stuff in IMG files. I have a hacked version of imgdecode that is somewhat more useful than the original and there is always the display code written by Steve. I am simply using cmp -l to do a byte-for-byte comparison of the files generated by various runs of mkgmap and then using imgdecode and display to locate where the differences reside. Cheers, Mark ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Re: [mkgmap-dev] [PATCH v10] make maps in parallel
Yes, get back to work, you slacker! Many thanks for the feedback. That's a good speedup for a 2 core machine. When you have the time, please also try the quick-distance patch. real4m13.508s user4m39.725s sys 0m16.205s Time duration: 254 secs. real3m25.462s user5m21.408s sys 0m15.697s Time duration: 205 secs. The first is without --max-jobs and the second is with and both are a considerable improvement than before the quick-distance patch Cheers Paul ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Re: [mkgmap-dev] [PATCH v10] make maps in parallel
Hi Paul, Thanks for the figures. real4m13.508s user4m39.725s sys 0m16.205s Time duration: 254 secs. real3m25.462s user5m21.408s sys 0m15.697s Time duration: 205 secs. The first is without --max-jobs and the second is with and both are a considerable improvement than before the quick-distance patch Yes, it's a good time saver and when it's combined with the multicore patch I'm seeing around 350% speedup! Cheers, Mark ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Re: [mkgmap-dev] [PATCH v1] quick distance calculation
On Tue, May 19, 2009 at 12:57 AM, Mark Burton ma...@ordern.com wrote: With all the postings to the list in the last 24H or so, it's easy to miss stuff - so just in case you missed this patch, please give it a go as I believe it provides a useful performance boost. I've applied and tested this patch: at first glance, it seems OK. I didn't notice breakage or similar problems. I also did not do any performance tests, so I cannot say if the compilation time has improved significantly. I'll do more extensive tests later. Cheers. ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Re: [mkgmap-dev] [PATCH v10] make maps in parallel
Hi Clinton, I have also tested this patch on my Windows machine: the error which I previously reported regarding missing files no longer occurs. Sorry for not responding earlier, but I was away. No problem, glad it has fixed the issue. A superficial examination of the map revealed no noticeable differences or problems compared to maps compiled without the parallel code. I'll also test later on with Mac OS. OK. Thanks! The patch looks good so far. Excellent, we now have several reports that the missing file problem has been fixed (methinks that there is either a bug in the Java Futures stuff or it works somewhat differently than the documentation suggests). Cheers, Mark ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Re: [mkgmap-dev] [PATCH v10] make maps in parallel
Mark Burton wrote: Excellent, we now have several reports that the missing file problem has been fixed several++ Like everybody else: maps seem to be fine, with superficial testing... Regards, Martin ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Re: [mkgmap-dev] [PATCH v1] quick distance calculation
Hi Robert, Many thanks for the feedback. how much difference is tolerable...? enabling your diagnostic code, i've observed differences typically in the range of .25% to .4%, with some exceptions like... quickDistance() = 1.4536911305450404 slowDistance() = 1.446011378093334 (0.5310990333860841% difference) quickDistance() = 1.4535945914221295 slowDistance() = 1.446011378093334 (0.5244227980276771% difference) ... and a number of lines where the relative difference doesn't really compute: quickDistance() = 0.0 slowDistance() = 0.09493529796600342 (100.0% difference) Well, that's a good question. As distance() mostly gets called to determine which of a bunch of points is nearest, it probably doesn't matter at all that the result is slightly wrong. I don't believe that the quality of the values returned by distance() will affect the accuracy of the map. Perhaps it will make some difference to the routing but, at this time, I am not convinced that distance() needs to be super^2 accurate. If anyone knows otherwise, please say. b - performance increase/decrease i compiled my village four times with and without the patch: with: make basemap1 56.00s user 0.99s system 143% cpu 39.621 total without: make basemap1 65.33s user 1.02s system 132% cpu 49.988 total without: make basemap1 70.27s user 1.19s system 137% cpu 52.111 total with: make basemap1 58.36s user 1.07s system 144% cpu 41.124 total with: make basemap1 55.84s user 1.20s system 134% cpu 42.394 total without: make basemap1 66.21s user 0.90s system 131% cpu 51.153 total with: make basemap1 51.98s user 1.14s system 142% cpu 37.339 total without: make basemap1 69.27s user 1.03s system 137% cpu 51.306 total - with avg: 55.54s user - without avg: 67.77s user - the compile with the patch takes only some 81% of the time. (in case the type of CPU matters, /proc/cpuinfo on this laptop says AMD Turion(tm) 64 X2 Mobile Technology TL-52.) Well, that's not as good a speedup as I have been seeing. Without using the parallelism patch, i.e. just using 1 core, I get around x2 performance on a very quick machine when using the quick-distance patch. Anyway, thanks again for taking the time to give it a go. Cheers, Mark ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Re: [mkgmap-dev] [PATCH] - Code for Highway Symbols
On Mon, May 18, 2009 at 07:47:23AM +0200, Thilo Hannemann wrote: Attached is a patch by Toby Speight that allows to insert the code for highway symbols. I'm using it since a while with no adverse effects. Is there any chance that it gets integrated into trunk? Or if there is a problem with that code, what should be changed in order to make if fit for integration? Thanks for reminding me about this, I'll look back over the discussion and apply it soon. Regards, ..Steve ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Re: [mkgmap-dev] [PATCH v1] quick distance calculation
On Tue, 19 May 2009, Mark Burton wrote: Well, that's a good question. As distance() mostly gets called to determine which of a bunch of points is nearest, it probably doesn't matter at all that the result is slightly wrong. Really? In that case you could as well change the metric from Euclidean to something more simple like the Manhattan metric: dist = abs(lat2-lat1) + costab[lat1] * abs(lon2-lon1) where costab is a table lookup for cos(). This will still find a point that is close, but it is not guaranteed to be the same as with the Euclidean metric. Would that matter? -Wolfgang ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
