Re: [mkgmap-dev] Splitter Error
Hi, I did not find a simple solution regarding a change in mkgmap. But maybe a different approach: What about using two layers like we do with contour lines? I mean: create a transparent map with the polygons for seamark:type=fairway and a normal map with all other elements and give them different draw-orders. Would that work for you? Gerd RheinSkipper wrote The use of the prepro changes something in the output file, but that seems not to change the draw order. Reversing the order of the lines in wayorder also changes something in the img file, but not the draw order. Did you run the prepro after splitter on each tile? I use a script with a for loop for that. Splitter output must be set to xml. I attach some test data. With this the prepro works perfectly. It seems that the Garmin algo choses the larger polygon if it finds two overlapping polygons in one sub division. Both riverbank and fairway are divided into segments. So the fairway polygons are not always smaller than the riverbank. We were already thinking about a prepro that combines all the riverbank segments to one big polygon for each river. If fairway is always completely inside riverbank, one could use it as an inner multipolygon and cut out holes in the riverbank. But this was considered to be too complicated. ___ mkgmap-dev mailing list mkgmap-dev@.org http://lists.mkgmap.org.uk/mailman/listinfo/mkgmap-dev test3.osm (2K) lt;http://gis.19327.n5.nabble.com/attachment/5753112/0/test3.osmgt; -- View this message in context: http://gis.19327.n5.nabble.com/Splitter-Error-tp5752745p5753952.html Sent from the Mkgmap Development mailing list archive at Nabble.com. ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://lists.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Re: [mkgmap-dev] Splitter Error
I already tried this method a while ago. As marine devices support only a single gmapsupp.img container, I had to combine both layers into one file at the end. The result looked good on my marine device, but on handheld models the upper layer did not show up at all. I feared there will be more unexpected or random behavior again, and gave up this approach. But maybe I did something wrong. I can try it again after my Easter holiday and maybe with your help we can find a method that works predictably. -Ursprüngliche Nachricht- Von: mkgmap-dev-boun...@lists.mkgmap.org.uk [mailto:mkgmap-dev- boun...@lists.mkgmap.org.uk] Im Auftrag von GerdP Gesendet: Mittwoch, 20. März 2013 08:22 An: mkgmap-dev@lists.mkgmap.org.uk Betreff: Re: [mkgmap-dev] Splitter Error Hi, I did not find a simple solution regarding a change in mkgmap. But maybe a different approach: What about using two layers like we do with contour lines? I mean: create a transparent map with the polygons for seamark:type=fairway and a normal map with all other elements and give them different draw-orders. Would that work for you? Gerd RheinSkipper wrote The use of the prepro changes something in the output file, but that seems not to change the draw order. Reversing the order of the lines in wayorder also changes something in the img file, but not the draw order. Did you run the prepro after splitter on each tile? I use a script with a for loop for that. Splitter output must be set to xml. I attach some test data. With this the prepro works perfectly. It seems that the Garmin algo choses the larger polygon if it finds two overlapping polygons in one sub division. Both riverbank and fairway are divided into segments. So the fairway polygons are not always smaller than the riverbank. We were already thinking about a prepro that combines all the riverbank segments to one big polygon for each river. If fairway is always completely inside riverbank, one could use it as an inner multipolygon and cut out holes in the riverbank. But this was considered to be too complicated. ___ mkgmap-dev mailing list mkgmap-dev@.org http://lists.mkgmap.org.uk/mailman/listinfo/mkgmap-dev test3.osm (2K) lt;http://gis.19327.n5.nabble.com/attachment/5753112/0/test3.osmgt; -- View this message in context: http://gis.19327.n5.nabble.com/Splitter-Error- tp5752745p5753952.html Sent from the Mkgmap Development mailing list archive at Nabble.com. ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://lists.mkgmap.org.uk/mailman/listinfo/mkgmap-dev ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://lists.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Re: [mkgmap-dev] Splitter Error
Hi, I think I have enough info now to continue the work. I use a variant of the default style for testing, the last for lines look like this: #include 'inc/water_polygons'; include 'inc/landuse_polygons'; seamark:type=fairway [0x10302 resolution 20] waterway=riverbank [0x10303 resolution 20] My test file contains the bbox bounds minlat=49.7021484 minlon=7.8222656 maxlat=50.1855468 maxlon=8.4375/ My results are a bit different, but I also see better results with the option switched off. The use of the prepro changes something in the output file, but that seems not to change the draw order. Reversing the order of the lines in wayorder also changes something in the img file, but not the draw order. The reason seems to be this: 1) Option switched off Some sub divisions contain only polygons with type 0x10302, some only contain 0x10303, some contain both. 2) Option switched off All subdivisions that contain a fairway (0x10302 ) do also contain the riverbank polygon (0x10303) I tried also with swapped types to make sure that it's not the higher type value that matters, but that just changed the color to a different blue. It seems that the Garmin algo choses the larger polygon if it finds two overlapping polygons in one sub division. Conclusion: Either you need a) an algo that makes sure that overlapping polygons are not placed within the same sub division or b) which cuts the smaller polygon out of the larger one. I don't know yet where to implement that. I have to think about this again... Gerd RheinSkipper wrote I don't have Mapsource, only Basecamp. I assume that works as well? Yes, same results. I tried it. Where should I find an overlapping polygon which is not correctly displayed? Do you have an OSM id? 162118093 and 174328025 have tob e on top of 143749272 and 4499851 and 30759835 Be sure to use garmin types of equal draw priority! 0x10302 to 0x10306 are good examples. Some types (like 0x10105) have higher draw priority (defined in the device´s firmware/mapsource/basecamp) and are ALWAYS drawn on top. I wonder what it means when you say that you cannot control the order with the patch. At least it seams to have an effect, and always wrong seems to mean that you have to reverse the order in the input to get a correct result? I also tried preprocessing in the opposite order. Preprocessing had no effect at all with the option enabled. But only with big data. With 3 small polygons it worked. You can find the preprocessor for sorting xml here: http://openseamap.svn.sourceforge.net/viewvc/openseamap/garmin/wayorder http://openseamap.svn.sourceforge.net/viewvc/openseamap/garmin/gsort/gsort.j ar Here is a map with option off and random behavior: http://files.mkgmap.org.uk/download/88/random.JPG In the upper right rectangle the light blue polygon is covered by the dark blue polygon, which is incorrect. The other parts of the map are correct (light blue on top of dark blue). There is no tile boarder there, so this rectangle must be one of those sub divisions. And this is the same map with option on: http://files.mkgmap.org.uk/download/89/option_on.JPG Here all light blue polygons are hidden below the darker ones. Preprocessing has no influence. ___ mkgmap-dev mailing list mkgmap-dev@.org http://lists.mkgmap.org.uk/mailman/listinfo/mkgmap-dev -- View this message in context: http://gis.19327.n5.nabble.com/Splitter-Error-tp5752745p5753107.html Sent from the Mkgmap Development mailing list archive at Nabble.com. ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://lists.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Re: [mkgmap-dev] Splitter Error
The use of the prepro changes something in the output file, but that seems not to change the draw order. Reversing the order of the lines in wayorder also changes something in the img file, but not the draw order. Did you run the prepro after splitter on each tile? I use a script with a for loop for that. Splitter output must be set to xml. I attach some test data. With this the prepro works perfectly. It seems that the Garmin algo choses the larger polygon if it finds two overlapping polygons in one sub division. Both riverbank and fairway are divided into segments. So the fairway polygons are not always smaller than the riverbank. We were already thinking about a prepro that combines all the riverbank segments to one big polygon for each river. If fairway is always completely inside riverbank, one could use it as an inner multipolygon and cut out holes in the riverbank. But this was considered to be too complicated. test3.osm Description: Binary data ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://lists.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Re: [mkgmap-dev] Splitter Error
Hi, please try the attached patch (which is just a quick hack). It implements a special mode in the split algo (only for polygons). Activate it with --x-marine-device=true and use it in combination with --preserve-element-order and your prepro. Does that improve your map? marine_device_v1.patch http://gis.19327.n5.nabble.com/file/n5753014/marine_device_v1.patch Gerd RheinSkipper wrote Ah, I begin to understand. With a few polygons of test data (which might fit into a single sub division), changing order in the input had the desired effect on the drawing order. But with bigger input the results were somehow random. I hope a practical way can be found to deal with this problem. If one could specify a layer or priority value for several tags within the style file, maybe mkgmap could collect polygons of equal priority in different sub divisions and sort the whole sub divisions in corresponding order. If it would be necessary to break the map into areas that fit into sub divisions, this could be made optional just for marine maps which do not support routing anyway. Maybe it´s easier without the routing stuff. -- View this message in context: http://gis.19327.n5.nabble.com/Splitter-Error-tp5752745p5753014.html Sent from the Mkgmap Development mailing list archive at Nabble.com. ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://lists.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Re: [mkgmap-dev] Splitter Error
RheinSkipper wrote Many, many thanks for working on this problem. Unfortunately I am not familiar with patches. I tried to apply it using GNU patch 2.5.9 under Windows. I suggest to use TortoiseSVN on windows. It is an easy interface for svn and lets you apply patches with a mouse click. I've also found a small error in the patch (some obsolete imports). You can download the patched mkgmap.jar here: http://files.mkgmap.org.uk/download/87/mkgmap.jar The new patch is here: marine_device_v2.patch http://gis.19327.n5.nabble.com/file/n5753024/marine_device_v2.patch Gerd -- View this message in context: http://gis.19327.n5.nabble.com/Splitter-Error-tp5752745p5753024.html Sent from the Mkgmap Development mailing list archive at Nabble.com. ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://lists.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Re: [mkgmap-dev] Splitter Error
RheinSkipper wrote Tests can be done very easily as no marine device is needed for that. Mapsource renders exactly the same draw order as a marine device (as long as there is no typ file of course). I don't have Mapsource, only Basecamp. I assume that works as well? Where should I find an overlapping polygon which is not correctly displayed? Do you have an OSM id? I wonder what it means when you say that you cannot control the order with the patch. At least it seams to have an effect, and always wrong seems to mean that you have to reverse the order in the input to get a correct result? Ciao, Gerd -- View this message in context: http://gis.19327.n5.nabble.com/Splitter-Error-tp5752745p5753037.html Sent from the Mkgmap Development mailing list archive at Nabble.com. ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://lists.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Re: [mkgmap-dev] Splitter Error
Do you see a chance to implement an option to control the order of data in the img file? Good question. I don't have any knowledge about the algorithm that is used in your device, so I have no idea what order we should keep (or establish). Gerd ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://lists.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Re: [mkgmap-dev] Splitter Error
I want to establish the order I need via a preprocessing. This preprocessor (programmed by Malcolm) is already working perfectly. All I need from mkgmap is not to mix up this order. I just hope that the Garmin device does not mix up the order, also. I am not sure yet , but chances are not bad. I tested a map with several overlapping polygons on a lot of different models from the Garmin marine lineup. All models showed exactly the same draw order. Ursprüngliche Nachricht Von: Gerd Petermann gpetermann_muenc...@hotmail.com Datum: An: mkgmap-dev@lists.mkgmap.org.uk Betreff: Re: [mkgmap-dev] Splitter Error Do you see a chance to implement an option to control the order of data in the img file? Good question. I don't have any knowledge about the algorithm that is used in your device, so I have no idea what order we should keep (or establish). Gerd ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://lists.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Re: [mkgmap-dev] Splitter Error
RheinSkipper wrote I want to establish the order I need via a preprocessing. This preprocessor (programmed by Malcolm) is already working perfectly. All I need from mkgmap is not to mix up this order. OK, that is understood. With --preserve-element-order and mkgmap = r2507 the order is not mixed in the program, means, two executions of the program with the same input should give the same output (besides some time stamp values in the img file). The problem remains: The garmin img format is not flat, it requires us to build sub divisions with a lot of restrictions regarding the size and number of elements in it. Thus, two polygons maybe be placed in the same sub division or not. I see only one possible solution: We could change the algo that creates sub divisions completely so that we first calculate the subdiv rectangle, next place all objects into it. If that rectangle doesn't fullfil the restrictions, divide it into two or more rectangles and place all objects into the parts by cutting anything that is not completely within the smaller rectangle. The current algo doesn't cut anything, so this change will probably require a lot more time and spcecial routines to care about routing and rounding errors caused by the cutting. I don't know if that was already tried before? Gerd -- View this message in context: http://gis.19327.n5.nabble.com/Splitter-Error-tp5752745p5752909.html Sent from the Mkgmap Development mailing list archive at Nabble.com. ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://lists.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Re: [mkgmap-dev] Splitter Error
Ah, I begin to understand. With a few polygons of test data (which might fit into a single sub division), changing order in the input had the desired effect on the drawing order. But with bigger input the results were somehow random. I hope a practical way can be found to deal with this problem. If one could specify a layer or priority value for several tags within the style file, maybe mkgmap could collect polygons of equal priority in different sub divisions and sort the whole sub divisions in corresponding order. If it would be necessary to break the map into areas that fit into sub divisions, this could be made optional just for marine maps which do not support routing anyway. Maybe it´s easier without the routing stuff. -Ursprüngliche Nachricht- Von: mkgmap-dev-boun...@lists.mkgmap.org.uk [mailto:mkgmap-dev- boun...@lists.mkgmap.org.uk] Im Auftrag von GerdP Gesendet: Dienstag, 12. März 2013 11:24 An: mkgmap-dev@lists.mkgmap.org.uk Betreff: Re: [mkgmap-dev] Splitter Error RheinSkipper wrote I want to establish the order I need via a preprocessing. This preprocessor (programmed by Malcolm) is already working perfectly. All I need from mkgmap is not to mix up this order. OK, that is understood. With --preserve-element-order and mkgmap = r2507 the order is not mixed in the program, means, two executions of the program with the same input should give the same output (besides some time stamp values in the img file). The problem remains: The garmin img format is not flat, it requires us to build sub divisions with a lot of restrictions regarding the size and number of elements in it. Thus, two polygons maybe be placed in the same sub division or not. I see only one possible solution: We could change the algo that creates sub divisions completely so that we first calculate the subdiv rectangle, next place all objects into it. If that rectangle doesn't fullfil the restrictions, divide it into two or more rectangles and place all objects into the parts by cutting anything that is not completely within the smaller rectangle. The current algo doesn't cut anything, so this change will probably require a lot more time and spcecial routines to care about routing and rounding errors caused by the cutting. I don't know if that was already tried before? Gerd -- View this message in context: http://gis.19327.n5.nabble.com/Splitter-Error- tp5752745p5752909.html Sent from the Mkgmap Development mailing list archive at Nabble.com. ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://lists.mkgmap.org.uk/mailman/listinfo/mkgmap-dev ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://lists.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Re: [mkgmap-dev] Splitter Error
I updated JRE from 1.6 to 1.7, updated osmosis to latest version and gave it 16 GB of memory. With a small bbox left=8 right=9 bottom=49.5 top=50.5 osmosis still produces 6 GB of bbox data. And running splitter on this bbox always results in: Exception in thread main java.lang.OutOfMemoryError: Java heap space at uk.me.parabola.splitter.DensityMap.addNode(DensityMap.java:132) at uk.me.parabola.splitter.DensityMapCollector.processNode(DensityMapCollector. java:58) at uk.me.parabola.splitter.BinaryMapParser.parseDense(BinaryMapParser.java:113) at crosby.binary.BinaryParser.parse(BinaryParser.java:124) at crosby.binary.BinaryParser.handleBlock(BinaryParser.java:68) at crosby.binary.file.FileBlock.process(FileBlock.java:135) at crosby.binary.file.BlockInputStream.process(BlockInputStream.java:34) at uk.me.parabola.splitter.PrecompSeaReader.processMap(PrecompSeaReader.java:85 ) at uk.me.parabola.splitter.Main.calculateAreas(Main.java:479) at uk.me.parabola.splitter.Main.split(Main.java:205) at uk.me.parabola.splitter.Main.start(Main.java:157) at uk.me.parabola.splitter.Main.main(Main.java:146) Good luck! I fear this might be a hardware problem. Or maybe some kind of multitasking problem in the JRE. If you cannot reproduce it for sure it is unlikely that I will be able to do it. Gerd -- View this message in context: http://gis.19327.n5.nabble.com/Splitter-Error- tp5752745p5752812.html Sent from the Mkgmap Development mailing list archive at Nabble.com. ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://lists.mkgmap.org.uk/mailman/listinfo/mkgmap-dev ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://lists.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Re: [mkgmap-dev] Splitter Error
Hi, if you have no good reason to use resolution=16, remove this parm. That will reduce the required memory in this phase. Gerd RheinSkipper wrote I updated JRE from 1.6 to 1.7, updated osmosis to latest version and gave it 16 GB of memory. With a small bbox left=8 right=9 bottom=49.5 top=50.5 osmosis still produces 6 GB of bbox data. And running splitter on this bbox always results in: Exception in thread main java.lang.OutOfMemoryError: Java heap space at uk.me.parabola.splitter.DensityMap.addNode(DensityMap.java:132) at uk.me.parabola.splitter.DensityMapCollector.processNode(DensityMapCollector. java:58) at uk.me.parabola.splitter.BinaryMapParser.parseDense(BinaryMapParser.java:113) at crosby.binary.BinaryParser.parse(BinaryParser.java:124) at crosby.binary.BinaryParser.handleBlock(BinaryParser.java:68) at crosby.binary.file.FileBlock.process(FileBlock.java:135) at crosby.binary.file.BlockInputStream.process(BlockInputStream.java:34) at uk.me.parabola.splitter.PrecompSeaReader.processMap(PrecompSeaReader.java:85 ) at uk.me.parabola.splitter.Main.calculateAreas(Main.java:479) at uk.me.parabola.splitter.Main.split(Main.java:205) at uk.me.parabola.splitter.Main.start(Main.java:157) at uk.me.parabola.splitter.Main.main(Main.java:146) Good luck! I fear this might be a hardware problem. Or maybe some kind of multitasking problem in the JRE. If you cannot reproduce it for sure it is unlikely that I will be able to do it. Gerd -- View this message in context: http://gis.19327.n5.nabble.com/Splitter-Error- tp5752745p5752812.html Sent from the Mkgmap Development mailing list archive at Nabble.com. ___ mkgmap-dev mailing list mkgmap-dev@.org http://lists.mkgmap.org.uk/mailman/listinfo/mkgmap-dev ___ mkgmap-dev mailing list mkgmap-dev@.org http://lists.mkgmap.org.uk/mailman/listinfo/mkgmap-dev -- View this message in context: http://gis.19327.n5.nabble.com/Splitter-Error-tp5752745p5752962.html Sent from the Mkgmap Development mailing list archive at Nabble.com. ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://lists.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Re: [mkgmap-dev] Splitter Error
Hi, probably an error in splitter. It seems to be caused by a long string in the data. Can you tell me how to reproduce it? Gerd RheinSkipper wrote I receive an error when running the current splitter version: Exception in thread worker-6 java.lang.StringIndexOutOfBoundsException: String index out of range: -3 at java.lang.String.getChars(String.java:860) at uk.me.parabola.splitter.OSMXMLWriter.writeString(OSMXMLWriter.java:199) at uk.me.parabola.splitter.OSMXMLWriter.write(OSMXMLWriter.java:96) at uk.me.parabola.splitter.SplitProcessor$OSMWriterWorker.processElement(SplitP rocessor.java:407) at uk.me.parabola.splitter.SplitProcessor$OSMWriterWorker.run(SplitProcessor.ja va:442) at java.lang.Thread.run(Thread.java:679) What does this mean? ___ mkgmap-dev mailing list mkgmap-dev@.org http://lists.mkgmap.org.uk/mailman/listinfo/mkgmap-dev -- View this message in context: http://gis.19327.n5.nabble.com/Splitter-Error-tp5752745p5752748.html Sent from the Mkgmap Development mailing list archive at Nabble.com. ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://lists.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Re: [mkgmap-dev] Splitter Error
First I extract my bbox from the planet: read=planet/planet-latest.osm.pbf write=planet/bbox.osm.pbf omitmetadata=true zipname=osm_europe_central bbox=left=-8 right=25 bottom=38 top=61 JAVACMD_OPTIONS=-Djava.io.tmpdir=$HOME/tmp -Xmx6G osmosis/bin/osmosis --read-pbf $read --tee 1 outPipe.0=a --bounding-box inPipe.0=a completeWays=yes $bbox --write-pbf $write Then I run the splitter: java -XX:+UseCompressedOops -Xmx6500M -jar splitter/splitter.jar --max-nodes=150 --mixed=yes --no-trim --keep-complete --precomp-sea=sea --overlap=0 --resolution=16 --geonames-file=cities15000.zip --description=INT-Offshore --output=xml --output-dir=tiles planet/bbox.osm.pbf 2err1.txt -Ursprüngliche Nachricht- Von: mkgmap-dev-boun...@lists.mkgmap.org.uk [mailto:mkgmap-dev- boun...@lists.mkgmap.org.uk] Im Auftrag von GerdP Gesendet: Montag, 11. März 2013 14:47 An: mkgmap-dev@lists.mkgmap.org.uk Betreff: Re: [mkgmap-dev] Splitter Error Hi, probably an error in splitter. It seems to be caused by a long string in the data. Can you tell me how to reproduce it? Gerd RheinSkipper wrote I receive an error when running the current splitter version: Exception in thread worker-6 java.lang.StringIndexOutOfBoundsException: String index out of range: -3 at java.lang.String.getChars(String.java:860) at uk.me.parabola.splitter.OSMXMLWriter.writeString(OSMXMLWriter.java:199 ) at uk.me.parabola.splitter.OSMXMLWriter.write(OSMXMLWriter.java:96) at uk.me.parabola.splitter.SplitProcessor$OSMWriterWorker.processElement( SplitP rocessor.java:407) at uk.me.parabola.splitter.SplitProcessor$OSMWriterWorker.run(SplitProces sor.ja va:442) at java.lang.Thread.run(Thread.java:679) What does this mean? ___ mkgmap-dev mailing list mkgmap-dev@.org http://lists.mkgmap.org.uk/mailman/listinfo/mkgmap-dev -- View this message in context: http://gis.19327.n5.nabble.com/Splitter-Error- tp5752745p5752748.html Sent from the Mkgmap Development mailing list archive at Nabble.com. ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://lists.mkgmap.org.uk/mailman/listinfo/mkgmap-dev ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://lists.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Re: [mkgmap-dev] Splitter Error
Okay, I'll try to reproduce that. Would it be a problem to use o5m or pbf for output? Gerd RheinSkipper wrote First I extract my bbox from the planet: read=planet/planet-latest.osm.pbf write=planet/bbox.osm.pbf omitmetadata=true zipname=osm_europe_central bbox=left=-8 right=25 bottom=38 top=61 JAVACMD_OPTIONS=-Djava.io.tmpdir=$HOME/tmp -Xmx6G osmosis/bin/osmosis --read-pbf $read --tee 1 outPipe.0=a --bounding-box inPipe.0=a completeWays=yes $bbox --write-pbf $write Then I run the splitter: java -XX:+UseCompressedOops -Xmx6500M -jar splitter/splitter.jar --max-nodes=150 --mixed=yes --no-trim --keep-complete --precomp-sea=sea --overlap=0 --resolution=16 --geonames-file=cities15000.zip --description=INT-Offshore --output=xml --output-dir=tiles planet/bbox.osm.pbf 2err1.txt -Ursprüngliche Nachricht- Von: mkgmap-dev-bounces@.org [mailto:mkgmap-dev- bounces@.org ] Im Auftrag von GerdP Gesendet: Montag, 11. März 2013 14:47 An: mkgmap-dev@.org Betreff: Re: [mkgmap-dev] Splitter Error Hi, probably an error in splitter. It seems to be caused by a long string in the data. Can you tell me how to reproduce it? Gerd RheinSkipper wrote I receive an error when running the current splitter version: Exception in thread worker-6 java.lang.StringIndexOutOfBoundsException: String index out of range: -3 at java.lang.String.getChars(String.java:860) at uk.me.parabola.splitter.OSMXMLWriter.writeString(OSMXMLWriter.java:199 ) at uk.me.parabola.splitter.OSMXMLWriter.write(OSMXMLWriter.java:96) at uk.me.parabola.splitter.SplitProcessor$OSMWriterWorker.processElement( SplitP rocessor.java:407) at uk.me.parabola.splitter.SplitProcessor$OSMWriterWorker.run(SplitProces sor.ja va:442) at java.lang.Thread.run(Thread.java:679) What does this mean? ___ mkgmap-dev mailing list mkgmap-dev@.org http://lists.mkgmap.org.uk/mailman/listinfo/mkgmap-dev -- View this message in context: http://gis.19327.n5.nabble.com/Splitter-Error- tp5752745p5752748.html Sent from the Mkgmap Development mailing list archive at Nabble.com. ___ mkgmap-dev mailing list mkgmap-dev@.org http://lists.mkgmap.org.uk/mailman/listinfo/mkgmap-dev ___ mkgmap-dev mailing list mkgmap-dev@.org http://lists.mkgmap.org.uk/mailman/listinfo/mkgmap-dev -- View this message in context: http://gis.19327.n5.nabble.com/Splitter-Error-tp5752745p5752762.html Sent from the Mkgmap Development mailing list archive at Nabble.com. ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://lists.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Re: [mkgmap-dev] Splitter Error
Hi RheinSkipper, I traced through the code and I fear I am not able to reproduce the problem. Are you able to reproduce it? Maybe also with smaller bboxes? If yes, can you post a link to your input file for splitter? Else we may have to add some debugging code to your splitter version. Gerd RheinSkipper wrote First I extract my bbox from the planet: read=planet/planet-latest.osm.pbf write=planet/bbox.osm.pbf omitmetadata=true zipname=osm_europe_central bbox=left=-8 right=25 bottom=38 top=61 JAVACMD_OPTIONS=-Djava.io.tmpdir=$HOME/tmp -Xmx6G osmosis/bin/osmosis --read-pbf $read --tee 1 outPipe.0=a --bounding-box inPipe.0=a completeWays=yes $bbox --write-pbf $write Then I run the splitter: java -XX:+UseCompressedOops -Xmx6500M -jar splitter/splitter.jar --max-nodes=150 --mixed=yes --no-trim --keep-complete --precomp-sea=sea --overlap=0 --resolution=16 --geonames-file=cities15000.zip --description=INT-Offshore --output=xml --output-dir=tiles planet/bbox.osm.pbf 2err1.txt -Ursprüngliche Nachricht- Von: mkgmap-dev-bounces@.org [mailto:mkgmap-dev- bounces@.org ] Im Auftrag von GerdP Gesendet: Montag, 11. März 2013 14:47 An: mkgmap-dev@.org Betreff: Re: [mkgmap-dev] Splitter Error Hi, probably an error in splitter. It seems to be caused by a long string in the data. Can you tell me how to reproduce it? Gerd RheinSkipper wrote I receive an error when running the current splitter version: Exception in thread worker-6 java.lang.StringIndexOutOfBoundsException: String index out of range: -3 at java.lang.String.getChars(String.java:860) at uk.me.parabola.splitter.OSMXMLWriter.writeString(OSMXMLWriter.java:199 ) at uk.me.parabola.splitter.OSMXMLWriter.write(OSMXMLWriter.java:96) at uk.me.parabola.splitter.SplitProcessor$OSMWriterWorker.processElement( SplitP rocessor.java:407) at uk.me.parabola.splitter.SplitProcessor$OSMWriterWorker.run(SplitProces sor.ja va:442) at java.lang.Thread.run(Thread.java:679) What does this mean? ___ mkgmap-dev mailing list mkgmap-dev@.org http://lists.mkgmap.org.uk/mailman/listinfo/mkgmap-dev -- View this message in context: http://gis.19327.n5.nabble.com/Splitter-Error- tp5752745p5752748.html Sent from the Mkgmap Development mailing list archive at Nabble.com. ___ mkgmap-dev mailing list mkgmap-dev@.org http://lists.mkgmap.org.uk/mailman/listinfo/mkgmap-dev ___ mkgmap-dev mailing list mkgmap-dev@.org http://lists.mkgmap.org.uk/mailman/listinfo/mkgmap-dev -- View this message in context: http://gis.19327.n5.nabble.com/Splitter-Error-tp5752745p5752783.html Sent from the Mkgmap Development mailing list archive at Nabble.com. ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://lists.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Re: [mkgmap-dev] Splitter Error
Well, I used xml output just for some experiments. I tried to sort the output to control the draw priority of polygons. According to http://www.cferrero.net/maps/mkgmap_tiddlywiki.html#--preserve-element-order this should work, but it didn’t. So I can switch back to pbf again for now. Not being able to control what is drawn on top of what is still my biggest problem in generating marine maps. Nautical chartplotters do not support TYP files, so controlling draw-priority via TYP is not an option for the Garmin marine lineup. Okay, I'll try to reproduce that. Would it be a problem to use o5m or pbf for output? ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://lists.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Re: [mkgmap-dev] Splitter Error
I had the problem a couple of times. But when I run splitter again with same input and same script, the problem is sometimes away. I will try to reproduce it with smaller input. -Ursprüngliche Nachricht- Von: mkgmap-dev-boun...@lists.mkgmap.org.uk [mailto:mkgmap-dev- boun...@lists.mkgmap.org.uk] Im Auftrag von GerdP Gesendet: Montag, 11. März 2013 16:37 An: mkgmap-dev@lists.mkgmap.org.uk Betreff: Re: [mkgmap-dev] Splitter Error Hi RheinSkipper, I traced through the code and I fear I am not able to reproduce the problem. Are you able to reproduce it? Maybe also with smaller bboxes? If yes, can you post a link to your input file for splitter? Else we may have to add some debugging code to your splitter version. Gerd ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://lists.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Re: [mkgmap-dev] Splitter Error
RheinSkipper wrote I had the problem a couple of times. But when I run splitter again with same input and same script, the problem is sometimes away. I will try to reproduce it with smaller input. Good luck! I fear this might be a hardware problem. Or maybe some kind of multitasking problem in the JRE. If you cannot reproduce it for sure it is unlikely that I will be able to do it. Gerd -- View this message in context: http://gis.19327.n5.nabble.com/Splitter-Error-tp5752745p5752812.html Sent from the Mkgmap Development mailing list archive at Nabble.com. ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://lists.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Re: [mkgmap-dev] Splitter Error
RheinSkipper wrote Well, I used xml output just for some experiments. I tried to sort the output to control the draw priority of polygons. According to http://www.cferrero.net/maps/mkgmap_tiddlywiki.html#--preserve-element-order this should work, but it didn’t. So I can switch back to pbf again for now. Not being able to control what is drawn on top of what is still my biggest problem in generating marine maps. Nautical chartplotters do not support TYP files, so controlling draw-priority via TYP is not an option for the Garmin marine lineup. Hmm, --preserve-element-order only makes sure that (most of) the data is always processed in the same order. I doubt that you can really influnce the order of data in the img file. The algorithm that is used to distribute data into the sub regions is likely to randomize the result for each small change in the style or osm data. Gerd -- View this message in context: http://gis.19327.n5.nabble.com/Splitter-Error-tp5752745p5752814.html Sent from the Mkgmap Development mailing list archive at Nabble.com. ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://lists.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Re: [mkgmap-dev] Splitter Error
Do you see a chance to implement an option to control the order of data in the img file? In our experiments tweaking the order of data in the osm file did have an influence on draw order. But as you described the influence was more or less random. For rendering shaded depth areas in a sea map it is absolutely necessary to make sure that the depth area polygons are drawn on top of sea or riverbank polygons and not below them. As another approach to this problem I was thinking about leaving away the sea polygons at all (so they do not cover my depth shading) and creating Bluechart type maps instead of topo maps. As far as I understand, the difference between Bluechart and topo is that blank unmapped areas are rendered as water on a Bluechart but rendered as land on a topo map. So we need sea and riverbank polygons on topos but on a Bluechart we need land polygons. Unfortunately as mkgmap can only generate sea polygons and no land polygons, this approach is out of reach at the moment. Hmm, --preserve-element-order only makes sure that (most of) the data is always processed in the same order. I doubt that you can really influnce the order of data in the img file. The algorithm that is used to distribute data into the sub regions is likely to randomize the result for each small change in the style or osm data. Gerd -- View this message in context: http://gis.19327.n5.nabble.com/Splitter-Error- tp5752745p5752814.html Sent from the Mkgmap Development mailing list archive at Nabble.com. ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://lists.mkgmap.org.uk/mailman/listinfo/mkgmap-dev ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://lists.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Re: [mkgmap-dev] Splitter Error
For rendering shaded depth areas in a sea map it is absolutely necessary to make sure that the depth area polygons are drawn on top of sea or riverbank polygons and not below them. If you use a typ file you can set the draworder of every polygon ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://lists.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Re: [mkgmap-dev] Splitter error
Hello Francois, it seems that you use an out-aged version of the fastutil.jar lib. The correct version is in the package, see http://www.mkgmap.org.uk/download/splitter-r289.zip Gerd Francois wrote Hello, Running splitter with the following command line parameters, java -Xmx896m -jar $FILEDIR/splitter.jar --cache=$cache --output=$output --geonames-file=$geonames_file --max-areas=$max_areas --max-nodes=$max_nodes --description=$description_splitter --mapid=$mapid $CARTESDIR/$geofabrik_map . Parameter $cache = /usr/local/misc/cartes . Parameter $output = xml . Parameter $geonames_file = cities1000.zip . Parameter $max_areas = 30 . Parameter $max_nodes = 80 . Parameter $description_splitter = Carte Geographique OSM . Parameter $mapid = 0001 . Parameter $CARTESDIR/$geofabrik_map = /usr/local/misc/cartes/auvergne.osm.bz2 Gives me the following error : http://paste.opensuse.org/14715469 Maybe one or more parameters I use, are no longer available... thank you. Francois -- ___ mkgmap-dev mailing list mkgmap-dev@.org http://lists.mkgmap.org.uk/mailman/listinfo/mkgmap-dev -- View this message in context: http://gis.19327.n5.nabble.com/Splitter-error-tp5749963p5749965.html Sent from the Mkgmap Development mailing list archive at Nabble.com. ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://lists.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Re: [mkgmap-dev] Splitter error
On 19/02/2013 20:39, GerdP wrote: Hello, it seems that you use an out-aged version of the fastutil.jar lib. The correct version is in the package, see http://www.mkgmap.org.uk/download/splitter-r289.zip Bingo. You're right. It works fine now. Thank you. Francois ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://lists.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Re: [mkgmap-dev] Splitter error
Hello Francois, thanks for the feedback. If you are interested in the latest parms, see http://wiki.openstreetmap.org/wiki/Mkgmap/help/splitter Gerd Francois wrote On 19/02/2013 20:39, GerdP wrote: Hello, it seems that you use an out-aged version of the fastutil.jar lib. The correct version is in the package, see http://www.mkgmap.org.uk/download/splitter-r289.zip Bingo. You're right. It works fine now. Thank you. Francois ___ mkgmap-dev mailing list mkgmap-dev@.org http://lists.mkgmap.org.uk/mailman/listinfo/mkgmap-dev -- View this message in context: http://gis.19327.n5.nabble.com/Splitter-error-tp5749963p5749978.html Sent from the Mkgmap Development mailing list archive at Nabble.com. ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://lists.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Re: [mkgmap-dev] Splitter Error
Hello Roger, you are running an out-aged version of splitter. Please update to the latest stable version: r202 http://www.mkgmap.org.uk/splitter/ Ciao, Gerd Roger Calvert wrote In running a macro I have used many times before with a new download from geofabrik, splitter gave me the following error. (I realise that there is no need to split the file in this particular case, but it has never caused problems before.) I am not sure of the version of splitter I am using, but the files are dated 11/11/2011. Any advice would be appreciated. Roger --- cache=DataTest description=Test geonames-file=..\Resources\cities5000.zip legacy-mode=false mapid=60555001 max-areas=255 max-nodes=250 max-threads=4 (auto) mixed=false no-trim=false output=pbf output-dir= overlap=2000 resolution=13 split-file= status-freq=120 write-kml= Elapsed time: 0s Memory: Current 59MB (2MB used, 57MB free) Max 2666MB Time started: Wed Nov 14 10:48:30 GMT 2012 Map is being split for resolution 13: - area boundaries are aligned to 0x800 map units - areas are multiples of 0x1000 map units wide and high Processing DataTest\cumbria.osm.pbf Bounding box -3.907471 53.900631 -2.15958803 55.18863 in 1 file Time: Wed Nov 14 10:48:31 GMT 2012 Exact map coverage is (53.90062093734741,-3.907463550567627) to (55.188617706298 83,-2.159585952758789) Trimmed and rounded map coverage is (53.876953125,-3.9111328125) to (55.1953125, -2.1533203125) Splitting nodes into areas containing a maximum of 2,500,000 nodes each... Area (53.876953125,-3.9111328125) to (55.1953125,-2.1533203125) contains 699,912 nodes. DONE! 1 areas: Area 60555001 covers (0x265000,0xfffd3800) to (0x274000,0xfffe7800) GB-Carlisle Writing out split osm files Wed Nov 14 10:48:32 GMT 2012 Processing 1 areas in a single pass (53.876953125,-3.9111328125) to (55.1953125,-2.1533203125) Starting pass 1 of 1, processing 1 areas (60555001 to 60555001) Making SparseMultiMap Making SparseMultiMap Processing DataTest\cumbria.osm.pbf Bounding box -3.907471 53.900631 -2.15958803 55.18863 Exception in thread main java.lang.IndexOutOfBoundsException: Index (31257321) is greater than or equal to list size (31250001) at it.unimi.dsi.fastutil.objects.ObjectArrayList.get(ObjectArrayList.jav a:258) at uk.me.parabola.splitter.SparseInt2ShortMapInline.put(SparseInt2ShortM apInline.java:128) at uk.me.parabola.splitter.SparseInt2ShortMultiMap$Inner.put(SparseInt2S hortMultiMap.java:81) at uk.me.parabola.splitter.SparseInt2ShortMultiMap.put(SparseInt2ShortMu ltiMap.java:31) at uk.me.parabola.splitter.SplitProcessor.writeNode(SplitProcessor.java: 209) at uk.me.parabola.splitter.SplitProcessor.processNode(SplitProcessor.jav a:118) at uk.me.parabola.splitter.BinaryMapParser.parseDense(BinaryMapParser.ja va:49) at crosby.binary.BinaryParser.parse(BinaryParser.java:124) at crosby.binary.BinaryParser.handleBlock(BinaryParser.java:68) at crosby.binary.file.FileBlock.process(FileBlock.java:135) at crosby.binary.file.BlockInputStream.process(BlockInputStream.java:34) at uk.me.parabola.splitter.Main.processMap(Main.java:403) at uk.me.parabola.splitter.Main.writeAreas(Main.java:368) at uk.me.parabola.splitter.Main.split(Main.java:190) at uk.me.parabola.splitter.Main.start(Main.java:118) at uk.me.parabola.splitter.Main.main(Main.java:107) -- Roger Calvert ___ mkgmap-dev mailing list mkgmap-dev@.org http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev -- View this message in context: http://gis.19327.n5.nabble.com/Splitter-Error-tp5735749p5735776.html Sent from the Mkgmap Development mailing list archive at Nabble.com. ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Re: [mkgmap-dev] Splitter Error
Thanks, Gerd. That solved the problem. I have another, more general, query: will the improvements currently being made to Splitter to handle problem polygons remove or reduce the need for pre-compiled sea (whose purpose seems to be to prevent flooding resulting from faulty coastlines) in mkgmap? Roger On 14/11/2012 12:26, GerdP wrote: Hello Roger, you are running an out-aged version of splitter. Please update to the latest stable version: r202 http://www.mkgmap.org.uk/splitter/ Ciao, Gerd Roger Calvert wrote In running a macro I have used many times before with a new download from geofabrik, splitter gave me the following error. (I realise that there is no need to split the file in this particular case, but it has never caused problems before.) I am not sure of the version of splitter I am using, but the files are dated 11/11/2011. Any advice would be appreciated. Roger --- cache=DataTest description=Test geonames-file=..\Resources\cities5000.zip legacy-mode=false mapid=60555001 max-areas=255 max-nodes=250 max-threads=4 (auto) mixed=false no-trim=false output=pbf output-dir= overlap=2000 resolution=13 split-file= status-freq=120 write-kml= Elapsed time: 0s Memory: Current 59MB (2MB used, 57MB free) Max 2666MB Time started: Wed Nov 14 10:48:30 GMT 2012 Map is being split for resolution 13: - area boundaries are aligned to 0x800 map units - areas are multiples of 0x1000 map units wide and high Processing DataTest\cumbria.osm.pbf Bounding box -3.907471 53.900631 -2.15958803 55.18863 in 1 file Time: Wed Nov 14 10:48:31 GMT 2012 Exact map coverage is (53.90062093734741,-3.907463550567627) to (55.188617706298 83,-2.159585952758789) Trimmed and rounded map coverage is (53.876953125,-3.9111328125) to (55.1953125, -2.1533203125) Splitting nodes into areas containing a maximum of 2,500,000 nodes each... Area (53.876953125,-3.9111328125) to (55.1953125,-2.1533203125) contains 699,912 nodes. DONE! 1 areas: Area 60555001 covers (0x265000,0xfffd3800) to (0x274000,0xfffe7800) GB-Carlisle Writing out split osm files Wed Nov 14 10:48:32 GMT 2012 Processing 1 areas in a single pass (53.876953125,-3.9111328125) to (55.1953125,-2.1533203125) Starting pass 1 of 1, processing 1 areas (60555001 to 60555001) Making SparseMultiMap Making SparseMultiMap Processing DataTest\cumbria.osm.pbf Bounding box -3.907471 53.900631 -2.15958803 55.18863 Exception in thread main java.lang.IndexOutOfBoundsException: Index (31257321) is greater than or equal to list size (31250001) at it.unimi.dsi.fastutil.objects.ObjectArrayList.get(ObjectArrayList.jav a:258) at uk.me.parabola.splitter.SparseInt2ShortMapInline.put(SparseInt2ShortM apInline.java:128) at uk.me.parabola.splitter.SparseInt2ShortMultiMap$Inner.put(SparseInt2S hortMultiMap.java:81) at uk.me.parabola.splitter.SparseInt2ShortMultiMap.put(SparseInt2ShortMu ltiMap.java:31) at uk.me.parabola.splitter.SplitProcessor.writeNode(SplitProcessor.java: 209) at uk.me.parabola.splitter.SplitProcessor.processNode(SplitProcessor.jav a:118) at uk.me.parabola.splitter.BinaryMapParser.parseDense(BinaryMapParser.ja va:49) at crosby.binary.BinaryParser.parse(BinaryParser.java:124) at crosby.binary.BinaryParser.handleBlock(BinaryParser.java:68) at crosby.binary.file.FileBlock.process(FileBlock.java:135) at crosby.binary.file.BlockInputStream.process(BlockInputStream.java:34) at uk.me.parabola.splitter.Main.processMap(Main.java:403) at uk.me.parabola.splitter.Main.writeAreas(Main.java:368) at uk.me.parabola.splitter.Main.split(Main.java:190) at uk.me.parabola.splitter.Main.start(Main.java:118) at uk.me.parabola.splitter.Main.main(Main.java:107) -- Roger Calvert ___ mkgmap-dev mailing list mkgmap-dev@.org http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev -- View this message in context: http://gis.19327.n5.nabble.com/Splitter-Error-tp5735749p5735776.html Sent from the Mkgmap Development mailing list archive at Nabble.com. ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev -- Roger Calvert ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Re: [mkgmap-dev] Splitter Error
Hello Roger, I really don't know. If I got that right, flooding is caused by different reasons: 1) wrong OSM data 2) incomplete OSM data caused by splitting planet using e.g. osmosis or other tools 3) incomplete OSM data caused by splitter Even with a perfect error free splitter you will have point 1) left. My understanding is that the precompiled sea data is somehow verified to be ok. Besides that the precompiled files save CPU time. Ciao, Gerd Date: Wed, 14 Nov 2012 14:48:39 + From: ro...@rogercalvert.me.uk To: mkgmap-dev@lists.mkgmap.org.uk Subject: Re: [mkgmap-dev] Splitter Error Thanks, Gerd. That solved the problem. I have another, more general, query: will the improvements currently being made to Splitter to handle problem polygons remove or reduce the need for pre-compiled sea (whose purpose seems to be to prevent flooding resulting from faulty coastlines) in mkgmap? Roger On 14/11/2012 12:26, GerdP wrote: Hello Roger, you are running an out-aged version of splitter. Please update to the latest stable version: r202 http://www.mkgmap.org.uk/splitter/ Ciao, Gerd Roger Calvert wrote In running a macro I have used many times before with a new download from geofabrik, splitter gave me the following error. (I realise that there is no need to split the file in this particular case, but it has never caused problems before.) I am not sure of the version of splitter I am using, but the files are dated 11/11/2011. Any advice would be appreciated. Roger --- cache=DataTest description=Test geonames-file=..\Resources\cities5000.zip legacy-mode=false mapid=60555001 max-areas=255 max-nodes=250 max-threads=4 (auto) mixed=false no-trim=false output=pbf output-dir= overlap=2000 resolution=13 split-file= status-freq=120 write-kml= Elapsed time: 0s Memory: Current 59MB (2MB used, 57MB free) Max 2666MB Time started: Wed Nov 14 10:48:30 GMT 2012 Map is being split for resolution 13: - area boundaries are aligned to 0x800 map units - areas are multiples of 0x1000 map units wide and high Processing DataTest\cumbria.osm.pbf Bounding box -3.907471 53.900631 -2.15958803 55.18863 in 1 file Time: Wed Nov 14 10:48:31 GMT 2012 Exact map coverage is (53.90062093734741,-3.907463550567627) to (55.188617706298 83,-2.159585952758789) Trimmed and rounded map coverage is (53.876953125,-3.9111328125) to (55.1953125, -2.1533203125) Splitting nodes into areas containing a maximum of 2,500,000 nodes each... Area (53.876953125,-3.9111328125) to (55.1953125,-2.1533203125) contains 699,912 nodes. DONE! 1 areas: Area 60555001 covers (0x265000,0xfffd3800) to (0x274000,0xfffe7800) GB-Carlisle Writing out split osm files Wed Nov 14 10:48:32 GMT 2012 Processing 1 areas in a single pass (53.876953125,-3.9111328125) to (55.1953125,-2.1533203125) Starting pass 1 of 1, processing 1 areas (60555001 to 60555001) Making SparseMultiMap Making SparseMultiMap Processing DataTest\cumbria.osm.pbf Bounding box -3.907471 53.900631 -2.15958803 55.18863 Exception in thread main java.lang.IndexOutOfBoundsException: Index (31257321) is greater than or equal to list size (31250001) at it.unimi.dsi.fastutil.objects.ObjectArrayList.get(ObjectArrayList.jav a:258) at uk.me.parabola.splitter.SparseInt2ShortMapInline.put(SparseInt2ShortM apInline.java:128) at uk.me.parabola.splitter.SparseInt2ShortMultiMap$Inner.put(SparseInt2S hortMultiMap.java:81) at uk.me.parabola.splitter.SparseInt2ShortMultiMap.put(SparseInt2ShortMu ltiMap.java:31) at uk.me.parabola.splitter.SplitProcessor.writeNode(SplitProcessor.java: 209) at uk.me.parabola.splitter.SplitProcessor.processNode(SplitProcessor.jav a:118) at uk.me.parabola.splitter.BinaryMapParser.parseDense(BinaryMapParser.ja va:49) at crosby.binary.BinaryParser.parse(BinaryParser.java:124) at crosby.binary.BinaryParser.handleBlock(BinaryParser.java:68) at crosby.binary.file.FileBlock.process(FileBlock.java:135) at crosby.binary.file.BlockInputStream.process(BlockInputStream.java:34) at uk.me.parabola.splitter.Main.processMap(Main.java:403) at uk.me.parabola.splitter.Main.writeAreas(Main.java:368) at uk.me.parabola.splitter.Main.split(Main.java:190) at uk.me.parabola.splitter.Main.start(Main.java:118) at uk.me.parabola.splitter.Main.main(Main.java:107) -- Roger Calvert ___ mkgmap-dev mailing list mkgmap-dev@.org http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Re: [mkgmap-dev] splitter error : java.lang.NoClassDefFoundError: crosby/binary/file/BlockReaderAdapter
Le 27/02/2011 18:24, Henning Scholland a écrit : I've downloaded splitter -r164, but got the same result. Do not know where my mistake is Francois Did you also copied lib directory next to splitter.jar? Bingo, you did it. Now it works as before. Great and thank you. Francois -- ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Re: [mkgmap-dev] splitter error message
Hello Francois, It looks like this is a bug that was introduced by some patches I applied for Scott last night. I've rolled back the changes for now until I get a chance to debug the problem further. If you grab the latest codebase (r121), or alternatively use r118 or older, your problem should disappear. Sorry about that! Chris Hello, Today, running splitter against the french osm.bz2 geofabrik file, I got this error : Exception in thread main java.lang.NullPointerException at uk.me.parabola.splitter.SplittableDensityArea.split(SplittableDensityA rea.java:73) ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Re: [mkgmap-dev] splitter error message
On Tue, Aug 17, 2010 at 7:12 AM, Chris Miller chris_overs...@hotmail.com wrote: Hello Francois, It looks like this is a bug that was introduced by some patches I applied for Scott last night. I've rolled back the changes for now until I get a chance to debug the problem further. If you grab the latest codebase (r121), or alternatively use r118 or older, your problem should disappear. Sorry about that! I'm really sorry too. I have found the broken code in my historical archives; it appears to be untested code that was finished, tested, and fixed on May 25th, weeks before I emailled the list. My local branch was good. (I'm attaching a working diff off of r110). Chris, can you check over my new patch? Again, my apologies, Scott PATCH Description: Binary data ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Re: [mkgmap-dev] splitter error message
Hello Scott, Thanks for the quick patch, saves me digging in to figure out where things were getting derailed. I'll try and take a look at your updated patch later tonight. Cheers, Chris On Tue, Aug 17, 2010 at 7:12 AM, Chris Miller chris_overs...@hotmail.com wrote: Hello Francois, It looks like this is a bug that was introduced by some patches I applied for Scott last night. I've rolled back the changes for now until I get a chance to debug the problem further. If you grab the latest codebase (r121), or alternatively use r118 or older, your problem should disappear. Sorry about that! I'm really sorry too. I have found the broken code in my historical archives; it appears to be untested code that was finished, tested, and fixed on May 25th, weeks before I emailled the list. My local branch was good. (I'm attaching a working diff off of r110). Chris, can you check over my new patch? Again, my apologies, Scott ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Re: [mkgmap-dev] splitter error message
Le 17/08/2010 15:25, Scott Crosby a écrit : Hello, Again, my apologies, Don't worry, there is no problem at all. that's so great to see you guys giving your knowledge and time to help to give us the best program, and you're so fast to give a fix. No, you don't have to apologize, I have to thank you all ;-). Francois -- ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Re: [mkgmap-dev] Splitter Error
On 09/08/2009 09:21 PM, Chris Miller wrote: It looks like there is a latitude value in your file that contains a large number of significant figures (far more than is actually required) and the parsing is failing on it. I just checked in a change that will hopefully solve the problem for you (r88) Great, thanks. r88 solved this issue. ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Re: [mkgmap-dev] Splitter Error
Chris Miller wrote: Thanks Paul, that would explain it for sure. The problem Ralf hit was definitely a bug in the splitter though, I had made one too many assumptions in some custom code for parsing floating point numbers (Java's double parsing is very slow). I figured if the code could parse the planet file it could parse anything. Apparently not! :) The mixture of node and way IDs should be OK as long as the --mixed parameter is specified for the splitter when it generates the cache. Using --mixed together with the fix I put in to r88 means there should be no need for your perl script. Once you have a cache generated, you can rerun the splitter as many times as you like without --mixed by using the (much faster) cache instead of the osm file. There's no need to even specify the osm file on the command line if you have a populated cache. eg first run: java -Xmx2000m splitter.jar --cache=cache/srtm_test --mixed --max-nodes=120 srtm_test.osm Sorry Chris but unless I've missed something then. p...@paul-pc:~/Maps/tile-splitter/splitter-r88/dist java -jar splitter.jar --mapid=70020001 --cache=./ --mixed ../../../srtm2osm/Srtm2Osm-1.7.10.0/fullukcontours.osm Time started: Wed Sep 09 11:48:04 BST 2009 mapid = 70020001 cache = ./ Checking for an existing cache and verifying contents... No suitable cache was found. A new cache will be created to speed up the splitting stage Map is being split for resolution 13: - area boundaries are aligned to 0x800 map units - areas are multiples of 0x1000 map units wide and high Processing ../../../srtm2osm/Srtm2Osm-1.7.10.0/fullukcontours.osm Exception in thread main java.lang.ArrayIndexOutOfBoundsException: 19 at uk.me.parabola.splitter.Convert.parseDouble(Convert.java:171) at uk.me.parabola.splitter.OSMParser.startNode(OSMParser.java:138) at uk.me.parabola.splitter.OSMParser.startElement(OSMParser.java:98) at uk.me.parabola.splitter.AbstractXppParser.parse(AbstractXppParser.java:42) at uk.me.parabola.splitter.Main.processOsmFiles(Main.java:346) at uk.me.parabola.splitter.Main.processMap(Main.java:335) at uk.me.parabola.splitter.Main.calculateAreas(Main.java:240) at uk.me.parabola.splitter.Main.split(Main.java:148) at uk.me.parabola.splitter.Main.main(Main.java:93) The input file is 3.9GB if that helps and r88 runs against the very small test file I created when testing my script. If you need any other info just ask. Cheers Paul ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Re: [mkgmap-dev] Splitter Error
Hi Paul, I'd used Ralf's osm file as a test case to solve this problem, but seems like even that didn't go far enough. I have no idea why but the numbers in your file contain several digits more precision than a 64bit floating point number can even represent. I've added another check so that any digits beyond what is sane are now safely ignored. Sorry for the trouble and thanks for letting me know. It should be solid now but if you still have a problem please send through your test file and I'll take another look. Chris P Sorry Chris but unless I've missed something then. P P p...@paul-pc:~/Maps/tile-splitter/splitter-r88/dist java -jar P splitter.jar --mapid=70020001 --cache=./ --mixed P ../../../srtm2osm/Srtm2Osm-1.7.10.0/fullukcontours.osm P Time started: Wed Sep 09 11:48:04 BST 2009 P mapid = 70020001 P cache = ./ P Checking for an existing cache and verifying contents... P No suitable cache was found. A new cache will be created to speed up P the P splitting stage P Map is being split for resolution 13: P - area boundaries are aligned to 0x800 map units P - areas are multiples of 0x1000 map units wide and high P Processing ../../../srtm2osm/Srtm2Osm-1.7.10.0/fullukcontours.osm P Exception in thread main java.lang.ArrayIndexOutOfBoundsException: P 19 P at P uk.me.parabola.splitter.Convert.parseDouble(Convert.java:171) P at P uk.me.parabola.splitter.OSMParser.startNode(OSMParser.java:138) P at P uk.me.parabola.splitter.OSMParser.startElement(OSMParser.java:98) P at P uk.me.parabola.splitter.AbstractXppParser.parse(AbstractXppParser.jav P a:42) P at P uk.me.parabola.splitter.Main.processOsmFiles(Main.java:346) P at uk.me.parabola.splitter.Main.processMap(Main.java:335) P at uk.me.parabola.splitter.Main.calculateAreas(Main.java:240) P at uk.me.parabola.splitter.Main.split(Main.java:148) P at uk.me.parabola.splitter.Main.main(Main.java:93) P The input file is 3.9GB if that helps and r88 runs against the very P small test file I created when testing my script. If you need any P other info just ask. P P Cheers P P Paul P ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Re: [mkgmap-dev] Splitter Error
Chris Miller wrote: Hi Paul, I'd used Ralf's osm file as a test case to solve this problem, but seems like even that didn't go far enough. I have no idea why but the numbers in your file contain several digits more precision than a 64bit floating point number can even represent. I've added another check so that any digits beyond what is sane are now safely ignored. Sorry for the trouble and thanks for letting me know. It should be solid now but if you still have a problem please send through your test file and I'll take another look. Chris That did the trick. Whilst running I received quite a few of the following Node encountered with missing data. Bad/corrupt osm file? id=1000446803, lat=52.5529168, lon=null. Ignoring this node Examination of this node reveals that the data is indeed corrupt as it has no longitude value:- node id=1000446803 lat=52.5529168 / Cheers Paul ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Re: [mkgmap-dev] Splitter Error
Heh, you're doing a great job testing various corner-cases in the splitter - keep up the good work :) I have to wonder quite how that osm file you have was generated though. Are there really nodes in the osm database that are missing lat and/or lon values? Seems a bit worrying if so. P That did the trick. Whilst running I received quite a few of the P following P P Node encountered with missing data. Bad/corrupt osm file? P id=1000446803, lat=52.5529168, lon=null. Ignoring this node P P Examination of this node reveals that the data is indeed corrupt as P it has no longitude value:- P P node id=1000446803 lat=52.5529168 / P P Cheers P P Paul ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Re: [mkgmap-dev] Splitter Error
It looks like there is a latitude value in your file that contains a large number of significant figures (far more than is actually required) and the parsing is failing on it. I just checked in a change that will hopefully solve the problem for you (r88) however I'd be interested in seeing the osm file that caused this problem. Is it possible to get a copy of your osm file? Chris RK Hi, RK RK when I try to split a OSM file which contains only contour lines RK made with srtm2osm (1.8.14.10) I get this error: RK RK [splitter-r83]$java -Xmx2048M -jar splitter.jar --mapid=2101 RK --mixed=yes --cache=./cache/ --max-nodes=5 srtm_test.osm RK Time started: Tue Sep 08 20:03:15 CEST 2009 RK mapid = 2101 RK mixed = yes RK cache = ./cache/ RK max-nodes = 5 RK Checking for an existing cache and verifying contents... RK No suitable cache was found. A new cache will be created to speed up RK the RK splitting stage RK Map is being split for resolution 13: RK - area boundaries are aligned to 0x800 map units RK - areas are multiples of 0x1000 map units wide and high RK Processing srtm_test.osm RK Exception in thread main java.lang.ArrayIndexOutOfBoundsException: RK 14 RK at RK uk.me.parabola.splitter.Convert.parseDouble(Convert.java:163) RK at RK uk.me.parabola.splitter.OSMParser.startNode(OSMParser.java:137) RK at RK uk.me.parabola.splitter.OSMParser.startElement(OSMParser.java:98) RK at RK uk.me.parabola.splitter.AbstractXppParser.parse(AbstractXppParser.ja RK va:42) RK at RK uk.me.parabola.splitter.Main.processOsmFiles(Main.java:334) RK at uk.me.parabola.splitter.Main.processMap(Main.java:323) RK at RK uk.me.parabola.splitter.Main.calculateAreas(Main.java:225) RK at uk.me.parabola.splitter.Main.split(Main.java:132) RK at uk.me.parabola.splitter.Main.main(Main.java:91) RK The file is rather small, just 1x1 deg, 165 MB. RK RK Any ideas? RK ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Re: [mkgmap-dev] Splitter Error
Chris Miller wrote: It looks like there is a latitude value in your file that contains a large number of significant figures (far more than is actually required) and the parsing is failing on it. I just checked in a change that will hopefully solve the problem for you (r88) however I'd be interested in seeing the osm file that caused this problem. Is it possible to get a copy of your osm file? srt2osm uses about 15dp and mixes node ids and way ids throughout the file. I came across this myself recently and so wrote the attached to reduce the data to 7dp and to place all the nodes before the ways. N.B. I am not a programmer in any sense of the word and this is my first and therefore only perl script so use at your own risk Cheers Paul trimsrtm.pl Description: Perl program ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Re: [mkgmap-dev] Splitter Error
Thanks Paul, that would explain it for sure. The problem Ralf hit was definitely a bug in the splitter though, I had made one too many assumptions in some custom code for parsing floating point numbers (Java's double parsing is very slow). I figured if the code could parse the planet file it could parse anything. Apparently not! :) The mixture of node and way IDs should be OK as long as the --mixed parameter is specified for the splitter when it generates the cache. Using --mixed together with the fix I put in to r88 means there should be no need for your perl script. Once you have a cache generated, you can rerun the splitter as many times as you like without --mixed by using the (much faster) cache instead of the osm file. There's no need to even specify the osm file on the command line if you have a populated cache. eg first run: java -Xmx2000m splitter.jar --cache=cache/srtm_test --mixed --max-nodes=120 srtm_test.osm second and subsequent runs: java -Xmx2000m splitter.jar --cache=cache/srtm_test --split-file=areas.list --mapid=12340001 or perhaps java -Xmx2000m splitter.jar --cache=cache/srtm_test --max-nodes=100 --description=My Map etc Chris P Chris Miller wrote: P It looks like there is a latitude value in your file that contains a large number of significant figures (far more than is actually required) and the parsing is failing on it. I just checked in a change that will hopefully solve the problem for you (r88) however I'd be interested in seeing the osm file that caused this problem. Is it possible to get a copy of your osm file? P srt2osm uses about 15dp and mixes node ids and way ids throughout the P file. I came across this myself recently and so wrote the attached to P reduce the data to 7dp and to place all the nodes before the ways. P P N.B. I am not a programmer in any sense of the word and this is my P first and therefore only perl script so use at your own risk P P Cheers P P Paul ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Re: [mkgmap-dev] splitter error using --cache option
Hi Carlos, Try splitter r77, it should now behave the way you expect with caching and --split-file. I decided to make it print a (perhaps too threatening?) warning message if it detects that you want to build a cache even though it only needs to make a single pass over the .osm file. I figured it's better to make people aware of the performance impact of doing this than for the splitter to just silently take longer than it should without the user realising something might be amiss. Chris CD Currently I only run splitter once on europe.osm to extract Spain, CD Portugal and South of France, but depending on the performance with CD the cache may be I run it later to extract other countries. Thanks CD for the explanation. ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Re: [mkgmap-dev] splitter error using --cache option
Hi Carlos, This is something I'm aware of but haven't yet had time to address. It's because (currently) the cache can only be generated during the first stage when the areas.list file is also being generated. If you supply areas.list and use --cache at the same time, the splitter assumes the cache has already been generated, hence the reason you're seeing this error. It's on my todo list to allow cache generation to occur during the second stage too if need be, I just didn't think anyone would encounter this situation before I had a solution in place. Guess I was wrong! :) To be honest though if you already have an areas.list file and are only making one pass over the XML file during the splitting stage, there's no benefit to using the --cache parameter at all unless you perform multiple runs of the splitter (even with different areas.list files - all that matters is the original .osm file being split is the same). Is that what you want to do? For now your options are to either generate the cache by running the splitter once with no --split-file parameter, or running without the --cache option. Sorry for any inconvenience! Chris CD I have used successfully new splitter when no --split_file is given, CD but CD if I use an existing areas.list splitter looks for cached files and CD they CD don't exist; see below: ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Re: [mkgmap-dev] splitter error using --cache option
Chris Miller escribió: Hi Carlos, This is something I'm aware of but haven't yet had time to address. It's because (currently) the cache can only be generated during the first stage when the areas.list file is also being generated. If you supply areas.list and use --cache at the same time, the splitter assumes the cache has already been generated, hence the reason you're seeing this error. It's on my todo list to allow cache generation to occur during the second stage too if need be, I just didn't think anyone would encounter this situation before I had a solution in place. Guess I was wrong! :) To be honest though if you already have an areas.list file and are only making one pass over the XML file during the splitting stage, there's no benefit to using the --cache parameter at all unless you perform multiple runs of the splitter (even with different areas.list files - all that matters is the original .osm file being split is the same). Is that what you want to do? Currently I only run splitter once on europe.osm to extract Spain, Portugal and South of France, but depending on the performance with the cache may be I run it later to extract other countries. Thanks for the explanation. For now your options are to either generate the cache by running the splitter once with no --split-file parameter, or running without the --cache option. Sorry for any inconvenience! No problem Carlos Chris CD I have used successfully new splitter when no --split_file is given, CD but CD if I use an existing areas.list splitter looks for cached files and CD they CD don't exist; see below: ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
