Re: [mkgmap-dev] Embedding raster maps
Hi Oliver, that's an interesting subject, you are working on. You probably know this free map, I think this is an example of required minimum: http://static.garmin.com/shared/aus/HTML/_pages/isle-of-man.html -- Best regards, Andrzej ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Re: [mkgmap-dev] Embedding raster maps
Hi Andrzej yes this is the map that was used to get first information about the format. But it's still quite big and has a lot of stuff that's probably not related to the raster map layer. My question targets more to what is the minimum required to get a map listed on the device. Without any raster map etc. Is it ok to have sub files with no elements at all? Are there some elements required like the map boundary polygon? Is there the need of some other mandatory stuff that is not so obvious? Before I can concentrate on my raster map encoding I need something that works without. Cheers Oliver > Gesendet: Samstag, 02. Januar 2021 um 17:12 Uhr > Von: "Andrzej Popowski" > An: mkgmap-dev@lists.mkgmap.org.uk > Betreff: Re: [mkgmap-dev] Embedding raster maps > > Hi Oliver, > > that's an interesting subject, you are working on. You probably know > this free map, I think this is an example of required minimum: > http://static.garmin.com/shared/aus/HTML/_pages/isle-of-man.html > > -- > Best regards, > Andrzej > ___ > mkgmap-dev mailing list > mkgmap-dev@lists.mkgmap.org.uk > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Re: [mkgmap-dev] Embedding raster maps
Hi Oliver > What is the minimum data an IMG file has to contain to be listed > on the device? Its an interesting question. I don't remember having too many problems getting a map recognised in the beginning. At r20 it worked according to the commit comment but I only added the test map builder at r24. However on my etrex 30 the map produced by r24 does not work. The first one that does work is r79. With a bit of experimenting I discovered that adding a sufficiently large gap between the end of the TRE header and the first data would make r78 work too. Going back to r24 and adding more padding also produced a map that was recognized. I was able to remove the second copyright message too and all map data. The required gap was between 40 and 101, depending on the rest of the code. So this all makes very little sense and it is possible that this is covering for something else that we don't understand. I've uploaded the minimal IMG file at: http://files.mkgmap.org.uk/download/494/gmapsupp.img (After this I also managed to remove the first copyright message, so you do not need any copyright messages and the label section can be empty) With this version there is no message displayed on startup, which is something that was fixed in r79 I think. There is just a single zoom and subdivision in the TRE section and no points, line or polygons. Cheers, Steve ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Re: [mkgmap-dev] Embedding raster maps
Hi Steve, I tested your map. It's not listed either. I have a GPSMap64s and placed it into the internal as well as into the external memory. I removed all other maps on the device, except the basemap. The basemap is the only one listed. I made a minimum test.img, too. It contains no points, lines or areas. It has 3 zoom levels 22..24 with a sub-division each. I also uploaded a log from my test decoder. https://www.magentacloud.de/share/s429h1z-r6 I uploaded a second one with the version in the header set to 2. So far the difference to be another offset for the FAT. Maybe you can cross check if your device is willing to list them. Cheers, Oliver > Gesendet: Sonntag, 03. Januar 2021 um 23:08 Uhr > Von: "Steve Ratcliffe" > An: mkgmap-dev@lists.mkgmap.org.uk > Betreff: Re: [mkgmap-dev] Embedding raster maps > > > Hi Oliver > > > What is the minimum data an IMG file has to contain to be listed > > on the device? > > Its an interesting question. I don't remember having too many problems > getting a map recognised in the beginning. At r20 it worked > according to the commit comment but I only added the test map > builder at r24. > > However on my etrex 30 the map produced by r24 does not work. > > The first one that does work is r79. With a bit of experimenting > I discovered that adding a sufficiently large gap between the > end of the TRE header and the first data would make r78 work too. > > Going back to r24 and adding more padding also produced a map that > was recognized. I was able to remove the second copyright message > too and all map data. > > The required gap was between 40 and 101, depending on the rest of > the code. > > So this all makes very little sense and it is possible that this is > covering for something else that we don't understand. > > I've uploaded the minimal IMG file at: >http://files.mkgmap.org.uk/download/494/gmapsupp.img > > (After this I also managed to remove the first copyright message, so > you do not need any copyright messages and the label section can be > empty) > > With this version there is no message displayed on startup, which is > something that was fixed in r79 I think. > > There is just a single zoom and subdivision in the TRE section and > no points, line or polygons. > > Cheers, > Steve > ___ > mkgmap-dev mailing list > mkgmap-dev@lists.mkgmap.org.uk > http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev > ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Re: [mkgmap-dev] Embedding raster maps
On 04/01/2021 13:07, Oliver Eichler wrote: Hi Steve, I tested your map. It's not listed either. I have a GPSMap64s and placed it into the internal as well as into the external memory. Attempting to use the old code was probably a mistake, I've uploaded an empty map using a modern version of mkgmap: http://files.mkgmap.org.uk/detail/495 This is also in the proper gmapsupp format unlike the previous one. I made a minimum test.img, too. It contains no points, lines or areas. It has 3 zoom levels 22..24 with a sub-division each. I also uploaded a log from my test decoder. Maybe you can cross check if your device is willing to list them. My etrex30 does not list either of them. Re-packing the GMP file into a gmapsupp using mkgmap did not get it listed either. I believe that the byte at 0x40 is the start block of the FAT so in test.img it should be 2. Patching that byte did not make it load though. Cheers, Steve ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Re: [mkgmap-dev] Embedding raster maps
Hi Steve, that was really helpful. I managed to get my map listed. However the raster layer did not show up :( Well I didn't really expect it. ;) > > I believe that the byte at 0x40 is the start block of the FAT so in > test.img it should be 2. Patching that byte did not make it load > though. Indeed, it seems to be an offset in number of 0x200 blocks. In all my example maps it's set to 8 and FAT starts at 0x1000. In your map it's set to 2 and FAT starts at 0x400. Another mystery solved. I guess the next step is to get a rectangular extended polygon without the raster data to show up. The polygon type for the raster map tiles used is 0x10613. Usually it only contains a single point as the extended data referencing the jpeg tile has the four boundaries coded as 32 bit integers. My goal is to remove the extended data and to code all 4 corner coordinates. This should give me a matrix of all rectangles. Again this raises the question about the minimum effort necessary to add a single polygon. In my naive point of view it would be just encoding it into the extended polygon section of the RGN file. Of course I tried it and it does not work. So I am missing something, or have a couple of more bugs in my code. Would it be possible to code another file with a single polygon QPolygonF( QPointF(11.7914,46.9628) QPointF(11.8021,46.9628) QPointF(11.8021,46.9555) QPointF(11.7914,46.9555) ) coded in three map levels: 0/24bit, 1/23bit and 2/22bit. This would give me a nice working reference to compare. Thanks & Cheers Oliver ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Re: [mkgmap-dev] Embedding raster maps
Hi Steve, thanks for the file. On my device nothing is visible at all. But that's kind of an information, too. There is no way to circle down bugs by simply skipping the extended part. Probably it is still some missing encoding. In the Isle of Man map (and others) there is much more information in the RGN header. Note the values in the "filling bytes". And there seems to be an additional section after the extended point2 section. I called it offset1/length2. --- RGN header --- size 007D type 47 41 52 4D 49 4E 20 52 47 4E | GARMIN RGN byte0x000C 01 flag 00 year 2010 month 7 day 21 hour 9 minute 53 second 6 offset1 D54E length1 offset_polyg2 D54E length_polyg2 1A1F byte0x0025_0x0038 02 00 00 00 00 00 00 00 FF 00 00 20 FD FC 03 00 00 00 00 00 offset_polyl2 EF6D length_polyl2 byte0x0041_0x0054 00 00 00 00 00 00 00 00 3F 00 00 20 FD 0F 00 00 00 00 00 00 offset_point2 EF6D length_point2 byte0x005D_0x0070 00 00 00 00 00 00 00 00 FF 07 00 20 3F F7 3F 00 00 00 00 00 offset2 EF6D length2 020B unknown 00E3 This section has a variable length in different sub-files. The unknown value E3 can be E5, too. It doesn't seem to be a record size. I had a look at several sections. They are not completely random, thus there seem to be no "compression". However the patterns do not ring a bell for me. I copied a couple of them into a file and loaded it into my cloud share: https://www.magentacloud.de/share/s429h1z-r6 Maybe one of you has an idea what it's about. I also uploaded my current version of the test map with a single raster map tile. It does not show. Top left coordinate is N46° 57.768 E011° 47.484. Sadly I am stuck here and I am running out of quality time to concentrate on the topic. I also have to update QMapShack's decoding. While reading the mkgmap source code I noticed a lot of new stuff that is not covered by the decoder. Cheers Oliver ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Re: [mkgmap-dev] Embedding raster maps
Hi Oliver > thanks for the file. On my device nothing is visible at all. But > that's kind of an information, too. There is no way to circle down > bugs by simply skipping the extended part. To be clear, nothing is visible on my device either with that polygon type. Only if I change the type to something else (I used 0x10101) do I see anything. As far as I know extended polygons work in mkgmap and I just assumed that it is a reserved type for the images or something similar. > Probably it is still some missing encoding. In the Isle of Man map > (and others) there is much more information in the RGN header. Note > the values in the "filling bytes". And there seems to be an > additional section after the extended point2 section. I called it > offset1/length2. mkgmap sets all of these to zero and seems to work. Of course it may not work for your purpose with the images. I would try taking a copy of a working map and zeroing out some of the unknown things until it stops working. You will then know if something is essential or not. I have no ideas about what byte0x0025_0x0038 could be. They look like bitmaps or flags to me. > Maybe one of you has an idea what it's about. I also uploaded my I've had a go at displaying this section and I have uploaded a sample to: http://files.mkgmap.org.uk/detail/497 There is an initial part followed by two lists, the first has five byte records and the second 4 bytes. My guess is that the first one is related to polygon types and the second is some kind of index sorted by type or by zoom and/or division. Presumably it points into the extended polygon section, but I do not have anything that displays that section at the moment so I can't yet match anything up. The code I used is in the method RngDisplay::displayRgn5() in the display project. (The particular file is at https://svn.mkgmap.org.uk/mkgmap/display/trunk/src/test/display/RgnDisplay.java) This works with the files that are available to me, maybe not on anything else :) Cheers, Steve ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Re: [mkgmap-dev] Embedding raster maps
Hi Steve, > mkgmap sets all of these to zero and seems to work. Of course it may > not work for your purpose with the images. I would try taking a > copy of a working map and zeroing out some of the unknown things > until it stops working. You will then know if something is essential > or not. Done that. Removing the "flags" seem to have no impact. Removing the size of this 5th RGN section does. Therefor it is important :) > Presumably it points into the extended polygon section, but I do not > have anything that displays that section at the moment so I can't > yet match anything up. Ok, I will think about it. Just from my understanding also the information how these raster tiles fit into the draw order is missing. Maybe it's hard coded or set by that section. I also noted that QMapShack is probably decoding the extended polylines completely wrong. (Just wonder why no one complained so far ^^ ). I saw in the code that there is some test code. Is there already some way to create a test img for extended polylines and polgons that I can use as a reference to fix my code? This would be of great help. Cheers Oliver ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Re: [mkgmap-dev] Embedding raster maps
Hi Steve, if I got that right data in RGN5 is like MDR16, means, it contains Huffman decoding data. At least a lot of bytes look similar and GPXSee also gets the decoding data from it. Most of the code that displays MDR16 can be used to read RGN5, too. Gerd Von: mkgmap-dev im Auftrag von Steve Ratcliffe Gesendet: Donnerstag, 7. Januar 2021 11:31 An: mkgmap-dev@lists.mkgmap.org.uk Betreff: Re: [mkgmap-dev] Embedding raster maps Hi Oliver > thanks for the file. On my device nothing is visible at all. But > that's kind of an information, too. There is no way to circle down > bugs by simply skipping the extended part. To be clear, nothing is visible on my device either with that polygon type. Only if I change the type to something else (I used 0x10101) do I see anything. As far as I know extended polygons work in mkgmap and I just assumed that it is a reserved type for the images or something similar. > Probably it is still some missing encoding. In the Isle of Man map > (and others) there is much more information in the RGN header. Note > the values in the "filling bytes". And there seems to be an > additional section after the extended point2 section. I called it > offset1/length2. mkgmap sets all of these to zero and seems to work. Of course it may not work for your purpose with the images. I would try taking a copy of a working map and zeroing out some of the unknown things until it stops working. You will then know if something is essential or not. I have no ideas about what byte0x0025_0x0038 could be. They look like bitmaps or flags to me. > Maybe one of you has an idea what it's about. I also uploaded my I've had a go at displaying this section and I have uploaded a sample to: http://files.mkgmap.org.uk/detail/497 There is an initial part followed by two lists, the first has five byte records and the second 4 bytes. My guess is that the first one is related to polygon types and the second is some kind of index sorted by type or by zoom and/or division. Presumably it points into the extended polygon section, but I do not have anything that displays that section at the moment so I can't yet match anything up. The code I used is in the method RngDisplay::displayRgn5() in the display project. (The particular file is at https://svn.mkgmap.org.uk/mkgmap/display/trunk/src/test/display/RgnDisplay.java) This works with the files that are available to me, maybe not on anything else :) Cheers, Steve ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk http://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Re: [mkgmap-dev] Embedding raster maps
Hi Gerd When I looked at this (not very closely) a couple of weeks ago, it seemed as if LBL could be huffman encoded, with ENCODING_FORMAT set to 11, and the decoding table somewhere in RGN Ticker On Tue, 2022-01-25 at 08:10 +, Gerd Petermann wrote: > Hi Steve, > > if I got that right data in RGN5 is like MDR16, means, it contains > Huffman decoding data. At least a lot of bytes look similar > and GPXSee also gets the decoding data from it. > > Most of the code that displays MDR16 can be used to read RGN5, too. > > Gerd ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Re: [mkgmap-dev] Embedding raster maps
Hi Ticker, yes, that's right. I am working on an implementation, but it will take a while ... The Huffman table in the Ile of man map is probably used to decode the raster image. I has a symbol width of 32. Just starting to learn how it works, GPXSee is a great help but the code is hard to read. Gerd Von: mkgmap-dev im Auftrag von Ticker Berkin Gesendet: Dienstag, 25. Januar 2022 09:45 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] Embedding raster maps Hi Gerd When I looked at this (not very closely) a couple of weeks ago, it seemed as if LBL could be huffman encoded, with ENCODING_FORMAT set to 11, and the decoding table somewhere in RGN Ticker On Tue, 2022-01-25 at 08:10 +, Gerd Petermann wrote: > Hi Steve, > > if I got that right data in RGN5 is like MDR16, means, it contains > Huffman decoding data. At least a lot of bytes look similar > and GPXSee also gets the decoding data from it. > > Most of the code that displays MDR16 can be used to read RGN5, too. > > Gerd ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Re: [mkgmap-dev] Embedding raster maps
Hi developers, yes, RGN5 contains one or more Huffman decoding sections. For the Ile of man map there is only one table with symbol width 32, in a CN Europe NTU tile I find 5 different tables, two with symbol width 8 and three with 32. A lot more to learn here.. Gerd Von: mkgmap-dev im Auftrag von Gerd Petermann Gesendet: Dienstag, 25. Januar 2022 09:53 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] Embedding raster maps Hi Ticker, yes, that's right. I am working on an implementation, but it will take a while ... The Huffman table in the Ile of man map is probably used to decode the raster image. I has a symbol width of 32. Just starting to learn how it works, GPXSee is a great help but the code is hard to read. Gerd Von: mkgmap-dev im Auftrag von Ticker Berkin Gesendet: Dienstag, 25. Januar 2022 09:45 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] Embedding raster maps Hi Gerd When I looked at this (not very closely) a couple of weeks ago, it seemed as if LBL could be huffman encoded, with ENCODING_FORMAT set to 11, and the decoding table somewhere in RGN Ticker On Tue, 2022-01-25 at 08:10 +, Gerd Petermann wrote: > Hi Steve, > > if I got that right data in RGN5 is like MDR16, means, it contains > Huffman decoding data. At least a lot of bytes look similar > and GPXSee also gets the decoding data from it. > > Most of the code that displays MDR16 can be used to read RGN5, too. > > Gerd ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
