Re: [mkgmap-dev] [mkgmap-svn] Commit r572: MDR16 is some kind of codebook.
Hi Ticker, sorry, just noticed that something went wrong with copy because of the \0 character. Anyway, I think I understand the meaning of the part with the prefixes. I assume that Garmin reads the first 5 bits and uses this value as an index into a table which directly follows the first array. This 2nd table is a 32x2 lookup table, where the 2nd byte gives the value and the first byte some kind of status info which is used to position the bit reader. This would explain the repeating characters. Something like this: - MDR 16 (decompression codebook Huffman tree) - 02b6 | 00 | 4a | ??? 02b7 | 01 | 03 | ??? 02b8 | 02 | 15 | ??? 02b9 | 03 | 14 | 1st level with struct: 20 02ba | 04 | 0f 08 85| ??? (2nd byte always 8?) 02bd | 07 | 00 00 00| struct for 19 02c0 | 0a | 14 | struct level: 20 02c1 | 0b | 00 | count byte 0, delta = 0 02c2 | 0c | 06 00 00| struct for 19 02c5 | 0f | 13 | struct level: 19 02c6 | 10 | 06 | count byte 6, delta = 6 02c7 | 11 | 18 00 00| struct for 18 02ca | 14 | 12 | struct level: 18 02cb | 15 | 0f | count byte 15, delta = 9 02cc | 16 | 30 00 00| struct for 17 02cf | 19 | 11 | struct level: 17 02d0 | 1a | 15 | count byte 21, delta = 6 02d1 | 1b | 50 00 00| struct for 16 02d4 | 1e | 10 | struct level: 16 02d5 | 1f | 19 | count byte 25, delta = 4 02d6 | 20 | 80 00 00| struct for 15 02d9 | 23 | 0f | struct level: 15 02da | 24 | 1c | count byte 28, delta = 3 02db | 25 | c0 00 00| struct for 14 02de | 28 | 0e | struct level: 14 02df | 29 | 1e | count byte 30, delta = 2 02e0 | 2a | 00 02 00| struct for 13 02e3 | 2d | 0d | struct level: 13 02e4 | 2e | 23 | count byte 35, delta = 5 02e5 | 2f | 00 03 00| struct for 12 02e8 | 32 | 0c | struct level: 12 02e9 | 33 | 25 | count byte 37, delta = 2 02ea | 34 | 00 06 00| struct for 11 02ed | 37 | 0b | struct level: 11 02ee | 38 | 28 | count byte 40, delta = 3 02ef | 39 | 00 0c 00| struct for 10 02f2 | 3c | 0a | struct level: 10 02f3 | 3d | 2b | count byte 43, delta = 3 02f4 | 3e | 00 10 00| struct for 9 02f7 | 41 | 09 | struct level: 9 02f8 | 42 | 2c | count byte 44, delta = 1 02f9 | 43 | 00 30 00| struct for 8 02fc | 46 | 08 | struct level: 8 02fd | 47 | 30 | count byte 48, delta = 4 02fe | 48 | 00 c0 00| struct for 7 0301 | 4b | 07 | struct level: 7 0302 | 4c | 39 | count byte 57, delta = 9 0303 | 4d | 00 40 01| struct for 6 0306 | 50 | 06 | struct level: 6 0307 | 51 | 3d | count byte 61, delta = 4 0308 | 52 | 00 0c | prefix 0 : stat=0 v=' ', 0xc 030a | 54 | 18 0d | prefix 1 : stat=24 v=' ', 0xd 030c | 56 | 1a 0e | prefix 00010 : stat=26 v='', 0xe 030e | 58 | 1c 0e | prefix 00011 : stat=28 v='', 0xe 0310 | 5a | 1c 0e | prefix 00100 : stat=28 v='', 0xe 0312 | 5c | 0b 43 | prefix 00101 : stat=11 v='C', 0x43 0314 | 5e | 0b 53 | prefix 00110 : stat=11 v='S', 0x53 0316 | 60 | 0b 44 | prefix 00111 : stat=11 v='D', 0x44 0318 | 62 | 0b 54 | prefix 01000 : stat=11 v='T', 0x54 031a | 64 | 0b 4d | prefix 01001 : stat=11 v='M', 0x4d 031c | 66 | 0b 56 | prefix 01010 : stat=11 v='V', 0x56 031e | 68 | 0b 4a | prefix 01011 : stat=11 v='J', 0x4a 0320 | 6a | 0b 4b | prefix 01100 : stat=11 v='K', 0x4b 0322 | 6c | 0b 4c | prefix 01101 : stat=11 v='L', 0x4c 0324 | 6e | 09 00 | prefix 01110 : stat=9
Re: [mkgmap-dev] [mkgmap-svn] Commit r572: MDR16 is some kind of codebook.
Hi Ticker, I think those are the two that I didn't post before. Gerd Von: mkgmap-dev im Auftrag von Ticker Berkin Gesendet: Mittwoch, 22. Dezember 2021 12:32 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] [mkgmap-svn]Commit r572: MDR16 is somekindof codebook. Hi Gerd Can you send me the Mdr16 display of some of the other maps you've been looking at. I'd like to try and find some meaning for bytes 0..2 and the prefix before the level 5 data. Thanks Ticker On Wed, 2021-12-22 at 08:43 +, Gerd Petermann wrote: > Hi Ticker, > > I also thought that some bytes may tells us the uncompressed size or > frequencies. I don't see any clear pattern so far... > > Some number for the Adria Topo Map: (Mdr 16 length 212) > Uncompressed size: 1.582.655 (0x18263e) , compressed: 1029077 > (0x0fb3d5) > Frequencies of the tree nodes: > > '' 0x3 freq: 1 (0x1) > '%' 0x25 freq: 1 (0x1) > ':' 0x3a freq: 1 (0x1) ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev - MDR 16 (decompression codebook Huffman tree) - 030e | 00 | 8e | ??? 030f | 01 | 02 | ??? 0310 | 02 | 15 | ??? 0311 | 03 | 10 | 1st level with struct: 16 0312 | 04 | 0a 08 6d 00 00 | ??? 0317 | 09 | 10 | struct level: 16 0318 | 0a | 00 | count byte 0, delta = 0 0319 | 0b | 06 00 | struct for 16 031b | 0d | 0f | struct level: 15 031c | 0e | 06 | count byte 6, delta = 6 031d | 0f | 14 00 | struct for 15 031f | 11 | 0e | struct level: 14 0320 | 12 | 0d | count byte 13, delta = 7 0321 | 13 | 38 00 | struct for 14 0323 | 15 | 0d | struct level: 13 0324 | 16 | 16 | count byte 22, delta = 9 0325 | 17 | 70 00 | struct for 13 0327 | 19 | 0c | struct level: 12 0328 | 1a | 1d | count byte 29, delta = 7 0329 | 1b | a0 00 | struct for 12 032b | 1d | 0b | struct level: 11 032c | 1e | 20 | count byte 32, delta = 3 032d | 1f | 00 01 | struct for 11 032f | 21 | 09 | struct level: 9 0330 | 22 | 23 | count byte 35, delta = 3 0331 | 23 | 00 02 | struct for 9 0333 | 25 | 08 | struct level: 8 0334 | 26 | 25 | count byte 37, delta = 2 0335 | 27 | 00 04 | struct for 8 0337 | 29 | 07 | struct level: 7 0338 | 2a | 27 | count byte 39, delta = 2 0339 | 2b | 00 0c | struct for 7 033b | 2d | 06 | struct level: 6 033c | 2e | 2b | count byte 43, delta = 4 033d | 2f | 00 08 | struct for 6 033f | 31 | 10 09 12 09 12 09 12 09 | ??? 0347 | 39 | 12 09 12 09 | 034b | 3d | 0b | prefix b 034c | 3e | 59 | value in tree 'Y' (0x59) 034d | 3f | 0b | prefix b 034e | 40 | 47 | value in tree 'G' (0x47) 034f | 41 | 0b | prefix b 0350 | 42 | 49 | value in tree 'I' (0x49) 0351 | 43 | 0b | prefix b 0352 | 44 | c1 | value in tree 'Ã' (0xc1) 0353 | 45 | 0b | prefix b 0354 | 46 | 5a | value in tree 'Z' (0x5a) 0355 | 47 | 0b | prefix b 0356 | 48 | 4b | value in tree 'K' (0x4b) 0357 | 49 | 0b | prefix b 0358 | 4a | 44 | value in tree 'D' (0x44) 0359 | 4b | 0b | prefix b 035a | 4c | 4f | value in tree 'O' (0x4f) 035b | 4d | 0b | prefix b 035c | 4e | 4e | value in tree 'N' (0x4e) 035d | 4f | 09 | prefix 9 035e | 50 | 54 | value in tree 'T' (0x54) 035f | 51 | 09 | prefix 9 0360 | 52 | 54 | value in tree 'T' (0x54) 0361 | 53 | 09 | prefix 9 0362 | 54
Re: [mkgmap-dev] [mkgmap-svn] Commit r572: MDR16 is some kind of codebook.
Hi Gerd Can you send me the Mdr16 display of some of the other maps you've been looking at. I'd like to try and find some meaning for bytes 0..2 and the prefix before the level 5 data. Thanks Ticker On Wed, 2021-12-22 at 08:43 +, Gerd Petermann wrote: > Hi Ticker, > > I also thought that some bytes may tells us the uncompressed size or > frequencies. I don't see any clear pattern so far... > > Some number for the Adria Topo Map: (Mdr 16 length 212) > Uncompressed size: 1.582.655 (0x18263e) , compressed: 1029077 > (0x0fb3d5) > Frequencies of the tree nodes: > > '' 0x3 freq: 1 (0x1) > '%' 0x25 freq: 1 (0x1) > ':' 0x3a freq: 1 (0x1) ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Re: [mkgmap-dev] [mkgmap-svn] Commit r572: MDR16 is some kind of codebook.
Hi Ticker,
I also thought that some bytes may tells us the uncompressed size or
frequencies. I don't see any clear pattern so far...
Some number for the Adria Topo Map: (Mdr 16 length 212)
Uncompressed size: 1.582.655 (0x18263e) , compressed: 1029077 (0x0fb3d5)
Frequencies of the tree nodes:
'' 0x3 freq: 1 (0x1)
'%' 0x25 freq: 1 (0x1)
':' 0x3a freq: 1 (0x1)
';' 0x3b freq: 1 (0x1)
'�' 0xfd freq: 1 (0x1)
'¸' 0xb8 freq: 1 (0x1)
'?' 0x3f freq: 2 (0x2)
'`' 0x60 freq: 2 (0x2)
'„' 0x1e freq: 2 (0x2)
'Ž' 0x7d freq: 3 (0x3)
'’' 0x19 freq: 2 (0x2)
'Ä' 0xc4 freq: 2 (0x2)
'Ø' 0xd8 freq: 2 (0x2)
'Ú' 0xda freq: 1 (0x1)
'Û' 0xdb freq: 2 (0x2)
'_' 0x5f freq: 4 (0x4)
'{' 0x7b freq: 6 (0x6)
'“' 0x1c freq: 4 (0x4)
'É' 0xc9 freq: 4 (0x4)
'Ð' 0xd0 freq: 5 (0x5)
'Ö' 0xd6 freq: 5 (0x5)
'+' 0x2b freq: 7 (0x7)
'–' 0x13 freq: 12 (0xc)
'Á' 0xc1 freq: 6 (0x6)
'Ü' 0xdc freq: 9 (0x9)
'' 0x4 freq: 19 (0x13)
'!' 0x21 freq: 18 (0x12)
'ž' 0x7e freq: 15 (0xf)
'Æ' 0xc6 freq: 43 (0x2b)
'È' 0xc8 freq: 34 (0x22)
'Q' 0x51 freq: 80 (0x50)
'Š' 0x60 freq: 49 (0x31)
''' 0x27 freq: 95 (0x5f)
'Ë' 0xcb freq: 51 (0x33)
'/' 0x2f freq: 64 (0x40)
'&' 0x26 freq: 153 (0x99)
' ' 0x1f freq: 195 (0xc3)
'W' 0x57 freq: 294 (0x126)
'X' 0x58 freq: 422 (0x1a6)
'' 0x5 freq: 242 (0xf2)
'Y' 0x59 freq: 719 (0x2cf)
'"' 0x22 freq: 614 (0x266)
'' 0x6 freq: 842 (0x34a)
'*' 0x2a freq: 1905 (0x771)
'F' 0x46 freq: 4481 (0x1181)
',' 0x2c freq: 2126 (0x84e)
'(' 0x28 freq: 2537 (0x9e9)
')' 0x29 freq: 2558 (0x9fe)
'5' 0x35 freq: 6778 (0x1a7a)
'6' 0x36 freq: 6634 (0x19ea)
'-' 0x2d freq: 6588 (0x19bc)
'7' 0x37 freq: 5972 (0x1754)
'3' 0x33 freq: 7478 (0x1d36)
'8' 0x38 freq: 5832 (0x16c8)
'9' 0x39 freq: 5314 (0x14c2)
'0' 0x30 freq: 6539 (0x198b)
'4' 0x34 freq: 7203 (0x1c23)
'H' 0x48 freq: 12992 (0x32c0)
'1' 0x31 freq: 13281 (0x33e1)
'2' 0x32 freq: 8713 (0x2209)
'.' 0x2e freq: 10184 (0x27c8)
'G' 0x47 freq: 24930 (0x6162)
'Z' 0x5a freq: 23284 (0x5af4)
'U' 0x55 freq: 31985 (0x7cf1)
'P' 0x50 freq: 34866 (0x8832)
'B' 0x42 freq: 33286 (0x8206)
'C' 0x43 freq: 66619 (0x1043b)
'S' 0x53 freq: 71047 (0x11587)
'D' 0x44 freq: 37462 (0x9256)
'T' 0x54 freq: 53719 (0xd1d7)
'M' 0x4d freq: 40220 (0x9d1c)
'V' 0x56 freq: 63113 (0xf689)
'J' 0x4a freq: 44778 (0xaeea)
'K' 0x4b freq: 63856 (0xf970)
'L' 0x4c freq: 63057 (0xf651)
'' 0x0 freq: 108543 (0x1a7ff)
' ' 0x20 freq: 139214 (0x21fce)
'E' 0x45 freq: 99070 (0x182fe)
'N' 0x4e freq: 79383 (0x13617)
'O' 0x4f freq: 101405 (0x18c1d)
'I' 0x49 freq: 113012 (0x1b974)
'R' 0x52 freq: 95298 (0x17442)
'A' 0x41 freq: 181900 (0x2c68c)
Maybe you find something new?
Gerd
Von: mkgmap-dev im Auftrag von Gerd
Petermann
Gesendet: Dienstag, 21. Dezember 2021 17:49
An: Development list for mkgmap
Betreff: Re: [mkgmap-dev] [mkgmap-svn]Commit r572: MDR16 is
somekindof codebook.
Hi Ticker,
looks good.
With your hints I was able to write a patch for MdrDisplay
that seems able to decode compressed mdr15 for my
maps which have mdr 16 section.
Maybe you have another map...
It also changes some details for mdr 30/31 and 32/33.
Open problem is the length of the gap between the end of data for the 6th level
and the first bytes for level 5 (prefix 0b).
I see either 7 or 12 or 5 bytes. Check the code that calculdates value for
variable skip.
The values might not at all depend on the depth of the tree.
Code is not at all optimized, it just started to work for me.
Lots of bytes in MDR 16 are ignored.
Gerd
Von: mkgmap-dev im Auftrag von Gerd
Petermann
Gesendet: Montag, 20. Dezember 2021 11:29
An: Development list for mkgmap
Betreff: Re: [mkgmap-dev] [mkgmap-svn]Commit r572: MDR16 is
somekindof codebook.
Hi Ticker,
yes, with your theory I found errors in my code. Here's the corrected version
// depth 3, prefix 0x07
addHuffmanNode("111", 'A');
// depth 4, prefix 0x09
addHuffmanNode("0111", '\0'); // 0x00
addHuffmanNode("1000", ' '); //0x20
addHuffmanNode("1001", 'E');
addHuffmanNode("1010", 'N');
addHuffmanNode("1011", 'O');
addHuffmanNode("1100", 'I');
addHuffmanNode("1101", 'R');
// depth 5, prefix 0x0b
addHuffmanNode("00101", 'C');
addHuffmanNode("00110", 'S');
addHuffmanNode("00111", 'D');
addHuffmanNode("01000", 'T');
addHuffmanNode("01001", 'M');
addHuffmanNode("01010", 'V');
addHuffmanNode("01011", 'J');
