Re: [mkgmap-dev] Commit r4836: Cut labels which are longer than 170 characters, but not the copyright message or labels from ExtTypeAttributes
Hi all, I've checked a few old Garmin maps and it seems that Garmin often used 50 bytes as a max length for labels and obviously names are cut to that length, I see truncated names like "BIBLIOTHÈQUE MUNICIPALE ET INTERUNIVERSITAIRE DE C" or "CANAL D'IRRIGATION DE REMOULINS À FOURQUES OU LE C" in the Topo France map (2008) Exception: The Adria Topo 2.4 (2011) has 90 bytes : "STUDIJ RESTAURACIJA I KONZERVACIJA ARTEFAKATA OD DRVA, PAPIRA, TEKSTILA, METALA I KERAMIKE" These are the lengths of labels in the index (MDR15). The search for these names works well. So, 170 is probably still far too high. Gerd Von: mkgmap-dev im Auftrag von svn commit Gesendet: Montag, 13. Dezember 2021 11:40 An: mkgmap-...@lists.mkgmap.org.uk; mkgmap-dev@lists.mkgmap.org.uk Betreff: [mkgmap-dev] Commit r4836: Cut labels which are longer than 170 characters, but not the copyright message or labels from ExtTypeAttributes Version mkgmap-r4836 was committed by gerd on Mon, 13 Dec 2021 Cut labels which are longer than 170 characters, but not the copyright message or labels from ExtTypeAttributes modified label_len170.patch http://www.mkgmap.org.uk/websvn/revision.php?repname=mkgmap=4836 ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Re: [mkgmap-dev] [mkgmap-svn] Commit r572: MDR16 is some kind of codebook.
Hi Ticker,
reg. "stat": my rule is a bit different but the result is the same:
for all odd values:
depth = usedBits = stat >> 1;
val is the 2nd byte.
for even stat values and rows where stat == val << 1 the decoder
has to read exactly 1 more bit to decide which character was encoded.
I've not yet understood how the stat or the val field can be used to find the
correct entry
in the byte array with the characters.
So, maybe the stat value should be shifted to the right to make some sense.
Gerd
Von: mkgmap-dev im Auftrag von Ticker
Berkin
Gesendet: Donnerstag, 23. Dezember 2021 15:56
An: Development list for mkgmap
Betreff: Re: [mkgmap-dev] [mkgmap-svn]Commit r572: MDR16 is
somekindof codebook.
Hi Gerd
Maybe the "is always 8" is the character width.
I think I understand parts of it.
Read 5 bits and look up, giving 2 bytes.
if first 3/5/6/9/b the final char in the second byte and return (b-
first)/2 to the bit stream. Otherwise the combination somehow indicate
a minimum number of more bits to read and where to start searching.
maybe the 'struct for level' 1 or 2 bytes is a bit flag of which levels
to look in. I notice (in topo_fr) it is normally a single or adjacent
bits except for 0xa, but there are no chars at level 19
prefix 0 doesn't quite follow the rules
Ticker
On Thu, 2021-12-23 at 09:40 +, Gerd Petermann wrote:
> Hi Ticker,
>
> ouch, didn't even read that one :( Would have saved me a lot
> of time as my first idea was a "jump table".
> Attached is the new patch and the corrected mdr16 outputs produced
> with this patch.
>
> So, besides the fields with ??? the only open question is the meaning
> of the struct bytes in the first table.
> I guess it will become clear when I try to use the lookup table in
> the decoder.
>
> Gerd
>
>
>
> Von: mkgmap-dev im Auftrag
> von Ticker Berkin
> Gesendet: Donnerstag, 23. Dezember 2021 09:31
> An: Development list for mkgmap
> Betreff: Re: [mkgmap-dev] [mkgmap-svn] Commit r572:
> MDR16 is somekindof codebook.
>
> Hi Gerd
>
> I guessed that it was the \0 that cut the file off.
>
> You mean something like answer 5 here:
>
> https://stackoverflow.com/questions/759707/efficient-way-of-storing-huffman-tree
>
>
> I looked at this earlier trying to work out if it was relevant but
> didn't make it fit - I should have tried harder.
>
> I still haven't spotted anything to determine the length of the 2 /3
> byte in before and in the struct for {length}
>
> Ticker
>
> On Thu, 2021-12-23 at 01:17 +, Gerd Petermann wrote:
> > Hi Ticker,
> >
> > sorry, just noticed that something went wrong with copy
> > because
> > of the \0 character.
> > Anyway, I think I understand the meaning of the part with the
> > prefixes.
> > I assume that Garmin reads the first 5 bits and uses this value as
> > an
> > index into a table
> > which directly follows the first array. This 2nd table is a 32x2
> > lookup
> > table, where the 2nd byte gives the value
> > and the first byte some kind of status info which is used to
> > position
> > the bit reader.
> > This would explain the repeating characters. Something like this:
> >
> > - MDR 16 (decompression codebook Huffman tree)
> >
> > -
> > 02b6 | 00 | 4a | ???
>
>
> ___
> mkgmap-dev mailing list
> mkgmap-dev@lists.mkgmap.org.uk
> https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
> ___
> mkgmap-dev mailing list
> mkgmap-dev@lists.mkgmap.org.uk
> https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
___
mkgmap-dev mailing list
mkgmap-dev@lists.mkgmap.org.uk
https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
___
mkgmap-dev mailing list
mkgmap-dev@lists.mkgmap.org.uk
https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Re: [mkgmap-dev] [mkgmap-svn] Commit r572: MDR16 is some kind of codebook.
Hi Gerd
Maybe the "is always 8" is the character width.
I think I understand parts of it.
Read 5 bits and look up, giving 2 bytes.
if first 3/5/6/9/b the final char in the second byte and return (b-
first)/2 to the bit stream. Otherwise the combination somehow indicate
a minimum number of more bits to read and where to start searching.
maybe the 'struct for level' 1 or 2 bytes is a bit flag of which levels
to look in. I notice (in topo_fr) it is normally a single or adjacent
bits except for 0xa, but there are no chars at level 19
prefix 0 doesn't quite follow the rules
Ticker
On Thu, 2021-12-23 at 09:40 +, Gerd Petermann wrote:
> Hi Ticker,
>
> ouch, didn't even read that one :( Would have saved me a lot
> of time as my first idea was a "jump table".
> Attached is the new patch and the corrected mdr16 outputs produced
> with this patch.
>
> So, besides the fields with ??? the only open question is the meaning
> of the struct bytes in the first table.
> I guess it will become clear when I try to use the lookup table in
> the decoder.
>
> Gerd
>
>
>
> Von: mkgmap-dev im Auftrag
> von Ticker Berkin
> Gesendet: Donnerstag, 23. Dezember 2021 09:31
> An: Development list for mkgmap
> Betreff: Re: [mkgmap-dev] [mkgmap-svn] Commit r572:
> MDR16 is some kind of codebook.
>
> Hi Gerd
>
> I guessed that it was the \0 that cut the file off.
>
> You mean something like answer 5 here:
>
> https://stackoverflow.com/questions/759707/efficient-way-of-storing-huffman-tree
>
>
> I looked at this earlier trying to work out if it was relevant but
> didn't make it fit - I should have tried harder.
>
> I still haven't spotted anything to determine the length of the 2 /3
> byte in before and in the struct for {length}
>
> Ticker
>
> On Thu, 2021-12-23 at 01:17 +, Gerd Petermann wrote:
> > Hi Ticker,
> >
> > sorry, just noticed that something went wrong with copy
> > because
> > of the \0 character.
> > Anyway, I think I understand the meaning of the part with the
> > prefixes.
> > I assume that Garmin reads the first 5 bits and uses this value as
> > an
> > index into a table
> > which directly follows the first array. This 2nd table is a 32x2
> > lookup
> > table, where the 2nd byte gives the value
> > and the first byte some kind of status info which is used to
> > position
> > the bit reader.
> > This would explain the repeating characters. Something like this:
> >
> > - MDR 16 (decompression codebook Huffman tree)
> >
> > -
> > 02b6 | 00 | 4a | ???
>
>
> ___
> mkgmap-dev mailing list
> mkgmap-dev@lists.mkgmap.org.uk
> https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
> ___
> mkgmap-dev mailing list
> mkgmap-dev@lists.mkgmap.org.uk
> https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
___
mkgmap-dev mailing list
mkgmap-dev@lists.mkgmap.org.uk
https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
Re: [mkgmap-dev] [mkgmap-svn] Commit r572: MDR16 is some kind of codebook.
Hi Ticker,
ouch, didn't even read that one :( Would have saved me a lot
of time as my first idea was a "jump table".
Attached is the new patch and the corrected mdr16 outputs produced with this
patch.
So, besides the fields with ??? the only open question is the meaning of the
struct bytes in the first table.
I guess it will become clear when I try to use the lookup table in the decoder.
Gerd
Von: mkgmap-dev im Auftrag von Ticker
Berkin
Gesendet: Donnerstag, 23. Dezember 2021 09:31
An: Development list for mkgmap
Betreff: Re: [mkgmap-dev] [mkgmap-svn]Commit r572: MDR16 is
somekindof codebook.
Hi Gerd
I guessed that it was the \0 that cut the file off.
You mean something like answer 5 here:
https://stackoverflow.com/questions/759707/efficient-way-of-storing-huffman-tree
I looked at this earlier trying to work out if it was relevant but
didn't make it fit - I should have tried harder.
I still haven't spotted anything to determine the length of the 2 /3
byte in before and in the struct for {length}
Ticker
On Thu, 2021-12-23 at 01:17 +, Gerd Petermann wrote:
> Hi Ticker,
>
> sorry, just noticed that something went wrong with copy because
> of the \0 character.
> Anyway, I think I understand the meaning of the part with the prefixes.
> I assume that Garmin reads the first 5 bits and uses this value as an
> index into a table
> which directly follows the first array. This 2nd table is a 32x2 lookup
> table, where the 2nd byte gives the value
> and the first byte some kind of status info which is used to position
> the bit reader.
> This would explain the repeating characters. Something like this:
>
> - MDR 16 (decompression codebook Huffman tree)
> -
> 02b6 | 00 | 4a | ???
___
mkgmap-dev mailing list
mkgmap-dev@lists.mkgmap.org.uk
https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
mdr-experimental-v2.patch
Description: mdr-experimental-v2.patch
- MDR 16 (decompression codebook Huffman tree) -
030e | 00 | 8e | ???
030f | 01 | 02 | ???
0310 | 02 | 15 | ???
0311 | 03 | 10 | 1st level with struct: 16
0312 | 04 | 0a 08 6d| ??? (2nd byte always 8?)
0315 | 07 | 00 00 | struct for 16
0317 | 09 | 10 | level: 16
0318 | 0a | 00 | count byte 0, delta = 0
0319 | 0b | 06 00 | struct for 16
031b | 0d | 0f | level: 15
031c | 0e | 06 | count byte 6, delta = 6
031d | 0f | 14 00 | struct for 15
031f | 11 | 0e | level: 14
0320 | 12 | 0d | count byte 13, delta = 7
0321 | 13 | 38 00 | struct for 14
0323 | 15 | 0d | level: 13
0324 | 16 | 16 | count byte 22, delta = 9
0325 | 17 | 70 00 | struct for 13
0327 | 19 | 0c | level: 12
0328 | 1a | 1d | count byte 29, delta = 7
0329 | 1b | a0 00 | struct for 12
032b | 1d | 0b | level: 11
032c | 1e | 20 | count byte 32, delta = 3
032d | 1f | 00 01 | struct for 11
032f | 21 | 09 | level: 9
0330 | 22 | 23 | count byte 35, delta = 3
0331 | 23 | 00 02 | struct for 9
0333 | 25 | 08 | level: 8
0334 | 26 | 25 | count byte 37, delta = 2
0335 | 27 | 00 04 | struct for 8
0337 | 29 | 07 | level: 7
0338 | 2a | 27 | count byte 39, delta = 2
0339 | 2b | 00 0c | struct for 7
033b | 2d | 06 | level: 6
033c | 2e | 2b | count byte 43, delta = 4
|| |
lookup table
033d | 2f | 00 08 | prefix 0 : stat=0 v='.', 0x8
033f | 31 | 10 09 | prefix 1 : stat=16 v='.', 0x9
0341 | 33 | 12 09 | prefix 00010 : stat=18 v='.', 0x9
0343 | 35 | 12 09 | prefix 00011 : stat=18 v='.', 0x9
0345 | 37 | 12 09 | prefix 00100 : stat=18 v='.', 0x9
0347 | 39 | 12 09 | prefix 00101 : stat=18 v='.', 0x9
0349 | 3b | 12 09 | prefix 00110 : stat=18 v='.', 0x9
034b | 3d | 0b 59 |
Re: [mkgmap-dev] [mkgmap-svn] Commit r572: MDR16 is some kind of codebook.
Hi Gerd
I guessed that it was the \0 that cut the file off.
You mean something like answer 5 here:
https://stackoverflow.com/questions/759707/efficient-way-of-storing-huffman-tree
I looked at this earlier trying to work out if it was relevant but
didn't make it fit - I should have tried harder.
I still haven't spotted anything to determine the length of the 2 /3
byte in before and in the struct for {length}
Ticker
On Thu, 2021-12-23 at 01:17 +, Gerd Petermann wrote:
> Hi Ticker,
>
> sorry, just noticed that something went wrong with copy because
> of the \0 character.
> Anyway, I think I understand the meaning of the part with the prefixes.
> I assume that Garmin reads the first 5 bits and uses this value as an
> index into a table
> which directly follows the first array. This 2nd table is a 32x2 lookup
> table, where the 2nd byte gives the value
> and the first byte some kind of status info which is used to position
> the bit reader.
> This would explain the repeating characters. Something like this:
>
> - MDR 16 (decompression codebook Huffman tree)
> -
> 02b6 | 00 | 4a | ???
___
mkgmap-dev mailing list
mkgmap-dev@lists.mkgmap.org.uk
https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
