Re: [mkgmap-dev] New assertion, now with code-page=632 and Japan tile

[Gerd Petermann]

Hi Ticker,

no problem, still trying to find a case where mdrUnicode_v9b.patch changes 
something...

Gerd


Von: mkgmap-dev  im Auftrag von Ticker 
Berkin 
Gesendet: Freitag, 19. November 2021 08:46
An: Development list for mkgmap
Betreff: Re: [mkgmap-dev] New assertion, now with code-page=632 and Japan tile

Hi Gerd

Please ignore the CodeFunctions.java change in this patch.

A better / hybrid TableTranslitorator is needed before
SparseTranslitorator can go.

Ticker

On Thu, 2021-11-18 at 17:47 +, Ticker Berkin wrote:
> Hi Gerd
>
> For any code-page except Japanese/cp932, AnyCharSetEncoder takes
> anything that can't be represented, tries to find a reasonable ascii
> representation or "?", then writes this to the output. This is a big
> assumption for far-eastern charsets, most likely generating garbage
> with possible invalid shift-in/out requests...
>
> SparseTranslitorator is a very strange special case, without any
> explanation. Doing a bit of searching, it was submitted as a change
> because user had map that needed to be in Japanese/cp932 and it also
> contained latin characters. The characters with macrons couldn't be
> encoded. Many others could. The rest of Unicode that can't be encoded
> resulted in garbage.
>
> Your patch fixes the "rest of Unicode" problem for cp932. It misses
> any
> ability of the 'latin1' transliterator to provide reasonable
> replacement chars that can be encoded. It doesn't deal with possible
> problems for other (non-european) charsets.
>
> I've attached cs932-V3.patch that addresses both of these issues.
>
> SparseTranslitorator.java can the be removed.
>
> Ticker
>
> On Wed, 2021-11-17 at 18:00 +, Gerd Petermann wrote:
> > Hi Ticker,
> >
> > > For some other character sets the result could be invalid or
> > > garbage.
> > OK, I assumed that '?' is always at the same position, might be
> > wrong
> > with that.
> > SparseTransliterator is only used for cs932.
> >
> > Gerd
>
> ___
> mkgmap-dev mailing list
> mkgmap-dev@lists.mkgmap.org.uk
> https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev


___
mkgmap-dev mailing list
mkgmap-dev@lists.mkgmap.org.uk
https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
___
mkgmap-dev mailing list
mkgmap-dev@lists.mkgmap.org.uk
https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

Re: [mkgmap-dev] New assertion, now with code-page=632 and Japan tile

[Ticker Berkin]

Hi Gerd

Please ignore the CodeFunctions.java change in this patch.

A better / hybrid TableTranslitorator is needed before
SparseTranslitorator can go. 

Ticker

On Thu, 2021-11-18 at 17:47 +, Ticker Berkin wrote:
> Hi Gerd
>  
> For any code-page except Japanese/cp932, AnyCharSetEncoder takes
> anything that can't be represented, tries to find a reasonable ascii
> representation or "?", then writes this to the output. This is a big
> assumption for far-eastern charsets, most likely generating garbage
> with possible invalid shift-in/out requests...
> 
> SparseTranslitorator is a very strange special case, without any
> explanation. Doing a bit of searching, it was submitted as a change
> because user had map that needed to be in Japanese/cp932 and it also
> contained latin characters. The characters with macrons couldn't be
> encoded. Many others could. The rest of Unicode that can't be encoded
> resulted in garbage.
> 
> Your patch fixes the "rest of Unicode" problem for cp932. It misses
> any
> ability of the 'latin1' transliterator to provide reasonable
> replacement chars that can be encoded. It doesn't deal with possible
> problems for other (non-european) charsets.
> 
> I've attached cs932-V3.patch that addresses both of these issues.
> 
> SparseTranslitorator.java can the be removed.
> 
> Ticker
> 
> On Wed, 2021-11-17 at 18:00 +, Gerd Petermann wrote:
> > Hi Ticker,
> > 
> > > For some other character sets the result could be invalid or
> > > garbage.
> > OK, I assumed that '?' is always at the same position, might be
> > wrong
> > with that.
> > SparseTransliterator is only used for cs932.
> > 
> > Gerd
> 
> ___
> mkgmap-dev mailing list
> mkgmap-dev@lists.mkgmap.org.uk
> https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev


___
mkgmap-dev mailing list
mkgmap-dev@lists.mkgmap.org.uk
https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

Re: [mkgmap-dev] New assertion, now with code-page=632 and Japan tile

[Ticker Berkin]

Hi Gerd
 
For any code-page except Japanese/cp932, AnyCharSetEncoder takes
anything that can't be represented, tries to find a reasonable ascii
representation or "?", then writes this to the output. This is a big
assumption for far-eastern charsets, most likely generating garbage
with possible invalid shift-in/out requests...

SparseTranslitorator is a very strange special case, without any
explanation. Doing a bit of searching, it was submitted as a change
because user had map that needed to be in Japanese/cp932 and it also
contained latin characters. The characters with macrons couldn't be
encoded. Many others could. The rest of Unicode that can't be encoded
resulted in garbage.

Your patch fixes the "rest of Unicode" problem for cp932. It misses any
ability of the 'latin1' transliterator to provide reasonable
replacement chars that can be encoded. It doesn't deal with possible
problems for other (non-european) charsets.

I've attached cs932-V3.patch that addresses both of these issues.

SparseTranslitorator.java can the be removed.

Ticker

On Wed, 2021-11-17 at 18:00 +, Gerd Petermann wrote:
> Hi Ticker,
> 
> > For some other character sets the result could be invalid or
> > garbage.
> OK, I assumed that '?' is always at the same position, might be wrong
> with that.
> SparseTransliterator is only used for cs932.
> 
> Gerd

Index: src/uk/me/parabola/imgfmt/app/labelenc/AnyCharsetEncoder.java
===
--- src/uk/me/parabola/imgfmt/app/labelenc/AnyCharsetEncoder.java	(revision 4817)
+++ src/uk/me/parabola/imgfmt/app/labelenc/AnyCharsetEncoder.java	(working copy)
@@ -80,8 +80,9 @@
 if (result.length() == 1) {
 	s0 = String.valueOf(charBuffer.get());
 } else {
-	// Don't know under what circumstances this will be called and may not be the
-	// correct thing to do when it does happen.
+	// Probably a UTF-16 surrogate pair (represents a single unicode point)
+	// Handle as the general case, but the translitorator will need a extra code and
+	// tables to handle these.
 	StringBuilder sb = new StringBuilder();
 	for (int i = 0; i < result.length(); i++)
 		sb.append(charBuffer.get());
@@ -90,18 +91,27 @@
 }
 
 String s = transliterator.transliterate(s0);
-
-// Make sure that there is enough space for the transliterated string
-while (outBuf.limit() < outBuf.position() + s.length())
-	outBuf = reallocBuf(outBuf);
-
-if (s.equals(s0)) {
-	// string is still unmappable
-	outBuf.put(encoder.replacement()); //typically '?'
+if (s.equals(s0)) { // string is still unmappable
+	// however, at moment, TableTranslitorator returns a "?" so won't happen
+	// Make sure that there is enough space for the replacement
+	while (outBuf.limit() < outBuf.position() + encoder.replacement().length)
+		outBuf = reallocBuf(outBuf);
+	outBuf.put(encoder.replacement()); // typically '?'
 } else {
-	for (int i = 0; i < s.length(); i++) {
-		outBuf.put((byte) s.charAt(i));
-	}
+	// some transliteration has happened so see if this can be encoded
+	CharBuffer translitBuffer = CharBuffer.wrap(s);
+	do {
+		result = encoder.encode(translitBuffer, outBuf, true);
+		if (result.isUnmappable()) { // some can't be encoded
+			while (outBuf.limit() < outBuf.position() + encoder.replacement().length)
+outBuf = reallocBuf(outBuf);
+			outBuf.put(encoder.replacement());
+			break;  // and give up with this inner part
+		} else if (result == CoderResult.OVERFLOW) {
+			// Ran out of space in the output
+			outBuf = reallocBuf(outBuf);
+		}
+	} while (result != CoderResult.UNDERFLOW);
 }
 
 			} else if (result == CoderResult.OVERFLOW) {
Index: src/uk/me/parabola/imgfmt/app/labelenc/CodeFunctions.java
===
--- src/uk/me/parabola/imgfmt/app/labelenc/CodeFunctions.java	(revision 4817)
+++ src/uk/me/parabola/imgfmt/app/labelenc/CodeFunctions.java	(working copy)
@@ -101,10 +101,17 @@
 		case "cp932":
 		case "ms932":
 			funcs.setEncodingType(ENCODING_FORMAT10);
-			funcs.setEncoder(new AnyCharsetEncoder("ms932", new SparseTransliterator("nomacron")));
+			//funcs.setEncoder(new AnyCharsetEncoder("ms932", new SparseTransliterator("nomacron")));
+			// Note: above was a strange special case - it just removed macrons because it was known that
+			// cp932 didn't support them.
+			// However, there are many other unicode values that cp932 doesnt support so use more
+			// intelligent login in AnyCharSetCoder to find more complete tables of mapping from latin1
+			funcs.setEncoder(new AnyCharsetEncoder("ms932", new TableTransliterator("latin1")));
 			funcs.setDecoder(new AnyCharsetDecoder("ms932"));
 			funcs.setCodepage(932);
 			break;
+	//	case "other far-eastern code pages":
+			// probably ENCODING_FORMAT10 if variable width and generally like above
 		

Re: [mkgmap-dev] New assertion, now with code-page=632 and Japan tile

[Gerd Petermann]

Hi Ticker,

> For some other character sets the result could be invalid or garbage.
OK, I assumed that '?' is always at the same position, might be wrong with that.
SparseTransliterator is only used for cs932.

Gerd


Von: mkgmap-dev  im Auftrag von Ticker 
Berkin 
Gesendet: Mittwoch, 17. November 2021 18:23
An: Development list for mkgmap
Betreff: Re: [mkgmap-dev] New assertion, now with code-page=632 and Japan tile

Hi Gerd

It makes a big assumption that transliterated chars can have their low
byte written as a sequence that will be taken as some other charset.

For cp932, because so few chars are transliterated, and, by chance,
these have the same representation, it won't crash or give an invalid
sequence, but will be strange that only macron chars, with the macron
removed, will show amongst the "?"s.

For some other character sets the result could be invalid or garbage.

Ticker

On Wed, 2021-11-17 at 16:40 +, Gerd Petermann wrote:
> Hi Ticker,
>
> yes, sure, a lot of unicode characters cannot be represented in
> cs932. SparseTransliterator only handles 5 of them.
> All others are now translated to ? instead of a more or less
> unpredictable character.
> My patch doesn't try to implement a good transliteration, just a
> better handling of unmapped chars.
>
> Gerd
>
> 
> Von: mkgmap-dev  im Auftrag
> von Ticker Berkin 
> Gesendet: Mittwoch, 17. November 2021 17:35
> An: Development list for mkgmap
> Betreff: Re: [mkgmap-dev] New assertion, now with code-page=632 and
> Japan tile
>
> Hi Gerd
>
> Not quite - The transliteration / "?" doesn't get encoded into the
> target charset.
>
> In this case with cp932, there seems to be an assumption that
> SparseTranslitorator will convert all unicode chars that are not in
> CP932. There must be lots of these.
>
> Ticker
>
> On Wed, 2021-11-17 at 16:00 +, Gerd Petermann wrote:
> > Hi Ticker,
> >
> > result.length() works and most times returns 1, sometimes higher
> > values for unicode characters which cannot be represented by a
> > single
> > char.
> >
> > OK to commit v2?
> > Gerd
> >
> > ____________________
> > Von: mkgmap-dev  im Auftrag
> > von Ticker Berkin 
> > Gesendet: Mittwoch, 17. November 2021 15:37
> > An: Development list for mkgmap
> > Betreff: Re: [mkgmap-dev] New assertion, now with code-page=632 and
> > Japan tile
> >
> > Hi Gerd
> >
> > My description didn't quite mean what I hoped it did - sorry. I was
> > thinking that there would be a single attempt at encoding the whole
> > string, and if that fails, start again but char-by-char.
> >
> > But, assuming result.length() works and charBuffer.get() and
> > outBuff.put() maintain positions used by main encoder, within the
> > loop
> > the failed component needs to be processed input char-by-char,
> > transliterated (if no change replaced by "?") and encoded with
> > another
> > encoder.
> >
> > Any variable length nature of the output charset shouldn't be a
> > problem. The variable length input UTF-16 will need care.
> >
> > Ticker
> >
> >
> > On Wed, 2021-11-17 at 11:16 +, Gerd Petermann wrote:
> > > Hi Ticker,
> > >
> > > remember that cs932 is a double-byte character set.
> > > With your code only a few unmappable utf-16 characters are
> > > replaced,
> > > for the rest one of cs932 is used, but without any good reason.
> > > The
> > > result is typically garbage.
> > >
> > > I've modified the patch to replace any  unmappable character that
> > > was
> > > not transliterated by '?' .
> > > I've also attached a debug version that shows what goes on.
> > > A possible change in SparseTransliterator would be to add a
> > > mapping
> > > for the MATH MINUS, the other FULLWIDTH digits are supported in
> > > cs932.
> > >
> > > Gerd
> >
> >
> > ___
> > mkgmap-dev mailing list
> > mkgmap-dev@lists.mkgmap.org.uk
> > https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
> > ___
> > mkgmap-dev mailing list
> > mkgmap-dev@lists.mkgmap.org.uk
> > https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
>
>
> ___
> mkgmap-dev mailing list
> mkgmap-dev@lists.mkgmap.org.uk
> https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
> ___
> mkgmap-dev mailing list
> mkgmap-dev@lists.mkgmap.org.uk
> https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev


___
mkgmap-dev mailing list
mkgmap-dev@lists.mkgmap.org.uk
https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
___
mkgmap-dev mailing list
mkgmap-dev@lists.mkgmap.org.uk
https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

Re: [mkgmap-dev] New assertion, now with code-page=632 and Japan tile

[Ticker Berkin]

Hi Gerd

It makes a big assumption that transliterated chars can have their low
byte written as a sequence that will be taken as some other charset.

For cp932, because so few chars are transliterated, and, by chance,
these have the same representation, it won't crash or give an invalid
sequence, but will be strange that only macron chars, with the macron
removed, will show amongst the "?"s.

For some other character sets the result could be invalid or garbage.

Ticker

On Wed, 2021-11-17 at 16:40 +, Gerd Petermann wrote:
> Hi Ticker,
> 
> yes, sure, a lot of unicode characters cannot be represented in
> cs932. SparseTransliterator only handles 5 of them.
> All others are now translated to ? instead of a more or less
> unpredictable character.
> My patch doesn't try to implement a good transliteration, just a
> better handling of unmapped chars.
> 
> Gerd
> 
> 
> Von: mkgmap-dev  im Auftrag
> von Ticker Berkin 
> Gesendet: Mittwoch, 17. November 2021 17:35
> An: Development list for mkgmap
> Betreff: Re: [mkgmap-dev] New assertion, now with code-page=632 and
> Japan tile
> 
> Hi Gerd
> 
> Not quite - The transliteration / "?" doesn't get encoded into the
> target charset.
> 
> In this case with cp932, there seems to be an assumption that
> SparseTranslitorator will convert all unicode chars that are not in
> CP932. There must be lots of these.
> 
> Ticker
> 
> On Wed, 2021-11-17 at 16:00 +, Gerd Petermann wrote:
> > Hi Ticker,
> > 
> > result.length() works and most times returns 1, sometimes higher
> > values for unicode characters which cannot be represented by a
> > single
> > char.
> > 
> > OK to commit v2?
> > Gerd
> > 
> > ____________________
> > Von: mkgmap-dev  im Auftrag
> > von Ticker Berkin 
> > Gesendet: Mittwoch, 17. November 2021 15:37
> > An: Development list for mkgmap
> > Betreff: Re: [mkgmap-dev] New assertion, now with code-page=632 and
> > Japan tile
> > 
> > Hi Gerd
> > 
> > My description didn't quite mean what I hoped it did - sorry. I was
> > thinking that there would be a single attempt at encoding the whole
> > string, and if that fails, start again but char-by-char.
> > 
> > But, assuming result.length() works and charBuffer.get() and
> > outBuff.put() maintain positions used by main encoder, within the
> > loop
> > the failed component needs to be processed input char-by-char,
> > transliterated (if no change replaced by "?") and encoded with
> > another
> > encoder.
> > 
> > Any variable length nature of the output charset shouldn't be a
> > problem. The variable length input UTF-16 will need care.
> > 
> > Ticker
> > 
> > 
> > On Wed, 2021-11-17 at 11:16 +, Gerd Petermann wrote:
> > > Hi Ticker,
> > > 
> > > remember that cs932 is a double-byte character set.
> > > With your code only a few unmappable utf-16 characters are
> > > replaced,
> > > for the rest one of cs932 is used, but without any good reason.
> > > The
> > > result is typically garbage.
> > > 
> > > I've modified the patch to replace any  unmappable character that
> > > was
> > > not transliterated by '?' .
> > > I've also attached a debug version that shows what goes on.
> > > A possible change in SparseTransliterator would be to add a
> > > mapping
> > > for the MATH MINUS, the other FULLWIDTH digits are supported in
> > > cs932.
> > > 
> > > Gerd
> > 
> > 
> > ___
> > mkgmap-dev mailing list
> > mkgmap-dev@lists.mkgmap.org.uk
> > https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
> > ___
> > mkgmap-dev mailing list
> > mkgmap-dev@lists.mkgmap.org.uk
> > https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
> 
> 
> ___
> mkgmap-dev mailing list
> mkgmap-dev@lists.mkgmap.org.uk
> https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
> ___
> mkgmap-dev mailing list
> mkgmap-dev@lists.mkgmap.org.uk
> https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev


___
mkgmap-dev mailing list
mkgmap-dev@lists.mkgmap.org.uk
https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

Re: [mkgmap-dev] New assertion, now with code-page=632 and Japan tile

[Gerd Petermann]

Hi Ticker,

yes, sure, a lot of unicode characters cannot be represented in cs932. 
SparseTransliterator only handles 5 of them.
All others are now translated to ? instead of a more or less unpredictable 
character.
My patch doesn't try to implement a good transliteration, just a better 
handling of unmapped chars.

Gerd


Von: mkgmap-dev  im Auftrag von Ticker 
Berkin 
Gesendet: Mittwoch, 17. November 2021 17:35
An: Development list for mkgmap
Betreff: Re: [mkgmap-dev] New assertion, now with code-page=632 and Japan tile

Hi Gerd

Not quite - The transliteration / "?" doesn't get encoded into the
target charset.

In this case with cp932, there seems to be an assumption that
SparseTranslitorator will convert all unicode chars that are not in
CP932. There must be lots of these.

Ticker

On Wed, 2021-11-17 at 16:00 +, Gerd Petermann wrote:
> Hi Ticker,
>
> result.length() works and most times returns 1, sometimes higher
> values for unicode characters which cannot be represented by a single
> char.
>
> OK to commit v2?
> Gerd
>
> 
> Von: mkgmap-dev  im Auftrag
> von Ticker Berkin 
> Gesendet: Mittwoch, 17. November 2021 15:37
> An: Development list for mkgmap
> Betreff: Re: [mkgmap-dev] New assertion, now with code-page=632 and
> Japan tile
>
> Hi Gerd
>
> My description didn't quite mean what I hoped it did - sorry. I was
> thinking that there would be a single attempt at encoding the whole
> string, and if that fails, start again but char-by-char.
>
> But, assuming result.length() works and charBuffer.get() and
> outBuff.put() maintain positions used by main encoder, within the
> loop
> the failed component needs to be processed input char-by-char,
> transliterated (if no change replaced by "?") and encoded with
> another
> encoder.
>
> Any variable length nature of the output charset shouldn't be a
> problem. The variable length input UTF-16 will need care.
>
> Ticker
>
>
> On Wed, 2021-11-17 at 11:16 +, Gerd Petermann wrote:
> > Hi Ticker,
> >
> > remember that cs932 is a double-byte character set.
> > With your code only a few unmappable utf-16 characters are
> > replaced,
> > for the rest one of cs932 is used, but without any good reason. The
> > result is typically garbage.
> >
> > I've modified the patch to replace any  unmappable character that
> > was
> > not transliterated by '?' .
> > I've also attached a debug version that shows what goes on.
> > A possible change in SparseTransliterator would be to add a mapping
> > for the MATH MINUS, the other FULLWIDTH digits are supported in
> > cs932.
> >
> > Gerd
>
>
> ___
> mkgmap-dev mailing list
> mkgmap-dev@lists.mkgmap.org.uk
> https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
> ___
> mkgmap-dev mailing list
> mkgmap-dev@lists.mkgmap.org.uk
> https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev


___
mkgmap-dev mailing list
mkgmap-dev@lists.mkgmap.org.uk
https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
___
mkgmap-dev mailing list
mkgmap-dev@lists.mkgmap.org.uk
https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

Re: [mkgmap-dev] New assertion, now with code-page=632 and Japan tile

[Ticker Berkin]

Hi Gerd

Not quite - The transliteration / "?" doesn't get encoded into the
target charset.

In this case with cp932, there seems to be an assumption that
SparseTranslitorator will convert all unicode chars that are not in
CP932. There must be lots of these.

Ticker

On Wed, 2021-11-17 at 16:00 +, Gerd Petermann wrote:
> Hi Ticker,
> 
> result.length() works and most times returns 1, sometimes higher
> values for unicode characters which cannot be represented by a single
> char.
> 
> OK to commit v2?
> Gerd
> 
> 
> Von: mkgmap-dev  im Auftrag
> von Ticker Berkin 
> Gesendet: Mittwoch, 17. November 2021 15:37
> An: Development list for mkgmap
> Betreff: Re: [mkgmap-dev] New assertion, now with code-page=632 and
> Japan tile
> 
> Hi Gerd
> 
> My description didn't quite mean what I hoped it did - sorry. I was
> thinking that there would be a single attempt at encoding the whole
> string, and if that fails, start again but char-by-char.
> 
> But, assuming result.length() works and charBuffer.get() and
> outBuff.put() maintain positions used by main encoder, within the
> loop
> the failed component needs to be processed input char-by-char,
> transliterated (if no change replaced by "?") and encoded with
> another
> encoder.
> 
> Any variable length nature of the output charset shouldn't be a
> problem. The variable length input UTF-16 will need care.
> 
> Ticker
> 
> 
> On Wed, 2021-11-17 at 11:16 +, Gerd Petermann wrote:
> > Hi Ticker,
> > 
> > remember that cs932 is a double-byte character set.
> > With your code only a few unmappable utf-16 characters are
> > replaced,
> > for the rest one of cs932 is used, but without any good reason. The
> > result is typically garbage.
> > 
> > I've modified the patch to replace any  unmappable character that
> > was
> > not transliterated by '?' .
> > I've also attached a debug version that shows what goes on.
> > A possible change in SparseTransliterator would be to add a mapping
> > for the MATH MINUS, the other FULLWIDTH digits are supported in
> > cs932.
> > 
> > Gerd
> 
> 
> ___
> mkgmap-dev mailing list
> mkgmap-dev@lists.mkgmap.org.uk
> https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
> ___
> mkgmap-dev mailing list
> mkgmap-dev@lists.mkgmap.org.uk
> https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev


___
mkgmap-dev mailing list
mkgmap-dev@lists.mkgmap.org.uk
https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

Re: [mkgmap-dev] New assertion, now with code-page=632 and Japan tile

[Gerd Petermann]

Hi Ticker,

result.length() works and most times returns 1, sometimes higher values for 
unicode characters which cannot be represented by a single char.

OK to commit v2?
Gerd


Von: mkgmap-dev  im Auftrag von Ticker 
Berkin 
Gesendet: Mittwoch, 17. November 2021 15:37
An: Development list for mkgmap
Betreff: Re: [mkgmap-dev] New assertion, now with code-page=632 and Japan tile

Hi Gerd

My description didn't quite mean what I hoped it did - sorry. I was
thinking that there would be a single attempt at encoding the whole
string, and if that fails, start again but char-by-char.

But, assuming result.length() works and charBuffer.get() and
outBuff.put() maintain positions used by main encoder, within the loop
the failed component needs to be processed input char-by-char,
transliterated (if no change replaced by "?") and encoded with another
encoder.

Any variable length nature of the output charset shouldn't be a
problem. The variable length input UTF-16 will need care.

Ticker


On Wed, 2021-11-17 at 11:16 +, Gerd Petermann wrote:
> Hi Ticker,
>
> remember that cs932 is a double-byte character set.
> With your code only a few unmappable utf-16 characters are replaced,
> for the rest one of cs932 is used, but without any good reason. The
> result is typically garbage.
>
> I've modified the patch to replace any  unmappable character that was
> not transliterated by '?' .
> I've also attached a debug version that shows what goes on.
> A possible change in SparseTransliterator would be to add a mapping
> for the MATH MINUS, the other FULLWIDTH digits are supported in
> cs932.
>
> Gerd


___
mkgmap-dev mailing list
mkgmap-dev@lists.mkgmap.org.uk
https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
___
mkgmap-dev mailing list
mkgmap-dev@lists.mkgmap.org.uk
https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

Re: [mkgmap-dev] New assertion, now with code-page=632 and Japan tile

[Ticker Berkin]

Hi Gerd

My description didn't quite mean what I hoped it did - sorry. I was
thinking that there would be a single attempt at encoding the whole
string, and if that fails, start again but char-by-char.

But, assuming result.length() works and charBuffer.get() and
outBuff.put() maintain positions used by main encoder, within the loop
the failed component needs to be processed input char-by-char,
transliterated (if no change replaced by "?") and encoded with another
encoder.

Any variable length nature of the output charset shouldn't be a
problem. The variable length input UTF-16 will need care.

Ticker


On Wed, 2021-11-17 at 11:16 +, Gerd Petermann wrote:
> Hi Ticker,
> 
> remember that cs932 is a double-byte character set.
> With your code only a few unmappable utf-16 characters are replaced,
> for the rest one of cs932 is used, but without any good reason. The
> result is typically garbage.
> 
> I've modified the patch to replace any  unmappable character that was
> not transliterated by '?' .
> I've also attached a debug version that shows what goes on.
> A possible change in SparseTransliterator would be to add a mapping
> for the MATH MINUS, the other FULLWIDTH digits are supported in
> cs932.
> 
> Gerd


___
mkgmap-dev mailing list
mkgmap-dev@lists.mkgmap.org.uk
https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

Re: [mkgmap-dev] New assertion, now with code-page=632 and Japan tile

[Gerd Petermann]

Hi Ticker,

remember that cs932 is a double-byte character set.
With your code only a few unmappable utf-16 characters are replaced, for the 
rest one of cs932 is used, but without any good reason. The result is typically 
garbage.

I've modified the patch to replace any  unmappable character that was not 
transliterated by '?' .
I've also attached a debug version that shows what goes on.
A possible change in SparseTransliterator would be to add a mapping for the 
MATH MINUS, the other FULLWIDTH digits are supported in cs932.

Gerd










Von: mkgmap-dev  im Auftrag von Ticker 
Berkin 
Gesendet: Dienstag, 16. November 2021 17:33
An: Development list for mkgmap
Betreff: Re: [mkgmap-dev] New assertion, now with code-page=632 and Japan tile

Hi

wouldn't:
if ((c & 0xff) == 0)
c = "?";
be safer

I don't understand the point of sparseTranslitorator and why it is only
used for cp932 (japanese), unless this charset includes quite a few
european accented character.

If this is the case then wouldn't it be much better to do as I
described, the essence of which is not to transliterate the complete
string into the small ascii/latin1 set just because some chars can't be
mapped. The TableTranslitorator (ascii & latin1) map these FULLWIDTH
digits (and letters). MATHS MINUS isn't defined but easy to add.

Handling char at a time might allow removal of the 'ascii' table - if
transliteration changes char to [string of] another, for each of these,
if can't be represented, transliterate them.

Ticker


On Tue, 2021-11-16 at 15:48 +, Gerd Petermann wrote:
> Hi all,
>
> this small patch would be my approach. It replaces those characters
> which don't fit into a byte by '?'
> This fixes the problems with japanese codepage 932.
>
> Gerd
> BTW: SparseTransliterator is very sparse. We could add a few more
> character mappings, for example there is a housenumber that contains
> "１２３７−１" instead of "1237-1".
> https://www.fontspace.com/unicode/analyzer#e=77yR77yS77yT77yX4oiS77yR
>
> 
> Von: mkgmap-dev  im Auftrag
> von Ticker Berkin 
> Gesendet: Montag, 15. November 2021 15:59
> An: Development list for mkgmap
> Betreff: Re: [mkgmap-dev] New assertion, now with code-page=632 and
> Japan tile
>
> Hi
>
> How about something like:
>
> If the full string fails to encode in the target charset, process
> char
> at a time.
>
> If a char can't be represented, try transliteration on it and, if
> none
> defined, use "?", then go through the resultant string char at a
> time,
> and if this can't be represented, drop it.
>
> Maybe a final warning at end if no transliteration for a char or
> transliteration couldn't be represented.
>
> Ticker
>
> On Mon, 2021-11-15 at 13:04 +, Gerd Petermann wrote:
> > Hi all,
> >
> > > Maybe we should simply stop transliteration when this happens and
> > > return an empty string for the label?
> >
> > any thoughts on this?
> >
> > Gerd
> >
> > ________________
> > Von: mkgmap-dev  im Auftrag
> > von Gerd Petermann 
> > Gesendet: Mittwoch, 10. November 2021 11:17
> > An: Development list for mkgmap
> > Betreff: Re: [mkgmap-dev] New assertion,now with code-
> > page=632 and Japan tile
> >
> > Hi devs,
> >
> > the problem occurs with node https://www.osm.org/node/5692472121
> > name=키타가키 고로케
> > Google translate says the name is Korean. The (utf8) name cannot be
> > translated into code-page 932 (japanese) and thus mkgmap converts
> > the
> > internal utf16 representation of the name to bytes.  This happens
> > in
> > method AnyCharsetEncoder.encodeText(String text) in this loop:
> > for (int i = 0; i < s.length();
> > i++)
> > outBuf.put((byte)
> > s.charAt(i));
> > The name 키타가키 고로케 ends with  케 and the char value is \ucf00, so it
> > is
> > converted to \0x00.
> > Maybe we should simply stop transliteration when this happens and
> > return an empty string for the label?
> >
> > If mkgmap is executed without the -ea run time option the map shows
> > name ､ﾀ for the restaurant which is just wrong.
> > Gerd
> >
> > 
> > Von: mkgmap-dev  im Auftrag
> > von Gerd Petermann 
> > Gesendet: Mittwoch, 10. November 2021 09:43
> > An: Development list for mkgmap
> > Betreff: Re: [mkgmap-dev] New assertion,now with code-
> > page=632 and Japan tile
> >
> > Hi Carlos,
> >
> > I'll 

Re: [mkgmap-dev] New assertion, now with code-page=632 and Japan tile

[Ticker Berkin]

Hi

wouldn't:
if ((c & 0xff) == 0)
c = "?";
be safer

I don't understand the point of sparseTranslitorator and why it is only
used for cp932 (japanese), unless this charset includes quite a few
european accented character.

If this is the case then wouldn't it be much better to do as I
described, the essence of which is not to transliterate the complete
string into the small ascii/latin1 set just because some chars can't be
mapped. The TableTranslitorator (ascii & latin1) map these FULLWIDTH
digits (and letters). MATHS MINUS isn't defined but easy to add.

Handling char at a time might allow removal of the 'ascii' table - if
transliteration changes char to [string of] another, for each of these,
if can't be represented, transliterate them.

Ticker 


On Tue, 2021-11-16 at 15:48 +, Gerd Petermann wrote:
> Hi all,
> 
> this small patch would be my approach. It replaces those characters
> which don't fit into a byte by '?'
> This fixes the problems with japanese codepage 932.
> 
> Gerd
> BTW: SparseTransliterator is very sparse. We could add a few more
> character mappings, for example there is a housenumber that contains
> "１２３７−１" instead of "1237-1".
> https://www.fontspace.com/unicode/analyzer#e=77yR77yS77yT77yX4oiS77yR
> 
> 
> Von: mkgmap-dev  im Auftrag
> von Ticker Berkin 
> Gesendet: Montag, 15. November 2021 15:59
> An: Development list for mkgmap
> Betreff: Re: [mkgmap-dev] New assertion, now with code-page=632 and
> Japan tile
> 
> Hi
> 
> How about something like:
> 
> If the full string fails to encode in the target charset, process
> char
> at a time.
> 
> If a char can't be represented, try transliteration on it and, if
> none
> defined, use "?", then go through the resultant string char at a
> time,
> and if this can't be represented, drop it.
> 
> Maybe a final warning at end if no transliteration for a char or
> transliteration couldn't be represented.
> 
> Ticker
> 
> On Mon, 2021-11-15 at 13:04 +, Gerd Petermann wrote:
> > Hi all,
> > 
> > > Maybe we should simply stop transliteration when this happens and
> > > return an empty string for the label?
> > 
> > any thoughts on this?
> > 
> > Gerd
> > 
> > ________________
> > Von: mkgmap-dev  im Auftrag
> > von Gerd Petermann 
> > Gesendet: Mittwoch, 10. November 2021 11:17
> > An: Development list for mkgmap
> > Betreff: Re: [mkgmap-dev] New assertion,    now with code-
> > page=632 and Japan tile
> > 
> > Hi devs,
> > 
> > the problem occurs with node https://www.osm.org/node/5692472121
> > name=키타가키 고로케
> > Google translate says the name is Korean. The (utf8) name cannot be
> > translated into code-page 932 (japanese) and thus mkgmap converts
> > the
> > internal utf16 representation of the name to bytes.  This happens
> > in
> > method AnyCharsetEncoder.encodeText(String text) in this loop:
> >     for (int i = 0; i < s.length();
> > i++)
> >     outBuf.put((byte)
> > s.charAt(i));
> > The name 키타가키 고로케 ends with  케 and the char value is \ucf00, so it
> > is
> > converted to \0x00.
> > Maybe we should simply stop transliteration when this happens and
> > return an empty string for the label?
> > 
> > If mkgmap is executed without the -ea run time option the map shows
> > name ､ﾀ for the restaurant which is just wrong.
> > Gerd
> > 
> > 
> > Von: mkgmap-dev  im Auftrag
> > von Gerd Petermann 
> > Gesendet: Mittwoch, 10. November 2021 09:43
> > An: Development list for mkgmap
> > Betreff: Re: [mkgmap-dev] New assertion,    now with code-
> > page=632 and Japan tile
> > 
> > Hi Carlos,
> > 
> > I'll try to debug this.
> > 
> > BTW: I see you use *.o5m for the tiles (output from splitter). I
> > think this is no longer a good choice, pbf is a lot smaller and
> > almost as fast. Esp. when it comes to the goal of reducing disk I/O
> > (as with --gmapi-minimal)
> > 
> > Gerd
> > 
> > 
> > Von: mkgmap-dev  im Auftrag
> > von Carlos Dávila 
> > Gesendet: Dienstag, 9. November 2021 22:54
> > An: mkgmap-dev@lists.mkgmap.org.uk
> > Betreff: Re: [mkgmap-dev] New assertion,    now with code-
> > page=632 and Japan tile
> > 
> > Hi Ticker
> > 
> > Not sure if relevant, but note in this case assertion occurs while

Re: [mkgmap-dev] New assertion, now with code-page=632 and Japan tile

[Gerd Petermann]

Hi all,

this small patch would be my approach. It replaces those characters which don't 
fit into a byte by '?'
This fixes the problems with japanese codepage 932.

Gerd
BTW: SparseTransliterator is very sparse. We could add a few more character 
mappings, for example there is a housenumber that contains "１２３７−１" instead of 
"1237-1".
https://www.fontspace.com/unicode/analyzer#e=77yR77yS77yT77yX4oiS77yR


Von: mkgmap-dev  im Auftrag von Ticker 
Berkin 
Gesendet: Montag, 15. November 2021 15:59
An: Development list for mkgmap
Betreff: Re: [mkgmap-dev] New assertion, now with code-page=632 and Japan tile

Hi

How about something like:

If the full string fails to encode in the target charset, process char
at a time.

If a char can't be represented, try transliteration on it and, if none
defined, use "?", then go through the resultant string char at a time,
and if this can't be represented, drop it.

Maybe a final warning at end if no transliteration for a char or
transliteration couldn't be represented.

Ticker

On Mon, 2021-11-15 at 13:04 +, Gerd Petermann wrote:
> Hi all,
>
> > Maybe we should simply stop transliteration when this happens and
> > return an empty string for the label?
>
> any thoughts on this?
>
> Gerd
>
> 
> Von: mkgmap-dev  im Auftrag
> von Gerd Petermann 
> Gesendet: Mittwoch, 10. November 2021 11:17
> An: Development list for mkgmap
> Betreff: Re: [mkgmap-dev] New assertion,now with code-
> page=632 and Japan tile
>
> Hi devs,
>
> the problem occurs with node https://www.osm.org/node/5692472121
> name=키타가키 고로케
> Google translate says the name is Korean. The (utf8) name cannot be
> translated into code-page 932 (japanese) and thus mkgmap converts the
> internal utf16 representation of the name to bytes.  This happens in
> method AnyCharsetEncoder.encodeText(String text) in this loop:
> for (int i = 0; i < s.length(); i++)
> outBuf.put((byte)
> s.charAt(i));
> The name 키타가키 고로케 ends with  케 and the char value is \ucf00, so it is
> converted to \0x00.
> Maybe we should simply stop transliteration when this happens and
> return an empty string for the label?
>
> If mkgmap is executed without the -ea run time option the map shows
> name ､ﾀ for the restaurant which is just wrong.
> Gerd
>
> ________
> Von: mkgmap-dev  im Auftrag
> von Gerd Petermann 
> Gesendet: Mittwoch, 10. November 2021 09:43
> An: Development list for mkgmap
> Betreff: Re: [mkgmap-dev] New assertion,now with code-
> page=632 and Japan tile
>
> Hi Carlos,
>
> I'll try to debug this.
>
> BTW: I see you use *.o5m for the tiles (output from splitter). I
> think this is no longer a good choice, pbf is a lot smaller and
> almost as fast. Esp. when it comes to the goal of reducing disk I/O
> (as with --gmapi-minimal)
>
> Gerd
>
> ____________
> Von: mkgmap-dev  im Auftrag
> von Carlos Dávila 
> Gesendet: Dienstag, 9. November 2021 22:54
> An: mkgmap-dev@lists.mkgmap.org.uk
> Betreff: Re: [mkgmap-dev] New assertion,now with code-
> page=632 and Japan tile
>
> Hi Ticker
>
> Not sure if relevant, but note in this case assertion occurs while
> compiling the tile, not the index. In fact, --index is not included
> in
> the command.
>
> El 9/11/21 a las 21:55, Ticker Berkin escribió:
> > Hi
> >
> > I think this assertion could be removed from the code.
> >
> > Looking through the definition of Shift-JIS, I read it as saying
> > the
> > second byte shouldn't be zero, so I don't know why this happens.
> >
> > As with the Chinese code-pages, mkgmap has places where multi-byte
> > encodings are not handled correctly in the --index generation and
> > unknown meanings of flags to the Garmin software.
> >
> > Ticker
> >
> >
> >
> > On 09/11/2021 19:43, Carlos Dávila wrote:
> > > code-page=932, sorry for the typo.
> > >
> > > El 9/11/21 a las 20:36, Carlos Dávila escribió:
> > > > The command below produces an assertion while compiling this
> > > > tile
> > > > <https://files.mkgmap.org.uk/download/526/31191025.o5m> from
> > > > Japan.
> > > > Process continues with remaining tiles and finishes without
> > > > "Number
> > > > of MapFailedExceptions: 1" as expected. This is with r4813, but
> > > > I
> > > > also tried with an old version of mkgmap with the same result.
> > > >
> > > >

Re: [mkgmap-dev] New assertion, now with code-page=632 and Japan tile

[Ticker Berkin]

Hi

How about something like:

If the full string fails to encode in the target charset, process char
at a time.

If a char can't be represented, try transliteration on it and, if none
defined, use "?", then go through the resultant string char at a time,
and if this can't be represented, drop it.

Maybe a final warning at end if no transliteration for a char or
transliteration couldn't be represented.

Ticker

On Mon, 2021-11-15 at 13:04 +, Gerd Petermann wrote:
> Hi all,
> 
> > Maybe we should simply stop transliteration when this happens and
> > return an empty string for the label?
> 
> any thoughts on this?
> 
> Gerd
> 
> 
> Von: mkgmap-dev  im Auftrag
> von Gerd Petermann 
> Gesendet: Mittwoch, 10. November 2021 11:17
> An: Development list for mkgmap
> Betreff: Re: [mkgmap-dev] New assertion,    now with code-
> page=632 and Japan tile
> 
> Hi devs,
> 
> the problem occurs with node https://www.osm.org/node/5692472121
> name=키타가키 고로케
> Google translate says the name is Korean. The (utf8) name cannot be
> translated into code-page 932 (japanese) and thus mkgmap converts the
> internal utf16 representation of the name to bytes.  This happens in
> method AnyCharsetEncoder.encodeText(String text) in this loop:
>     for (int i = 0; i < s.length(); i++)
>     outBuf.put((byte)
> s.charAt(i));
> The name 키타가키 고로케 ends with  케 and the char value is \ucf00, so it is
> converted to \0x00.
> Maybe we should simply stop transliteration when this happens and
> return an empty string for the label?
> 
> If mkgmap is executed without the -ea run time option the map shows
> name ､ﾀ for the restaurant which is just wrong.
> Gerd
> 
> 
> Von: mkgmap-dev  im Auftrag
> von Gerd Petermann 
> Gesendet: Mittwoch, 10. November 2021 09:43
> An: Development list for mkgmap
> Betreff: Re: [mkgmap-dev] New assertion,    now with code-
> page=632 and Japan tile
> 
> Hi Carlos,
> 
> I'll try to debug this.
> 
> BTW: I see you use *.o5m for the tiles (output from splitter). I
> think this is no longer a good choice, pbf is a lot smaller and
> almost as fast. Esp. when it comes to the goal of reducing disk I/O
> (as with --gmapi-minimal)
> 
> Gerd
> 
> ____________________
> Von: mkgmap-dev  im Auftrag
> von Carlos Dávila 
> Gesendet: Dienstag, 9. November 2021 22:54
> An: mkgmap-dev@lists.mkgmap.org.uk
> Betreff: Re: [mkgmap-dev] New assertion,    now with code-
> page=632 and Japan tile
> 
> Hi Ticker
> 
> Not sure if relevant, but note in this case assertion occurs while
> compiling the tile, not the index. In fact, --index is not included
> in
> the command.
> 
> El 9/11/21 a las 21:55, Ticker Berkin escribió:
> > Hi
> > 
> > I think this assertion could be removed from the code.
> > 
> > Looking through the definition of Shift-JIS, I read it as saying
> > the
> > second byte shouldn't be zero, so I don't know why this happens.
> > 
> > As with the Chinese code-pages, mkgmap has places where multi-byte
> > encodings are not handled correctly in the --index generation and
> > unknown meanings of flags to the Garmin software.
> > 
> > Ticker
> > 
> > 
> > 
> > On 09/11/2021 19:43, Carlos Dávila wrote:
> > > code-page=932, sorry for the typo.
> > > 
> > > El 9/11/21 a las 20:36, Carlos Dávila escribió:
> > > > The command below produces an assertion while compiling this
> > > > tile
> > > > <https://files.mkgmap.org.uk/download/526/31191025.o5m> from
> > > > Japan.
> > > > Process continues with remaining tiles and finishes without
> > > > "Number
> > > > of MapFailedExceptions: 1" as expected. This is with r4813, but
> > > > I
> > > > also tried with an old version of mkgmap with the same result.
> > > > 
> > > > java -Xmx27G -ea -jar mkgmap.jar--code-page=632 31191025.o5m
> > > > Mkgmap version 4813
> > > > Time started: Tue Nov 09 20:18:16 CET 2021
> > > > WARNING (global): Setting max-jobs to 8
> > > > Exception in thread "main" java.lang.AssertionError: found
> > > > trailing
> > > > 0 in chars
> > > >     at
> > > > uk.me.parabola.imgfmt.app.labelenc.EncodedText.(EncodedTe
> > > > xt.java:39)
> > > > 
> > > >     at
> > > > uk.me.parabola.imgfmt.app.labelenc.AnyCharsetEncoder.en

Re: [mkgmap-dev] New assertion, now with code-page=632 and Japan tile

[Gerd Petermann]

Hi all,

> Maybe we should simply stop transliteration when this happens and return an 
> empty string for the label?

any thoughts on this?

Gerd


Von: mkgmap-dev  im Auftrag von Gerd 
Petermann 
Gesendet: Mittwoch, 10. November 2021 11:17
An: Development list for mkgmap
Betreff: Re: [mkgmap-dev] New assertion,        now with code-page=632 and 
Japan tile

Hi devs,

the problem occurs with node https://www.osm.org/node/5692472121
name=키타가키 고로케
Google translate says the name is Korean. The (utf8) name cannot be translated 
into code-page 932 (japanese) and thus mkgmap converts the internal utf16 
representation of the name to bytes.  This happens in method 
AnyCharsetEncoder.encodeText(String text) in this loop:
for (int i = 0; i < s.length(); i++)
outBuf.put((byte) s.charAt(i));
The name 키타가키 고로케 ends with  케 and the char value is \ucf00, so it is converted 
to \0x00.
Maybe we should simply stop transliteration when this happens and return an 
empty string for the label?

If mkgmap is executed without the -ea run time option the map shows name ､ﾀ for 
the restaurant which is just wrong.
Gerd


Von: mkgmap-dev  im Auftrag von Gerd 
Petermann 
Gesendet: Mittwoch, 10. November 2021 09:43
An: Development list for mkgmap
Betreff: Re: [mkgmap-dev] New assertion,    now with code-page=632 and 
Japan tile

Hi Carlos,

I'll try to debug this.

BTW: I see you use *.o5m for the tiles (output from splitter). I think this is 
no longer a good choice, pbf is a lot smaller and almost as fast. Esp. when it 
comes to the goal of reducing disk I/O (as with --gmapi-minimal)

Gerd


Von: mkgmap-dev  im Auftrag von Carlos 
Dávila 
Gesendet: Dienstag, 9. November 2021 22:54
An: mkgmap-dev@lists.mkgmap.org.uk
Betreff: Re: [mkgmap-dev] New assertion,    now with code-page=632 and 
Japan tile

Hi Ticker

Not sure if relevant, but note in this case assertion occurs while
compiling the tile, not the index. In fact, --index is not included in
the command.

El 9/11/21 a las 21:55, Ticker Berkin escribió:
> Hi
>
> I think this assertion could be removed from the code.
>
> Looking through the definition of Shift-JIS, I read it as saying the
> second byte shouldn't be zero, so I don't know why this happens.
>
> As with the Chinese code-pages, mkgmap has places where multi-byte
> encodings are not handled correctly in the --index generation and
> unknown meanings of flags to the Garmin software.
>
> Ticker
>
>
>
> On 09/11/2021 19:43, Carlos Dávila wrote:
>> code-page=932, sorry for the typo.
>>
>> El 9/11/21 a las 20:36, Carlos Dávila escribió:
>>> The command below produces an assertion while compiling this tile
>>> <https://files.mkgmap.org.uk/download/526/31191025.o5m> from Japan.
>>> Process continues with remaining tiles and finishes without "Number
>>> of MapFailedExceptions: 1" as expected. This is with r4813, but I
>>> also tried with an old version of mkgmap with the same result.
>>>
>>> java -Xmx27G -ea -jar mkgmap.jar--code-page=632 31191025.o5m
>>> Mkgmap version 4813
>>> Time started: Tue Nov 09 20:18:16 CET 2021
>>> WARNING (global): Setting max-jobs to 8
>>> Exception in thread "main" java.lang.AssertionError: found trailing
>>> 0 in chars
>>> at
>>> uk.me.parabola.imgfmt.app.labelenc.EncodedText.(EncodedText.java:39)
>>>
>>> at
>>> uk.me.parabola.imgfmt.app.labelenc.AnyCharsetEncoder.encodeText(AnyCharsetEncoder.java:112)
>>>
>>> at
>>> uk.me.parabola.imgfmt.app.lbl.LBLFile.newLabel(LBLFile.java:132)
>>> at
>>> uk.me.parabola.imgfmt.app.lbl.PlacesFile.createPOI(PlacesFile.java:253)
>>> at
>>> uk.me.parabola.imgfmt.app.lbl.LBLFile.createPOI(LBLFile.java:172)
>>> at
>>> uk.me.parabola.mkgmap.build.MapBuilder.processPOIs(MapBuilder.java:670)
>>> at
>>> uk.me.parabola.mkgmap.build.MapBuilder.makeMap(MapBuilder.java:325)
>>> at
>>> uk.me.parabola.mkgmap.main.MapMaker.makeMap(MapMaker.java:114)
>>> at
>>> uk.me.parabola.mkgmap.main.MapMaker.makeMap(MapMaker.java:62)
>>> at
>>> uk.me.parabola.mkgmap.main.Main.lambda$processFilename$1(Main.java:291)
>>> at
>>> java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
>>> at
>>> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
>>>
>>> at
>>> java.base/java.util

Re: [mkgmap-dev] New assertion, now with code-page=632 and Japan tile

[Gerd Petermann]

Hi Carlos,

yes, I only suggested to use pbf for the output of splitter.

Gerd


Von: mkgmap-dev  im Auftrag von Carlos 
Dávila 
Gesendet: Donnerstag, 11. November 2021 19:22
An: mkgmap-dev@lists.mkgmap.org.uk
Betreff: Re: [mkgmap-dev] New assertion,now with code-page=632 and 
Japan tile

Following are the results processing Portugal:


Size
osmupdate
split into o5m
split into pbf
mkgmap
portugal.o5m
713 MB
0:00:36
0:00:38
0:00:46
o5m: 0:01:25
portugal.pbf
410 MB
0:01:02
0:02:09
0:02:20
pbf: 0:01:27

With these figures the only reasonable option for me would be to output
pbf from splitter to reduce disk I/O. Time penalty for the other steps
is too high for me. Currently my server is working some 12 hours/day
updating maps.

El 10/11/21 a las 22:59, Felix Hartmann escribió:
> O5m is always faster, but needs more disk space. Converting geofabrik
> osm.pbf to o5m makes sense however (especially if you can write to
> Ramdisk or slow Harddisk)
>
> On Wed, 10 Nov 2021, 17:38 Carlos Dávila
> mailto:car...@alternativaslibres.org>>
> wrote:
>
> I'm aware of the supposed advantages of pbf, but I did some tests
> in the
> past and I found o5m to be quite faster than pbf, but I'll test again.
>
> El 10/11/21 a las 9:43, Gerd Petermann escribió:
> > Hi Carlos,
> >
> > I'll try to debug this.
> >
> > BTW: I see you use *.o5m for the tiles (output from splitter). I
> think this is no longer a good choice, pbf is a lot smaller and
> almost as fast. Esp. when it comes to the goal of reducing disk
> I/O (as with --gmapi-minimal)
> >
> > Gerd
> >
> > 
> > Von: mkgmap-dev  <mailto:mkgmap-dev-boun...@lists.mkgmap.org.uk>> im Auftrag von
> Carlos Dávila  <mailto:car...@alternativaslibres.org>>
> > Gesendet: Dienstag, 9. November 2021 22:54
>     > An: mkgmap-dev@lists.mkgmap.org.uk
> <mailto:mkgmap-dev@lists.mkgmap.org.uk>
> > Betreff: Re: [mkgmap-dev] New assertion,now with
> code-page=632 and Japan tile
> >
> > Hi Ticker
> >
> > Not sure if relevant, but note in this case assertion occurs while
> > compiling the tile, not the index. In fact, --index is not
> included in
> > the command.
> >
> > El 9/11/21 a las 21:55, Ticker Berkin escribió:
> >> Hi
> >>
> >> I think this assertion could be removed from the code.
> >>
> >> Looking through the definition of Shift-JIS, I read it as
> saying the
> >> second byte shouldn't be zero, so I don't know why this happens.
> >>
> >> As with the Chinese code-pages, mkgmap has places where multi-byte
> >> encodings are not handled correctly in the --index generation and
> >> unknown meanings of flags to the Garmin software.
> >>
> >> Ticker
> >>
> >>
> >>
> >> On 09/11/2021 19:43, Carlos Dávila wrote:
> >>> code-page=932, sorry for the typo.
> >>>
> >>> El 9/11/21 a las 20:36, Carlos Dávila escribió:
> >>>> The command below produces an assertion while compiling this tile
> >>>> <https://files.mkgmap.org.uk/download/526/31191025.o5m
> <https://files.mkgmap.org.uk/download/526/31191025.o5m>> from Japan.
> >>>> Process continues with remaining tiles and finishes without
> "Number
> >>>> of MapFailedExceptions: 1" as expected. This is with r4813, but I
> >>>> also tried with an old version of mkgmap with the same result.
> >>>>
> >>>> java -Xmx27G -ea -jar mkgmap.jar--code-page=632 31191025.o5m
> >>>> Mkgmap version 4813
> >>>> Time started: Tue Nov 09 20:18:16 CET 2021
> >>>> WARNING (global): Setting max-jobs to 8
> >>>> Exception in thread "main" java.lang.AssertionError: found
> trailing
> >>>> 0 in chars
> >>>>  at
> >>>>
> uk.me.parabola.imgfmt.app.labelenc.EncodedText.(EncodedText.java:39)
> >>>>
> >>>>  at
> >>>>
> 
> uk.me.parabola.imgfmt.app.labelenc.AnyCharsetEncoder.encodeText(AnyCharsetEncoder.java:112)
> >>>>
>

Re: [mkgmap-dev] New assertion, now with code-page=632 and Japan tile

[Carlos Dávila]

Following are the results processing Portugal:


Size
osmupdate
split into o5m
split into pbf
mkgmap
portugal.o5m
713 MB
0:00:36
0:00:38
0:00:46
o5m: 0:01:25
portugal.pbf
410 MB
0:01:02
0:02:09
0:02:20
pbf: 0:01:27

With these figures the only reasonable option for me would be to output 
pbf from splitter to reduce disk I/O. Time penalty for the other steps 
is too high for me. Currently my server is working some 12 hours/day 
updating maps.


El 10/11/21 a las 22:59, Felix Hartmann escribió:
O5m is always faster, but needs more disk space. Converting geofabrik 
osm.pbf to o5m makes sense however (especially if you can write to 
Ramdisk or slow Harddisk)


On Wed, 10 Nov 2021, 17:38 Carlos Dávila 
mailto:car...@alternativaslibres.org>> 
wrote:


I'm aware of the supposed advantages of pbf, but I did some tests
in the
past and I found o5m to be quite faster than pbf, but I'll test again.

El 10/11/21 a las 9:43, Gerd Petermann escribió:
> Hi Carlos,
>
> I'll try to debug this.
>
> BTW: I see you use *.o5m for the tiles (output from splitter). I
think this is no longer a good choice, pbf is a lot smaller and
almost as fast. Esp. when it comes to the goal of reducing disk
I/O (as with --gmapi-minimal)
>
> Gerd
>
> 
> Von: mkgmap-dev mailto:mkgmap-dev-boun...@lists.mkgmap.org.uk>> im Auftrag von
Carlos Dávila mailto:car...@alternativaslibres.org>>
> Gesendet: Dienstag, 9. November 2021 22:54
> An: mkgmap-dev@lists.mkgmap.org.uk
<mailto:mkgmap-dev@lists.mkgmap.org.uk>
> Betreff: Re: [mkgmap-dev] New assertion,        now with
code-page=632 and Japan tile
>
> Hi Ticker
>
> Not sure if relevant, but note in this case assertion occurs while
> compiling the tile, not the index. In fact, --index is not
included in
> the command.
>
> El 9/11/21 a las 21:55, Ticker Berkin escribió:
>> Hi
>>
>> I think this assertion could be removed from the code.
>>
>> Looking through the definition of Shift-JIS, I read it as
saying the
>> second byte shouldn't be zero, so I don't know why this happens.
>>
>> As with the Chinese code-pages, mkgmap has places where multi-byte
>> encodings are not handled correctly in the --index generation and
>> unknown meanings of flags to the Garmin software.
>>
>> Ticker
>>
>>
>>
>> On 09/11/2021 19:43, Carlos Dávila wrote:
>>> code-page=932, sorry for the typo.
>>>
>>> El 9/11/21 a las 20:36, Carlos Dávila escribió:
>>>> The command below produces an assertion while compiling this tile
>>>> <https://files.mkgmap.org.uk/download/526/31191025.o5m
<https://files.mkgmap.org.uk/download/526/31191025.o5m>> from Japan.
>>>> Process continues with remaining tiles and finishes without
"Number
>>>> of MapFailedExceptions: 1" as expected. This is with r4813, but I
>>>> also tried with an old version of mkgmap with the same result.
>>>>
>>>> java -Xmx27G -ea -jar mkgmap.jar--code-page=632 31191025.o5m
>>>> Mkgmap version 4813
>>>> Time started: Tue Nov 09 20:18:16 CET 2021
>>>> WARNING (global): Setting max-jobs to 8
>>>> Exception in thread "main" java.lang.AssertionError: found
trailing
>>>> 0 in chars
>>>>          at
>>>>
uk.me.parabola.imgfmt.app.labelenc.EncodedText.(EncodedText.java:39)
>>>>
>>>>          at
>>>>

uk.me.parabola.imgfmt.app.labelenc.AnyCharsetEncoder.encodeText(AnyCharsetEncoder.java:112)
>>>>
>>>>          at
>>>> uk.me.parabola.imgfmt.app.lbl.LBLFile.newLabel(LBLFile.java:132)
>>>>          at
>>>>
uk.me.parabola.imgfmt.app.lbl.PlacesFile.createPOI(PlacesFile.java:253)
>>>>          at
>>>> uk.me.parabola.imgfmt.app.lbl.LBLFile.createPOI(LBLFile.java:172)
>>>>          at
>>>>
uk.me.parabola.mkgmap.build.MapBuilder.processPOIs(MapBuilder.java:670)
>>>>          at
>>>>
uk.me.parabola.mkgmap.build.MapBuilder.makeMap(MapBuilder.java:325)
>>>>          at
>>>> uk.me.parabola.mkgmap.main.MapMaker.makeMap(MapMaker.java:114)
>>>>          at

Re: [mkgmap-dev] New assertion, now with code-page=632 and Japan tile

[Carlos Dávila]

I don't use Geofabrik extracts but my own ones. I keep country.o5m files 
that are updated with osmupdate as a first step for map updates.


El 10/11/21 a las 22:59, Felix Hartmann escribió:
O5m is always faster, but needs more disk space. Converting geofabrik 
osm.pbf to o5m makes sense however (especially if you can write to 
Ramdisk or slow Harddisk)


On Wed, 10 Nov 2021, 17:38 Carlos Dávila 
mailto:car...@alternativaslibres.org>> 
wrote:


I'm aware of the supposed advantages of pbf, but I did some tests
in the
past and I found o5m to be quite faster than pbf, but I'll test again.

El 10/11/21 a las 9:43, Gerd Petermann escribió:
> Hi Carlos,
>
> I'll try to debug this.
>
> BTW: I see you use *.o5m for the tiles (output from splitter). I
think this is no longer a good choice, pbf is a lot smaller and
almost as fast. Esp. when it comes to the goal of reducing disk
I/O (as with --gmapi-minimal)
>
> Gerd
>
> 
> Von: mkgmap-dev mailto:mkgmap-dev-boun...@lists.mkgmap.org.uk>> im Auftrag von
Carlos Dávila mailto:car...@alternativaslibres.org>>
> Gesendet: Dienstag, 9. November 2021 22:54
> An: mkgmap-dev@lists.mkgmap.org.uk
<mailto:mkgmap-dev@lists.mkgmap.org.uk>
> Betreff: Re: [mkgmap-dev] New assertion,        now with
code-page=632 and Japan tile
>
> Hi Ticker
>
> Not sure if relevant, but note in this case assertion occurs while
> compiling the tile, not the index. In fact, --index is not
included in
> the command.
>
> El 9/11/21 a las 21:55, Ticker Berkin escribió:
>> Hi
>>
>> I think this assertion could be removed from the code.
>>
>> Looking through the definition of Shift-JIS, I read it as
saying the
>> second byte shouldn't be zero, so I don't know why this happens.
>>
>> As with the Chinese code-pages, mkgmap has places where multi-byte
>> encodings are not handled correctly in the --index generation and
>> unknown meanings of flags to the Garmin software.
>>
>> Ticker
>>
>>
>>
>> On 09/11/2021 19:43, Carlos Dávila wrote:
>>> code-page=932, sorry for the typo.
>>>
>>> El 9/11/21 a las 20:36, Carlos Dávila escribió:
>>>> The command below produces an assertion while compiling this tile
>>>> <https://files.mkgmap.org.uk/download/526/31191025.o5m
<https://files.mkgmap.org.uk/download/526/31191025.o5m>> from Japan.
>>>> Process continues with remaining tiles and finishes without
"Number
>>>> of MapFailedExceptions: 1" as expected. This is with r4813, but I
>>>> also tried with an old version of mkgmap with the same result.
>>>>
>>>> java -Xmx27G -ea -jar mkgmap.jar--code-page=632 31191025.o5m
>>>> Mkgmap version 4813
>>>> Time started: Tue Nov 09 20:18:16 CET 2021
>>>> WARNING (global): Setting max-jobs to 8
>>>> Exception in thread "main" java.lang.AssertionError: found
trailing
>>>> 0 in chars
>>>>          at
>>>>
uk.me.parabola.imgfmt.app.labelenc.EncodedText.(EncodedText.java:39)
>>>>
>>>>          at
>>>>

uk.me.parabola.imgfmt.app.labelenc.AnyCharsetEncoder.encodeText(AnyCharsetEncoder.java:112)
>>>>
>>>>          at
>>>> uk.me.parabola.imgfmt.app.lbl.LBLFile.newLabel(LBLFile.java:132)
>>>>          at
>>>>
uk.me.parabola.imgfmt.app.lbl.PlacesFile.createPOI(PlacesFile.java:253)
>>>>          at
>>>> uk.me.parabola.imgfmt.app.lbl.LBLFile.createPOI(LBLFile.java:172)
>>>>          at
>>>>
uk.me.parabola.mkgmap.build.MapBuilder.processPOIs(MapBuilder.java:670)
>>>>          at
>>>>
uk.me.parabola.mkgmap.build.MapBuilder.makeMap(MapBuilder.java:325)
>>>>          at
>>>> uk.me.parabola.mkgmap.main.MapMaker.makeMap(MapMaker.java:114)
>>>>          at
>>>> uk.me.parabola.mkgmap.main.MapMaker.makeMap(MapMaker.java:62)
>>>>          at
>>>>
uk.me.parabola.mkgmap.main.Main.lambda$processFilename$1(Main.java:291)
>>>>          at
>>>>
java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
>>>>          at
>>>>

java.base/java.util.concurrent.ThreadPoolExec

Re: [mkgmap-dev] New assertion, now with code-page=632 and Japan tile

[Felix Hartmann]

O5m is always faster, but needs more disk space. Converting geofabrik
osm.pbf to o5m makes sense however (especially if you can write to Ramdisk
or slow Harddisk)

On Wed, 10 Nov 2021, 17:38 Carlos Dávila 
wrote:

> I'm aware of the supposed advantages of pbf, but I did some tests in the
> past and I found o5m to be quite faster than pbf, but I'll test again.
>
> El 10/11/21 a las 9:43, Gerd Petermann escribió:
> > Hi Carlos,
> >
> > I'll try to debug this.
> >
> > BTW: I see you use *.o5m for the tiles (output from splitter). I think
> this is no longer a good choice, pbf is a lot smaller and almost as fast.
> Esp. when it comes to the goal of reducing disk I/O (as with
> --gmapi-minimal)
> >
> > Gerd
> >
> > 
> > Von: mkgmap-dev  im Auftrag von
> Carlos Dávila 
> > Gesendet: Dienstag, 9. November 2021 22:54
> > An: mkgmap-dev@lists.mkgmap.org.uk
> > Betreff: Re: [mkgmap-dev] New assertion,now with code-page=632
> and Japan tile
> >
> > Hi Ticker
> >
> > Not sure if relevant, but note in this case assertion occurs while
> > compiling the tile, not the index. In fact, --index is not included in
> > the command.
> >
> > El 9/11/21 a las 21:55, Ticker Berkin escribió:
> >> Hi
> >>
> >> I think this assertion could be removed from the code.
> >>
> >> Looking through the definition of Shift-JIS, I read it as saying the
> >> second byte shouldn't be zero, so I don't know why this happens.
> >>
> >> As with the Chinese code-pages, mkgmap has places where multi-byte
> >> encodings are not handled correctly in the --index generation and
> >> unknown meanings of flags to the Garmin software.
> >>
> >> Ticker
> >>
> >>
> >>
> >> On 09/11/2021 19:43, Carlos Dávila wrote:
> >>> code-page=932, sorry for the typo.
> >>>
> >>> El 9/11/21 a las 20:36, Carlos Dávila escribió:
> >>>> The command below produces an assertion while compiling this tile
> >>>> <https://files.mkgmap.org.uk/download/526/31191025.o5m> from Japan.
> >>>> Process continues with remaining tiles and finishes without "Number
> >>>> of MapFailedExceptions: 1" as expected. This is with r4813, but I
> >>>> also tried with an old version of mkgmap with the same result.
> >>>>
> >>>> java -Xmx27G -ea -jar mkgmap.jar--code-page=632 31191025.o5m
> >>>> Mkgmap version 4813
> >>>> Time started: Tue Nov 09 20:18:16 CET 2021
> >>>> WARNING (global): Setting max-jobs to 8
> >>>> Exception in thread "main" java.lang.AssertionError: found trailing
> >>>> 0 in chars
> >>>>  at
> >>>>
> uk.me.parabola.imgfmt.app.labelenc.EncodedText.(EncodedText.java:39)
> >>>>
> >>>>  at
> >>>>
> uk.me.parabola.imgfmt.app.labelenc.AnyCharsetEncoder.encodeText(AnyCharsetEncoder.java:112)
> >>>>
> >>>>  at
> >>>> uk.me.parabola.imgfmt.app.lbl.LBLFile.newLabel(LBLFile.java:132)
> >>>>  at
> >>>>
> uk.me.parabola.imgfmt.app.lbl.PlacesFile.createPOI(PlacesFile.java:253)
> >>>>  at
> >>>> uk.me.parabola.imgfmt.app.lbl.LBLFile.createPOI(LBLFile.java:172)
> >>>>  at
> >>>>
> uk.me.parabola.mkgmap.build.MapBuilder.processPOIs(MapBuilder.java:670)
> >>>>  at
> >>>> uk.me.parabola.mkgmap.build.MapBuilder.makeMap(MapBuilder.java:325)
> >>>>  at
> >>>> uk.me.parabola.mkgmap.main.MapMaker.makeMap(MapMaker.java:114)
> >>>>  at
> >>>> uk.me.parabola.mkgmap.main.MapMaker.makeMap(MapMaker.java:62)
> >>>>  at
> >>>>
> uk.me.parabola.mkgmap.main.Main.lambda$processFilename$1(Main.java:291)
> >>>>  at
> >>>> java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> >>>>  at
> >>>>
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> >>>>
> >>>>  at
> >>>>
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> >>>>
> >>>>  at java.base/java.lang.Thread.run(Thread.java:829)
> >>>>
> >>>>

Re: [mkgmap-dev] New assertion, now with code-page=632 and Japan tile

[Felix Hartmann]

Only as input for splitter it changes a lot. As input for mkgmap it's very
little difference.

On Wed, 10 Nov 2021, 17:38 Carlos Dávila 
wrote:

> I'm aware of the supposed advantages of pbf, but I did some tests in the
> past and I found o5m to be quite faster than pbf, but I'll test again.
>
> El 10/11/21 a las 9:43, Gerd Petermann escribió:
> > Hi Carlos,
> >
> > I'll try to debug this.
> >
> > BTW: I see you use *.o5m for the tiles (output from splitter). I think
> this is no longer a good choice, pbf is a lot smaller and almost as fast.
> Esp. when it comes to the goal of reducing disk I/O (as with
> --gmapi-minimal)
> >
> > Gerd
> >
> > 
> > Von: mkgmap-dev  im Auftrag von
> Carlos Dávila 
> > Gesendet: Dienstag, 9. November 2021 22:54
> > An: mkgmap-dev@lists.mkgmap.org.uk
> > Betreff: Re: [mkgmap-dev] New assertion,now with code-page=632
> and Japan tile
> >
> > Hi Ticker
> >
> > Not sure if relevant, but note in this case assertion occurs while
> > compiling the tile, not the index. In fact, --index is not included in
> > the command.
> >
> > El 9/11/21 a las 21:55, Ticker Berkin escribió:
> >> Hi
> >>
> >> I think this assertion could be removed from the code.
> >>
> >> Looking through the definition of Shift-JIS, I read it as saying the
> >> second byte shouldn't be zero, so I don't know why this happens.
> >>
> >> As with the Chinese code-pages, mkgmap has places where multi-byte
> >> encodings are not handled correctly in the --index generation and
> >> unknown meanings of flags to the Garmin software.
> >>
> >> Ticker
> >>
> >>
> >>
> >> On 09/11/2021 19:43, Carlos Dávila wrote:
> >>> code-page=932, sorry for the typo.
> >>>
> >>> El 9/11/21 a las 20:36, Carlos Dávila escribió:
> >>>> The command below produces an assertion while compiling this tile
> >>>> <https://files.mkgmap.org.uk/download/526/31191025.o5m> from Japan.
> >>>> Process continues with remaining tiles and finishes without "Number
> >>>> of MapFailedExceptions: 1" as expected. This is with r4813, but I
> >>>> also tried with an old version of mkgmap with the same result.
> >>>>
> >>>> java -Xmx27G -ea -jar mkgmap.jar--code-page=632 31191025.o5m
> >>>> Mkgmap version 4813
> >>>> Time started: Tue Nov 09 20:18:16 CET 2021
> >>>> WARNING (global): Setting max-jobs to 8
> >>>> Exception in thread "main" java.lang.AssertionError: found trailing
> >>>> 0 in chars
> >>>>  at
> >>>>
> uk.me.parabola.imgfmt.app.labelenc.EncodedText.(EncodedText.java:39)
> >>>>
> >>>>  at
> >>>>
> uk.me.parabola.imgfmt.app.labelenc.AnyCharsetEncoder.encodeText(AnyCharsetEncoder.java:112)
> >>>>
> >>>>  at
> >>>> uk.me.parabola.imgfmt.app.lbl.LBLFile.newLabel(LBLFile.java:132)
> >>>>  at
> >>>>
> uk.me.parabola.imgfmt.app.lbl.PlacesFile.createPOI(PlacesFile.java:253)
> >>>>  at
> >>>> uk.me.parabola.imgfmt.app.lbl.LBLFile.createPOI(LBLFile.java:172)
> >>>>  at
> >>>>
> uk.me.parabola.mkgmap.build.MapBuilder.processPOIs(MapBuilder.java:670)
> >>>>  at
> >>>> uk.me.parabola.mkgmap.build.MapBuilder.makeMap(MapBuilder.java:325)
> >>>>  at
> >>>> uk.me.parabola.mkgmap.main.MapMaker.makeMap(MapMaker.java:114)
> >>>>  at
> >>>> uk.me.parabola.mkgmap.main.MapMaker.makeMap(MapMaker.java:62)
> >>>>  at
> >>>>
> uk.me.parabola.mkgmap.main.Main.lambda$processFilename$1(Main.java:291)
> >>>>  at
> >>>> java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> >>>>  at
> >>>>
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> >>>>
> >>>>  at
> >>>>
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> >>>>
> >>>>  at java.base/java.lang.Thread.run(Thread.java:829)
> >>>>
> >>>>
> >>>> ___
> >>>> mkgmap-dev 

Re: [mkgmap-dev] New assertion, now with code-page=632 and Japan tile

[Carlos Dávila]

I'm aware of the supposed advantages of pbf, but I did some tests in the 
past and I found o5m to be quite faster than pbf, but I'll test again.


El 10/11/21 a las 9:43, Gerd Petermann escribió:

Hi Carlos,

I'll try to debug this.

BTW: I see you use *.o5m for the tiles (output from splitter). I think this is 
no longer a good choice, pbf is a lot smaller and almost as fast. Esp. when it 
comes to the goal of reducing disk I/O (as with --gmapi-minimal)

Gerd


Von: mkgmap-dev  im Auftrag von Carlos Dávila 

Gesendet: Dienstag, 9. November 2021 22:54
An: mkgmap-dev@lists.mkgmap.org.uk
Betreff: Re: [mkgmap-dev] New assertion,now with code-page=632 and 
Japan tile

Hi Ticker

Not sure if relevant, but note in this case assertion occurs while
compiling the tile, not the index. In fact, --index is not included in
the command.

El 9/11/21 a las 21:55, Ticker Berkin escribió:

Hi

I think this assertion could be removed from the code.

Looking through the definition of Shift-JIS, I read it as saying the
second byte shouldn't be zero, so I don't know why this happens.

As with the Chinese code-pages, mkgmap has places where multi-byte
encodings are not handled correctly in the --index generation and
unknown meanings of flags to the Garmin software.

Ticker



On 09/11/2021 19:43, Carlos Dávila wrote:

code-page=932, sorry for the typo.

El 9/11/21 a las 20:36, Carlos Dávila escribió:

The command below produces an assertion while compiling this tile
<https://files.mkgmap.org.uk/download/526/31191025.o5m> from Japan.
Process continues with remaining tiles and finishes without "Number
of MapFailedExceptions: 1" as expected. This is with r4813, but I
also tried with an old version of mkgmap with the same result.

java -Xmx27G -ea -jar mkgmap.jar--code-page=632 31191025.o5m
Mkgmap version 4813
Time started: Tue Nov 09 20:18:16 CET 2021
WARNING (global): Setting max-jobs to 8
Exception in thread "main" java.lang.AssertionError: found trailing
0 in chars
 at
uk.me.parabola.imgfmt.app.labelenc.EncodedText.(EncodedText.java:39)

 at
uk.me.parabola.imgfmt.app.labelenc.AnyCharsetEncoder.encodeText(AnyCharsetEncoder.java:112)

 at
uk.me.parabola.imgfmt.app.lbl.LBLFile.newLabel(LBLFile.java:132)
 at
uk.me.parabola.imgfmt.app.lbl.PlacesFile.createPOI(PlacesFile.java:253)
 at
uk.me.parabola.imgfmt.app.lbl.LBLFile.createPOI(LBLFile.java:172)
 at
uk.me.parabola.mkgmap.build.MapBuilder.processPOIs(MapBuilder.java:670)
 at
uk.me.parabola.mkgmap.build.MapBuilder.makeMap(MapBuilder.java:325)
 at
uk.me.parabola.mkgmap.main.MapMaker.makeMap(MapMaker.java:114)
 at
uk.me.parabola.mkgmap.main.MapMaker.makeMap(MapMaker.java:62)
 at
uk.me.parabola.mkgmap.main.Main.lambda$processFilename$1(Main.java:291)
 at
java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
 at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)

 at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)

 at java.base/java.lang.Thread.run(Thread.java:829)


___
mkgmap-dev mailing list
mkgmap-dev@lists.mkgmap.org.uk
https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

___
mkgmap-dev mailing list
mkgmap-dev@lists.mkgmap.org.uk
https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

___
mkgmap-dev mailing list
mkgmap-dev@lists.mkgmap.org.uk
https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

___
mkgmap-dev mailing list
mkgmap-dev@lists.mkgmap.org.uk
https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
___
mkgmap-dev mailing list
mkgmap-dev@lists.mkgmap.org.uk
https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev


___
mkgmap-dev mailing list
mkgmap-dev@lists.mkgmap.org.uk
https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

Re: [mkgmap-dev] New assertion, now with code-page=632 and Japan tile

[Gerd Petermann]

Hi devs,

the problem occurs with node https://www.osm.org/node/5692472121
name=키타가키 고로케
Google translate says the name is Korean. The (utf8) name cannot be translated 
into code-page 932 (japanese) and thus mkgmap converts the internal utf16 
representation of the name to bytes.  This happens in method 
AnyCharsetEncoder.encodeText(String text) in this loop:
for (int i = 0; i < s.length(); i++)
outBuf.put((byte) s.charAt(i));
The name 키타가키 고로케 ends with  케 and the char value is \ucf00, so it is converted 
to \0x00.
Maybe we should simply stop transliteration when this happens and return an 
empty string for the label?

If mkgmap is executed without the -ea run time option the map shows name ､ﾀ for 
the restaurant which is just wrong.
Gerd


Von: mkgmap-dev  im Auftrag von Gerd 
Petermann 
Gesendet: Mittwoch, 10. November 2021 09:43
An: Development list for mkgmap
Betreff: Re: [mkgmap-dev] New assertion,    now with code-page=632 and 
Japan tile

Hi Carlos,

I'll try to debug this.

BTW: I see you use *.o5m for the tiles (output from splitter). I think this is 
no longer a good choice, pbf is a lot smaller and almost as fast. Esp. when it 
comes to the goal of reducing disk I/O (as with --gmapi-minimal)

Gerd


Von: mkgmap-dev  im Auftrag von Carlos 
Dávila 
Gesendet: Dienstag, 9. November 2021 22:54
An: mkgmap-dev@lists.mkgmap.org.uk
Betreff: Re: [mkgmap-dev] New assertion,    now with code-page=632 and 
Japan tile

Hi Ticker

Not sure if relevant, but note in this case assertion occurs while
compiling the tile, not the index. In fact, --index is not included in
the command.

El 9/11/21 a las 21:55, Ticker Berkin escribió:
> Hi
>
> I think this assertion could be removed from the code.
>
> Looking through the definition of Shift-JIS, I read it as saying the
> second byte shouldn't be zero, so I don't know why this happens.
>
> As with the Chinese code-pages, mkgmap has places where multi-byte
> encodings are not handled correctly in the --index generation and
> unknown meanings of flags to the Garmin software.
>
> Ticker
>
>
>
> On 09/11/2021 19:43, Carlos Dávila wrote:
>> code-page=932, sorry for the typo.
>>
>> El 9/11/21 a las 20:36, Carlos Dávila escribió:
>>> The command below produces an assertion while compiling this tile
>>> <https://files.mkgmap.org.uk/download/526/31191025.o5m> from Japan.
>>> Process continues with remaining tiles and finishes without "Number
>>> of MapFailedExceptions: 1" as expected. This is with r4813, but I
>>> also tried with an old version of mkgmap with the same result.
>>>
>>> java -Xmx27G -ea -jar mkgmap.jar--code-page=632 31191025.o5m
>>> Mkgmap version 4813
>>> Time started: Tue Nov 09 20:18:16 CET 2021
>>> WARNING (global): Setting max-jobs to 8
>>> Exception in thread "main" java.lang.AssertionError: found trailing
>>> 0 in chars
>>> at
>>> uk.me.parabola.imgfmt.app.labelenc.EncodedText.(EncodedText.java:39)
>>>
>>> at
>>> uk.me.parabola.imgfmt.app.labelenc.AnyCharsetEncoder.encodeText(AnyCharsetEncoder.java:112)
>>>
>>> at
>>> uk.me.parabola.imgfmt.app.lbl.LBLFile.newLabel(LBLFile.java:132)
>>> at
>>> uk.me.parabola.imgfmt.app.lbl.PlacesFile.createPOI(PlacesFile.java:253)
>>> at
>>> uk.me.parabola.imgfmt.app.lbl.LBLFile.createPOI(LBLFile.java:172)
>>> at
>>> uk.me.parabola.mkgmap.build.MapBuilder.processPOIs(MapBuilder.java:670)
>>> at
>>> uk.me.parabola.mkgmap.build.MapBuilder.makeMap(MapBuilder.java:325)
>>> at
>>> uk.me.parabola.mkgmap.main.MapMaker.makeMap(MapMaker.java:114)
>>> at
>>> uk.me.parabola.mkgmap.main.MapMaker.makeMap(MapMaker.java:62)
>>> at
>>> uk.me.parabola.mkgmap.main.Main.lambda$processFilename$1(Main.java:291)
>>> at
>>> java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
>>> at
>>> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
>>>
>>> at
>>> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
>>>
>>> at java.base/java.lang.Thread.run(Thread.java:829)
>>>
>>>
>>> ___
>>> mkgmap-dev mailing list
>>> mkgmap-dev@lists.mkgmap.org.uk
>>> https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
>>
>> ___

Re: [mkgmap-dev] New assertion, now with code-page=632 and Japan tile

[Gerd Petermann]

Hi Carlos,

I'll try to debug this.

BTW: I see you use *.o5m for the tiles (output from splitter). I think this is 
no longer a good choice, pbf is a lot smaller and almost as fast. Esp. when it 
comes to the goal of reducing disk I/O (as with --gmapi-minimal)

Gerd


Von: mkgmap-dev  im Auftrag von Carlos 
Dávila 
Gesendet: Dienstag, 9. November 2021 22:54
An: mkgmap-dev@lists.mkgmap.org.uk
Betreff: Re: [mkgmap-dev] New assertion,now with code-page=632 and 
Japan tile

Hi Ticker

Not sure if relevant, but note in this case assertion occurs while
compiling the tile, not the index. In fact, --index is not included in
the command.

El 9/11/21 a las 21:55, Ticker Berkin escribió:
> Hi
>
> I think this assertion could be removed from the code.
>
> Looking through the definition of Shift-JIS, I read it as saying the
> second byte shouldn't be zero, so I don't know why this happens.
>
> As with the Chinese code-pages, mkgmap has places where multi-byte
> encodings are not handled correctly in the --index generation and
> unknown meanings of flags to the Garmin software.
>
> Ticker
>
>
>
> On 09/11/2021 19:43, Carlos Dávila wrote:
>> code-page=932, sorry for the typo.
>>
>> El 9/11/21 a las 20:36, Carlos Dávila escribió:
>>> The command below produces an assertion while compiling this tile
>>> <https://files.mkgmap.org.uk/download/526/31191025.o5m> from Japan.
>>> Process continues with remaining tiles and finishes without "Number
>>> of MapFailedExceptions: 1" as expected. This is with r4813, but I
>>> also tried with an old version of mkgmap with the same result.
>>>
>>> java -Xmx27G -ea -jar mkgmap.jar--code-page=632 31191025.o5m
>>> Mkgmap version 4813
>>> Time started: Tue Nov 09 20:18:16 CET 2021
>>> WARNING (global): Setting max-jobs to 8
>>> Exception in thread "main" java.lang.AssertionError: found trailing
>>> 0 in chars
>>> at
>>> uk.me.parabola.imgfmt.app.labelenc.EncodedText.(EncodedText.java:39)
>>>
>>> at
>>> uk.me.parabola.imgfmt.app.labelenc.AnyCharsetEncoder.encodeText(AnyCharsetEncoder.java:112)
>>>
>>> at
>>> uk.me.parabola.imgfmt.app.lbl.LBLFile.newLabel(LBLFile.java:132)
>>> at
>>> uk.me.parabola.imgfmt.app.lbl.PlacesFile.createPOI(PlacesFile.java:253)
>>> at
>>> uk.me.parabola.imgfmt.app.lbl.LBLFile.createPOI(LBLFile.java:172)
>>> at
>>> uk.me.parabola.mkgmap.build.MapBuilder.processPOIs(MapBuilder.java:670)
>>> at
>>> uk.me.parabola.mkgmap.build.MapBuilder.makeMap(MapBuilder.java:325)
>>> at
>>> uk.me.parabola.mkgmap.main.MapMaker.makeMap(MapMaker.java:114)
>>> at
>>> uk.me.parabola.mkgmap.main.MapMaker.makeMap(MapMaker.java:62)
>>> at
>>> uk.me.parabola.mkgmap.main.Main.lambda$processFilename$1(Main.java:291)
>>> at
>>> java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
>>> at
>>> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
>>>
>>> at
>>> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
>>>
>>> at java.base/java.lang.Thread.run(Thread.java:829)
>>>
>>>
>>> ___
>>> mkgmap-dev mailing list
>>> mkgmap-dev@lists.mkgmap.org.uk
>>> https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
>>
>> ___
>> mkgmap-dev mailing list
>> mkgmap-dev@lists.mkgmap.org.uk
>> https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
> ___
> mkgmap-dev mailing list
> mkgmap-dev@lists.mkgmap.org.uk
> https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

___
mkgmap-dev mailing list
mkgmap-dev@lists.mkgmap.org.uk
https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev
___
mkgmap-dev mailing list
mkgmap-dev@lists.mkgmap.org.uk
https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

Re: [mkgmap-dev] New assertion, now with code-page=632 and Japan tile

[Carlos Dávila]

Hi Ticker

Not sure if relevant, but note in this case assertion occurs while 
compiling the tile, not the index. In fact, --index is not included in 
the command.


El 9/11/21 a las 21:55, Ticker Berkin escribió:

Hi

I think this assertion could be removed from the code.

Looking through the definition of Shift-JIS, I read it as saying the 
second byte shouldn't be zero, so I don't know why this happens.


As with the Chinese code-pages, mkgmap has places where multi-byte 
encodings are not handled correctly in the --index generation and 
unknown meanings of flags to the Garmin software.


Ticker



On 09/11/2021 19:43, Carlos Dávila wrote:

code-page=932, sorry for the typo.

El 9/11/21 a las 20:36, Carlos Dávila escribió:
The command below produces an assertion while compiling this tile 
 from Japan. 
Process continues with remaining tiles and finishes without "Number 
of MapFailedExceptions: 1" as expected. This is with r4813, but I 
also tried with an old version of mkgmap with the same result.


java -Xmx27G -ea -jar mkgmap.jar--code-page=632 31191025.o5m
Mkgmap version 4813
Time started: Tue Nov 09 20:18:16 CET 2021
WARNING (global): Setting max-jobs to 8
Exception in thread "main" java.lang.AssertionError: found trailing 
0 in chars
    at 
uk.me.parabola.imgfmt.app.labelenc.EncodedText.(EncodedText.java:39) 

    at 
uk.me.parabola.imgfmt.app.labelenc.AnyCharsetEncoder.encodeText(AnyCharsetEncoder.java:112) 

    at 
uk.me.parabola.imgfmt.app.lbl.LBLFile.newLabel(LBLFile.java:132)
    at 
uk.me.parabola.imgfmt.app.lbl.PlacesFile.createPOI(PlacesFile.java:253)
    at 
uk.me.parabola.imgfmt.app.lbl.LBLFile.createPOI(LBLFile.java:172)
    at 
uk.me.parabola.mkgmap.build.MapBuilder.processPOIs(MapBuilder.java:670)
    at 
uk.me.parabola.mkgmap.build.MapBuilder.makeMap(MapBuilder.java:325)
    at 
uk.me.parabola.mkgmap.main.MapMaker.makeMap(MapMaker.java:114)
    at 
uk.me.parabola.mkgmap.main.MapMaker.makeMap(MapMaker.java:62)
    at 
uk.me.parabola.mkgmap.main.Main.lambda$processFilename$1(Main.java:291)
    at 
java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at 
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) 

    at 
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) 


    at java.base/java.lang.Thread.run(Thread.java:829)


___
mkgmap-dev mailing list
mkgmap-dev@lists.mkgmap.org.uk
https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev


___
mkgmap-dev mailing list
mkgmap-dev@lists.mkgmap.org.uk
https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

___
mkgmap-dev mailing list
mkgmap-dev@lists.mkgmap.org.uk
https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev


___
mkgmap-dev mailing list
mkgmap-dev@lists.mkgmap.org.uk
https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

Re: [mkgmap-dev] New assertion, now with code-page=632 and Japan tile

[Ticker Berkin]

Hi

I think this assertion could be removed from the code.

Looking through the definition of Shift-JIS, I read it as saying the 
second byte shouldn't be zero, so I don't know why this happens.


As with the Chinese code-pages, mkgmap has places where multi-byte 
encodings are not handled correctly in the --index generation and 
unknown meanings of flags to the Garmin software.


Ticker



On 09/11/2021 19:43, Carlos Dávila wrote:

code-page=932, sorry for the typo.

El 9/11/21 a las 20:36, Carlos Dávila escribió:
The command below produces an assertion while compiling this tile 
 from Japan. 
Process continues with remaining tiles and finishes without "Number of 
MapFailedExceptions: 1" as expected. This is with r4813, but I also 
tried with an old version of mkgmap with the same result.


java -Xmx27G -ea -jar mkgmap.jar--code-page=632 31191025.o5m
Mkgmap version 4813
Time started: Tue Nov 09 20:18:16 CET 2021
WARNING (global): Setting max-jobs to 8
Exception in thread "main" java.lang.AssertionError: found trailing 0 
in chars
    at 
uk.me.parabola.imgfmt.app.labelenc.EncodedText.(EncodedText.java:39) 

    at 
uk.me.parabola.imgfmt.app.labelenc.AnyCharsetEncoder.encodeText(AnyCharsetEncoder.java:112) 

    at 
uk.me.parabola.imgfmt.app.lbl.LBLFile.newLabel(LBLFile.java:132)
    at 
uk.me.parabola.imgfmt.app.lbl.PlacesFile.createPOI(PlacesFile.java:253)
    at 
uk.me.parabola.imgfmt.app.lbl.LBLFile.createPOI(LBLFile.java:172)
    at 
uk.me.parabola.mkgmap.build.MapBuilder.processPOIs(MapBuilder.java:670)
    at 
uk.me.parabola.mkgmap.build.MapBuilder.makeMap(MapBuilder.java:325)

    at uk.me.parabola.mkgmap.main.MapMaker.makeMap(MapMaker.java:114)
    at uk.me.parabola.mkgmap.main.MapMaker.makeMap(MapMaker.java:62)
    at 
uk.me.parabola.mkgmap.main.Main.lambda$processFilename$1(Main.java:291)
    at 
java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at 
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) 

    at 
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) 


    at java.base/java.lang.Thread.run(Thread.java:829)


___
mkgmap-dev mailing list
mkgmap-dev@lists.mkgmap.org.uk
https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev


___
mkgmap-dev mailing list
mkgmap-dev@lists.mkgmap.org.uk
https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

___
mkgmap-dev mailing list
mkgmap-dev@lists.mkgmap.org.uk
https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

Re: [mkgmap-dev] New assertion, now with code-page=632 and Japan tile

[Carlos Dávila]

code-page=932, sorry for the typo.

El 9/11/21 a las 20:36, Carlos Dávila escribió:
The command below produces an assertion while compiling this tile 
 from Japan. 
Process continues with remaining tiles and finishes without "Number of 
MapFailedExceptions: 1" as expected. This is with r4813, but I also 
tried with an old version of mkgmap with the same result.


java -Xmx27G -ea -jar mkgmap.jar--code-page=632 31191025.o5m
Mkgmap version 4813
Time started: Tue Nov 09 20:18:16 CET 2021
WARNING (global): Setting max-jobs to 8
Exception in thread "main" java.lang.AssertionError: found trailing 0 
in chars
    at 
uk.me.parabola.imgfmt.app.labelenc.EncodedText.(EncodedText.java:39)
    at 
uk.me.parabola.imgfmt.app.labelenc.AnyCharsetEncoder.encodeText(AnyCharsetEncoder.java:112)
    at 
uk.me.parabola.imgfmt.app.lbl.LBLFile.newLabel(LBLFile.java:132)
    at 
uk.me.parabola.imgfmt.app.lbl.PlacesFile.createPOI(PlacesFile.java:253)
    at 
uk.me.parabola.imgfmt.app.lbl.LBLFile.createPOI(LBLFile.java:172)
    at 
uk.me.parabola.mkgmap.build.MapBuilder.processPOIs(MapBuilder.java:670)
    at 
uk.me.parabola.mkgmap.build.MapBuilder.makeMap(MapBuilder.java:325)

    at uk.me.parabola.mkgmap.main.MapMaker.makeMap(MapMaker.java:114)
    at uk.me.parabola.mkgmap.main.MapMaker.makeMap(MapMaker.java:62)
    at 
uk.me.parabola.mkgmap.main.Main.lambda$processFilename$1(Main.java:291)
    at 
java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at 
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at 
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)

    at java.base/java.lang.Thread.run(Thread.java:829)


___
mkgmap-dev mailing list
mkgmap-dev@lists.mkgmap.org.uk
https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev


___
mkgmap-dev mailing list
mkgmap-dev@lists.mkgmap.org.uk
https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev