Re: [mkgmap-dev] java.lang.AssertionError while building index from unicode tiles

2021-10-22 Thread Steve Ratcliffe
Hi Ticker > Problem is that resources/sort/cp65001.txt doesn't give ordering to > lots of characters; it looks like it covers only about 10,500 of the > 1,112,064 possible code-points. Many of these non-ordered characters > are being used by the names in the tile in question. I used the program

Re: [mkgmap-dev] java.lang.AssertionError while building index from unicode tiles

2021-10-22 Thread Ticker Berkin
Gerd Petermann > Gesendet: Donnerstag, 21. Oktober 2021 15:48 > An: Development list for mkgmap > Betreff: Re: [mkgmap-dev] java.lang.AssertionError while building index > from unicode tiles > > Hi Ticker, > > I agree that the original code isn't clear, what I don't und

Re: [mkgmap-dev] java.lang.AssertionError while building index from unicode tiles

2021-10-22 Thread Gerd Petermann
analyse the content further. Gerd Von: mkgmap-dev im Auftrag von Gerd Petermann Gesendet: Donnerstag, 21. Oktober 2021 15:48 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] java.lang.AssertionError while building index from unicode tiles Hi

Re: [mkgmap-dev] java.lang.AssertionError while building index from unicode tiles

2021-10-21 Thread Gerd Petermann
has. Gerd Von: mkgmap-dev im Auftrag von Ticker Berkin Gesendet: Mittwoch, 20. Oktober 2021 12:32 An: Development list for mkgmap; Steve Ratcliffe Betreff: Re: [mkgmap-dev] java.lang.AssertionError while building index from unicode tiles Hi In the changes I've

Re: [mkgmap-dev] java.lang.AssertionError while building index from unicode tiles

2021-10-21 Thread Ticker Berkin
2021 12:32 > An: Development list for mkgmap; Steve Ratcliffe > Betreff: Re: [mkgmap-dev] java.lang.AssertionError while building index > from unicode tiles > > Hi > > In the changes I've just made, I hope I've been consistent and fixed > all instances to use collator.

Re: [mkgmap-dev] java.lang.AssertionError while building index from unicode tiles

2021-10-21 Thread Gerd Petermann
mkgmap-dev] java.lang.AssertionError while building index from unicode tiles Hi In the changes I've just made, I hope I've been consistent and fixed all instances to use collator.compare() where scanning the results of a sort on the same table for a change. Also consistently setting strength to SECONDARY (gene

Re: [mkgmap-dev] java.lang.AssertionError while building index from unicode tiles

2021-10-20 Thread Ticker Berkin
Hi In the changes I've just made, I hope I've been consistent and fixed all instances to use collator.compare() where scanning the results of a sort on the same table for a change. Also consistently setting strength to SECONDARY (generally case-insensitive). There may be places where an indirect

Re: [mkgmap-dev] java.lang.AssertionError while building index from unicode tiles

2021-10-20 Thread Ticker Berkin
Hi Gerd I didn't understand this either - Mdr29 with lowest refs to Mdr17, Mdr22, Mdr24, Mdr25 and Mdr26 is beyond me so I thought it best leave that part untouched. Ticker On Wed, 2021-10-20 at 07:59 +, Gerd Petermann wrote: > Hi Ticker, > > please double check Mdr25: > I just wonder why

Re: [mkgmap-dev] java.lang.AssertionError while building index from unicode tiles

2021-10-20 Thread Gerd Petermann
heck and the classes in mkgmap. Gerd Von: mkgmap-dev im Auftrag von Gerd Petermann Gesendet: Mittwoch, 20. Oktober 2021 09:59 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] java.lang.AssertionError while building index from unicode tiles Hi Ticker, pl

Re: [mkgmap-dev] java.lang.AssertionError while building index from unicode tiles

2021-10-20 Thread Gerd Petermann
An: Development list for mkgmap Betreff: Re: [mkgmap-dev] java.lang.AssertionError while building index from unicode tiles Hi Gerd Here it is Ticker On Tue, 2021-10-19 at 09:22 +, Gerd Petermann wrote: > Hi Ticker, > > yes, please remove all unrelated optimizations.

Re: [mkgmap-dev] java.lang.AssertionError while building index from unicode tiles

2021-10-19 Thread Ticker Berkin
et: Dienstag, 19. Oktober 2021 11:03 > An: Development list for mkgmap > Betreff: Re: [mkgmap-dev] java.lang.AssertionError while building > index from unicode tiles > > Hi Gerd > > I'd removed the change relating to clearing the reference to the Sort > object to all

Re: [mkgmap-dev] java.lang.AssertionError while building index from unicode tiles

2021-10-19 Thread Gerd Petermann
ent list for mkgmap > Betreff: Re: [mkgmap-dev] java.lang.AssertionError while building > index from unicode tiles > > Hi Gerd > > Here is first version of the changes to improve MDR unicode and stop > the crash. > > It always provides a PRIMARY strength sort value, both in the ke

Re: [mkgmap-dev] java.lang.AssertionError while building index from unicode tiles

2021-10-19 Thread Ticker Berkin
h > patch mdrSort.patch in May, subject "MDR building out-of-memory". > > Gerd > > > Von: mkgmap-dev im Auftrag > von Ticker Berkin > Gesendet: Montag, 18. Oktober 2021 16:36 > An: Development list for mkgmap > Betreff

Re: [mkgmap-dev] java.lang.AssertionError while building index from unicode tiles

2021-10-19 Thread Gerd Petermann
2021 16:36 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] java.lang.AssertionError while building index from unicode tiles Hi Gerd Here is first version of the changes to improve MDR unicode and stop the crash. It always provides a PRIMARY strength sort value, both in the key for sorting and direct

Re: [mkgmap-dev] java.lang.AssertionError while building index from unicode tiles

2021-10-18 Thread Ticker Berkin
Hi Gerd Here is first version of the changes to improve MDR unicode and stop the crash. It always provides a PRIMARY strength sort value, both in the key for sorting and direct comparison when using the collator. Previously neither of these would have anything for a unicode character not

Re: [mkgmap-dev] java.lang.AssertionError while building index from unicode tiles

2021-10-18 Thread Gerd Petermann
] java.lang.AssertionError while building index from unicode tiles Hi Gerd In imgfmt/app/srt/Sort.java around line 853: // Get the first non-ignorable at this level int c = chars[pos++ & 0xff]; if (!hasPage(c >>> 8)) { I'm at a loss to understand the 0xff mask!

Re: [mkgmap-dev] java.lang.AssertionError while building index from unicode tiles

2021-10-18 Thread Ticker Berkin
Hi Gerd In imgfmt/app/srt/Sort.java around line 853: // Get the first non-ignorable at this level int c = chars[pos++ & 0xff]; if (!hasPage(c >>> 8)) { I'm at a loss to understand the 0xff mask! am I missing something? Ticker

Re: [mkgmap-dev] java.lang.AssertionError while building index from unicode tiles

2021-10-18 Thread Ticker Berkin
ther names like those for highways, POI etc? > > Gerd > > > Von: mkgmap-dev im Auftrag > von Ticker Berkin > Gesendet: Montag, 18. Oktober 2021 09:58 > An: Development list for mkgmap > Betreff: Re: [mkgmap-dev] java.lang.As

Re: [mkgmap-dev] java.lang.AssertionError while building index from unicode tiles

2021-10-18 Thread Gerd Petermann
for other names like those for highways, POI etc? Gerd Von: mkgmap-dev im Auftrag von Ticker Berkin Gesendet: Montag, 18. Oktober 2021 09:58 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] java.lang.AssertionError while building index from

Re: [mkgmap-dev] java.lang.AssertionError while building index from unicode tiles

2021-10-18 Thread Ticker Berkin
Hi Although 2 16-bit items (surrogate pairs in UTF-16 speak) are required to represent many Chinese characters, this isn't the significant problem in this case. Problem is that resources/sort/cp65001.txt doesn't give ordering to lots of characters; it looks like it covers only about 10,500 of

Re: [mkgmap-dev] java.lang.AssertionError while building index from unicode tiles

2021-10-17 Thread Gerd Petermann
Gesendet: Sonntag, 17. Oktober 2021 13:48 An: mkgmap-dev@lists.mkgmap.org.uk Betreff: Re: [mkgmap-dev] java.lang.AssertionError while building index from unicode tiles In that case, it seems estrange that only 2 of 67 tiles of China map cause problems, doesn't it? El 17/10/21 a las 12:16, Ticker

Re: [mkgmap-dev] java.lang.AssertionError while building index from unicode tiles

2021-10-17 Thread Carlos Dávila
In that case, it seems estrange that only 2 of 67 tiles of China map cause problems, doesn't it? El 17/10/21 a las 12:16, Ticker Berkin escribió: Hi It is most likely that this problem is because Chinese requires 2 UTF16 chars to encode many of its characters - see

Re: [mkgmap-dev] java.lang.AssertionError while building index from unicode tiles

2021-10-17 Thread Ticker Berkin
Hi It is most likely that this problem is because Chinese requires 2 UTF16 chars to encode many of its characters - see https://softwareengineering.stackexchange.com/questions/102205/should-utf-16-be-considered-harmful I think it is only  --index processing where this is a problem mkgmap. I'll

Re: [mkgmap-dev] java.lang.AssertionError while building index from unicode tiles

2021-10-15 Thread Ticker Berkin
Hi I can also reproduce this. I'll investigate, but am no expert on java sort/collation. Ticker ___ mkgmap-dev mailing list mkgmap-dev@lists.mkgmap.org.uk https://www.mkgmap.org.uk/mailman/listinfo/mkgmap-dev

Re: [mkgmap-dev] java.lang.AssertionError while building index from unicode tiles

2021-10-15 Thread Gerd Petermann
sendet: Freitag, 15. Oktober 2021 10:09 An: Development list for mkgmap Betreff: Re: [mkgmap-dev] java.lang.AssertionError while building index from unicode tiles Hi Carlos, I can reproduce the crash. Not sure where to fix this yet... Gerd Von: mkgmap-d

Re: [mkgmap-dev] java.lang.AssertionError while building index from unicode tiles

2021-10-15 Thread Gerd Petermann
Hi Carlos, I can reproduce the crash. Not sure where to fix this yet... Gerd Von: mkgmap-dev im Auftrag von Carlos Dávila Gesendet: Donnerstag, 14. Oktober 2021 18:33 An: Development list for mkgmap Betreff: [mkgmap-dev] java.lang.AssertionError while