Author: atsushi
Date: 2005-05-26 06:09:02 -0400 (Thu, 26 May 2005)
New Revision: 45035
Modified:
branches/atsushi/mcs/class/corlib/Mono.Globalization.Unicode/ChangeLog
branches/atsushi/mcs/class/corlib/Mono.Globalization.Unicode/Collation-notes.txt
branches/atsushi/mcs/class/corlib/Mono.Globalization.Unicode/create-mscompat-collation-table.cs
Log:
2005-05-26 Atsushi Enomoto <[EMAIL PROTECTED]>
* Collation-notes.txt : more analysis on "letters".
* create-mscompat-collation-table.cs : more proof of concepts.
Modified: branches/atsushi/mcs/class/corlib/Mono.Globalization.Unicode/ChangeLog
===================================================================
--- branches/atsushi/mcs/class/corlib/Mono.Globalization.Unicode/ChangeLog
2005-05-26 07:47:00 UTC (rev 45034)
+++ branches/atsushi/mcs/class/corlib/Mono.Globalization.Unicode/ChangeLog
2005-05-26 10:09:02 UTC (rev 45035)
@@ -1,3 +1,8 @@
+2005-05-26 Atsushi Enomoto <[EMAIL PROTECTED]>
+
+ * Collation-notes.txt : more analysis on "letters".
+ * create-mscompat-collation-table.cs : more proof of concepts.
+
2005-05-25 Atsushi Enomoto <[EMAIL PROTECTED]>
* Collation-notes.txt : more info. Started letter sortkey analysis
Modified:
branches/atsushi/mcs/class/corlib/Mono.Globalization.Unicode/Collation-notes.txt
===================================================================
---
branches/atsushi/mcs/class/corlib/Mono.Globalization.Unicode/Collation-notes.txt
2005-05-26 07:47:00 UTC (rev 45034)
+++
branches/atsushi/mcs/class/corlib/Mono.Globalization.Unicode/Collation-notes.txt
2005-05-26 10:09:02 UTC (rev 45035)
@@ -701,27 +701,80 @@
10 culture dependent letters (general)
0F: 386-3F2 ... Greek and Coptic
- 386-3CF: 0F 02 - 0F 19 (consider primary equivalents)
- 3D0-3EF: 0F 40 - 0F 54
+ 386-3CF: [0F 02] - [0F 19] (consider primary equivalents)
+ 3D0-3EF: [0F 40] - [0F 54]
10: 400-4E9 ... Cyrillic.
For 400-45F and 4B1, they are mostly UCA DUCET order.
After that 460-481 follows, by codepoint.
(490-4FF except for 4B1 and Cyrillic supplementary are unused.)
11: 531-586 ... Armenian.
Simply sorted by codepoint (handle case).
- 12: 5D0-5F2 ... Hebrew
+ 12: 5D0-5F2 ... Hebrew.
Codepoint order (handle case).
13: 621-6D5 plus 670 (NonSpacingMark) ... Arabic
+ Area 1:
+ They look like ordered by Arabic Presentation Form B except
+ for FE95, and considering diacritical equivalents maybe based
+ on the primary character area (621-6D5).
+ There are still some special characters: 67E,686,698,6AF ...
+ which might not have equivalent characters (I wonder how they
+ are inserted into the presentation form B map).
+ Area 2:
+ 674-6D5 : by codepoint from [13 84].
+ 14: 901-963 exc. 93A-93D 950-954 ... Devanagari.
+ For <905 codepoint order, x2 from [14 04].
+ For 905-939 codepoint order, x4 from [14 0B].
+ For 93E-94D codepoint order, x2 from [14 DA].
+ 15: 982-9FA ... Bengali. Actually all UnicodeCategories except for
+ NonSpacingMark, DecimalDigitNumber and OtherNumber.
+ For <9E0 simple codepoint order from [15 02].
+ For >9E0 simple codepoint order from [15 3B].
+ 16: A05-A74 exc. A3C A4D A66-A71 ... Gurmukhi.
+ The same as UCA order, x4 from [16 04].
+ 17: A81-AE0 exc. ABC-ABD ... Gujarati.
+ Mostly equivalent to UCA, but insert {AB3,A81-A83} before AB9,
+ x4 from [17 04].
+ 18: B00-B70 ... Oriya
+ All but NonSpacingMark and DecimalDigitNumber, by codepoint.
+ 19: B80-BFF ... Tamil
+ BD7 is special : [19 02].
+ B82-B93 (vowels) : x2 from [19 0A].
+ B94 (vowel AU) : [19 24]
+ For consonant order Windows has native Tamil order which is
+ different from UCA.
+ http://www.nationmaster.com/encyclopedia/Tamil-alphabet
+ (The order is still different in "Grantha" order from TAM.)
+ So, we should just hold constant array for consonants.
+ And put them in order, x4 form [19 26].
+ 1A: C00-C61 ... Telugu.
+ C55 and C56 are ignored (C5x line and remaining part of C6x
+ line just look like ignored).
+ C60 and C61 are specially placed. C60 after C0B, C61 after C0C.
+ Except for above, by codepoint, x3 from [1A 04].
+ 1B: C80-CE5 ... Kannada.
+ CD5,CD6 (and CE6-CEF: DecimalDigitNumber) are ignored.
+ by codepoint, 3x from [1B 04].
+ 1C: D02-D40 ... Malayalam.
+ by simple codepoint from [1C 02].
+ (1D: Sinhala ... totally ignored?)
+ 1E: E00-E44 ... Thai.
+ preceding vowels (E40-E44) by codepoint [1E 02 - 1E 06]
+ consonants (E01-E2A) by codepoint, x6 from [1E 07].
+ 1F: E2B-E5B,E80-EDF ... Thai / Lao. (Thai breaks the category wall.)
+ Thai:
+ remaining consonants (E2B-E2E) by codepoint, x6 from [1E 07].
+ remaining vowels (E2F-E3A) by codepoint.
+ E45,E46,E4E,E4F,E5A,E5B
+ Lao:
+ E80-EDF by codepoint from [1F 02].
+ 21: 10A0-10FF ... Georgian
+ Mostly equal to UCA order, but swap 10E3 <-> 10F3,
+ x5 from [21 05].
- 14: 901-963 exc. 93C-93D 950-954
- 15: 982-9FA exc. NonSpacingMark DecimalDigitNumber OtherNumber
- 16: A05-A74 exc. A3C A4D A66-A71
- 17: A81-AE0 exc. ABC-ABD
+ (Tibetan, Limbu, Tagalog, Hanunoo, Buhid, Tagbanwa, Myanmar, Kumer,
+ Tai-Le, Mongolian, Cherokee, Canadian-Aboriginal, Ogham, Runic
+ are ignored)
- ...
-
- (21) georgian letters
-
11 (22) japanese kana letters and symbols, not in codepoint order
For single character, the sortkeys look like:
Modified:
branches/atsushi/mcs/class/corlib/Mono.Globalization.Unicode/create-mscompat-collation-table.cs
===================================================================
---
branches/atsushi/mcs/class/corlib/Mono.Globalization.Unicode/create-mscompat-collation-table.cs
2005-05-26 07:47:00 UTC (rev 45034)
+++
branches/atsushi/mcs/class/corlib/Mono.Globalization.Unicode/create-mscompat-collation-table.cs
2005-05-26 10:09:02 UTC (rev 45035)
@@ -230,7 +230,172 @@
AddLetterMap ((char) i, 0x12, true);
// Arabic
+ fillIndex [0x13] = 0x3;
+ /*
+ FIXME: I still need more love on presentation form B
+ */
+ fillIndex [0x13] = 0x84;
+ for (int i = 0x0674; i < 0x06D6; i++)
+ if (Char.IsLetter ((char) i))
+ AddLetterMap ((char) i, 0x13, true);
+ // Devanagari
+ for (int i = 0x0901; i < 0x0905; i++) {
+ if (Char.IsLetter ((char) i)) {
+ AddLetterMap ((char) i, 0x14, false);
+ fillIndex [0x14] += 2;
+ }
+ }
+ for (int i = 0x0905; i < 0x093A; i++) {
+ if (Char.IsLetter ((char) i)) {
+ AddLetterMap ((char) i, 0x14, false);
+ fillIndex [0x14] += 4;
+ }
+ }
+ for (int i = 0x093E; i < 0x094F; i++) {
+ if (Char.IsLetter ((char) i)) {
+ AddLetterMap ((char) i, 0x14, false);
+ fillIndex [0x14] += 2;
+ }
+ }
+
+ // Bengali
+ fillIndex [0x15] = 02;
+ for (int i = 0x0980; i < 0x9FF; i++) {
+ if (i == 0x09E0)
+ fillIndex [0x15] = 0x3B;
+ switch (Char.GetUnicodeCategory ((char) i)) {
+ case NonSpacingMark:
+ case DecimalDigitNumber:
+ case OtherNumber:
+ continue;
+ }
+ AddLetterMap ((char) i, 0x15, true);
+ }
+
+ // Gurmukhi
+ fillIndex [0x16] = 02;
+ // FIXME: orderedGurmukhi needed from UCA
+ for (int i = 0; i < orderedGurmukhi.Length; i++) {
+ char c = orderedGurmukhi [i];
+ if (c == '\u0A3C' || c == '\u0A4D' ||
+ '\u0A66' <= c && c <= '\u0A71')
+ continue;
+ AddLetterMap (c, 0x16, false);
+ fillIndex [0x16] += 4;
+ }
+
+ // Gujarati
+ fillIndex [0x17] = 02;
+ // FIXME: orderedGujarati needed from UCA
+ for (int i = 0; i < orderedGujarati.Length; i++) {
+ char c = orderedGujarati [i];
+ AddLetterMap (c, 0x17, false);
+ fillIndex [0x17] += 4;
+ }
+
+ // Oriya
+ fillIndex [0x18] = 02;
+ for (int i = 0x0B00; i < 0x0B7F; i++) {
+ switch (Char.GetUnicodeCategory ((char) i)) {
+ case NonSpacingMark:
+ case DecimalDigitNumber:
+ continue;
+ }
+ AddLetterMap ((char) i, 0x18, true);
+ }
+
+ // Tamil
+ fillIndex [0x19] = 2;
+ AddCharMap ('\u0BD7', 0x19, false);
+ fillIndex [0x19] = 0xA;
+ // vowels
+ for (int i = 0x0BD7; i < 0x0B94; i++) {
+ if (Char.IsLetter ((char) i) {
+ AddCharMap ((char) i, 0x19, false);
+ fillIndex [0x19] += 2;
+ }
+ }
+ // special vowel
+ fillIndex [0x19] = 0x24;
+ AddCharMap ('\u0B94', 0x19, false);
+ fillIndex [0x19] = 0x26;
+ // FIXME: we need to have constant array for Tamil
+ // consonants. Windows have almost similar sequence
+ // to TAM from tamilnet but a bit different in Grantha
+ for (int i = 0; i < orderedTamil.Length; i++) {
+ char c = orderedGujarati [i];
+ AddLetterMap (c, 0x19, false);
+ fillIndex [0x19] += 4;
+ }
+
+ // Telugu
+ fillIndex [0x1A] = 0x4;
+ for (int i = 0x0C00; i < 0x0C62; i++) {
+ if (i == 0x0C55 || i == 0x0C56)
+ continue; // skip
+ AddCharMap ((char) i, 0x1A, false);
+ fillIndex [0x1A] += 3;
+ char supp = (i == 0x0C0B) ? '\u0C60':
+ i == 0x0C0C ? '\u0C61' : char.MinValue;
+ if (supp == char.MinValue)
+ continue;
+ AddCharMap (supp, 0x1A, false);
+ fillIndex [0x1A] += 3;
+ }
+
+ // Kannada
+ fillIndex [0x1B] = 4;
+ for (int i = 0x0C80; i < 0x0CE5; i++) {
+ if (i == 0x0CD5 || i == 0x0CD6)
+ continue; // ignore
+ AddCharMap ((char) i, 0x1B, false);
+ fillIndex [0x1B] += 3;
+ }
+
+ // Malayalam
+ fillIndex [0x1C] = 2;
+ for (int i = 0x0D02; i < 0x0D61; i++)
+ if (!IsIgnorable ((char) i))
+ AddCharMap ((char) i, 0x1C, true);
+
+ // Thai ... note that it breaks 0x1E wall after E2B!
+ // Also, all Thai characters have level 2 value 3.
+ fillIndex [0x1E] = 2;
+ for (int i = 0xE44; i < 0xE48; i++)
+ AddThaiCharMap ((char) i, 0x1E, true);
+ for (int i = 0xE01; i < 0xE2B; i++) {
+ AddThaiCharMap ((char) i, 0x1E, false);
+ fillIndex [0x1E] += 6;
+ }
+ fillIndex [0x1F] = 5;
+ for (int i = 0xE2B; i < 0xE30; i++) {
+ AddThaiCharMap ((char) i, 0x1F, false);
+ fillIndex [0x1F] += 6;
+ }
+ for (int i = 0xE30; i < 0xE3B; i++)
+ AddThaiCharMap ((char) i, 0x1F, true);
+ // some Thai characters remains.
+ char [] specialThai = new char [] {'\u0E45', '\u0E46',
+ '\u0E4E', '\u0E4F', '\u0E5A', '\u0E5B'};
+ foreach (char c in specialThai)
+ AddThaiCharMap (c, 0x1F, true);
+
+ // Lao
+ fillIndex [0x1F] = 2;
+ for (int i = 0xE80; i < 0xEDF; i++)
+ if (Char.IsLetter ((char) i))
+ AddCharMap ((char) i, 0x1F, true);
+
+ // Georgian
+ // FIXME: we need an array in UCA order.
+ fillIndex [0x21] = 5;
+ for (int i = 0; i < orderedGeorgian.Length; i++) {
+ char c = orderedGeorgian [i];
+ AddLetterMap (c, 0x21, false);
+ fillIndex [0x21] += 5;
+ }
+
#endregion
}
_______________________________________________
Mono-patches maillist - [email protected]
http://lists.ximian.com/mailman/listinfo/mono-patches
