I’ve found out that sort doesn’t sort strings for many non-Latin scripts at all if the locale you’re using is one of en_US.UTF-8, fr_FR.UTF-8 or fi_FI.UTF-8 (probably others, too, but these are the ones I have tested). For locales ”C” and ko_KR.UTF-8, things work as expected. Here’s a test case:
Open xterm, launch sort and input some lines of Syriac, Ethiopic, Korean, Japanese (Hiragana or Katakana, not Han) or Thai text repeating one of the lines twice. Here’s an example in Syriac: ܡܠܬܐ ܒܝܬܐ ܒܪܢܫܐ ܡܠܬܐ Sort produces the following: ܡܠܬܐ ܒܝܬܐ ܡܠܬܐ ܒܪܢܫܐ Here strings are ordered only according to their length but not characters. Even the two instances of the word ܡܠܬܐ are found on non-adjacent lines (1 and 3). The expected sort order based on Unicode points would be: ܒܝܬܐ ܒܪܢܫܐ ܡܠܬܐ ܡܠܬܐ If you further pass sort’s output to uniq, it produces the following: ܡܠܬܐ ܒܪܢܫܐ Here the word on line 2 ܒܝܬܐ is completely lost since, like sort, uniq seems to consider all Syriac strings of equal length as the same. Although this issue affects locale, I think it is not a locale issue per se, since perl seems to handle similar cases as expected. For instance, the following command produces the expected result: perl -CDS -e 'use locale; use utf8; @str = ("ܡܠܬܐ", "ܒܝܬܐ", "ܒܪܢܫܐ", "ܡܠܬܐ"); foreach $i (sort @str) { print "$i\n"; }' Curiously enough, codepoints in Plane 1 seem to count as two codepoints of the basic plane, so that if you sort | uniq the following (six codepoints of Syriac and three codepoints of Phoenician): ܥܠܝܟܘܢ 𐤁𐤉𐤕 you get ”ܥܠܝܟܘܢ" as the result whereas ”𐤁𐤉𐤕” is lost. This is of course due to the UTF-8 representation of Plane 1 characters as two surrogate characters on the basic plane. Also curiously, LTR scripts seem to conflate with each other and RTL scripts among themselves but not across the directionality line, so that if you sort | uniq the following (three codepoints each in Ethiopic, Hangul, Syriac, Hiragana and Thai): ዘመን 스물셋 ܐܢܐ わたし ฟ้า you are left with: ܐܢܐ ዘመን That’s one line of Syriac and one line of Ethiopic; everything else was lost. This issue does not seem to affect most Indic scripts (Devanagari, Bengali, Telugu etc.) or Arabic. For CJK, things work as expected for the main Unicode block (4E00..9FFF) but not for Extension A (3400..4DBF, such as 㗖 or 㡘 or 㰋). For Greek, monotonic accents work fine but all polytonic letters are conflated (αὐλὸς and αὐλῆς conflate to αὐλῆς). For Hebrew, letters and vowel marks work fine but cantillation marks are conflated. I'm using coreutils 8.28 on Ubuntu 18.04. I first reported this bug on Launchpad at https://bugs.launchpad.net/ubuntu/+source/coreutils/+bug/1774857 but since nobody hasn't reacted for a couple of months, I decided to post the report here.