sepuri sai krishna created SPARK-61622:
------------------------------------------

             Summary: Joins and IN on UNICODE_AI strings match a sharp s with 
"ss", but = does not
                 Key: SPARK-61622
                 URL: https://issues.apache.org/jira/browse/SPARK-61622
             Project: Spark
          Issue Type: Bug
          Components: SQL
    Affects Versions: 4.0.3
            Reporter: sepuri sai krishna


Under the case-sensitive, accent-insensitive ICU collations ({{UNICODE_AI}}, 
{{en_AI}}, {{de_AI}}, ..., and their {{_RTRIM}} variants), an equi-join returns 
rows for which its own join condition is false, and equality is not transitive. 
Default configuration, reproduced on 4.0.3 (ICU 76.1) and on master (ICU 78.3).

{code:sql}
CREATE TEMPORARY VIEW t AS SELECT id, CAST(s AS STRING COLLATE de_AI) AS s
  FROM VALUES (1, 'Straße'), (2, 'Strasse') AS v(id, s);

SELECT a.id, b.id, a.s = b.s FROM t a CROSS JOIN t b;
-- (1,1,true) (1,2,false) (2,1,false) (2,2,true)

SELECT a.id, b.id FROM t a JOIN t b ON a.s = b.s;
-- (1,1) (1,2) (2,1) (2,2): the pairs (1,2) and (2,1) do not satisfy a.s = b.s

SELECT count(DISTINCT s) FROM t;                    -- 2
SELECT id FROM t WHERE s IN ('Strasse', 'x');       -- 2
SELECT id FROM t WHERE s IN ('Strasse', 'x1', 'x2', 'x3', 'x4', 'x5', 'x6', 
'x7', 'x8', 'x9', 'x10');
-- 1 and 2 on master (InSet); 2 on 4.0.3

-- equality is not transitive: U+FF53 is the fullwidth s
SELECT 'ß' COLLATE UNICODE_AI = 'ss' COLLATE UNICODE_AI,   -- false
  'ß' COLLATE UNICODE_AI = 'ss' COLLATE UNICODE_AI,       -- true
  'ss' COLLATE UNICODE_AI = 'ss' COLLATE UNICODE_AI;      -- true
{code}

*Cause*

Spark compares the strings of an ICU collation in two ways: with 
{{Collator.compare}} (the comparator of the collation: {{=}}, {{<}}, sorting, 
sort-based aggregation, {{In}}), and by their collation keys 
({{Collator.getCollationKey}}, the sort key of the collation: hash joins, hash 
aggregation, shuffles, {{InSet}}). The two have to agree.

The CS_AI collations are built with primary strength and the case level 
({{ks=level1}}, {{kc=true}} in 
{{CollationFactory.CollationSpecICU.buildCollation}}). For this setting the two 
do not agree in ICU4J when one of the strings contains U+00DF (sharp s). Plain 
icu4j 78.3, without Spark:

{noformat}
ks=level1 kc=true
  "ß"  vs "ss"    compare = 1    collation keys equal
  "ß"  vs "SS"    compare = 1    collation key of "ß" is less
  "ßあ" vs "ssあ"  compare = 0    collation keys equal
  "ßあ" vs "SSあ"  compare = -1   collation key of "ßあ" is less
{noformat}

This is a bug of the fast path that {{Collator.compare}} takes for Latin text 
({{com.ibm.icu.impl.coll.CollationFastLatin}}). A character that maps to two 
collation elements is held there as two 16-bit "mini CEs" in one {{int}}, and 
{{getCases}} (also {{getSecondaries}}, {{getTertiaries}} and 
{{getQuaternaries}}) tells one element from two with {{if (pair <= 0xffff)}}. 
The code is a port of ICU4C, where {{pair}} is a {{uint32_t}}. In Java the 
{{int}} is negative when the second mini CE has its top bit set, which is the 
case for a sharp s (s + s) in the fast Latin table of every locale, so the two 
elements are taken for one and their case weights are compared unmasked. ICU4C 
returns equal for the same strings ({{ucol_strcoll}}, checked with 74.2), and 
so does ICU4J when a character outside the fast path is appended to both 
strings (the last two lines above). So the collation keys are right: at the 
primary level a sharp s is "ss", and it is lowercase. The primary level of the 
fast path masks the value first and is not affected.

Among all pairs of one- and two-character strings over the letters up to U+024F 
(36.0 million pairs), {{compare}} and the collation keys disagree on the 
equality of two strings only for this setting, and only when one of them 
contains U+00DF. With the other strengths (UNICODE, UNICODE_CI, UNICODE_CI_AI) 
equality always agrees.

SPARK-49667 already disallowed the CS_AI collations in the functions that use 
the ICU {{StringSearch}}. Equality, ordering, grouping, joins and IN still 
accept them.

*Fix*

For the CS_AI collations, order the strings with a collator of primary strength 
(the one the CI_AI collations use, whose fast path is right), and compare the 
strings that are equal at the primary level by their collation keys. Then the 
comparator and the sort key agree by construction, and only the strings that 
differ by case at most pay for the collation keys. The other collations keep 
{{Collator.compare}}.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to