[
https://issues.apache.org/jira/browse/SPARK-61622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-61622:
-----------------------------------
Labels: correctness pull-request-available (was: correctness)
> Joins and IN on UNICODE_AI strings match a sharp s with "ss", but = does not
> ----------------------------------------------------------------------------
>
> Key: SPARK-61622
> URL: https://issues.apache.org/jira/browse/SPARK-61622
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 4.0.3
> Reporter: sepuri sai krishna
> Priority: Major
> Labels: correctness, pull-request-available
>
> Under the case-sensitive, accent-insensitive ICU collations ({{UNICODE_AI}},
> {{en_AI}}, {{de_AI}}, ..., and their {{_RTRIM}} variants), an equi-join
> returns rows for which its own join condition is false, and equality is not
> transitive. Default configuration, reproduced on 4.0.3 (ICU 76.1) and on
> master (ICU 78.3).
> {code:sql}
> CREATE TEMPORARY VIEW t AS SELECT id, CAST(s AS STRING COLLATE de_AI) AS s
> FROM VALUES (1, 'Straße'), (2, 'Strasse') AS v(id, s);
> SELECT a.id, b.id, a.s = b.s FROM t a CROSS JOIN t b;
> -- (1,1,true) (1,2,false) (2,1,false) (2,2,true)
> SELECT a.id, b.id FROM t a JOIN t b ON a.s = b.s;
> -- (1,1) (1,2) (2,1) (2,2): the pairs (1,2) and (2,1) do not satisfy a.s = b.s
> SELECT count(DISTINCT s) FROM t; -- 2
> SELECT id FROM t WHERE s IN ('Strasse', 'x'); -- 2
> SELECT id FROM t WHERE s IN ('Strasse', 'x1', 'x2', 'x3', 'x4', 'x5', 'x6',
> 'x7', 'x8', 'x9', 'x10');
> -- 1 and 2 on master (InSet); 2 on 4.0.3
> -- equality is not transitive: U+FF53 is the fullwidth s
> SELECT 'ß' COLLATE UNICODE_AI = 'ss' COLLATE UNICODE_AI, -- false
> 'ß' COLLATE UNICODE_AI = 'ss' COLLATE UNICODE_AI, -- true
> 'ss' COLLATE UNICODE_AI = 'ss' COLLATE UNICODE_AI; -- true
> {code}
> *Cause*
> Spark compares the strings of an ICU collation in two ways: with
> {{Collator.compare}} (the comparator of the collation: {{=}}, {{<}}, sorting,
> sort-based aggregation, {{In}}), and by their collation keys
> ({{Collator.getCollationKey}}, the sort key of the collation: hash joins,
> hash aggregation, shuffles, {{InSet}}). The two have to agree.
> The CS_AI collations are built with primary strength and the case level
> ({{ks=level1}}, {{kc=true}} in
> {{CollationFactory.CollationSpecICU.buildCollation}}). For this setting the
> two do not agree in ICU4J when one of the strings contains U+00DF (sharp s).
> Plain icu4j 78.3, without Spark:
> {noformat}
> ks=level1 kc=true
> "ß" vs "ss" compare = 1 collation keys equal
> "ß" vs "SS" compare = 1 collation key of "ß" is less
> "ßあ" vs "ssあ" compare = 0 collation keys equal
> "ßあ" vs "SSあ" compare = -1 collation key of "ßあ" is less
> {noformat}
> This is a bug of the fast path that {{Collator.compare}} takes for Latin text
> ({{com.ibm.icu.impl.coll.CollationFastLatin}}). A character that maps to two
> collation elements is held there as two 16-bit "mini CEs" in one {{int}}, and
> {{getCases}} (also {{getSecondaries}}, {{getTertiaries}} and
> {{getQuaternaries}}) tells one element from two with {{if (pair <= 0xffff)}}.
> The code is a port of ICU4C, where {{pair}} is a {{uint32_t}}. In Java the
> {{int}} is negative when the second mini CE has its top bit set, which is the
> case for a sharp s (s + s) in the fast Latin table of every locale, so the
> two elements are taken for one and their case weights are compared unmasked.
> ICU4C returns equal for the same strings ({{ucol_strcoll}}, checked with
> 74.2), and so does ICU4J when a character outside the fast path is appended
> to both strings (the last two lines above). So the collation keys are right:
> at the primary level a sharp s is "ss", and it is lowercase. The primary
> level of the fast path masks the value first and is not affected.
> Among all pairs of one- and two-character strings over the letters up to
> U+024F (36.0 million pairs), {{compare}} and the collation keys disagree on
> the equality of two strings only for this setting, and only when one of them
> contains U+00DF. With the other strengths (UNICODE, UNICODE_CI,
> UNICODE_CI_AI) equality always agrees.
> SPARK-49667 already disallowed the CS_AI collations in the functions that use
> the ICU {{StringSearch}}. Equality, ordering, grouping, joins and IN still
> accept them.
> *Fix*
> For the CS_AI collations, order the strings with a collator of primary
> strength (the one the CI_AI collations use, whose fast path is right), and
> compare the strings that are equal at the primary level by their collation
> keys. Then the comparator and the sort key agree by construction, and only
> the strings that differ by case at most pay for the collation keys. The other
> collations keep {{Collator.compare}}.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]