srielau commented on code in PR #58080:
URL: https://github.com/apache/spark/pull/58080#discussion_r3816056956
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CollationTypeCoercion.scala:
##########
@@ -427,14 +446,13 @@ object CollationTypeCoercion extends SQLConfHelper {
}
}
- (left.strength.priority, right.strength.priority) match {
- case (leftPriority, rightPriority) if leftPriority == rightPriority =>
- if (left.sameType(right)) left
- else handleMismatch()
+ val winner =
+ if (left.strength.priority <= right.strength.priority) left else right
- case (leftPriority, rightPriority) =>
- if (leftPriority < rightPriority) left
- else right
+ StringHelper.tightestCommonString(left.stringType, right.stringType) match
{
+ case Some(lct) => StringTypeWithContext(lct, winner.strength)
Review Comment:
Fixed in 4e9205b.
`changeType` now nests `Cast(existingCast, lct)` when the source is CHAR and
the LCT is not. CHAR-to-wider-CHAR still retargets (padding is preserved).
Analyzer goldens for the collated CHAR vs VARCHAR compare/IN now show the
nested `cast(cast(a as char(2) collate UTF8_LCASE) as varchar(2) collate
UTF8_LCASE)` plan, same shape as the uncollated case. That pair is also in the
SPARK-58802 dual-run matrix (expects `false`).
##########
sql/core/src/test/resources/sql-tests/inputs/charvarchar-standard-semantics.sql:
##########
@@ -66,22 +66,105 @@ SELECT typeof(reverse(array(1, 2)));
SELECT typeof(str_to_map(cast('a:1,b:2' AS CHAR(7))));
SELECT typeof(c0) FROM (SELECT json_tuple(cast('{"a":"1"}' AS CHAR(9)), 'a')
AS c0);
--- R2 with collation. A declared collation survives the CAST and an LCT over
equally constrained
--- operands. The mixed-length case (CHAR(2) with CHAR(4), same collation) is
deliberately not
--- covered here: CollationTypeCoercion reads the differing lengths as a
collation mismatch and
--- yields an indeterminate collation. That predates this change (it reproduces
under
--- spark.sql.preserveCharVarcharTypeInfo) and is tracked separately, so
goldening it would
--- normalize the bug.
+-- Collation survives CAST and LCT. Mixed lengths with the same collation
widen to max(n, m);
+-- they must not collapse to an indeterminate collation.
SELECT typeof(cast('a' AS CHAR(2) COLLATE UTF8_LCASE));
SELECT typeof(coalesce(
cast('a' AS CHAR(2) COLLATE UTF8_LCASE), cast('bb' AS CHAR(2) COLLATE
UTF8_LCASE)));
+SELECT typeof(coalesce(
+ cast('a' AS CHAR(2) COLLATE UTF8_LCASE), cast('bb' AS CHAR(4) COLLATE
UTF8_LCASE)));
+SELECT concat('<', coalesce(
+ cast('a' AS CHAR(2) COLLATE UTF8_LCASE), cast('bb' AS CHAR(4) COLLATE
UTF8_LCASE)), '>');
+SELECT typeof(coalesce(
+ cast('a' AS CHAR(2) COLLATE UTF8_LCASE), cast('bb' AS VARCHAR(4) COLLATE
UTF8_LCASE)));
+-- Mixed strength, same collation: Implicit string CAST CHAR(2) vs Default
+-- non-string CAST CHAR(4). Length still widens to max(n, m); the COLLATE
+-- operator itself is STRING, so it is not used here.
+SELECT typeof(coalesce(
+ cast('a' AS CHAR(2) COLLATE UTF8_LCASE),
+ cast(1 AS CHAR(4) COLLATE UTF8_LCASE)));
+SELECT concat('<', coalesce(
+ cast('a' AS CHAR(2) COLLATE UTF8_LCASE),
+ cast(1 AS CHAR(4) COLLATE UTF8_LCASE)), '>');
--- UNION LCT
+-- Set operations and multi-row VALUES share the same LCT as COALESCE.
SELECT typeof(c) FROM (
SELECT cast('a' AS VARCHAR(3)) AS c
UNION ALL
SELECT cast('abcd' AS VARCHAR(8)) AS c
) t LIMIT 1;
+SELECT typeof(c) FROM (
+ SELECT cast('a' AS CHAR(2)) AS c
+ UNION ALL
+ SELECT cast('bb' AS CHAR(4)) AS c
+) t LIMIT 1;
+SELECT concat('<', c, '>') FROM (
+ SELECT cast('a' AS CHAR(2)) AS c
+ UNION ALL
+ SELECT cast('bb' AS CHAR(4)) AS c
+) t;
+SELECT typeof(c) FROM (
+ SELECT cast('a' AS CHAR(2)) AS c
+ UNION
+ SELECT cast('a' AS CHAR(4)) AS c
+) t;
+SELECT concat('<', c, '>') FROM (
+ SELECT cast('a' AS CHAR(2)) AS c
+ UNION
+ SELECT cast('a' AS CHAR(4)) AS c
+) t;
+SELECT typeof(c) FROM (
+ SELECT cast('ab' AS CHAR(2)) AS c
+ INTERSECT
+ SELECT cast('ab' AS CHAR(4)) AS c
+) t;
+SELECT concat('<', c, '>') FROM (
+ SELECT cast('ab' AS CHAR(2)) AS c
+ INTERSECT
+ SELECT cast('ab' AS CHAR(4)) AS c
+) t;
+-- Non-empty EXCEPT: after widen, 'ab ' is not 'xy '.
+SELECT typeof(c) FROM (
+ SELECT cast('ab' AS CHAR(2)) AS c
+ EXCEPT
+ SELECT cast('xy' AS CHAR(4)) AS c
+) t;
+SELECT concat('<', c, '>') FROM (
+ SELECT cast('ab' AS CHAR(2)) AS c
+ EXCEPT
+ SELECT cast('xy' AS CHAR(4)) AS c
+) t;
+SELECT typeof(c) FROM (VALUES
+ (cast('a' AS CHAR(2))),
+ (cast('bb' AS CHAR(4)))
+) t(c);
+SELECT concat('<', c, '>') FROM (VALUES
+ (cast('a' AS CHAR(2))),
+ (cast('bb' AS CHAR(4)))
+) t(c);
+
+-- Comparison and IN: both sides (including the IN left-hand side) are cast to
the LCT of all
+-- participants. Casting to CHAR pads, so CHAR vs CHAR of different lengths
compares equal after
+-- widen; casting to VARCHAR/STRING keeps the CHAR pad, so CHAR 'a' (stored as
'a ') is not equal
+-- to VARCHAR/STRING 'a' unless the other side carries the same trailing
blank. Trailing-blank
+-- ignoring is a collation concern (RTRIM), not a type-level PAD SPACE policy.
+SELECT cast('a' AS CHAR(2)) = cast('a' AS CHAR(4));
+SELECT cast('a' AS CHAR(2)) = cast('a' AS VARCHAR(2));
+SELECT cast('a' AS CHAR(2)) = cast('a ' AS VARCHAR(2));
+SELECT cast('a' AS CHAR(2)) = 'a';
+SELECT cast('a' AS CHAR(2)) = 'a ';
+SELECT cast('a' AS CHAR(2) COLLATE UTF8_BINARY_RTRIM) = 'a';
+SELECT cast('a' AS CHAR(2) COLLATE UTF8_BINARY_RTRIM) =
+ cast('a' AS CHAR(4) COLLATE UTF8_BINARY_RTRIM);
+SELECT cast('a' AS CHAR(2)) IN (cast('a' AS CHAR(4)));
+SELECT cast('a' AS CHAR(2)) IN (cast('a' AS VARCHAR(2)));
+SELECT cast('a' AS CHAR(2)) IN (cast('a ' AS VARCHAR(2)));
+SELECT cast('a' AS CHAR(2)) IN ('a', 'b');
+SELECT cast('a' AS CHAR(2)) IN ('a ', 'b');
+-- Three-part IN: LHS CHAR(2) and list CHAR(4)/VARCHAR(3) all widen to
VARCHAR(4).
+SELECT cast('a' AS CHAR(2)) IN (cast('a' AS CHAR(4)), cast('b' AS VARCHAR(3)));
+SELECT cast('a' AS CHAR(2) COLLATE UTF8_LCASE) = cast('a' AS VARCHAR(2)
COLLATE UTF8_LCASE);
Review Comment:
Agreed -- that was the retargeted CAST, not intended equality. Both goldens
are `false` now, with a comment that UTF8_LCASE is not RTRIM and must keep the
nested CHAR pad.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
