[
https://issues.apache.org/jira/browse/CALCITE-7812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18120350#comment-18120350
]
Lino Rosa commented on CALCITE-7812:
------------------------------------
Maybe it helps when we're faced with what Spark does with casts of ROW types.
{code:java}
SELECT named_struct('x', 1) AS src,
-- x: nullable = false
CAST(named_struct('x', 1) AS STRUCT<x: BIGINT>) AS no_hint,
-- x: nullable = true
CAST(named_struct('x', 1) AS STRUCT<x: BIGINT NOT NULL>) AS not_null
-- x: nullable = false
root
|-- src: struct (nullable = false)
| |-- x: integer (nullable = false) -- originally x was nullable
|-- no_hint: struct (nullable = false)
| |-- x: long (nullable = true) -- when the CAST did not specify
nullability, x became nullable
|-- not_null: struct (nullable = false)
| |-- x: long (nullable = false) -- when the CAST specify NOT
NULL{code}
The point to take from this is that any CAST has to specify nullability.
There's no way to omit it because omission implies nullable.
If we try to solve this during unparse, we'd have to basically do type
inference during unparse in order to figure out whether the CAST should be into
a NULL-able or a NOT NULL. Doing that during unparse is a bit of a minefield.
---
This goes back to why I suggested the real solution here is probably allow
overriding the definition of built-in functions (e.g. SUM) so that type
coercion knows to set things to NULL or NOT NULL when injecting the
corresponding CAST. Then everything downstream from that just works.
Seems to be a persistent problem for users, it's possible to see usages trying
to solve this a number of ways:
* Here it's overriding `deriveSumType` to make it nullable
**
https://github.com/apache/impala/blob/3d4d47dc30c9913d34bd0ac0331fc15d31e5dab5/java/calcite-planner/src/main/java/org/apache/impala/calcite/type/ImpalaTypeSystemImpl.java#L177
**
[https://github.com/linkedin/coral/commit/8e134df7a849ef2be0e527f8f748d9009ce3398f]
* I think I've seen other projects solving this by wrapping another SUM on top
of the existing SUM, or overriding it entirely. But they add lots of trickery
to make this play nice with existing Calcite code.
> Implicit type coercion emits CASTs in unparsed SQL that the target dialect
> rejects
> ----------------------------------------------------------------------------------
>
> Key: CALCITE-7812
> URL: https://issues.apache.org/jira/browse/CALCITE-7812
> Project: Calcite
> Issue Type: Bug
> Reporter: Lino Rosa
> Assignee: krooswu
> Priority: Major
>
> h2. Problem
> When type coercion is on, {{TypeCoercionImpl}} rewrites the validated
> {{SqlNode}} tree. It wraps operands and select items in {{{}CAST(x AS T){}}},
> where {{T}} is the type Calcite's own rules chose. These casts become
> ordinary {{RexCall(CAST)}} nodes after {{{}SqlToRelConverter{}}}.
> {{RelToSqlConverter}} then unparses them into the SQL sent to the target
> engine.
> That is fine only when Calcite and the engine type the expression in a
> compatible way. Otherwise it generates invalid queries.
> h4. Example: struct fields and {{SUM}} nullability on Spark
> {code:java}
> -- target: t2 (s ROW(a DOUBLE))
> -- source: t1 (k INT NOT NULL, x BIGINT NOT NULL)
> INSERT INTO t2 (s)
> SELECT ROW(SUM(x)) FROM t1 GROUP BY k {code}
> # Calcite types {{SUM}} as {{{}BIGINT NOT NULL{}}}, because {{x}} is {{NOT
> NULL}} and the query is grouped.
> # {{coerceColumnType}} wraps the row in a {{CAST}} to the target struct type.
> # {{SqlTypeUtil.convertTypeToSpec}} loses per-field nullability
> (CALCITE-6932). The unparsed cast spells the field {{{}NOT NULL{}}}.
> # Spark types {{SUM}} as nullable. It refuses to cast a struct with a
> nullable field to one with a {{NOT NULL}} field, so the query fails at
> analysis. Fixing CALCITE-6932 alone would not settle this. The cast still
> carries Calcite's view of nullability, which is correct for Calcite and wrong
> for Spark. The engine would have accepted the uncast expression.
> h4. Existing workaround is not enough
> {{SqlDialect.supportsImplicitTypeCoercion}} together with
> {{SqlImplementor.stripCastFromString}} already recognizes that engines coerce
> for themselves. Its scope is limited but its existing is promising and may
> lead to a fix.
> h3. Proposals
> h4. 1. Carry the coerced type outside the tree
> h4. The validator would record the coerced type without rewriting the tree,
> and conversion would pick it up from there. This is the cleanest option but
> hard today, because the tree is the only place coercion is stored. Much of
> the downstream logic (e.g. {{FamilyOperandTypeChecker, }}{{SqlToRelConverter)
> rely on the existence of this CAST.}}
> h4. 2. Mark coercion casts, then let the dialect decide (preferred)
> h4. Keep inserting a cast, but make it distinguishable from a user cast. Two
> ways to do that:
> * A dedicated operator of kind {{{}CAST{}}}, for example
> {{{}SqlStdOperatorTable.IMPLICIT_CAST{}}}.
> * A flag on the cast.
> The marker has to survive {{{{{}SqlToRelConverter{}}}}} and appear on the
> {{{}RexCall{}}}. Because its kind stays {{{}CAST{}}}, {{RexSimplify}} and the
> rules keep treating it as a cast.
> And here we go back to {{SqlDialect.supportsImplicitTypeCoercion.}} We can
> now simply unparse this marker as either a \{{CAST }}or not unparse anything
> depending on that flag.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
