dongjoon-hyun opened a new pull request, #60038:
URL: https://github.com/apache/spark/pull/60038
### What changes were proposed in this pull request?
`ConstantFolding` folds `size(array(...))` and `size(map(...))` into the
number of elements when every element is free of side effects.
`hasNoSideEffect` treated a `Cast` as free of side effects whenever the session
had ANSI mode off, without looking at the cast's own eval mode. A cast keeps
the eval mode it was analyzed with, so an ANSI cast analyzed with ANSI on (in a
view, a SQL function, or an eagerly analyzed DataFrame) was folded away when
the optimizer ran under a non-ANSI session, and its error was lost.
This PR also checks the cast's own eval mode, so an ANSI cast is never
treated as free of side effects:
```scala
case c: Cast if !conf.ansiEnabled && c.evalMode != EvalMode.ANSI =>
hasNoSideEffect(c.child)
```
The session check stays. Some LEGACY casts can still fail, such as
`CAST(INTERVAL '1000' SECOND AS TINYINT)` (`CAST_OVERFLOW`). Checking only the
cast's eval mode would start folding them in ANSI sessions, where they are not
folded today, and drop those errors, for example for a view created with ANSI
off and queried with ANSI on.
### Why are the changes needed?
The last query below should fail with `CAST_INVALID_INPUT` like the one on
`v2`, but returns `1`.
```sql
CREATE TABLE src(a STRING) USING parquet;
INSERT INTO src VALUES ('abc');
SET spark.sql.ansi.enabled=true;
CREATE VIEW v1 AS SELECT size(array(CAST(a AS INT))) AS d FROM src;
CREATE VIEW v2 AS SELECT CAST(a AS INT) AS d FROM src;
SET spark.sql.ansi.enabled=false;
SELECT * FROM v2; -- CAST_INVALID_INPUT
SELECT * FROM v1; -- It should fail with CAST_INVALID_INPUT, but returns 1
```
A SQL function created with ANSI on and a DataFrame analyzed with ANSI on
behave the same when they run with ANSI off.
This was found in the review of #59965, which analyzes the generation
expression of a DSv2 generated column with ANSI on. With that PR, a column `d
INT GENERATED ALWAYS AS (size(array(CAST(a AS INT))))` still accepts `INSERT
INTO t (a) VALUES ('abc')` from a non-ANSI writer and stores `d = 1`, while an
ANSI writer gets `CAST_INVALID_INPUT`.
The session-only check was added by SPARK-37392 (#34823) and has been in
Spark since [v3.3.0](https://github.com/apache/spark/tree/v3.3.0) (2022-06-09).
### Does this PR introduce _any_ user-facing change?
Yes. Under a non-ANSI session, `size(array(...))` and `size(map(...))` over
an ANSI cast now raise the cast's error instead of returning the number of
elements. Nothing changes under an ANSI session, and LEGACY and TRY casts are
folded as before.
### How was this patch tested?
- Added `SPARK-60820: Do not fold the size of CreateArray over an ANSI cast
in a non-ANSI session` to `ConstantFoldingSuite`. It checks every eval mode of
a cast under ANSI on and off, and only LEGACY and TRY casts under a non-ANSI
session are folded. Without this change, the ANSI cast under a non-ANSI session
is folded to `1` and the test fails.
- Ran all suites in `org.apache.spark.sql.catalyst.optimizer`,
`SQLQueryTestSuite` for `array.sql`, `cast.sql`, `try_cast.sql`, `explain*.sql`
and the `nonansi` variants, `GeneratorFunctionSuite`, `SimpleSQLViewSuite` and
`DataFrameFunctionsSuite`. All pass and no golden files change.
- Ran the view, SQL function and DataFrame cases above before and after this
change. With this change, they fail with `CAST_INVALID_INPUT` under a non-ANSI
session. A view created with ANSI off that casts an overflowing interval to
`TINYINT` still fails with `CAST_OVERFLOW` under an ANSI session, as before.
- Applied this change on top of #59965 and ran the generated column case
above. Both writers now get `CAST_INVALID_INPUT`.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
