dtenedor opened a new pull request, #59681:
URL: https://github.com/apache/spark/pull/59681
### What changes were proposed in this pull request?
This PR allows VALUES lists (inline tables) to contain expressions that are
not constant-foldable, for example:
- nondeterministic functions such as `rand()` and `uuid()`;
- Scala/Java UDFs;
- higher-order functions with lambdas;
- `raise_error()`;
- uncorrelated scalar subqueries.
Such a VALUES list is planned as an `Expand` over a `OneRowRelation`, with
one projection per row, so its rows are evaluated by the existing `ExpandExec`
when the query runs.
VALUES lists that are valid today keep exactly their current behavior: they
go through the same validation, produce the same `LocalRelation` or
`ResolvedInlineTable`, and are evaluated by the unchanged `EvalInlineTables`
rule. The new code path only applies to VALUES lists that the current
validation rejects.
**Analysis**
- `EvaluateUnresolvedInlineTable.validateInputEvaluable` accepts every
expression it accepts today, plus expressions that can be evaluated by
themselves when the query runs. It still rejects the following with the
existing `INVALID_INLINE_TABLE.CANNOT_EVALUATE_EXPRESSION_IN_INLINE_TABLE`
error:
- aggregate, window, and generator functions;
- references to columns of an outer query, e.g. `LATERAL (VALUES (t.a))`;
- subqueries other than uncorrelated scalar subqueries, such as `EXISTS`
and `IN`;
- unevaluable expressions that would have to be rewritten into separate
operators, such as SQL UDFs and Python UDFs.
- If every expression of a VALUES list is foldable or contains a
CURRENT_LIKE expression, which is what the current validation requires, the
list takes the current code path unchanged. Otherwise,
`EvaluateUnresolvedInlineTable` turns it into `Expand(rows, output,
OneRowRelation())`:
- Foldable cells are evaluated into literals during analysis, so
evaluation errors in constant cells are still reported during analysis.
- CURRENT_LIKE cells are replaced with literals at the end of the analysis
phase, like today.
- `ResolveSubquery` now resolves subqueries in `UnresolvedInlineTable`.
- The single-pass resolver throws `ExplicitlyUnsupportedResolverFeature` for
scalar subqueries in VALUES lists, and for VALUES lists that become an
`Expand`. These queries fall back to the fixed-point analyzer.
**Execution**
- No new operators. `ExpandExec` applies the projections to the single row
of its child, so all the rows of the VALUES list are evaluated in one task,
with whole-stage codegen.
- Each projection is evaluated once, for partition 0. As a result, `VALUES
(rand(5)), (rand(5))` returns the same values as `SELECT rand(5) UNION ALL
SELECT rand(5)`.
- Scalar subqueries are planned and executed by `PlanSubqueries` and
`PlanAdaptiveSubqueries` before `ExpandExec` runs.
- Like in a `SELECT` list, `ColumnPruning` removes the expressions of
columns that are not read.
- `ExpandExec` fix: the generated code evaluated a column once per input row
when the column had equal expressions in all projections, even if they were
nondeterministic. For example, a nondeterministic UDF called the same way in
every row of a VALUES list returned the same value in every row, unlike the
interpreted path. Now only deterministic columns are shared.
**Configuration**
- New internal conf `spark.sql.inlineTable.nonFoldableExpressions.enabled`,
default `true`. Setting it to `false` restores the previous behavior.
**References to other DBMSes**
- These engines accept general expressions in VALUES lists, including scalar
subqueries: PostgreSQL, SQL Server, Oracle 23ai, MySQL 8.0.19+ and SQLite.
BigQuery documents scalar subqueries in `INSERT ... VALUES`.
- Snowflake and Hive keep restrictions similar to Spark's current behavior.
### Why are the changes needed?
Today, every expression in a VALUES list must be constant-foldable, with the
exception of expressions like `current_timestamp()`, because the analyzer
evaluates the rows on the driver. So queries such as the following fail:
- `INSERT INTO t VALUES (uuid(), ...)`
- `SELECT * FROM VALUES (rand())`
- `INSERT INTO t VALUES ((SELECT max(id) FROM t) + 1, ...)`
Users have to rewrite them as `INSERT ... SELECT` or as a `UNION ALL` of
`SELECT`s. Most other SQL engines accept these queries.
SPARK-28291 tracks UDFs in VALUES lists. This PR supports Scala/Java UDFs
there, but not SQL or Python UDFs.
### Does this PR introduce _any_ user-facing change?
Yes. Queries that failed before now succeed:
```sql
SELECT * FROM VALUES ('one', rand(5)), ('two', 3.0D) AS data(a, b);
-- Before: [INVALID_INLINE_TABLE.CANNOT_EVALUATE_EXPRESSION_IN_INLINE_TABLE]
-- Cannot evaluate the expression "rand(5)" in inline table
definition.
-- After:
-- one 0.02390696427502892
-- two 3.0
INSERT INTO t VALUES ((SELECT max(id) FROM t) + 1);
-- Before:
[UNSUPPORTED_SUBQUERY_EXPRESSION_CATEGORY.SCALAR_SUBQUERY_IN_VALUES]
-- Scalar subqueries in the VALUES clause.
-- After: inserts the next id.
```
Expressions that still can't be evaluated in a VALUES list fail with
`INVALID_INLINE_TABLE.CANNOT_EVALUATE_EXPRESSION_IN_INLINE_TABLE`. An inline
table with non-foldable expressions shows up as an `Expand` over a
`OneRowRelation` in plans. VALUES lists that are valid today are not affected.
### How was this patch tested?
New tests:
- `InlineTableNonFoldableExpressionsSuite` (22 tests), covering:
- evaluation on executors, in a single task;
- constant lists, which remain a `LocalTableScan`;
- lists that are valid today: the same analyzed plans and results with the
conf on and off, CURRENT_LIKE expressions inside a Scala UDF still evaluated on
the driver, and evaluation errors next to CURRENT_LIKE expressions still
reported by the optimizer;
- nondeterministic semantics: a distinct `uuid()` per row, `rand(5)`
matching `SELECT rand(5)`, stable values when a plan is re-executed, and equal
nondeterministic UDF calls evaluated separately for every row, with whole-stage
codegen on and off;
- `current_timestamp()` evaluated once per query;
- type coercion and nullability;
- uncorrelated scalar subqueries, including `INSERT ... VALUES` that reads
its target table;
- rejected cases: `LATERAL` outer references, `EXISTS`, `IN`, aggregate,
window and generator functions, SQL UDFs, and Python UDFs;
- runtime errors (`raise_error`), and unread columns that are not
evaluated;
- `INSERT` with `DEFAULT` and `uuid()`;
- DataFrame self-joins;
- whole-stage codegen and AQE on and off;
- SQL metrics;
- a 1000-row list;
- the conf set to `false`.
- `ResolveInlineTablesSuite`, covering:
- validation;
- the `Expand` that replaces lists with non-foldable expressions, with
constant cells folded;
- lists without non-foldable expressions resolving to the same plans with
the conf on and off;
- aliased (`DEFAULT`) cells;
- CURRENT_LIKE expressions;
- analysis-time errors in constant cells;
- `CheckAnalysis` of nondeterministic expressions.
- New queries in `inline-table.sql`. The `rand(5)` cases in
`inline-table.sql` and `udf/udf-inline-table.sql` now succeed.
- `SQLQuerySuite`: the SPARK-49659 test runs with the conf set to `false`
and checks that the query now works with the conf on.
Regenerated `scripting/randomly_generated_scripts.sql`:
- 22 scripts used to fail with `SCALAR_SUBQUERY_IN_VALUES` and exit through
their `EXIT HANDLER`. They contain `INSERT ... VALUES` statements with scalar
subqueries, and they now run to completion.
- These scripts share tables, so the data they now insert changes the
control flow of 6 later scripts:
- 3 now reach `UPDATE` statements, which are unsupported for these Parquet
tables.
- 3 no longer reach such statements.
- I verified every changed outcome by running the file with all `EXIT
HANDLER`s removed, once with the conf on and once with it off.
Existing suites:
- `Expand` execution: `WholeStageCodegenSuite`, `PlannerSuite`,
`AdaptivePartialAggregationSuite`, `DatasetUnpivotSuite`,
`DataFrameAggregateSuite`, and the `GROUPING SETS` and UNPIVOT golden files.
- Catalyst analyzer suites, including `AnalysisSuite`, `AnalysisErrorSuite`,
`ResolveSubquerySuite` and `PlanParserSuite`.
- `InlineTableParsingImprovementsSuite`, `ResolveDefaultColumnsSuite`, the
`*InsertSuite`s and the `*SubquerySuite`s.
- The single-pass resolver suites: `ResolverGuardSuite`,
`HybridAnalyzerSuite`, the `*ResolverSuite`s, `ResolutionValidatorSuite`,
`ExplicitlyUnsupportedResolverFeatureSuite`, `DataFrameAnalyzerTestGapsSuite`.
- The `SQLQueryTestSuite` golden files for inline tables, CTEs,
`create_view`, `insert`, session variables and scripting.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Cursor (Claude Opus 5.5)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
