srielau opened a new pull request, #59874:
URL: https://github.com/apache/spark/pull/59874
### What changes were proposed in this pull request?
Lets `parse_sql` `ANALYZE` know about objects that do not exist in the
catalog.
- New named arguments, all ignored in `PARSE`:
- `variables`: `ARRAY<STRUCT<name, type>>`. These are session or procedure
variables, for example `system.session.v` or `proc.parm1`.
- `relations`: `ARRAY<STRUCT<name, schema>>`. Each is a table or view with
a DDL column list. It eclipses the real object of that identity as a query
source or as an INSERT target.
- `parameter_markers`: `ARRAY<STRUCT<name, type>>`. Use `?` for positional
markers and a simple identifier for named markers. They are bound as typed
NULLs.
- Batch carry-forward: statements are analyzed in order without being
executed.
- `DECLARE VARIABLE` / `DROP TEMPORARY VARIABLE` add or remove a variable
for later statements.
- `CREATE [OR REPLACE] TABLE / VIEW / TEMP VIEW` (with a column list, AS
query, LIKE, or USING with a schema) adds a schema-only relation. `DROP TABLE /
VIEW` removes one.
- In-place schema changes (ALTER TABLE), USE, SET and dynamic SQL are not
simulated.
- The overlays are thread-local registries (`SuppliedVariableRegistry`,
`SuppliedRelationRegistry`) consulted by `VariableResolution`,
`RelationResolution` and the Analyzer's relation lookup. A new
`ResolveInsertIntoSuppliedRelation` rule handles INSERT into an overlay. The
session itself is never mutated: no temp views are registered and no variables
are created.
- A malformed overlay specification (a NULL element or an empty name, type
or schema) fails the query with `INVALID_PARAMETER_VALUE.PARSE_SQL_CONTEXT`. An
invalid DDL type is reported as JSON.
This is part 3 of the stack that splits
[#59844](https://github.com/apache/spark/pull/59844) (SPARK-60593). It is
stacked on [#59873](https://github.com/apache/spark/pull/59873) (SPARK-60621);
only the last commit is new here.
### Why are the changes needed?
Callers often analyze SQL outside its runtime context: the body of a
procedure with its parameters, a query over a table that is not created yet, or
a script that declares what it later uses. Without these overlays such
statements fail analysis even though they are valid in context.
### Does this PR introduce _any_ user-facing change?
Yes, for the unreleased `parse_sql` function, which is behind
`spark.sql.function.parseSql.enabled` (default `false`). It adds the
`variables`, `relations` and `parameter_markers` arguments, and batch
carry-forward in `ANALYZE`.
### How was this patch tested?
- New `ParseSqlAnalyzeSuite` tests:
- variable overlays by identity and through the SQL path, including
shadowing without mutation;
- relation overlays eclipsing temp views and tables, and INSERT with a
column list;
- parameter markers in subqueries and CTEs;
- invalid specifications;
- batch carry-forward.
- New golden cases in `parse-sql-analyze.sql`.
- Ran the parse_sql suites, `ExpressionsSchemaSuite`, `SQLQueryTestSuite -z
parse-sql` and the analysis suites.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Cursor Grok 4.6
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
