srielau opened a new pull request, #59873:
URL: https://github.com/apache/spark/pull/59873
### What changes were proposed in this pull request?
Adds an `ANALYZE` mode to `parse_sql`. Previously the function only parsed.
- New optional arguments, positional or named: `mode` (`PARSE` by default,
or `ANALYZE`, case-insensitive), and `current_catalog`, `current_schema` and
`current_path`.
- In `ANALYZE` mode each statement is parsed and then analyzed against the
active session without being executed:
- Each `select_list` entry gets a `type`, a `DESCRIBE TABLE EXTENDED AS
JSON` type object. `*` is expanded to the output columns.
- Resolved relations and routines are fully qualified. Local temp views
and temp functions become `system.session.<name>`, and builtins become
`system.builtin.<name>`.
- A new `analyze_success` field is added. An analysis failure is reported
as a STANDARD `error` object on that statement and does not stop later
statements.
- `current_catalog`, `current_schema` and `current_path` override the
invoker's context only for the duration of the call, and the invoker's values
are restored afterwards. An invalid value is reported as JSON with
`INVALID_PARAMETER_VALUE.PARSE_SQL_CONTEXT`.
- `ANALYZE` never runs anything:
- `CALL` is analyzed, but the procedure is not invoked
(`InvokeProcedures.withoutInvocation`).
- A SQL script that calls `parse_sql` does not lend it its local variables.
- Each statement is analyzed independently against the session as it is, so
a `DECLARE`, `CREATE`, `USE` or `SET` in the batch does not affect later
statements. That carry-forward comes in a follow-up PR.
- Errors:
- `ANALYZE` requires an active session
(`PARSE_SQL_ANALYZE_REQUIRES_SESSION`).
- All arguments must be constant (`DATATYPE_MISMATCH.NON_FOLDABLE_INPUT`).
- An unknown mode fails the query with
`INVALID_PARAMETER_VALUE.PARSE_SQL_MODE`.
- `INTERNAL_ERROR` still fails the query.
- `DescribeRelationJsonCommand`'s type-to-JSON rendering is factored out so
that `parse_sql` can reuse it.
This is part 2 of the stack that splits
[#59844](https://github.com/apache/spark/pull/59844) (SPARK-60593). It is
stacked on [#59868](https://github.com/apache/spark/pull/59868) (SPARK-60620);
only the last commit is new here.
### Why are the changes needed?
Tools that inspect SQL text (lineage, linting, editors, migration
assistants) need column types and fully qualified object names, not only the
parse tree. `ANALYZE` provides them without running the statements.
### Does this PR introduce _any_ user-facing change?
Yes, for the unreleased `parse_sql` function, which is behind
`spark.sql.function.parseSql.enabled` (default `false`). The function gains the
`mode`, `current_catalog`, `current_schema` and `current_path` arguments. The
default `PARSE` output is unchanged.
### How was this patch tested?
- New `ParseSqlAnalyzeSuite`, which covers typing, qualification, context
overrides and restore, invalid context, CALL not being run, outer script locals
being hidden, the constant-argument rule and code-point error indexes.
- New golden file `parse-sql-analyze.sql`.
- Ran `ParseSqlAnalyzeSuite`, `ParseSqlResultSuite`, `ParseSqlSuite`,
`ExpressionsSchemaSuite` and `SQLQueryTestSuite -z parse-sql`.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Cursor Grok 4.6
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
