dongjoon-hyun opened a new pull request, #59736:
URL: https://github.com/apache/spark/pull/59736
### What changes were proposed in this pull request?
This PR makes `PipelinesHandler.blockUnsupportedSqlCommand`, the guard for
`spark.sql(...)` in Spark Declarative Pipelines (SDP), reject three kinds of
statements. The parser does not produce a `Command` for them, but they run a
command or a procedure once they are analyzed.
| Statement | Top node of the parsed plan | What happens during analysis |
|---|---|---|
| `EXECUTE IMMEDIATE '<sql>'` | `UnresolvedExecuteImmediate` |
`ResolveExecuteImmediate` parses and analyzes `<sql>`, and a command payload
then runs as `ExecuteImmediateCommand`. |
| `CALL cat.ns.proc(...)` | `Call` (an `ExecutableDuringAnalysis`) |
`InvokeProcedures` runs the procedure. |
| A statement named by an `IDENTIFIER` clause that the parser does not build
yet, e.g. `USE SCHEMA IDENTIFIER('a' \|\| 'b')`, `CREATE TABLE IDENTIFIER('a'
\|\| 'b') (id INT)`, or `CREATE TEMPORARY VIEW IDENTIFIER('v') AS SELECT 1` |
`PlanWithUnresolvedIdentifier` | `ResolveIdentifierClause` evaluates the clause
and builds the actual command. |
The guard now does the following.
- It blocks `UnresolvedExecuteImmediate` and `ExecutableDuringAnalysis`, the
trait that `Call` implements.
- For a top-level `PlanWithUnresolvedIdentifier`, it builds the statement
through the node's `planBuilder` with a placeholder name and checks that
statement with the same allowlist and blocklist. As a result, `USE SCHEMA
IDENTIFIER('a' || 'b')` is rejected as `SetNamespaceCommand`, like `USE SCHEMA
IDENTIFIER('ab')`, while read-only forms such as `SHOW COLUMNS FROM
IDENTIFIER(...)` and `TABLE IDENTIFIER(...)` are still allowed.
Because the change is in `blockUnsupportedSqlCommand`, it covers both a
plain SQL relation and the root SQL of a SQL-with-references `WITH_RELATIONS`
(SPARK-57189).
Like the existing checks, the new checks are applied to the top node of the
parsed plan. Nested cases such as `WITH ... INSERT ...` are out of scope here.
They are the top-node limitation discussed in
https://github.com/apache/spark/pull/59425.
### Why are the changes needed?
The guard checks only the class of the top node of the parsed plan. The
statements above pass that check, and `session.sql` then analyzes them, which
runs the command or the procedure. For example, the following line in a
pipeline definition file creates a table while the pipeline is being
registered, although `spark.sql("CREATE TABLE ...")` is rejected.
```python
spark.sql("EXECUTE IMMEDIATE 'CREATE TABLE spark_catalog.default.t (id
INT)'")
```
Similarly, `spark.sql("USE SCHEMA IDENTIFIER('some_' || 'schema')")` runs
`SetNamespaceCommand` although `USE SCHEMA some_schema` is rejected, and `CALL`
runs a procedure of a configured `ProcedureCatalog`. Inside a flow function,
the guard lets these statements pass as well.
### Does this PR introduce _any_ user-facing change?
Yes. In an SDP pipeline definition, `spark.sql(...)` with `EXECUTE
IMMEDIATE`, `CALL`, or a command named by such an `IDENTIFIER` clause now fails
with errors like the following instead of running.
```
[UNSUPPORTED_PIPELINE_SPARK_SQL_COMMAND] 'UnresolvedExecuteImmediate' is not
supported in spark.sql("...") API in Spark Declarative Pipeline. SQLSTATE: 0A000
[UNSUPPORTED_PIPELINE_SPARK_SQL_COMMAND] 'Call' is not supported in
spark.sql("...") API in Spark Declarative Pipeline. SQLSTATE: 0A000
[UNSUPPORTED_PIPELINE_SPARK_SQL_COMMAND] 'SetNamespaceCommand' is not
supported in spark.sql("...") API in Spark Declarative Pipeline. SQLSTATE: 0A000
```
This includes `EXECUTE IMMEDIATE` of a query such as `EXECUTE IMMEDIATE
'SELECT 1'`, because its SQL string is parsed only during analysis. This is a
bug fix for the SDP SQL guard, which has existed since Apache Spark
[4.1.0](https://github.com/apache/spark/releases/tag/v4.1.0) (2025-12-11).
### How was this patch tested?
Pass the CIs with the newly added test cases in `PythonPipelineSuite`.
- `unsupportedSqlCommandList`: `EXECUTE IMMEDIATE`, `CALL`, `USE SCHEMA`
with a non-literal `IDENTIFIER` clause, and `CREATE TEMPORARY VIEW` with a
literal `IDENTIFIER` clause. Each is checked outside and inside a query
function. Without the fix, these 8 test cases fail because the guard does not
reject the statements. For example, outside a query function, `EXECUTE
IMMEDIATE` creates the table, and `USE SCHEMA IDENTIFIER(...)` runs
`SetNamespaceCommand` and fails with `SCHEMA_NOT_FOUND`.
- `supportedSqlCommandList`: `SHOW COLUMNS FROM IDENTIFIER(...)` and `TABLE
IDENTIFIER(...)` with a non-literal `IDENTIFIER` clause, to make sure that the
new check does not block them.
```
$ build/sbt 'connect/testOnly
org.apache.spark.sql.connect.pipelines.PythonPipelineSuite -- -z "SQL command"'
```
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
