dongjoon-hyun opened a new pull request, #59739:
URL: https://github.com/apache/spark/pull/59739
### What changes were proposed in this pull request?
This PR makes `PipelinesHandler.blockUnsupportedSqlCommand` check the
statements under two parser wrappers at the top of a parsed plan.
- `UnresolvedWith`: `AstBuilder.visitDmlStatement` puts a DML statement with
a CTE prefix under `UnresolvedWith`, for example `WITH v AS (...) INSERT INTO t
SELECT * FROM v`. A CTE prefix is allowed on `INSERT INTO`, `INSERT OVERWRITE
[TABLE]`, `INSERT OVERWRITE [LOCAL] DIRECTORY`, `MERGE INTO`, `UPDATE`, `DELETE
FROM`, and a multi-insert.
- `Union`: `AstBuilder.visitMultiInsertQuery` builds a multi-insert with two
or more `INSERT` clauses, `FROM src INSERT INTO t1 SELECT ... INSERT INTO t2
SELECT ...`, as a `Union` of its `INSERT`s.
The guard now applies the same check to the child of `UnresolvedWith` and to
each child of `Union`. As a result, these statements fail with
`UNSUPPORTED_PIPELINE_SPARK_SQL_COMMAND`, like the same statements without the
wrapper. A read-only query with a CTE or a `UNION` still passes because its
children are queries.
This PR unwraps only these two wrappers instead of checking every node of
the plan. The grammar allows a CTE prefix only on a query or a DML statement,
and a CTE definition can only be a query. A multi-insert is the only statement
that the parser builds as a `Union` of DML statements. So, apart from commands
such as `EXPLAIN` that are already blocked, these are the only nodes that the
parser puts above a DML statement.
### Why are the changes needed?
The SDP guard checks only the class of the top node of the parsed plan. A
DML statement with a CTE prefix and a multi-insert have a non-command node at
the top, so the guard lets them pass. Outside a flow function, `spark.sql(...)`
then runs them eagerly. For example, each of the following lines in a pipeline
definition file inserts rows into `t` while the pipeline is being registered.
```python
spark.sql("WITH v AS (SELECT 1 AS id) INSERT INTO t SELECT * FROM v")
spark.sql("FROM (SELECT 1 AS id) s INSERT INTO t SELECT * INSERT INTO t
SELECT *")
```
Inside a flow function, the guard lets the same statements pass as well.
This is the top-node limitation that was discussed in
https://github.com/apache/spark/pull/59425 and left out of scope in
https://github.com/apache/spark/pull/59719.
### Does this PR introduce _any_ user-facing change?
Yes. These statements in `spark.sql(...)` of SDP now fail with the following
error instead of modifying tables or directories. This is a bug fix for the SDP
SQL guard, which has existed since Apache Spark
[4.1.0](https://github.com/apache/spark/releases/tag/v4.1.0) (2025-12-11).
```
[UNSUPPORTED_PIPELINE_SPARK_SQL_COMMAND] 'UnresolvedInsert' is not supported
in spark.sql("...") API in Spark Declarative Pipeline. SQLSTATE: 0A000
```
The error names the class of the wrapped statement, such as
`UnresolvedInsert` or `MergeIntoTable`, like the error for the same statement
without the wrapper.
### How was this patch tested?
Pass the CIs with the newly added test cases in `PythonPipelineSuite`.
- `unsupportedSqlCommandList` has five more statements: `WITH ... INSERT
INTO`, `WITH ... MERGE INTO`, `WITH ... UPDATE`, `WITH ... DELETE FROM`, and a
multi-insert. Like the existing entries, each of them runs both outside and
inside a pipeline flow function.
- `supportedSqlCommandList` has a read-only CTE query and a `UNION ALL`
query, to make sure that the new check does not block them.
Without the fix, the 10 new unsupported test cases fail. Outside a flow
function, the statement passes the guard and then fails with
`TABLE_OR_VIEW_NOT_FOUND` instead of `UNSUPPORTED_PIPELINE_SPARK_SQL_COMMAND`.
Inside a flow function, no exception is thrown. The 4 new supported test cases
pass with and without the fix.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
