HyukjinKwon opened a new pull request, #152: URL: https://github.com/apache/spark-connect-rust/pull/152
### What changes were proposed in this pull request? [SPARK-60909](https://issues.apache.org/jira/browse/SPARK-60909): `SparkSession.sql()` now sends the statement at once, as PySpark Connect 4.3.0 does, in both the Rust API and the Python client. - `sql()` sends the statement as a `SqlCommand`, and builds the DataFrame from the relation in the server's `SqlCommandResult`, as PySpark's `CachedRelation` does. That relation is sent back as is, with the DataFrame's own plan ID. - For a command, the relation is a local relation of the command's result, so the DataFrame is local. For a query, it is the query's relation, which runs at an action as before. - If the server returns no result relation, the DataFrame keeps the statement to run at an action, as in PySpark. ### Why are the changes needed? The client built a relation to run at the first action instead, so a command took effect only at an action, and ran again at every action: - After `spark.sql("CREATE OR REPLACE TEMP VIEW v AS SELECT 1 AS x")`, `spark.table("v")` failed with `TABLE_OR_VIEW_NOT_FOUND`, and after `spark.sql("SET spark.sql.shuffle.partitions=7")` the setting was unchanged. - `df = spark.sql("INSERT INTO t VALUES (1)")` inserted nothing, then a row at each `df.collect()`, and `df.count()` failed with `INTERNAL_ERROR`. - A statement that does not parse or resolve failed only at the first action, and `spark.sql("SHOW DATABASES").isLocal()` was false. ### Does this PR introduce _any_ user-facing change? Yes, these now match PySpark Connect 4.3.0: a command takes effect at `sql()` and runs once, an error in the statement is raised at `sql()`, and a command's DataFrame is local. A query's DataFrame still runs at an action. ### How was this patch tested? - Against a local Spark 4.3.0 server, probes of a temp view, `SET`, an `INSERT` followed by two actions, `args` given as a dict and as a list, and `isLocal()` of `SHOW DATABASES` give the same results as official PySpark 4.3.0. - New e2e tests, in Rust and Python, check the temp view, `SET` and `isLocal()` without an action, and that a syntax error is raised at `sql()`. The Python ones also pass with official PySpark 4.3.0. - A new unit test checks that a returned relation is sent back with the DataFrame's own plan ID. - The core lib tests and every integration test, e2e included, against the local server (630 passed), the build of all tests with `wasm-udf`, the offline Python suites (420 passed, 8 live cases skipped), and the live drop-in and Arrow suites (23 passed) and live cases (9 passed). Clippy reports no new diagnostics in the changed lines. The probes also showed differences that this PR does not change, now filed: server errors lack their exception class and details, so a syntax error raises `AnalysisException` rather than `ParseException` ([SPARK-60911](https://issues.apache.org/jira/browse/SPARK-60911)); `Row` prints as `[1]` rather than `Row(x=1)` ([SPARK-60912](https://issues.apache.org/jira/browse/SPARK-60912)); and the keyword arguments of `sql()` do not format the query ([SPARK-60913](https://issues.apache.org/jira/browse/SPARK-60913)). ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Opus 5.5 This pull request and its description were written by Isaac. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
