Hyukjin Kwon created SPARK-60909:
------------------------------------
Summary: Run spark.sql() statements eagerly in Spark Connect Rust
as PySpark does
Key: SPARK-60909
URL: https://issues.apache.org/jira/browse/SPARK-60909
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Hyukjin Kwon
In PySpark Connect, {{SparkSession.sql}} sends the statement to the server at
once, so a command takes effect when {{sql()}} is called. The Rust-backed
Python client builds a lazy relation instead, so a command only runs when an
action is taken on the returned DataFrame:
* After {{spark.sql("CREATE OR REPLACE TEMP VIEW v AS SELECT 1 AS x")}},
{{spark.table("v")}} fails with {{TABLE_OR_VIEW_NOT_FOUND}}; with PySpark 4.3.0
it returns the row.
* After {{spark.sql("SET spark.sql.shuffle.partitions=7")}},
{{spark.conf.get("spark.sql.shuffle.partitions")}} is still the default; with
PySpark it is {{7}}.
DDL such as {{CREATE TABLE}} and {{USE}} behaves the same way. Run the
statement at {{sql()}}, as PySpark does, and build the DataFrame from the
relation that the server returns.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
