[
https://issues.apache.org/jira/browse/SPARK-60909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon updated SPARK-60909:
---------------------------------
Fix Version/s: connect-rust-4.3.0
> Run spark.sql() statements eagerly in Spark Connect Rust as PySpark does
> ------------------------------------------------------------------------
>
> Key: SPARK-60909
> URL: https://issues.apache.org/jira/browse/SPARK-60909
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: Hyukjin Kwon
> Priority: Major
> Fix For: connect-rust-4.3.0
>
>
> In PySpark Connect, {{SparkSession.sql}} sends the statement to the server at
> once, so a command takes effect when {{sql()}} is called. The Rust-backed
> Python client builds a lazy relation instead, so a command only runs when an
> action is taken on the returned DataFrame:
> * After {{spark.sql("CREATE OR REPLACE TEMP VIEW v AS SELECT 1 AS x")}},
> {{spark.table("v")}} fails with {{TABLE_OR_VIEW_NOT_FOUND}}; with PySpark
> 4.3.0 it returns the row.
> * After {{spark.sql("SET spark.sql.shuffle.partitions=7")}},
> {{spark.conf.get("spark.sql.shuffle.partitions")}} is still the default; with
> PySpark it is {{7}}.
> DDL such as {{CREATE TABLE}} and {{USE}} behaves the same way. Run the
> statement at {{sql()}}, as PySpark does, and build the DataFrame from the
> relation that the server returns.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
