[
https://issues.apache.org/jira/browse/SPARK-60913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon updated SPARK-60913:
---------------------------------
Fix Version/s: connect-rust-4.3.0
> Format spark.sql() queries with keyword arguments in Spark Connect Rust as
> PySpark does
> ---------------------------------------------------------------------------------------
>
> Key: SPARK-60913
> URL: https://issues.apache.org/jira/browse/SPARK-60913
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: Hyukjin Kwon
> Priority: Major
> Fix For: connect-rust-4.3.0
>
>
> In PySpark Connect, the keyword arguments of {{SparkSession.sql(sqlQuery,
> args=None, **kwargs)}} format the query string, as {{str.format}} does: a
> DataFrame becomes a reference that the server resolves to it, a column
> reference becomes its name, a string becomes its SQL literal, and another
> value becomes its string. The Rust-backed Python client sends the query
> unformatted, and binds the keyword arguments as named parameters instead, so
> these PySpark 4.3.0 results differ:
> * {{spark.sql("SELECT {x} AS v", x=1)}} returns {{1}}. The client fails with
> {{PARSE_SYNTAX_ERROR}}.
> * {{spark.sql("SELECT count(*) AS n FROM {df}", df=spark.range(3))}} returns
> {{3}}. The client raises {{TypeError}}.
> * {{spark.sql("SELECT :p AS v", p=5)}} fails with {{UNBOUND_SQL_PARAMETER}},
> as {{p}} is not a parameter. The client returns {{5}}.
> Format the query with the keyword arguments as PySpark does, and send the
> DataFrames it references with it.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
