Hyukjin Kwon created SPARK-60913:
------------------------------------
Summary: Format spark.sql() queries with keyword arguments in
Spark Connect Rust as PySpark does
Key: SPARK-60913
URL: https://issues.apache.org/jira/browse/SPARK-60913
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Hyukjin Kwon
In PySpark Connect, the keyword arguments of {{SparkSession.sql(sqlQuery,
args=None, **kwargs)}} format the query string, as {{str.format}} does: a
DataFrame becomes a reference that the server resolves to it, a column
reference becomes its name, a string becomes its SQL literal, and another value
becomes its string. The Rust-backed Python client sends the query unformatted,
and binds the keyword arguments as named parameters instead, so these PySpark
4.3.0 results differ:
* {{spark.sql("SELECT {x} AS v", x=1)}} returns {{1}}. The client fails with
{{PARSE_SYNTAX_ERROR}}.
* {{spark.sql("SELECT count(*) AS n FROM {df}", df=spark.range(3))}} returns
{{3}}. The client raises {{TypeError}}.
* {{spark.sql("SELECT :p AS v", p=5)}} fails with {{UNBOUND_SQL_PARAMETER}}, as
{{p}} is not a parameter. The client returns {{5}}.
Format the query with the keyword arguments as PySpark does, and send the
DataFrames it references with it.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
