L. C. Hsieh created SPARK-60920:
-----------------------------------
Summary: Make reader/writer setters mutate in place like PySpark
in the Spark Connect Rust drop-in
Key: SPARK-60920
URL: https://issues.apache.org/jira/browse/SPARK-60920
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: L. C. Hsieh
Fix For: connect-rust-4.3.0
In the spark-connect-rust Python drop-in, {{DataFrameReader}},
{{DataStreamReader}}, {{DataFrameWriter}}, {{DataFrameWriterV2}},
{{DataStreamWriter}} and {{MergeIntoWriter}} are consuming builders. Each
setter ({{format}}, {{schema}}, {{option}}, {{options}}, {{mode}},
{{partitionBy}}, {{withSchemaEvolution}}, ...) takes the inner Rust builder and
returns a new wrapper, so the original Python object is left empty.
In PySpark these setters change {{self}} and return it. So the common PySpark
pattern
{code:python}
r = spark.read
r.option("a", "b")
r.format("jdbc").load()
{code}
fails with {{RuntimeError: DataFrameReader already consumed}}. The vendored
pandas-on-Spark {{read_sql_table}} calls {{reader.option(...)}},
{{reader.schema(...)}} and {{reader.options(...)}} on one reader and ignores
the return values, so it hits this error.
It also fails in these cases:
* reusing a reader or writer after a {{load()}} / {{save()}}, which PySpark
allows
* reusing one after a format method that failed argument validation, e.g.
{{csv(schema=...)}} with an invalid schema, since the reader was taken before
the arguments were checked
* a {{MergeIntoWriter}} clause ({{whenMatched()}} etc.). PySpark's clause
actions append to the parent writer and return that same writer, and a clause
can be used more than once.
The drop-in should mirror PySpark: setters mutate in place and return the same
object, and terminal methods leave the object reusable.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
