Hyukjin Kwon created SPARK-61056:
------------------------------------
Summary: Send the function of DataStreamWriter.foreach in the
Python client as PySpark does
Key: SPARK-61056
URL: https://issues.apache.org/jira/browse/SPARK-61056
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Hyukjin Kwon
In PySpark Connect, {{df.writeStream.foreach(f)}} sends a function of a
partition and its rows, which the Python worker of the server runs: for a
function {{f}}, one that calls {{f}} with each row, and for an object, one that
calls its {{open(partition_id, epoch_id)}}, its {{process(row)}} with each row,
and its {{close(error)}}. It raises {{ATTRIBUTE_NOT_CALLABLE}} for an object
without a callable {{process}}, or with an {{open}} or {{close}} that is not
callable.
The Rust-backed Python client sends {{f}} itself, which the worker calls with a
partition and its rows: the query of an object fails with {{TypeError: 'Writer'
object is not callable}}, and that of a function fails without a batch. Send it
as PySpark does.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
