[
https://issues.apache.org/jira/browse/SPARK-61056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon updated SPARK-61056:
---------------------------------
Fix Version/s: connect-rust-4.3.0
> Send the function of DataStreamWriter.foreach in the Python client as PySpark
> does
> ----------------------------------------------------------------------------------
>
> Key: SPARK-61056
> URL: https://issues.apache.org/jira/browse/SPARK-61056
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: Hyukjin Kwon
> Priority: Major
> Fix For: connect-rust-4.3.0
>
>
> In PySpark Connect, {{df.writeStream.foreach(f)}} sends a function of a
> partition and its rows, which the Python worker of the server runs: for a
> function {{f}}, one that calls {{f}} with each row, and for an object, one
> that calls its {{open(partition_id, epoch_id)}}, its {{process(row)}} with
> each row, and its {{close(error)}}. It raises {{ATTRIBUTE_NOT_CALLABLE}} for
> an object without a callable {{process}}, or with an {{open}} or {{close}}
> that is not callable.
> The Rust-backed Python client sends {{f}} itself, which the worker calls with
> a partition and its rows: the query of an object fails with {{TypeError:
> 'Writer' object is not callable}}, and that of a function fails without a
> batch. Send it as PySpark does.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
