[
https://issues.apache.org/jira/browse/SPARK-61014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon reassigned SPARK-61014:
------------------------------------
Assignee: Hyukjin Kwon
> Run foreach and foreachPartition, and return the DataFrame of toJSON, in the
> Python client as PySpark Connect does
> ------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61014
> URL: https://issues.apache.org/jira/browse/SPARK-61014
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: Hyukjin Kwon
> Assignee: Hyukjin Kwon
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-rust-4.3.0
>
>
> In PySpark Connect, {{df.foreach(f)}} calls {{f}} with each row, by a UDF of
> a struct of the row, {{df.foreachPartition(f)}} calls {{f}} with an iterator
> of the rows of each partition, by {{mapInArrow}}, and {{df.toJSON()}} returns
> the DataFrame of the JSON string of each row in the column {{value}},
> {{df.select(to_json(struct(col("*"))).alias("value"))}}.
> The Rust-backed Python client's {{foreach}} and {{foreachPartition}} fail on
> the worker with {{TypeError: cannot unpack non-iterable function object}}, as
> the command of their UDF is the function rather than the pair of the function
> and its return type; and {{toJSON()}} raises {{NotImplementedError}}. Run
> them as PySpark does.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
