Jungtaek Lim created SPARK-40414:
------------------------------------
Summary: Fix PythonArrowInput and PythonArrowOutput to be more
generic to handle complicated type/data
Key: SPARK-40414
URL: https://issues.apache.org/jira/browse/SPARK-40414
Project: Spark
Issue Type: Improvement
Components: PySpark
Affects Versions: 3.4.0
Reporter: Jungtaek Lim
During the work of flatMapGroupsWithState in PySpark, we figured out that we
are unable to reuse PythonArrowInput and PythonArrowOutput, as PythonArrowInput
and PythonArrowOutput are too specific to the strict input data (row) and
output data.
To reuse the implementations we should make these traits more general to handle
more generic type of data.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
