Hyukjin Kwon created SPARK-61862:
------------------------------------
Summary: Send the requests of toArrow, toPandas, toLocalIterator
and the map and group UDFs in PySpark's order, and yield calendar intervals as
PySpark's iterator does
Key: SPARK-61862
URL: https://issues.apache.org/jira/browse/SPARK-61862
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.4.0
Reporter: Hyukjin Kwon
Follow-ups of SPARK-61772 (createDataFrame reads its settings at PySpark's
point): other conversions of the Python drop-in send their requests, or run
their checks, in another order than PySpark 4.3.0, or give other values:
- toArrow executes the query before it analyzes the schema and checks that
Arrow can convert it (PySpark computes to_arrow_schema(self.schema) first, of
the cached schema), so a duplicated nested field name or an interval type runs
the query before the client error.
- toPandas and toLocalIterator read their settings before they build the plan,
so a plan that cannot be built sends Config requests where PySpark sends
nothing.
- toLocalIterator raises the year-month interval error before the server's
error, and yields a calendar interval as a str where PySpark yields pyarrow's
MonthDayNano.
- mapInPandas, mapInArrow, applyInPandas and applyInArrow analyze the input
before they parse the DDL of the output schema; cogroup sends an AnalyzePlan of
each side and binds their columns to the UDF, where PySpark's function has no
arguments; applyInPandasWithState parses its state schema, which PySpark sends
as given.
- toLocalIterator releases the GIL for every row, so a CPU-busy Python thread
slows the iteration (300k rows: 15-38 s here, 2.4 s on PySpark).
- spark.udf.register("f", fn, "long") parses the DDL once, where PySpark parses
it twice.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]