[
https://issues.apache.org/jira/browse/SPARK-61862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-61862:
-----------------------------------
Labels: pull-request-available (was: )
> Send the requests of toArrow, toPandas, toLocalIterator and the map and group
> UDFs in PySpark's order, and yield calendar intervals as PySpark's iterator
> does
> --------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61862
> URL: https://issues.apache.org/jira/browse/SPARK-61862
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.4.0
> Reporter: Hyukjin Kwon
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-rust-4.4.0
>
>
> Follow-ups of SPARK-61772 (createDataFrame reads its settings at PySpark's
> point): other conversions of the Python drop-in send their requests, or run
> their checks, in another order than PySpark 4.3.0, or give other values:
> - toArrow executes the query before it analyzes the schema and checks that
> Arrow can convert it (PySpark computes to_arrow_schema(self.schema) first, of
> the cached schema), so a duplicated nested field name or an interval type
> runs the query before the client error.
> - toPandas and toLocalIterator read their settings before they build the
> plan, so a plan that cannot be built sends Config requests where PySpark
> sends nothing.
> - toLocalIterator raises the year-month interval error before the server's
> error, and yields a calendar interval as a str where PySpark yields pyarrow's
> MonthDayNano.
> - mapInPandas, mapInArrow, applyInPandas and applyInArrow analyze the input
> before they parse the DDL of the output schema; cogroup sends an AnalyzePlan
> of each side and binds their columns to the UDF, where PySpark's function has
> no arguments; applyInPandasWithState parses its state schema, which PySpark
> sends as given.
> - toLocalIterator releases the GIL for every row, so a CPU-busy Python thread
> slows the iteration (300k rows: 15-38 s here, 2.4 s on PySpark).
> - spark.udf.register("f", fn, "long") parses the DDL once, where PySpark
> parses it twice.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]