Hyukjin Kwon created SPARK-61862:
------------------------------------

             Summary: Send the requests of toArrow, toPandas, toLocalIterator 
and the map and group UDFs in PySpark's order, and yield calendar intervals as 
PySpark's iterator does
                 Key: SPARK-61862
                 URL: https://issues.apache.org/jira/browse/SPARK-61862
             Project: Spark
          Issue Type: Bug
          Components: Connect
    Affects Versions: connect-rust-4.4.0
            Reporter: Hyukjin Kwon


Follow-ups of SPARK-61772 (createDataFrame reads its settings at PySpark's 
point): other conversions of the Python drop-in send their requests, or run 
their checks, in another order than PySpark 4.3.0, or give other values:
- toArrow executes the query before it analyzes the schema and checks that 
Arrow can convert it (PySpark computes to_arrow_schema(self.schema) first, of 
the cached schema), so a duplicated nested field name or an interval type runs 
the query before the client error.
- toPandas and toLocalIterator read their settings before they build the plan, 
so a plan that cannot be built sends Config requests where PySpark sends 
nothing.
- toLocalIterator raises the year-month interval error before the server's 
error, and yields a calendar interval as a str where PySpark yields pyarrow's 
MonthDayNano.
- mapInPandas, mapInArrow, applyInPandas and applyInArrow analyze the input 
before they parse the DDL of the output schema; cogroup sends an AnalyzePlan of 
each side and binds their columns to the UDF, where PySpark's function has no 
arguments; applyInPandasWithState parses its state schema, which PySpark sends 
as given.
- toLocalIterator releases the GIL for every row, so a CPU-busy Python thread 
slows the iteration (300k rows: 15-38 s here, 2.4 s on PySpark).
- spark.udf.register("f", fn, "long") parses the DDL once, where PySpark parses 
it twice.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to