cloud-fan commented on code in PR #57800: URL: https://github.com/apache/spark/pull/57800#discussion_r3736429392
########## python/docs/source/migration_guide/pyspark_upgrade.rst: ########## @@ -19,6 +19,10 @@ Upgrading PySpark ================== +Upgrading from PySpark 4.2 to 4.3 +--------------------------------- +* In Spark 4.3, a ``mapInPandas`` UDF must return an iterator of ``pandas.DataFrame``\s; returning any other iterable such as a ``list`` now raises ``UDF_RETURN_TYPE``, matching the existing ``mapInArrow`` behavior and the declared ``Iterator[...]`` signature. To restore the previous behavior of accepting any iterable for both ``mapInPandas`` and ``mapInArrow``, set ``spark.sql.execution.pythonUDF.mapInBatch.legacy.acceptAnyIterable.enabled`` to ``true``. Review Comment: Please update the Spark Connect callback aliases to require `Iterator` as well. `connect/_typing.py` still declares both `PandasMapIterFunction` and `ArrowMapIterFunction` with `Iterable` returns, but Connect dispatches these same strict evaluation types, so its public annotations currently tell users that returning a list is valid when the worker rejects it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
