Yicong-Huang opened a new pull request, #58436: URL: https://github.com/apache/spark/pull/58436
### What changes were proposed in this pull request? Follow-up to SPARK-58601, which tightened the `mapInPandas`/`mapInArrow` return-value contract to require a strict `Iterator` by default and added the internal escape-hatch config `spark.sql.execution.pythonUDF.mapInBatch.legacy.acceptAnyIterable.enabled` (default `false`). This PR flips that config's default to `true`, so the pre-4.3.0 lenient behavior (accepting any iterable such as a returned `list`) remains the default and can be disabled by setting the flag to `false` to opt into strict enforcement. The runtime code in `worker.py` and the config wiring are unchanged apart from the default. The 4.2-to-4.3 PySpark migration-guide note describing the breaking change is removed since there is no longer a default behavior change, and the negative tests that assert a returned `list` is rejected now set the flag to `false` explicitly. ### Why are the changes needed? Requiring a strict `Iterator` by default is a user-facing breaking change: a `mapInPandas`/`mapInArrow` UDF that returns a non-`Iterator` iterable (e.g. a `list`) previously worked and would start raising `UDF_RETURN_TYPE`. Defaulting the escape hatch to `true` preserves the established behavior for existing workloads while keeping the flag available for users who want the stricter contract, deferring any default change to a later decision. ### Does this PR introduce _any_ user-facing change? Yes, relative to the unreleased SPARK-58601 change on the unreleased branch. With this PR a `mapInPandas`/`mapInArrow` UDF that returns any iterable (e.g. a `list`) is accepted again by default, as it was before SPARK-58601. Setting `spark.sql.execution.pythonUDF.mapInBatch.legacy.acceptAnyIterable.enabled=false` restores the strict `Iterator`-only contract. Compared to released Spark versions there is no behavior change. ### How was this patch tested? Updated `test_pandas_map.py` and `test_arrow_map.py`: the negative cases asserting a returned `list` is rejected now run with the flag set to `false`; the existing `*_legacy_accept_any_iterable` tests continue to cover the lenient path. ### Was this patch authored or co-authored using generative AI tooling? No -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
