Yicong-Huang opened a new pull request, #58305: URL: https://github.com/apache/spark/pull/58305
### What changes were proposed in this pull request? This mirrors #58286 (branch-4.0) onto branch-4.2: it moves the `mapInPandas` and `mapInArrow` list-return (non-`Iterator` iterable) test cases under the `spark.sql.execution.pythonUDF.mapInBatch.legacy.acceptAnyIterable.enabled=true` flag, and leaves the default `test_map_in_pandas` / `test_map_in_arrow` cases returning a strict iterator. This matches the return-value contract introduced on master by SPARK-58601 (strict `Iterator` by default; any iterable accepted only under the legacy flag). ### Why are the changes needed? The connect old-client compatibility job runs a release branch's Python tests against a `master` Spark Connect server. On master, SPARK-58601 tightened `mapInPandas` / `mapInArrow` to require a strict `Iterator` return by default, so a release branch's `test_map_in_pandas`, which returned a bare `list`, fails against that server with `UDF_RETURN_TYPE`. #58286 fixed this for branch-4.0; this PR keeps branch-4.2 consistent so the same tests stay green if the compatibility job's target advances to this branch. The change is also safe on branch-4.2's own server: the flag is unknown there and ignored, and branch-4.2 accepts lists regardless, so the tests pass both on branch-4.2 and against a strict-Iterator server. ### Does this PR introduce _any_ user-facing change? No. Test-only change. ### How was this patch tested? Existing tests, restructured. `test_map_in_pandas` / `test_map_in_arrow` assert the default iterator path; the new `test_map_in_pandas_legacy_accept_any_iterable` / `test_map_in_arrow_legacy_accept_any_iterable` assert the list path under the legacy flag. ### Was this patch authored or co-authored using generative AI tooling? No. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
