Yicong-Huang opened a new pull request, #58286: URL: https://github.com/apache/spark/pull/58286
### What changes were proposed in this pull request? This moves the `mapInPandas` and `mapInArrow` list-return (non-`Iterator` iterable) test cases under the `spark.sql.execution.pythonUDF.mapInBatch.legacy.acceptAnyIterable.enabled=true` flag, and leaves the default `test_map_in_pandas` / `test_map_in_arrow` cases returning a strict iterator. This mirrors the return-value contract introduced on master by SPARK-58601 (strict `Iterator` by default; any iterable accepted only under the legacy flag). ### Why are the changes needed? The `pyspark-connect-old-client` CI job runs `branch-4.0`'s Python tests against a `master` Spark Connect server. On master, SPARK-58601 tightened `mapInPandas` / `mapInArrow` to require a strict `Iterator` return by default, so `branch-4.0`'s `test_map_in_pandas`, which returned a bare `list`, now fails against that server with `UDF_RETURN_TYPE`. Setting the legacy flag makes the server accept the list, keeping the old-client job green once this lands. The change is also safe on `branch-4.0`'s own server: the flag is unknown there and ignored, and `branch-4.0` accepts lists regardless, so the tests pass both 4.0-vs-4.0 and 4.0-vs-master. ### Does this PR introduce _any_ user-facing change? No. Test-only change. ### How was this patch tested? Existing tests, restructured. `test_map_in_pandas` / `test_map_in_arrow` assert the default iterator path; the new `test_map_in_pandas_legacy_accept_any_iterable` / `test_map_in_arrow_legacy_accept_any_iterable` assert the list path under the legacy flag. ### Was this patch authored or co-authored using generative AI tooling? No. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
