Yicong-Huang opened a new pull request, #58900: URL: https://github.com/apache/spark/pull/58900
### What changes were proposed in this pull request? Continues the umbrella refactor [SPARK-59415](https://issues.apache.org/jira/browse/SPARK-59415) that replaces the growing `if/elif` dispatcher in `read_udfs` with the `pyspark.eval_handlers` framework introduced in #58729. This PR migrates the five Arrow-native map/iter eval types to their own `EvalTypeHandler` subclasses in `python/pyspark/eval_handlers/_arrow.py`, each declaring its `eval_type` and self-registering at class definition: - `ArrowScalarIterUDFHandler` -- `SQL_SCALAR_ARROW_ITER_UDF` - `ArrowMapUDFHandler` -- `SQL_MAP_ARROW_ITER_UDF` (mapInArrow) - `ArrowGroupedMapUDFHandler` -- `SQL_GROUPED_MAP_ARROW_UDF` (applyInArrow) - `ArrowGroupedMapIterUDFHandler` -- `SQL_GROUPED_MAP_ARROW_ITER_UDF` - `ArrowCoGroupedMapUDFHandler` -- `SQL_COGROUPED_MAP_ARROW_UDF` The corresponding branches are removed from `read_udfs`, and the migrated types drop out of the now-dead serializer-selection tuples (mirroring the `SQL_SCALAR_ARROW_UDF` cleanup in #58729). Shared helpers used by both the handlers and the remaining `read_udfs` branches move to leaf modules to avoid an import cycle: `verify_return_type`, `verify_iterator_exhausted`, `verify_output_row_limit`, `verify_iter_result_row_count` (and `_top_level_package`) to `eval_handlers/verification.py`, and `extract_key_value_indexes` to `worker_util.py`. Handler logic is a direct move of the existing per-branch code. ### Why are the changes needed? To make each eval type's execution self-contained and testable and to remove the central `if/elif` dispatch, so eval types can be migrated one at a time. See the parent JIRA. ### Does this PR introduce _any_ user-facing change? No. Internal worker refactor; the migrated paths are behavior-identical, with no change to the UDF API or the on-the-wire format. ### How was this patch tested? Extended the `pyspark.tests.test_eval_type_handlers` unit suite with registration assertions and end-to-end `run()` tests for each new handler (including the scalar-iter row-count failure path). The existing per-eval-type integration suites cover the migrated paths and pass locally: `test_arrow_map`, `test_arrow_grouped_map`, `test_arrow_cogrouped_map` (+ `_misc`), and `test_arrow_udf_scalar`. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Opus 4.8 This pull request and its description were written by Isaac. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
