Yicong-Huang opened a new pull request, #59418:
URL: https://github.com/apache/spark/pull/59418
### What changes were proposed in this pull request?
Add variant registration to the eval type handler registry, then move
`SQL_ARROW_TABLE_UDF` onto it as two handlers.
- Registry (`_base.py`): a handler may declare `variant_conf` (a
`RunnerConf` property name) and `variant_value`. The registry maps `eval_type
-> (variant_conf, {variant_value: handler})`; handlers without `variant_conf`
register under `None`, so existing handlers are unchanged.
- `get_eval_type_handler(eval_type, runner_conf)` picks the variant per
lookup, since the conf is per task and workers are reused across tasks. A conf
value with no registered variant raises.
- Registration rejects a duplicate value and handlers of one eval type
keyed on different confs.
- `_udtf.py`: `ArrowTableUDTFHandler` base (per-row `eval`/`terminate` and
error wrapping), with two variants keyed on `use_legacy_pandas_udtf_conversion`:
- `ArrowTableUDTFLegacyPandasHandler` (`True`): the legacy pandas path.
- `ArrowTableUDTFArrowHandler` (`False`): the Arrow path.
- Move `verify_arrow_result`, used only by the Arrow path, from `worker.py`
to `eval_handlers/verification.py`.
- `read_udtf` keeps only `SQL_TABLE_UDF`; its body is dedented (review with
`?w=1`).
```
UDTFEvalTypeHandler
├─ ArrowUDTFHandler SQL_ARROW_UDTF
└─ ArrowTableUDTFHandler
├─ ArrowTableUDTFLegacyPandasHandler SQL_ARROW_TABLE_UDF, legacy
pandas conversion = True
└─ ArrowTableUDTFArrowHandler SQL_ARROW_TABLE_UDF, legacy
pandas conversion = False
```
### Why are the changes needed?
Step toward running UDTF and UDF eval types through one handler dispatch
path and removing `read_udtf` (SPARK-59415). `SQL_ARROW_BATCHED_UDF` has the
same legacy-conversion split and can reuse the variant registration later.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
New registry tests in `test_base_eval_type_handlers` and handler tests for
both variants in `test_udtf_eval_type_handlers`; existing `test_udtf` (incl.
`LegacyUDTFArrowTests` and `UDTFArrowTests`), `test_arrow_udtf` and
`test_parity_udtf` pass. ASV `ArrowTableUDFTimeBench`, master vs this PR
(`repeat=(5,10,10.0)`, one representative run; a second run agreed):
```text
scenario udtf arrow legacy_pandas
sm_batch_few_col identity_udtf 69.8+-0.4ms -> 70.8+-0.3ms 293+-7ms ->
289+-1ms
sm_batch_many_col stringify_udtf 24.5+-0.2ms -> 24.7+-0.1ms 76.1+-0.6ms
-> 76.2+-0.3ms
lg_batch_few_col filter_udtf 135+-1ms -> 137+-0.7ms 817+-9ms ->
816+-10ms
lg_batch_many_col explode_udtf 98.2+-0.3ms -> 98.6+-0.4ms 293+-3ms ->
296+-4ms
pure_ints identity_udtf 177+-0.9ms -> 180+-2ms 716+-3ms ->
719+-5ms
pure_strings stringify_udtf 187+-1ms -> 187+-1ms 737+-6ms ->
731+-4ms
```
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
This pull request and its description were written by Isaac.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
