Yicong-Huang opened a new pull request, #59604: URL: https://github.com/apache/spark/pull/59604
### What changes were proposed in this pull request? `read_single_udf` now returns one named `UDF` record (`func`, `raw_func`, `args_offsets`, `kwargs_offsets`, `return_type`) for every eval type, and `read_single_udtf` a `UDTF` record. Each eval type handler reads the fields it needs by name and derives its own values (`num_udf_args`, the unwrapped `Aggregator`, the SQL_BATCHED_UDF kwargs/`toInternal` wrapping) in `__init__`. The eval-type if/elif chain at the end of `read_single_udf` is removed. ### Why are the changes needed? Part of SPARK-59415. Each tuple shape was a private contract between `read_single_udf` and the one handler that consumes it; the chain only existed to pad a shared positional shape with `None` placeholders. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Updated and extended `pyspark.eval_handlers.tests` (record fields from `read_single_udf`, wrapped `func` vs. unwrapped `raw_func`, SQL_BATCHED_UDF with/without conversion, grouped map `num_udf_args`, incremental `Aggregator`). Ran the per-eval-type `pyspark.sql.tests` UDF/UDTF/Arrow/pandas/stateful modules locally. Worker microbenchmark vs. master (median +- stdev of 14 runs): | Case | Before | After | Diff | |---|---|---|---| | SQL_BATCHED_UDF, LongType | 72.0+-1.4ms | 64.2+-0.6ms | -10.9% | | SQL_BATCHED_UDF, DateType (toInternal) | 90.9+-0.8ms | 89.0+-3.4ms | -2.1% | | ScalarArrowUDFTimeBench (sm_batch_few_col, identity_udf) | 16.5+-0.7ms | 15.1+-1.0ms | -8.3% | | ScalarPandasUDFTimeBench (sm_batch_few_col, identity_udf) | 153.6+-2.4ms | 153.6+-3.3ms | +0.0% | | ArrowBatchedUDFTimeBench (sm_batch_few_col, identity_udf) | 29.0+-0.2ms | 29.4+-0.3ms | +1.3% | ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Opus 5.5 This pull request and its description were written by Isaac. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
