viirya opened a new pull request, #56168: URL: https://github.com/apache/spark/pull/56168
### What changes were proposed in this pull request? Fix the default `ret_type` resolution in `_MapArrowIterBenchMixin._write_scenario` in `python/benchmarks/bench_eval_type.py`. The three benchmark UDFs (`identity_udf`, `sort_udf`, `filter_udf`) all return whole `pa.RecordBatch`es with the input row schema, so their declared return type should be the inner row `StructType`, not the first nested field's data type. Before: ```python ret_type = schema.fields[0].dataType.fields[0].dataType # first column's type ``` After: ```python ret_type = schema.fields[0].dataType # the row's StructType ``` ### Why are the changes needed? `mapInArrow` UDFs are contractually `(Iterator[pa.RecordBatch]) -> Iterator[pa.RecordBatch]`, with the user-supplied `schema` describing the output rows as a whole. The benchmark UDFs return full batches but were declaring just one column's type, which is semantically inconsistent with the API. This did not surface as an error because `worker.py` discards `return_type` for `SQL_MAP_ARROW_ITER_UDF` in `read_single_udf` (returns `None`) and only checks `Iterator[pa.RecordBatch]` structurally via `verify_return_type`. Schema-level validation is currently absent, so the mismatched type was tolerated. If the worker ever adds schema validation for this eval type, the previous declaration would break the benchmark. ### Does this PR introduce _any_ user-facing change? No. Test-only change in the benchmark module. ### How was this patch tested? - Confirmed the new default resolves to a 5-field `StructType` matching the input row schema for `sm_batch_few_col`. - Ran `MapArrowIterUDFTimeBench.setup` + `time_worker` for `(sm_batch_few_col, pure_ints) x (identity_udf, sort_udf, filter_udf)`. - Ran `MapArrowIterUDFPeakmemBench.setup` + `peakmem_worker` for `sm_batch_few_col/identity_udf`. ### Was this patch authored or co-authored using generative AI tooling? Yes. Generated-by: Claude Code (claude-opus-4-7) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
