Yicong-Huang opened a new pull request, #58745: URL: https://github.com/apache/spark/pull/58745
### What changes were proposed in this pull request? This PR shrinks the runner contract that `read_udfs` and `read_udtf` hand back to the worker dispatch loop in `python/pyspark/worker.py`. Previously every branch of both functions returned a four-element tuple `(func, profiler, deserializer, serializer)` in which, for every UDF and UDTF eval type, `profiler` was always `None` and `deserializer` was always the very same serializer object. Both functions now return just `(func, serializer)`. The dispatch loop sets `profiler = None` and `deserializer = serializer` for the UDF/UDTF path, while the classic RDD command path (`NON_UDF`, via `read_command`) keeps the full four-element contract because its pickled command genuinely carries an independent profiler and input deserializer. The UDF and UDTF branches of the dispatch are also merged, since `read_udtf` and `read_udfs` share the same signature and now the same return shape. ### Why are the changes needed? This is preparatory refactoring for the extensible eval-type handler model tracked by the parent SPARK-59415. Shrinking the runner contract before per-eval-type handlers are introduced means each handler will produce a smaller `(func, serializer)` result instead of a four-element tuple, reducing the surface the migration has to carry. The dropped elements were already dead weight for UDF/UDTF: profiling is folded into `func` at construction time in `read_single_udf`, and input is read and output written through a single serializer object. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? No behavior change; the existing per-eval-type UDF/UDTF test suites cover these paths. Also verified locally across representative eval types routed through `read_udfs` (plain Python UDF, scalar pandas UDF, grouped-map and cogrouped-map `applyInPandas`, grouped-agg pandas UDF, `mapInPandas`), through `read_udtf` (a Python UDTF), and through the `NON_UDF` classic RDD path (`rdd.map`), all with identical results. ### Was this patch authored or co-authored using generative AI tooling? No. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
