dbtsai opened a new pull request, #119: URL: https://github.com/apache/spark-connect-rust/pull/119
### What changes were proposed in this pull request? Preserve string UDTF return types as `DataType::Unparsed` in the PyO3 binding and convert typed schemas through the existing Python-to-Rust type converter. Send only the cloudpickled handler class in `PythonUDTF.command`, matching Python Connect's wire format; the return schema is transmitted separately. Add offline Python coverage for direct/decorator calls, valid/nested and malformed/scalar DDL, StructType, omitted dynamic schemas and the handler payload, plus Rust assertions on the serialized return schema. ### Why are the changes needed? `udtf(Echo, returnType="value BIGINT")(lit(1))` currently raises a binding TypeError because the binding only accepts a DataType. Python Connect accepts the string and transmits it unparsed; even invalid DDL is validated by the server when the plan is executed. Eagerly parsing it in the client would change that timing. The existing `(handler, returnType)` command tuple also differs from Connect's handler-only UDTF payload. https://issues.apache.org/jira/browse/SPARK-60678 This patch is independent of the open Arrow/input/decoding PRs and has no PR dependencies. Python handler execution and dynamic analysis remain in Python workers. ### Does this PR introduce _any_ user-facing change? Yes. DDL-string UDTF calls construct a DataFrame and serialize a Connect-compatible return schema instead of failing with `TypeError: ... not DataType`. DDL parsing/errors remain server-side. StructType schemas and omitted schemas remain supported. ### How was this patch tested? - Release wheel: `maturin build --release --locked`. - `cargo test --locked -p apache-spark-connect --test coverage_plan_map_udtf --release`: 6 passed. - `python -m pytest python/tests/test_dropin_offline.py -k 'udtf or udf' -q`: 27 passed. - `cargo fmt --all -- --check` and `git diff --check` passed. Frozen harness `1bbdec9f3665`, unchanged workloads/validators, five fresh process samples per client/case with rotating order, release candidate vs master `d884e347452c` and official Python Connect: | Case | Master successes | Candidate successes | Candidate median | | --- | ---: | ---: | ---: | | UDTF construct | 0/5 | 5/5 | 0.127 ms | | UDTF first query | 0/5 | 5/5 | 0.501 ms | | UDTF repeat query | 0/5 | 5/5 | 0.358 ms | Scalar construct/first/repeat query and pandas first-query guards passed 5/5 for all three clients. The 15 baseline UDTF TypeErrors remain exposed in raw failures; there is no baseline speedup claim for failed work. Official construction was faster (0.047 ms); this fixes compatibility, not construction performance. Raw samples include wall/CPU time, RSS and RPC counts. The candidate used a separate environment with the same pinned NumPy/pandas/PyArrow/psutil dependencies; frozen baseline/reference installations were untouched. All CPU-heavy work used the shared lock. These are client-only fixture checks; no Spark server or worker execution is claimed. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: OpenAI Codex (GPT-6) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
