dbtsai opened a new pull request, #280: URL: https://github.com/apache/spark-connect-rust/pull/280
### What changes were proposed in this pull request? Preserve the existing row conversion of customized pandas float64 input in the fast path added by #207 ([SPARK-60727](https://issues.apache.org/jira/browse/SPARK-60727)). - A DataFrame constructed with `copy=False` can retain an ndarray subclass inside pandas' concrete NumPy wrapper. The column path skips that subclass's `astype` hook, which can change values or suppress an existing conversion error. Require exact ndarray storage before using this path. - Inspect public schema aliases through their namespace dictionary. A missing alias should decline the optimization without invoking a module's custom `__getattr__` before row conversion. Add three mock-service regression cases for transformed values, conversion errors, and a missing alias with a raising lazy hook. Ordinary ndarray input keeps the fast path; subclasses such as `np.memmap` take the row path. ### Why are the changes needed? These inputs are customized cases that #207 intended to keep on the existing row path. Both guard issues remain on current master. Official Spark 4.3 also bypasses the ndarray hook; this change preserves the Rust client's prior custom-input behavior rather than claiming an official parity fix. ### Does this PR introduce any user-facing change? It restores the prior values and errors for the affected customized input. Public signatures and inline transport policy are unchanged. ### How was this patch tested? - Reproduced both issues with pandas 2.3.3 and 3.0.6, NumPy 2.5.4 and PyArrow 26.0.0. - Supplemental helper checks pass on both pandas versions, including ordinary ndarray float bits and null masks. These use private copies of native 4.2 schema classes and do not prove current-master native dispatch. - Python syntax and `git diff --check` pass. - Genuine local Claude Code reviewed the landed patch and follow-up; the final full review and all findings are recorded in the local review evidence. **Draft: required validation remains pending.** The current environment has no usable Cargo/rustc toolchain, matching native extension, or protoc. The new public-API test attempt is blocked during collection; Rust compile/fmt/Clippy, complete native suites, API/official parity, and live Spark 4.3 checks have not passed for this candidate. Historical maintainer results are not claimed as validation of this follow-up. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: OpenAI Codex; reviewed with installed local Claude Code. This pull request and its description were written by Isaac. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
