dbtsai opened a new pull request, #128: URL: https://github.com/apache/spark-connect-rust/pull/128
### What changes were proposed in this pull request? Exact builtin `datetime.date` values currently pass through Decimal detection, integer/float extraction and other generic checks before being recognized as dates. Recognize them first with PyO3's supported `cast_exact::<PyDate>()`, retaining the existing `toordinal() - 719163` date32 encoding. Only exact dates take this path. Date subclasses retain numeric-protocol precedence and `toordinal` overrides; `datetime` and pandas `Timestamp` keep their existing timestamp conversion. No custom cache, unsafe code, numeric conversion or Arrow encoder changes are introduced. This is independently based on master, without the numeric fast path from PR111 or the encoding changes proposed in PR114. PyO3 supports this exact type check with the existing abi3-py39 wheel target. ### Why are the changes needed? For 10,000 × 3 date inputs, five fresh paired process samples with the unchanged client benchmark harness produced these median construction times: | Input | Baseline Rust | Candidate Rust | Speedup | Python reference | |---|---:|---:|---:|---:| | Tuples | 30.634 ms | 3.693 ms | 8.29× | 1.756 ms | | pandas | 31.011 ms | 4.730 ms | 6.56× | 1.757 ms | Rust still trails the reference in these cases. All 225 samples across 15 cases and three clients passed unchanged validation, including timestamp, Decimal, numeric, string, array, nested and nullable guards. Non-date input guard median differences range from -4.4% to +2.8%. An unchanged nullable-return guard initially showed +0.127 ms (+12.4%) with overlapping ranges; a ten-sample follow-up showed +0.048 ms (+4.7%), also with overlapping ranges, with all 30 samples passing. Both runs are retained. ### Does this PR introduce any user-facing change? No intended value or schema changes; faster exact builtin date input conversion. ### How was this patch tested? - 24 new semantic/protocol tests pass on both candidate and unchanged baseline. They cover range boundaries, epoch neighbors, leap/century dates, subclass numeric protocols and ordinal overrides, datetime/pandas Timestamp, nested values, nulls and unsupported impostors. - Eight separate-process localhost protocol cases independently decode emitted LocalRelation Arrow IPC through PyArrow for tuple/pandas inputs and inferred/named/explicit/DDL schemas. Each verifies year endpoints and leap days across years 1–9999, including signed epoch offsets and nulls. - All 274 existing offline Python tests pass on baseline and candidate (`--rootdir=python --import-mode=importlib`). - Release locked maturin wheel built; full default-member Rust test executables compiled; 392 Rust library tests passed; formatting and diff checks passed. - Frozen benchmark harness `1bbdec9f3665fcd2ec2ee5129bbcdb95ee4b00c4`; master baseline `d884e347452c6446c7bc54af151755c62801d8cd`; Python reference `ca1514a2b4e72f9529871f34d09881cd4921df82`. Same NumPy 2.5.4, pandas 2.3.3, PyArrow 26.0.0. Isolated candidate release environment; shared CPU lock used for all builds/tests/benchmarks. Original environments, harness, fixtures and validators preserved. Client-only localhost fixtures, with no JVM or real Spark server. Frozen date construction validators check DataFrame creation; the additional protocol tests verify actual payload correctness. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
