dbtsai opened a new pull request, #138: URL: https://github.com/apache/spark-connect-rust/pull/138
### What changes were proposed in this pull request? Build Arrow date32 arrays directly from ordinary pandas object-column DataFrames with at least 10,000 rows and an explicit all-DateType StructType, then reuse the existing Arrow importer. Validate every cell as an exact built-in datetime.date or None before conversion. Preserve explicit names, metadata and nullability. Keep the row path for smaller inputs, subclasses, customized native schema descriptors/access, other missing-value objects, inferred/name-only/DDL/cast schemas, unsupported names, and nulls in non-nullable fields. No workflow changes in the date-only patch. Stacked on exact heads of #121 (`e7e3094`), #128 (`b8940ed`) and #134 (`911e23c`), transitively #118/#108. The minimal dependency base is `549242a38f8dc1594c65004fdae4d90caa5861ea`; [review the date-only diff](https://github.com/dbtsai/spark-connect-rust/compare/549242a38f8dc1594c65004fdae4d90caa5861ea...896ca1a3602b5661cd4136376d2d1630587b92a7). The broader campaign integration is used only as a supplemental benchmark baseline. ### Why are the changes needed? Pandas date input still normalizes and materializes Python rows after the scalar date improvement in #128. Columnar conversion removes that work for the conservative supported case. Unchanged local harness `1bbdec9f3665fcd2ec2ee5129bbcdb95ee4b00c4`; five fresh processes/client/case, rotating client order, shared CPU lock. Baseline/candidate environments use identical dependencies (Python 3.12.14, pandas 2.3.3, NumPy 2.5.4, PyArrow 26.0.0). | pandas date 10,000×3 | Baseline ms (range) | Candidate ms (range) | Median reduction | |---|---:|---:|---:| | Minimal exact dependency base | 5.304 (5.084–5.491) | 2.303 (2.175–3.007) | 56.6% | | Full integration stack (measurement only) | 4.406 (4.260–4.720) | 2.603 (2.223–2.737) | 40.9% | All 195 minimal-base samples and 135 full-stack samples passed their unchanged validators, including Arrow, int64 and unsupported-type guards. Supplemental fresh-process boundary/subclass/null guards add 90 validated samples with every IPC cell checked. The full-stack baseline is immutable head `48a8b7a`; only the identical date patch is applied to its isolated candidate. ### Does this PR introduce _any_ user-facing change? Performance and transport behavior for the supported pandas date input: it uses the existing Arrow importer, including its configuration reads and cached-upload behavior. Public API signatures and date/schema values are unchanged. Fallback retains conversion hooks and local error ordering. The 10,000-row threshold avoids the measured RPC overhead on smaller date frames. ### How was this patch tested? - Selected CI Python suites: **454 passed, 9 existing skips** (eight live-server requirements and one existing active-session guard). - Supplemental full-stack Arrow/date suites: **174 passed**. - Unchanged baseline: **427 passed, 9 skipped** for the applicable tests. - Rust client/core libraries: **394/95 passed**. Full workspace `cargo test --release --locked --no-run` and WASM-feature integration compilation passed. - `cargo fmt --all --check`, `git diff --check`, and normal commit hooks passed. Clippy is unavailable in the offline toolchain. - No JVM/live Spark server was used; tests use localhost mock transport and independent IPC decoding. Tests independently decode every IPC cell, check years 1-9999 and leap-century boundaries, nullable/all-null columns, metadata and renamed schemas, and reject row materialization for supported input. Fallback tests cover date/DataFrame subclasses, late unsupported cells, alternate null forms, and the non-nullable-null error before RPC. Twenty-six fresh-process schema customization cases cover changes before/after helper import; the original two names-descriptor regressions fail on the preserved pre-guard candidate and pass on the unchanged baseline/final candidate. Tradeoff: exact-cell validation can add a scan before falling back for unsupported values; supplementary 10,000×3 subclass/NaN cases measured 0.612–1.355 ms of added median construction time. Smaller frames retain row conversion. Five-process ranges are descriptive, not a statistical equivalence claim. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: OpenAI Codex (GPT-6). This pull request and its description were written by Isaac. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
