dbtsai opened a new pull request, #121: URL: https://github.com/apache/spark-connect-rust/pull/121
### What changes were proposed in this pull request? Route supported pandas DataFrames through the existing Arrow Table importer using one Arrow array per Series, avoiding `astype(object).where(...).values.tolist()` and Rust row materialization. The initial path accepts NumPy `int64` and pandas nullable `Int64`, with inferred schemas, complete string-name schemas, or explicit all-`LongType` `StructType` schemas. It preserves masks, integer precision, field metadata, and nullability. Nullable inference also handles a null first value. NumPy-only frames below 1,000 rows retain the row path to avoid importer configuration overhead. Other dtypes, casts/DDL/atomic schemas, and unsupported names retain the existing conversion. Embedded NUL field names use IPC row conversion to preserve the complete name. `verifySchema` remains ignored for pandas Arrow input, matching Spark Connect. **Dependency:** stacked on #118 at `ce89b8dd496bbb1529f1b5531b1eaaa6d739bef0`, including its #108 dependency. The Arrow importer, cached uploads, and Arrow export implementations come from those PRs. The two Rust benchmark builds use the same importer and cached-upload source. This PR should wait for/rebase after #118. [Pandas-only diff](https://github.com/dbtsai/spark-connect-rust/compare/ce89b8dd496bbb1529f1b5531b1eaaa6d739bef0...a8774f0bc8289856b98d11ddbb6b0c0aff95eaee) ### Why are the changes needed? Preparing 1,000,000 × 3 integer pandas input currently materializes millions of Python values and Rust rows. In the paired client-only benchmark, the PR118 base takes about 1.24 seconds and reaches 854 MiB sampled peak RSS. Columnar pandas input reduces this to 6.29 ms and 258 MiB. ### Does this PR introduce _any_ user-facing change? Yes. Supported integer pandas inputs construct faster with substantially lower allocation, and nullable `Int64` inference keeps integer typing when the first value is null. Explicit schemas and renaming are retained. This is a narrow integer path; mixed, timestamp/timezone, date, decimal and nested inputs keep their existing behavior. ### How was this patch tested? - Release wheels built with `maturin build --release --locked` for the PR118 base and this tip, in isolated environments with pinned NumPy/pandas/PyArrow dependencies. - `test_pandas_input.py`, `test_arrow_handoff.py`, and `test_dropin_offline.py`: **350 passed, 9 skipped**. Eight skips require `SPARK_REMOTE`; one existing test requires no active session. Mock RPC tests validate exact integers above `2**53`, nullable masks, inferred/explicit schemas, metadata, names/NUL-name fallback, nullability errors, absence of row conversion, and 100/1,000/100,000/1,000,000-row uploads. - `cargo fmt --all -- --check` and `git diff --check` passed. - Frozen public benchmark harness `1bbdec9f3665`, unchanged workloads/validators: 18 cases × 3 clients × 5 fresh samples, rotating order, **270 successful / 0 failed**. All builds, suites and benchmarks held the shared CPU lock. Prepared inputs and one input warmup are outside timing. Local scripted gRPC fixtures; no JVM or Spark execution timing. Median construction time (ms); peak RSS is sampled externally at 2 ms and includes prepared input/warmup allocation: | Shape | Official Python | PR118 base | Candidate | Base peak MiB | Candidate peak MiB | |---|---:|---:|---:|---:|---:| | 100 × 3 | 0.809 | 0.490 | 0.506 | 102.3 | 102.2 | | 1,000 × 3 | 0.871 | 1.481 | 1.500 | 102.9 | 104.7 | | 100,000 × 3 | 1.733 | 116.214 | 2.209 | 180.8 | 123.4 | | 1,000,000 × 3 | 6.688 | 1241.807 | 6.291 | 854.1 | 258.4 | | 10,000 × 32 | 2.550 | 121.648 | 2.582 | 164.2 | 117.8 | Tuple, mixed pandas explicit/inferred/DDL, Arrow Table, UDF, Arrow/collect return, timestamp, date, decimal, array and string guards all validated. Official Python sampled peak RSS for the million-row case remains lower at 212 MiB. The Arrow Table guard median was 1.633 ms versus 1.385 ms, with overlapping ranges (candidate 1.319–6.159 ms, base 1.114–1.759 ms); a focused follow-up with ten samples/client passed all 30 samples: base 1.283 ms (1.125–1.699), candidate 1.392 ms (1.161–1.894). The remaining 0.109 ms difference is disclosed as a guard limitation. Both made four configuration RPCs; the Arrow importer and Arrow-only entry path are unchanged. No live Spark server was used for this fix. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: OpenAI Codex (GPT-6) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
