dbtsai opened a new pull request, #110: URL: https://github.com/apache/spark-connect-rust/pull/110
### What changes were proposed in this pull request? Decode Arrow `Duration(Microsecond)` as a distinct Rust `Value::DayTimeInterval(i64)` and materialize it as Python `datetime.timedelta` using integer arithmetic. Preserve exact signed microseconds across the full `i64` range, including nested values and nulls. Document the supported unit and range, retain the existing Arrow interval conversions, and add Rust decoder tests plus a local gRPC regression test collected by the existing offline Python CI suite. ### Why are the changes needed? [SPARK-60607](https://issues.apache.org/jira/browse/SPARK-60607): collecting a non-null Spark day-time interval currently fails with `Unsupported Arrow type Duration(Microsecond) - cannot convert to Value`, including intervals nested inside structs, lists or maps. ### Does this PR introduce _any_ user-facing change? Yes. `collect()` and `toLocalIterator()` return actual Python `datetime.timedelta` objects for day-time interval values, preserving the logical schema. Positive, negative, zero, multi-day and signed 64-bit boundary values are exact. Other Arrow duration units remain unsupported for non-null values; existing Arrow interval physical types keep their string conversions. ### How was this patch tested? - `cargo test -p apache-spark-connect --release --lib`: 388 passed. - All core integration tests compile, both with default features and with `--features wasm-udf`. - `cargo build -p pyspark-rs --release` and `cargo fmt --all --check`. - Three local gRPC regression tests exercise the compiled extension, actual Arrow IPC, `collect()`, and fully consumed `toLocalIterator()` with and without prefetch. Assertions cover Python type/value equality, both `i64` boundaries, nested structs/lists/maps, parent and leaf nulls, all-null arrays, slices, empty and multiple batches, and logical schema preservation. The full official Spark Connect suite has not been run for this patch: no local Spark Connect server is running. This PR is a draft pending that validation. Workspace Clippy could not complete because this toolchain lacks the `wasm32-unknown-unknown` target. A strict targeted Clippy run also reports existing lints in unchanged transport code. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: OpenAI Codex (GPT-6) This pull request and its description were written by Isaac. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
