gengliangwang opened a new pull request, #59520: URL: https://github.com/apache/spark/pull/59520
### What changes were proposed in this pull request? Add an experimental Rust SDK and a Range connector for the native data source C ABI in #59424, pinned to `51af65c63a9369d793aab1a198ddde4c90c8f85b`. This is a new companion to the FFM implementation; #59442 remains the earlier JNI prototype. The framework changes are not included here, and this draft depends on #59424. The connector API separates three lifetimes: - `BatchSource` handles driver options/schema discovery and creates an independent scan builder. - `ScanBuilder` negotiates pushdowns, then consumes itself in `plan()` to produce an immutable `ScanPlan<Factory>` with typed partition descriptors. - A serializable `BatchReaderFactory` creates a lazy executor reader from a partition and a lifetime-bound `TaskContext`. Live connections and driver handles do not travel to executors. The SDK exports only `spark_ds_init`, using the existing function table without changing the native ABI. A batch source declares only `SPARK_DS_BATCH_READ`; write and streaming function slots remain null. It provides sized-structure compatibility, task IDs and cooperative cancellation, error/panic conversion, cached planning results or failures, and Arrow schema/layout validation and cleanup. The Range example demonstrates disjoint partitions, bounded batches, projection including zero columns, exact integer predicates, and a global limit. Packaging and Spark smoke scripts are included. Documentation follows the pinned framework's semantics: Spark rechecks all filters and only pushes limits for scans without filters. The Cargo workspace is standalone and does not change Maven/SBT or add a Rust requirement to building Spark. ### Why are the changes needed? A Rust connector should be able to implement source-specific planning and reading without manually managing C handles, function tables, serialized state, and Arrow ownership. Making the reader factory explicit also separates driver planning from executor resources and makes task retries and snapshot configuration easier to reason about. The earlier SDK prototype targets JNI. The current framework uses a single C function-table entry point and exposes per-source capabilities and task cancellation, so it needs a matching SDK. ### Does this PR introduce _any_ user-facing change? Yes. It adds an experimental, unpublished Rust batch-read API and a runnable `rust_sdk_range` example. Running that example in Spark requires a distribution containing #59424 and Java 22 or later. It does not change existing Spark behavior or provide streaming/write APIs. ### How was this patch tested? From `sql/native/rust`, using the available local dependency cache: ```sh cargo build --workspace --locked cargo test --workspace --locked cargo clippy --workspace --all-targets --locked -- -D warnings cargo fmt --all -- --check ``` All 33 checks passed: 23 SDK unit tests, 8 example unit tests, one C integration test, and one compile-fail lifetime doctest. The C test compiles against an unmodified copy of the pinned framework header and loads the actual Rust dynamic library. It checks the complete function-table layout, capability flags and null unsupported slots, pushdown/partition planning, source and reader cleanup before executor reads, repeated serialization, cancellation with the original task context, short size-prefixed contexts, and Arrow buffers surviving stream release. Unit tests also cover terminal errors/panics, independent scans, schema mismatches, unsupported nested types, and nonzero nested array offsets. Python syntax checks and `.sparkpkg` packaging passed; the manifest and packaged library bytes were verified. The library's only exported symbol is `spark_ds_init`. **Not run:** the full Spark smoke script, because this branch does not contain the dependency's framework implementation. Rust/C tests validate the native boundary, not Spark's planner or Java FFM/task integration. The PR includes instructions for that remaining end-to-end validation. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: OpenAI Codex (GPT-6) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
