minni31 opened a new pull request, #12982: URL: https://github.com/apache/gluten/pull/12982
## What changes were proposed in this pull request? Adds support for offloading `RDDScanExec` to the Velox backend by converting the underlying `RDD[InternalRow]` into columnar batches through the native row-to-columnar path (the same JNI path used by `RowToVeloxColumnarExec`). Highlights: - **New `VeloxRDDScanTransformer`** implementing the `RDDScanTransformer` contract. Schemas not supported by the Arrow export path (e.g. `map` or interval types) are rejected during validation and fall back to vanilla Spark. - **Wiring in `VeloxSparkPlanExecApi`** (`isSupportRDDScanExec` / `getRDDScanTransform`). Offload is gated by a new config and skipped inside a Structured Streaming query (micro-batch / `foreachBatch`), where offloading a materialized per-batch source into a state-store pipeline can deadlock. - **New config** `spark.gluten.sql.columnar.backend.velox.rddScan.enabled` (default `true`), which also acts as a runtime kill-switch. - **Nullability-aware row conversion**: for schemas with non-nullable fields, rows are pre-projected with a nullability-honoring `UnsafeProjection` so a null in a non-nullable field yields the type default, matching Spark's `WholeStageCodegen` behavior (SPARK-35912). The guard recurses into nested struct/array/map types. - **`ColumnarPartialProjectExec` guard** against an empty `projectAttributes` set (a projection that references no child column), which can arise once `RDDScan` feeds downstream partial projection. - Zero-column `RDDScan` schemas are rejected so Spark's `OneRowRelation` stays on the row path, preserving existing plan assertions and parameterized `EXPLAIN` output. ## How was this patch tested? Added `VeloxRDDScanSuite` covering: type coverage (primitives, string, array, struct), aggregation downstream, empty/duplicate reads, null handling, checkpoint `BatchCarrierRow` reuse, map/interval fallback, nested nullability coercion, zero-column fallback, kill-switch fallback, and the streaming-query skip. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
