minni31 opened a new pull request, #12631: URL: https://github.com/apache/gluten/pull/12631
## What changes were proposed in this pull request? Offload `LocalTableScanExec` (a driver-side local collection) to native columnar execution on the Velox backend by converting its rows into columnar batches via the existing `RowToVeloxColumnar` JNI path. - Add gluten-substrait `LocalTableScanTransformer` base and `SparkPlanExecApi` hooks (`isSupportLocalTableScanExec` / `getLocalTableScanTransform`), wired into `OffloadOthers`. - Add `VeloxLocalTableScanTransformer` with schema/Arrow-compatibility validation (falls back for Map/Interval and other unsupported types). - Gate offload behind `spark.gluten.sql.columnar.localTableScan` (default `true`, consistent with other columnar operator toggles). - Skip offload for deserialized plans whose `@transient` rows became null (avoids an NPE when an AQE sub-plan is shipped across an RPC boundary), and for streaming sources via a new `SparkShims.getLocalTableScanStream` accessor (`None` on Spark 3.x, `plan.stream` on Spark 4.0+). - Add unit/integration tests and document the new config. ## How was this patch tested? - New unit/integration suite `VeloxLocalTableScanSuite` covering: successful offload, fallback when disabled, fallback for unsupported (Map/Interval) types, the deserialized-null-rows NPE guard, and streaming-source skip. - Config default is `true`, matching the convention used by other columnar operator toggles in `GlutenConfig`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
