sezruby opened a new pull request, #12958: URL: https://github.com/apache/gluten/pull/12958
## What changes were proposed in this pull request? Offloads read-only Lance `BatchScanExec` to the Velox backend through the Arrow C Data Interface. This is the Gluten consumer side of the three-step Lance integration; the two upstream pieces it builds on are already merged: - `LanceScanner#exportArrowStream(long)` in lance-core (Arrow C stream export) - `LanceArrowStreamScanner` forwarding in lance-spark (lance-spark#778) ### How it works - A new `OffloadLanceScan` rule matches a Lance `BatchScanExec` and rewrites it to `LanceScanTransformer`, which produces an `ArrowJavaBatchType` columnar RDD. - Per fragment, `LanceArrowStreamScanner.export(...)` hands back an Arrow C stream; Gluten wraps the stream address with `ArrowArrayStream.wrap(...)` / `Data.importArrayStream(...)` and loads the batches into `ArrowWritableColumnVector`. Only the Arrow C-struct address crosses the classloader boundary, so Gluten's Arrow version stays decoupled from lance-spark's. - Offload is gated: it engages only for a plain columnar Lance scan (no pushed aggregation, no full-text query). Anything else falls back to vanilla Spark. - A `velox-lance` `Component` registers the rule and activates only when lance-spark is on the classpath (`org.lance.spark.read.LanceScan` present). ### Where the code lives The read path is entirely Velox-specific (`ArrowWritableColumnVector`, `ArrowBufferAllocators`, `ValidatablePlan`), so it is placed under `backends-velox/src-lance` behind a new `lance` Maven profile — mirroring the `src-iceberg` Velox-integration layout rather than adding a top-level module. Build with `-Pbackends-velox -Pspark-3.5 -Plance`. **Open question for reviewers:** a top-level `gluten-lance` module (like `gluten-iceberg`) makes sense once there is engine-agnostic code to host — e.g. a write path. The read-only offload has none today, so this PR keeps it Velox-local. Happy to restructure if you'd prefer the module up front. ## How was this patch tested? The offload path was exercised end-to-end against a Lance table (scan offloaded to Velox, aggregation matched) on an amd64 runner. > **Draft status / CI:** this depends on a published lance-spark release that carries lance-spark#778. The latest release (`0.8.0-beta.1`) predates it, so the `lance` profile currently points at a not-yet-published `0.8.0-beta.2` placeholder and CI will be red until that dependency resolves. Kept as a draft until then; the version property will be bumped to the real release. Relates to #12263. 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
