sezruby commented on PR #12958: URL: https://github.com/apache/gluten/pull/12958#issuecomment-6528267137
@FelixYBW following up on your plan above (https://github.com/apache/gluten/pull/12958#issuecomment-6304202853): - **lance-spark side: lance-format/lance-spark#908** (in review). The new `arrow_batch_reads` option makes the standard columnar reader emit Spark `ArrowColumnVector` batches. Vectors become `List`s and row metadata becomes signed 64-bit, without copying values. `LanceScan.arrowBatchSchema()` reports at planning time whether a scan can do this, and `withArrowBatches()` turns it on. Batches are valid until the next `next()` and are allocated under `ArrowUtils.rootAllocator()`, matching your SPARK-57468 branch. - **Gluten side: a local prototype.** It's a rule that calls `withArrowBatches()`, plus a `convFuncOverride` marking the `BatchScanExec` as `ArrowJavaBatchType`. On 52 Arrow types every query matched vanilla Spark except one, and vectors and metadata columns went straight to Velox. Gluten issues found on this path: 1. `ArrowColumnVectors.adopt` throws for `NullVector` (no allocator). 2. `adopt` copies every batch until Gluten's allocator is under Spark's root. 3. There are "unreleased native columnar batches" warnings at task end. Questions: 1. Does #908 match what you had in mind? 2. Should the Gluten change go in now as a rule, or wait for SPARK-57468? 3. Is your allocator change planned for main? #13488 stays the short-term vector fix for 1.8 until then. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
