FelixYBW commented on PR #12958: URL: https://github.com/apache/gluten/pull/12958#issuecomment-6304202853
For recording In the long term, we should be able to reuse lance-spark's columnarScan, or just a thin layer of `LanceScanTransformer`. lance-spark keeps the standard BatchScanExec → DataSourceRDD → createColumnarReader path and adds an option to emit plain Arrow batches. That's what Spark's native data source PR (#59313) does: it imports an ArrowArrayStream into ArrowColumnVectors with supportColumnarReads = true. Gluten then only has to recognize the batches as Arrow, and all of LanceScanTransformer / LanceColumnarRDD / LanceFragmentIterator goes away. It also keeps everything the PR currently loses: Lance custom metrics, runtime filters, partitioning, and the OpenTelemetry setup. Three pieces are needed. 1. lance-spark: an "Arrow output" mode in the columnar reader, depending on [SPARK-57468](https://issues.apache.org/jira/browse/SPARK-57468) Today LanceFragmentColumnarBatchScanner.buildSparkOrderedVectors doesn't produce plain Arrow: - It wraps each column in lance's own LanceArrowColumnVector. - It adds JVM-only virtual columns: ConstantColumnVector for _fragid, and BlobPositionColumnVector / BlobSizeColumnVector for blob columns. An Arrow mode would: - Emit Spark's ArrowColumnVector (or anything exposing the underlying FieldVector), in Spark schema order. That reordering is cheap and already done here, so it covers the case export() currently rejects. - Turn _fragid into a real Arrow vector (a filled IntVector). That's easy. - For blob position/size columns, either build real Arrow vectors or mark the partition as not Arrow-capable, so the scan stays vanilla. That gives a clean fallback instead of the runtime throw in export(). - Document the batch lifetime. The reader's ArrowReader reuses one VectorSchemaRoot, so a batch is valid only until the next next(). Consumers must transfer the vectors to keep them. This is the same reuse issue we found with #59313. 2. The Arrow version/classloader problem has to be solved inside lance-spark This is the reason the PR went with passing a C-struct address. If lance-spark's batches carry lance's own (possibly newer or relocated) Arrow classes, Gluten can't treat them as Arrow on its side. Two ways out: - lance-spark builds the Arrow-mode output vectors with Spark's Arrow. It would do the C Data import itself, in Spark's classloader, from the stream it already exports. - Or the batch exposes only a C-struct export hook, and Gluten imports it. The first is cleaner and matches #59313. 3. Gluten: declare the batch type and accept Spark ArrowColumnVector, in progress https://github.com/apache/gluten/pull/13280 - Batch type. Without SPARK-57468, Spark labels a BatchScanExec as vanilla columnar. On Spark 3.5/4.0, Gluten needs a small rule that marks BatchScanExec(LanceScan) in Arrow mode as ArrowJavaBatchType, while still calling vanilla doExecuteColumnar(). On Spark 4.2+, the scan could declare ArrowBatchType itself through the convention API. - Offload input. ColumnarBatches.offload (gluten-arrow/.../ColumnarBatches.java) currently requires isHeavyBatch, meaning ArrowWritableColumnVector. It would need to accept Spark's ArrowColumnVector and export through C Data from its FieldVector. That's the same "Spark ArrowColumnVector instead of ArrowWritableColumnVector" change already on the Arrow plan. With those in place, the plan becomes BatchScanExec(Lance) → OffloadArrowDataExec → ArrowColumnarToVeloxColumnarExec → Velox, using only the existing transitions. That's zero-copy for fixed-width columns, and the only Lance-specific Gluten code is the batch-type rule. The same rule would also cover #59313 native sources and any other DSv2 source that emits Arrow. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
