malinjawi commented on issue #13487: URL: https://github.com/apache/gluten/issues/13487#issuecomment-6556181848
> Thanks for the detailed report. I opened [#13488](https://github.com/apache/gluten/pull/13488) to support vector columns in the offloaded scan instead of falling back: it rewrites `FixedSizeList` vectors as variable-size lists when the scan emits a batch, including vectors inside structs and lists of vectors. The values are not copied; each vector column only gets an offsets buffer. Velox's Arrow import has no `FixedSizeList` either, so the rewrite happens before the batch leaves the scan. > > For your plan-time fallback, could you leave `FixedSizeList` columns out of the fallback list, so the two changes don't conflict? Two notes on the rest: > > * Float16 vectors (`FixedSizeList<HALF>`, Spark 4.0+) still need the fallback: after the rewrite they are lists of `HALF`, which Spark's `ArrowColumnVector` can't read either. > * The other types in this issue are unaffected by [[GLUTEN-13487][VL] Offload Lance scans with vector columns #13488](https://github.com/apache/gluten/pull/13488). Thanks @sezruby, that makes sense. I've reworked the fallback on top of #13488 so it no longer falls back on `FixedSizeList` columns, while Float16 vectors still do: https://github.com/malinjawi/incubator-gluten/commit/34bef100e1a07a9bd788a6207de8bfd1d8e56e25. `VeloxLanceSuite`, including your vector tests, passes on Spark 3.5 and 4.0 against lance-spark 0.8.0. Like #13488, it's meant as the short-term fix for 1.8 and goes away once Gluten reads Lance through lance-spark's Arrow batches (lance-format/lance-spark#908). I can open the PR once #13488 is merged. One more finding: with lance-spark 0.8.0, `time` columns and timestamps stored in nanosecond, millisecond or second precision fail at export too, like unsigned integers and `LargeList`. The fallback can't detect them from the Spark schema, but planning from the dataset schema as lance-spark#908 does should catch them. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
