andygrove opened a new issue, #7298: URL: https://github.com/apache/datafusion-comet/issues/7298
### What is the problem the feature request solves? Part of #6834. `ArrowCachedBatchSerializer` does not override `vectorTypes`, so it returns Spark's default of `None`, and `CometInMemoryTableScanExec.vectorTypes` passes that through. Spark's generated columnar-to-row code therefore reads a cached batch through the `ColumnVector` interface instead of a concrete class, which costs a virtual call per value. ### Describe the potential solution Override `vectorTypes` in the serializer to report the `CometVector` subclasses that `convertCachedBatchToColumnarBatch` produces for each column, so the generated code can read them directly. The types must match what the read path really returns for every column, including the cases where it delegates to the fallback serializer, or the generated code will fail with a `ClassCastException`. A test that checks the reported type against the decoded batch for each supported type would guard that. ### Additional context Found in the cache audit (#6834). The effect should show up on cached reads that feed a Spark columnar-to-row, which is the case in #5485. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
