andygrove commented on issue #5484: URL: https://github.com/apache/datafusion-comet/issues/5484#issuecomment-5432139854
Fixed in #5051 rather than as a follow-up, since a cache format that loses to Spark's on narrow reads is not worth shipping even behind a disabled-by-default flag. `CometCachedBatch` now stores one compressed Arrow IPC stream per column and decodes only the selected ones, which is option 1 from the description. Two related cases went with it: an empty selection stays empty instead of expanding to every column, and `CometInMemoryTableScanExec` asks for one cheap column rather than the whole cache schema when a query needs only the row count. That widening was why `count(*)` was the worst case rather than the best. Results on the same 5M row, 6 column relation, min of 5 runs after a discarded warm-up: | Read shape | Spark cache | Before | After | |------------|------------:|-------:|------:| | `count(*)` | 62 ms | 241 ms | 77 ms | | 1 of 6 columns | 105 ms | 592 ms | 57 ms | | 3 of 6 columns | 339 ms | 655 ms | 204 ms | | 6 of 6 columns | 375 ms | 448 ms | 448 ms | Per-column framing costs a schema block and compression framing per column per batch and gives up cross-column compression: footprint grows 2.5% at 6 columns and 32% at 60, where the cached relation is still 22x smaller than Spark's format. Not fixed here, and worth a separate issue if anyone wants to pursue it: reads that feed Spark operators rather than Comet ones are still 1.7x to 2.5x behind Spark's cache, because they pay a row conversion that Spark's format avoids with a codegen'd path over its own layout. That is unrelated to projection. Closing when #5051 merges. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
