andygrove commented on PR #5634: URL: https://github.com/apache/datafusion-comet/pull/5634#issuecomment-5818129189
I audited the in-memory cache and filed what came out of it. #6202 is the one that bears most on this PR. With AQE on, once the table-cache stage materializes, the re-plan leaves the operators directly above it on Spark, so a plain `GROUP BY` or join over a cached table reads Comet's format through a `CometColumnarToRow` into Spark operators. That is the case @mbutrovich asks about in the review, and under AQE it is the common case rather than an edge case. The numbers in the cache guide come from a benchmark that turns AQE off, so they don't show it. It happens with the feature off too, but then the Spark operators read Spark's own format. #6203 is about the signal this PR collects. 23 of the 53 cache test definitions check a cached query with `checkSparkAnswer`, and both sides of that comparison read the same cached bytes, so a write-path or pruning bug passes. With two bugs injected on purpose, 54 of the 57 cache tests still passed. A differential check against uncached ground truth caught both, and on unmodified `main` it matched on every write path, so the format itself looks sound. I also added a cached-relation repro to #3079. A relation cached while Comet is on keeps native-shuffled partitions on a wide-decimal key, and a later join with Comet off returns 57 of 10000 rows. That doesn't depend on the cache format or on this PR, but caching is what carries it into queries that have Comet off. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
