LinSimon-901101 opened a new pull request, #6262: URL: https://github.com/apache/datafusion-comet/pull/6262
## Which issue does this PR close? Closes #6203. ## Rationale for this change Disabling Comet does not bypass a cached relation or its static serializer. Comparing a cached query with `checkSparkAnswer` can therefore accept the same corrupted values or incorrect pruning on both sides. These tests need expected answers computed independently of the cache. ## What changes are included in this PR? - Replace cached-query self-comparisons with explicit expected rows or Spark results collected before caching, including Kryo round trips and non-Arrow columnar inputs. - Exercise NaN equality/range predicates that use cached statistics and assert the number of retained batches. - Add differential tests for native Arrow, Spark columnar, and row input. Compare full values and 23 selective predicates against the original in-memory fixture, preserving independent signed-zero ground truth. Verify writer plans, small batches, and exact batch-pruning counts. - Register the new suite in Linux and macOS CI. ## How are these changes tested? - Rebuilt the native library and ran `CometInMemoryCacheSuite`, `CometInMemoryCacheKryoSuite`, and `CometInMemoryCachePruningSuite` locally across Spark 3.4–4.2: Spark 3.4 had 56 passed / 5 version-gated cancellations; Spark 3.5 had 59 / 2; Spark 4.0, 4.1, and 4.2 had 61 passed each. Spark 4.0 also passed semantic Scalafix CHECK. - Fault injection at commit `4924c36d3`: omitting NaN from double statistics makes the NaN test and all three differential writer tests fail (4/4). Adding 1.0 to cached doubles causes 9 failures, including non-Arrow columnar input, both Kryo storage levels, and all three new writer tests. Production source was restored afterward and all 61 Spark 4.1 cache tests passed again. - [Fork CI at the submitted commit](https://github.com/LinSimon-901101/datafusion-comet/actions/runs/36267399368) passed, including `Required Checks`, native build, Rust tests, all four Linux Spark 4.1 / JDK 17 test groups, static checks, TPC-H, and TPC-DS under three join configurations. The exec group passed 1,079 tests with 0 failures and 0 cancellations; its logs confirm all 61 cache tests ran, including the three new writer cases. Optional workflows followed the repository's normal PR conditions. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
