andygrove opened a new pull request, #5634: URL: https://github.com/apache/datafusion-comet/pull/5634
## Which issue does this PR close? Part of #5487. ## Rationale for this change `spark.comet.exec.inMemoryCache.enabled` has been off by default since #5051, so the native cache path only ever runs under `CometInMemoryCacheSuite` and `CometInMemoryCacheKryoSuite`, which exercise it deliberately. Nothing tells us how it behaves under the rest of the suite: the Spark SQL test diffs, the fuzz suites, the Iceberg and Delta jobs, and any test that calls `cache()`/`persist()` incidentally. This PR flips the default so a full CI run exercises the cache format everywhere caching happens. It is opened as a **draft to collect that signal**, not as a proposal to ship the feature on by default. A clean run is evidence the format is ready for that conversation; a red one is the list of things to fix. ## What changes are included in this PR? **This is stacked on #5543 and contains its commits.** Review #5543 first. GitHub will not let a PR against `apache/main` use a fork branch as its base, so the branch is opened against `main` to get a real CI run. Only the final commit is new here: - `spark.comet.exec.inMemoryCache.enabled` defaults to `true`. - `CometDriverPlugin.maybeSetCacheSerializer` read the config out of `SparkConf` with a hardcoded `false` fallback, so flipping the `ConfigEntry` alone would have left the cache serializer uninstalled for anyone who did not set the key explicitly. It now falls back to the entry's own default, the same way the plugin already reads `spark.comet.metrics.enabled`. - The in-memory cache user guide records the new default and shows how to turn the feature off. The feature is still described as experimental. ## How are these changes tested? The point of the PR is the CI run itself. Every job now builds cached tables in Comet's Arrow format wherever a test caches anything, rather than only in the two suites that opt in. The existing cache suites are unaffected: they set the config explicitly, including the two cases that set it to `false`. `CometInMemoryCacheSuite`'s driver-plugin test passes an explicit `true`, so it still covers the install path rather than relying on the new default. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
