felipepessoto opened a new issue, #13504: URL: https://github.com/apache/gluten/issues/13504
### Description #### Problem Correctness-only unit tests can remain green when Gluten stops offloading work and falls back to Spark. The results are still being checked, but the native implementation may no longer be exercised. Plugin registration, enabled configuration, and even a passing native smoke test are insufficient to detect a drop in native execution across the real test workload. Add CI reporting and regression detection for **native versus non-native execution coverage**, starting with Delta Spark UT and designing the evidence so it can also support other Gluten test pipelines. #### Concrete example: Delta 4.4 partitioned writes As investigated in [#13502](https://github.com/apache/gluten/pull/13502), Delta 4.4's writer defaults to physically writing partition columns into Parquet when `delta.writePartitionColumnsToParquet` is unset. This does not require the separate `materializePartitionColumns` protocol feature. Before that PR, Gluten's gate rejected partitioned writes requiring materialization. Under the default settings, those file writes used Delta's JVM writer even with native writes enabled. Reads and other operators could still execute in Velox. Consequently, passing partitioned-table tests and a `GlutenDelta` command in the plan did not establish native file-write coverage. Enabling native materialization exposed existing native limitations again. The current [Delta known-failures gate](https://github.com/apache/gluten/blob/main/.github/workflows/util/delta-spark-ut/README.md#how-the-gate-works) compares test outcomes, not how much work ran natively. Removing now-passing failures after fallback can make the baseline look better while native coverage has decreased. #### Requested validation - Record execution evidence with stable test/suite identities and the backend, Spark/Delta versions, and relevant configuration. Report counts and ratios for native, partially native, and fully fallback executions, with explicit handling of Spark-free/no-work tests and intentional disabling. - Break coverage down by operation category, especially **native scans versus native file writes**, rather than relying on one global percentage. Capture fallback reasons and identify which tests or operations lost native coverage. - Use final executed plans/runtime evidence where appropriate, including AQE. File writes need evidence of the actual writer used; neither a `GlutenDelta` wrapper nor native upstream input operators prove native file writing. Reuse existing metrics/reporting where possible. - Compare comparable workloads against a base revision or an explicitly reviewed coverage baseline. Account for added/removed tests, matrix/version differences, retries, and duplicate events. Report the numerator and denominator; do not treat a change in raw counts or one aggregate ratio as sufficient evidence by itself. - Fail CI for unexpected loss of native coverage according to documented, reviewed expectations/tolerances. Intentional fallbacks remain valid, but coverage reductions require an explicit explanation. Missing or incomplete instrumentation must not silently look like zero applicable work or successful coverage. - Keep this gate independent of correctness known-failure lists and flaky quarantine. Refreshing `known-failures.txt` must not automatically accept an offload regression. Canary queries should not count as evidence that the actual workload retained native coverage. #### Acceptance criteria - [ ] CI publishes native/fallback counts, ratios, operation-level breakdowns, and changed test identities alongside correctness results. - [ ] A negative control that forces previously native partitioned file writes to fall back fails the coverage gate even when results remain correct and native scans/smoke tests still pass. - [ ] A broad or complete loss of native execution cannot produce a green coverage result. - [ ] Intentional disabling, unsupported operations, and no-work tests have explicit treatment without requiring every test query to offload. - [ ] Missing/truncated evidence and duplicate events are detected or handled without inflating coverage; aggregation verifies required shard evidence is present. - [ ] Baseline comparison and approved coverage changes are documented and covered by positive and negative tests. #### Related work and scope [#13496](https://github.com/apache/gluten/issues/13496) and [#13499](https://github.com/apache/gluten/pull/13499) validate Gluten activation and per-JVM native smoke execution. This issue complements that work: **Gluten can be active while real workload coverage regresses**. It does not request that all unsupported operations become native or that legitimate fallback be removed. Drafted with assistance from GitHub Copilot (GPT-6 Astra). ### Gluten version main branch -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
