felipepessoto opened a new pull request, #13134: URL: https://github.com/apache/gluten/pull/13134
## What changes are proposed in this pull request? Related to apache/gluten#12743. Preserve Delta's existing transaction execution observer when running native Delta commands. The current command wrappers install a singleton `UseColumnarDeltaTransactionLog` observer whose lifecycle callbacks are no-ops. This replaces the phase-locking observer used by Delta's concurrency tests, so the test's barriers never advance even when the command runs. This accounts for the shared timeout mechanism in the 65 `Exceeded deadline waiting for check...` / `AtomicBarrier(state=Blocked)` failures identified in [this Delta Spark UT run](https://github.com/apache/gluten/actions/runs/35897644781). This patch: - Uses a per-command observer decorator shared by all three native command wrappers. - Creates `GlutenOptimisticTransaction` inside the existing observer's `startingTransaction` callback and forwards preparation, commit, backfill, post-commit, and abort callbacks. - Preserves Delta's existing next-observer advancement on commit/abort, including OPTIMIZE retries, without losing native transaction creation or restoring a stale observer on scope exit. - Preserves child observers, nested command scopes, and exception propagation. - Adds focused observer tests and a native DELETE regression under `gluten-ut`. The test module's Delta profile now includes `gluten-delta` at test runtime so the Delta offload rules are actually present. No timeout increases, fallback-only workaround, IncrementMetric changes, or test exclusions are included. The known-failure baseline is intentionally unchanged in this draft. The full Delta matrix needs to confirm which tests now pass before removing entries: fixing an unreachable barrier can expose later assertions, so this PR does not claim that all 65 tests are already green. ## How was this patch tested? Local validation used JDK 17, Scala 2.13.17, Spark 4.0.2, and Delta 4.0.1: - Built and installed the backend and `gluten-ut/test` reactor dependencies with `./build/mvn`, the `delta` / `spark-ut` profiles, and `fast-build`. - `org.apache.spark.sql.delta.DeltaTransactionObserverSuite`: **8 tests passed**, covering lifecycle forwarding, real Delta phase-observer advancement on commit and abort, nested scopes, child observers, explicit observer advancement, and exception handling. - `org.apache.spark.sql.delta.GlutenDeltaTransactionObserverSuite`: **1 test passed**, executing a native Delta DELETE and asserting that the real phase-locking observer advances. It also asserts that the observer sees a `GlutenOptimisticTransaction`, so falling back to a vanilla command cannot make the test pass. - Compiled the untouched `main` version of `DeltaWriteOperators.scala` into an isolated class directory and ran that identical DELETE regression: it failed with `observer.allPhasesHavePassed was false`. Removing the baseline classes and rerunning against the fix passed. - `./dev/format-scala-code.sh` and its `check` mode passed. - `scalastyle:check` passed for both changed modules. - The repository's CI license-header checker passed for all changed files. `dev/check.py header main --fix` currently points to a missing `dev/license-header.py`, so the existing `.github/workflows/util/license-header.py` helper was run directly. Native libraries were reused from the successful native-build job in the linked CI run. The existing local Velox checkout is older than the current Gluten native API and could not rebuild; no native source was changed for this patch. The complete Spark 4.1 / Delta v4.2.0 matrix has not been rerun locally. ## Was this patch authored or co-authored using generative AI tooling? Generated-by: GitHub Copilot CLI 1.0.88 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
