villebro opened a new pull request, #43563: URL: https://github.com/apache/superset/pull/43563
### SUMMARY Fixes a concurrency bug in GTF async task submission (surfaced on the epic branch after #43557 merged). **The race.** Concurrent chart-data submits for the same shared query resolve to the same `dedup_key` and are serialized by `task_lock` so all-but-one JOIN the winner's task. But the lock was released *inside* `run_with_info` while `@transaction` committed *afterward*. A waiter could acquire the lock and run `find_by_task_key` in the window between the winner's lock release and its commit — not yet seeing the winner's row (READ COMMITTED) — then INSERT a duplicate `dedup_key`, blocking until the winner committed and then failing with `UniqueViolation` on `idx_tasks_dedup_key` (a 500) instead of a clean join. Reproduces by loading a dashboard with charts that share a query (e.g. the deck.gl demo); it clears on reload once the row/cache is warm. **The fix.** Move the create-or-join into a dedicated `@transaction` method (`_create_or_join`) that the lock now **wraps**, so the commit happens before the lock is released and a waiter always observes the committed row. Validation runs before the lock so invalid input still fails fast (as `TaskInvalidError`). As a bonus, with `run_with_info` no longer decorated, the no-Redis KV-backed lock's own `@transaction` now commits and is durably visible during the critical section, so the metastore-fallback lock serializes properly too. **Contract guard.** This guarantee requires `SubmitTaskCommand` to own its transaction — if it runs inside an outer `@transaction`, the reentrancy guard would defer the commit past the lock release and silently reopen the race. All current callers submit outside any transaction; to keep it that way, `run_with_info` now raises if `g.in_transaction` is already set, so a future caller that wraps submit in a transaction fails loudly and reconsiders. ### BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF N/A (backend concurrency fix). ### TESTING INSTRUCTIONS ```bash pytest tests/unit_tests/tasks/test_submit.py -q ``` Manual: load a dashboard whose charts share a query with `GLOBAL_ASYNC_QUERIES` on; it no longer 500s on `idx_tasks_dedup_key`, and duplicate submits join the shared task. ### ADDITIONAL INFORMATION - [ ] Has associated issue: - [x] Required feature flags: `GLOBAL_ASYNC_QUERIES` / `GLOBAL_TASK_FRAMEWORK` (no new flags) - [ ] Changes UI - [ ] Includes DB Migration - [ ] Introduces new feature or API - [ ] Removes existing feature or API Targets the `gaq-to-gtf` epic branch. Part of #43407. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
