villebro opened a new pull request, #43316: URL: https://github.com/apache/superset/pull/43316
### SUMMARY Introduces `superset.coordination.CoordinationService`, a single entry point for the Redis/Valkey-backed coordination primitives Superset relies on — distributed **locks**, **pub/sub**, **streams**, and **key/value** — all backed by one connection (`DISTRIBUTED_COORDINATION_CONFIG`). Previously these primitives were wired up ad hoc: the Global Task Framework used `DISTRIBUTED_COORDINATION_CONFIG` for pub/sub and locking, while Global Async Queries used a separate `GLOBAL_ASYNC_QUERIES_CACHE_BACKEND` for its event streams — two configs and two access patterns over the same underlying backend classes. This PR consolidates them behind one service to modularize and simplify the architecture, and to pave the way for further cleanup of the Global Async Queries plumbing. It also covers the distributed lock. Highlights: - New `CoordinationService` exposing `lock()`, `publish()`/`pubsub()`, `get()`/`set()`/`delete()`, `xadd()`/`xrange()`, plus `get_backend()`/`is_backend_defined()`. - **Availability contract:** locks always work — Redis `SET NX EX` when a backend is configured, falling back to the DB-backed `DistributedLock` otherwise. The Valkey-only operations (pub/sub, streams, key/value) raise `CoordinationBackendUnavailableError` when no backend is defined, rather than silently no-op'ing; `is_backend_defined()` lets best-effort callers branch. - `AsyncQueryManager` and the GTF `TaskManager` / `task_lock` now route their low-level Redis access through the service. - `GLOBAL_ASYNC_QUERIES_CACHE_BACKEND` is **deprecated (non-breaking)**: when `DISTRIBUTED_COORDINATION_CONFIG` is unset it is still honored as a fallback with a one-time deprecation warning, and it will be removed in Superset 8.0. All of its parameters are supported identically under `DISTRIBUTED_COORDINATION_CONFIG` (same backend classes). No functional change to async-query behavior; this is an internal consolidation plus a config deprecation path. ### BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF N/A — internal refactor, no UI changes. ### TESTING INSTRUCTIONS - Unit tests: `pytest tests/unit_tests/coordination/test_service.py tests/unit_tests/async_events/async_query_manager_tests.py` - With `DISTRIBUTED_COORDINATION_CONFIG` set (Redis/Valkey): verify GTF task abort/completion notifications and Global Async Queries work end-to-end. - With only the legacy `GLOBAL_ASYNC_QUERIES_CACHE_BACKEND` set: verify GAQ still works and a one-time deprecation warning is logged. ### ADDITIONAL INFORMATION - [ ] Has associated issue: - [ ] Required feature flags: - [ ] Changes UI - [ ] Includes DB Migration (follow approval process in [SIP-59](https://github.com/apache/superset/issues/13351)) - [ ] Migration is atomic, supports rollback & is backwards-compatible - [ ] Confirm DB migration upgrade and downgrade tested - [ ] Runtime estimates and downtime expectations provided - [x] Introduces new feature or API - [ ] Removes existing feature or API -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
