villebro opened a new pull request, #43407:
URL: https://github.com/apache/superset/pull/43407
### SUMMARY
**Epic feature branch.** This PR is the integration target for the
multi-step migration of **Global Async Queries (GAQ)** onto the **Global Task
Framework (GTF)**. Individual step PRs are reviewed and merged into
`gaq-to-gtf`; this PR accumulates them and merges into `master` in **one go**
once the epic is complete. It is kept as a draft tracker until then.
Superset currently has two overlapping background-execution systems:
- **GAQ** (`GLOBAL_ASYNC_QUERIES`) — runs chart-data queries in a dedicated
Celery task (`load_chart_data_into_cache`) and notifies the browser through a
bespoke Redis Streams transport (`AsyncQueryManager` + the
`/api/v1/async_event/` polling endpoint, or the external `superset-websocket`
server). Results are handed back via a cached `qc-<hash>` query-context
descriptor and a `result_url`.
- **GTF** (`GLOBAL_TASK_FRAMEWORK`) — a newer unified background-task
abstraction: a `@task`/`.schedule()` API, a `tasks` table, dedup by `task_key`,
progress/timeouts/cancellation, abort handlers, `wait_for_completion`, a Task
List UI, and a REST API.
This epic **deprecates GAQ's internal plumbing and re-implements async chart
data on top of GTF**, while keeping `GLOBAL_ASYNC_QUERIES` as the
operator-facing switch for *whether* chart queries run asynchronously. Two
secondary goals: consolidate coordination primitives (locks, pub/sub, streams)
behind one service + one config (`DISTRIBUTED_COORDINATION_CONFIG`), and
harmonize `QueryObject` serialization on one canonical JSON-safe representation.
**Key architectural insight.** Per-query results are already the atomic
cached unit: `QueryContextProcessor.get_df_payload_result(query_obj)` executes
and caches exactly one `QueryObject` under its own `query_cache_key` (which
folds in datasource, `extra_cache_keys`, RLS, and impersonation). The
`qc-<hash>` entry holds no results — it is only a descriptor. So the atomic
async unit is the **`QueryObject` keyed by `query_cache_key`**, and per-query
dedup by that key is safe across users.
### Progress tracker
Every step PR targets `gaq-to-gtf`.
| Step | Scope | Status | PR |
|------|-------|--------|----|
| 0 | Coordination Service (locks/pub-sub/streams/KV/await consolidation;
non-breaking `GLOBAL_ASYNC_QUERIES_CACHE_BACKEND` deprecation) | ✅ Merged into
branch | #43316 |
| 1 | GTF task dependencies (DAG) via `task_dependencies` junction table | ⚪
Not started | — |
| 2 | Canonical `QueryObject` serialization | ⚪ Not started | — |
| 3 | GTF chart-data tasks + orchestrator (integration point) | ⚪ Not
started (needs 1+2) | — |
| 4 | `AsyncQueryManager` → notification layer + flag auto-enable | ⚪ Not
started (needs 3) | — |
| 5 | Frontend re-request result model | ⚪ Not started (needs 3) | — |
| 6 | Remove dead `qc-<hash>` path | ⚪ Not started (after 3–5) | — |
### Dependency graph
```
PR 0 Coordination Service ...................... (merged into branch)
├── PR 1 GTF task dependencies (DAG) ..... (independent of PR 2)
├── PR 2 Canonical QueryObject serialization
└── PR 3 GTF chart-data tasks + orchestrator ...... depends on PR 1
+ PR 2
├── PR 4 AsyncQueryManager → notification layer + flag
auto-enable
├── PR 5 Frontend: re-request result model
└── PR 6 Remove dead qc-<hash> path ..... last (after
PR 3–5)
```
### Design decisions (locked with product owner)
1. **Notification transport:** keep the Redis firehose. GTF emits completion
events into the existing per-channel + `async-events-full` streams, so the
external `superset-websocket` server and the `/api/v1/async_event/` polling
endpoint keep working.
2. **Feature-flag interaction:** `GLOBAL_ASYNC_QUERIES=on` force-enables
`GLOBAL_TASK_FRAMEWORK` at startup (with a log line), mirroring the
`DASHBOARD_RBAC` auto-migration precedent.
3. **Atomic unit / dedup:** one GTF task per `QueryObject`, `task_key =
query_cache_key`, `TaskScope.SHARED` (safe cross-user dedup — the key encodes
RLS/impersonation).
4. **Result reassembly:** drop the `qc-<hash>` descriptor. On completion the
client re-issues the same chart-data POST, which now hits the per-query DATA
cache and returns synchronously.
5. **Cross-query coupling:** add a general `depends_on` capability to GTF;
the chart-data orchestrator sets edges only where real coupling exists
(contribution dependents → the totals query). Independent queries still run in
parallel.
6. **Serialization:** standardize on a canonical, JSON-safe, self-contained
`QueryObject` representation (`to_dict()` + `json_int_dttm_ser` + datasource
ref + `result_type`/`result_format`/`force`).
7. **Coordination service (prerequisite, shipped in PR 0):** unify locks,
pub/sub, and streams behind one service + config.
`GLOBAL_ASYNC_QUERIES_CACHE_BACKEND` is deprecated, not removed — non-breaking.
### Security
No change to the role/capability matrix. Per-query dedup by
`query_cache_key` is safe across users because the key encodes RLS +
impersonation + datasource; workers run under `override_user`. Guest/embedded
channels remain HMAC-derived. Each step PR is reviewed against `SECURITY.md`.
### Backward compatibility
The operator switch stays `GLOBAL_ASYNC_QUERIES`; behavior is preserved.
WebSocket deployments keep working via the retained firehose. The JWT
channel/cookie contract and `/api/v1/async_event/` API are retained (only the
frontend result-fetch step changes to a re-request in PR 5).
`GLOBAL_ASYNC_QUERIES_CACHE_BACKEND` is deprecated (PR 0) but keeps working as
GAQ's dedicated backend when no coordinator is configured, and is removed in
Superset 8.0.
### TESTING INSTRUCTIONS
Per step PR (see each PR for specifics). End-to-end for the epic: with
`GLOBAL_ASYNC_QUERIES=on` (GTF auto-enabled), load a dashboard with mixed
multi-query + contribution charts under both polling and `ws` transport;
confirm tasks appear in the Task List UI, cancellation works, the firehose
still drives the websocket server, and a cached second load short-circuits to
200.
### ADDITIONAL INFORMATION
- [ ] Has associated issue
- [ ] Required feature flags: `GLOBAL_ASYNC_QUERIES` (auto-enables
`GLOBAL_TASK_FRAMEWORK`)
- [ ] Changes UI
- [x] Includes DB Migration (PR 1: `task_dependencies`)
- [ ] Introduces new feature or API
- [ ] Removes existing feature or API (PR 6: `qc-<hash>` path)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
