aminghadersohi opened a new pull request, #43941:
URL: https://github.com/apache/superset/pull/43941
### SUMMARY
Follow-up to #43407. Restore a narrow, documented downstream extension seam
without reverting GAQ's migration to GTF.
**Design decision:** use an app-local `QueryManager` selected by
`ASYNC_QUERY_MANAGER_CLASS`, plus a typed frontend polling reader. The default
manager delegates to the existing per-QueryObject GTF submitter and authorized
Task DAO. The chart route and protected status endpoint dispatch through it.
There is no second scheduler, Redis event engine, task table, or result cache.
This keeps deduplication, effective user/guest identity, contribution DAGs,
cancellation, and the task-UUID/cache-re-request contract with GTF.
- The manager supports app-scoped initialization, submission, status reads
and downstream request-channel parsing; initialization is idempotent and does
not share an instance across Flask apps.
- `asyncEvent.init(config, pollingTransport)` lets a frontend fork use a
custom authenticated polling endpoint/client while retaining the shared waiter,
backoff, cancellation and websocket catch-up logic. Omitting it preserves
existing behavior.
- Deprecated Python imports resolve lazily instead of failing at import. The
legacy chart submission adapters authorize QueryContext access before
delegating to GTF. They return **GTF task_ids/cursor**, not legacy
job_id/result_url.
- Deprecation window: retain the Python adapters and manager config alias
throughout the first major release containing this change; removal no earlier
than the following major release. Imports/config use emit `DeprecationWarning`.
`UPDATING.md` and the extension guide document the boundary.
**Not a drop-in restoration of the old protocol or an unchanged-downstream
deployment fix.** Recreating `init_job`/`update_job`/Redis stream replay would
undo the architectural simplification in #43407. Those lifecycle overrides must
move to GTF; the old methods are deliberately not silently implemented as
no-ops.
#### Manager-JWT channels and transport finding
A downstream override of `parse_channel_id_from_request` can continue
returning a signature-verified Manager JWT's `jti`, with no old async-token
cookie required. Custom manager/transport code can use it as its transport
identity. **GTF does not natively express an arbitrary opaque `jti` as a
built-in websocket routing principal.** Its producer validates routes against
task subscribers (`user:<id>` / `guest:<hmac>`, optionally per-client). This PR
does not weaken that validation or treat channel possession as task
authorization. A custom transport must map the authenticated identity to
authorized GTF state, or the downstream must adopt principal/per-tab routing.
The default GTF routes do not use the custom channel helper as an authorization
source.
#### Per-symbol disposition
| Removed surface | Disposition |
| --- | --- |
| `superset.async_events` | Deprecated import namespace only; no API or
Redis engine |
| `superset.extensions.async_query_manager_factory` | Lazy factory alias;
`init_app(app)` (or current app) initializes the GTF-backed manager once |
| `superset.extensions.async_query_manager` | Lazy app-local proxy; replace
with `get_query_manager()` |
| `AsyncQueryManager` | Subclassable `QueryManager` adapter;
`submit_chart_data_job` validates access and delegates to GTF |
| `AsyncQueryTokenException` | Alias for `AsyncQueryTokenError`; existing
exception catches retain identity |
| `create_async_job_command.CreateAsyncChartDataJobCommand` | `validate`
resolves the custom channel; `run` validates QueryContext access and returns
the GTF handshake |
| `init_job`, `update_job`, `read_events`, legacy cancellation and
`result_url` replay | Not emulated; use GTF task lifecycle, status reads, task
cancellation and chart-data re-request |
#### Per-config disposition
| Key | Replacement or disposition |
| --- | --- |
| `GLOBAL_ASYNC_QUERY_MANAGER_CLASS` | Deprecated fallback for
`ASYNC_QUERY_MANAGER_CLASS`; the new key takes precedence |
| `GLOBAL_ASYNC_QUERIES` feature flag | Retained; auto-enables GTF |
| `GLOBAL_ASYNC_QUERIES_POLLING_DELAY` | Retained, same base polling cadence
|
| `GLOBAL_ASYNC_QUERIES_CACHE_BACKEND` | Removed; use
`DISTRIBUTED_COORDINATION_CONFIG` for coordination and `DATA_CACHE_CONFIG` for
results |
| `GLOBAL_ASYNC_QUERIES_TRANSPORT` | Removed; `WEBSOCKET_ENABLE=False` uses
polling, or supply a client polling reader |
| `GLOBAL_ASYNC_QUERIES_WEBSOCKET_URL` | Removed; use `WEBSOCKET_URL` with
the new websocket server |
| `GLOBAL_ASYNC_QUERIES_JWT_SECRET` | Removed; explicitly configure
`WEBSOCKET_JWT_SECRET` only if using websocket transport |
| `GLOBAL_ASYNC_QUERIES_JWT_COOKIE_NAME` | Removed;
`WEBSOCKET_JWT_COOKIE_NAME` (new token claims, not old-token compatibility) |
| `GLOBAL_ASYNC_QUERIES_JWT_COOKIE_SECURE` | Removed;
`WEBSOCKET_JWT_COOKIE_SECURE` |
| `GLOBAL_ASYNC_QUERIES_JWT_COOKIE_SAMESITE` | Removed;
`WEBSOCKET_JWT_COOKIE_SAMESITE` |
| `GLOBAL_ASYNC_QUERIES_JWT_COOKIE_DOMAIN` | Removed;
`WEBSOCKET_JWT_COOKIE_DOMAIN` |
| `GLOBAL_ASYNC_QUERIES_JWT_EXPIRATION_SECONDS` | Removed;
`WEBSOCKET_JWT_EXPIRATION_SECONDS` for websocket token lifetime, not task
timeout |
| `GLOBAL_ASYNC_QUERIES_REGISTER_REQUEST_HANDLERS` | Removed; websocket
cookie handling follows `WEBSOCKET_ENABLE`; custom manager owns its transport
handlers |
| `GLOBAL_ASYNC_QUERIES_REDIS_STREAM_PREFIX` | Removed; no browser-facing
GAQ Redis streams; coordination uses its own namespace |
| `GLOBAL_ASYNC_QUERIES_REDIS_STREAM_LIMIT` | Removed; no GAQ per-channel
event log |
| `GLOBAL_ASYNC_QUERIES_REDIS_STREAM_LIMIT_FIREHOSE` | Removed; no GAQ
firehose; use GTF task observability/retention |
The GTF-era settings `GLOBAL_ASYNC_QUERIES_DEFAULT`,
`GLOBAL_ASYNC_QUERIES_MIN_CACHE_TTL`, `GLOBAL_ASYNC_QUERIES_QUERY_TIMEOUT`,
`GLOBAL_ASYNC_QUERIES_POLLING_MAX_DELAY` and
`GLOBAL_ASYNC_QUERIES_POLLING_STALE_TIMEOUT` are unchanged. Retired settings
other than the manager alias are not interpreted or silently translated.
#### Remaining downstream work
Port legacy manager hooks to `submit`/`status_changes`, consume the GTF
multi-task handshake and re-request chart data, supply the custom polling
reader, and migrate retired deployment settings. Lifecycle metrics tied to
Redis `xadd`/one legacy job must be adapted to GTF task lifecycle rather than
assumed to run through these shims.
A downstream legacy Explore worker also imports removed **private** helpers
from `superset.tasks.async_queries`: `_handle_soft_time_limit`,
`_load_user_from_job_metadata`, `query_timeout`, and `set_form_data`. Those
imports are not covered by this public-seam adapter and can still fail during
downstream manager initialization. That worker needs a separate downstream
migration; this PR must not be presented as sufficient to make the unchanged
shell boot. No downstream source, gitlinks, dependency pins, Helm templates or
CI configuration were modified.
### BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF
Not applicable (extension API/import compatibility; no visual change).
Before: downstream imports of the removed GAQ manager/factory fail, and
chart submission/status reads have no manager hook.
After: the listed imports emit deprecation warnings; configured managers and
polling readers are invoked while GTF remains the execution/state source.
### TESTING INSTRUCTIONS
Automated validation:
```bash
PYTHONPATH=superset-core/src SUPERSET_SECRET_KEY=test pytest \
tests/unit_tests/tasks/test_query_manager.py \
tests/unit_tests/charts/test_chart_data_api.py \
tests/unit_tests/tasks/test_async_queries.py \
tests/unit_tests/tasks/test_subscription.py \
tests/unit_tests/tasks/test_filters.py \
tests/unit_tests/tasks/test_guest.py -q
# 96 passed
cd superset-frontend
npm ci --ignore-scripts --legacy-peer-deps
npm run test -- --maxWorkers=2 src/middleware/asyncEvent.test.ts
# 34 passed, including custom polling and websocket reconciliation
npx tsc --build packages/superset-ui-core
cd ..
PYTHONPATH=superset-core/src pre-commit run
```
The documented changed-files pre-commit subset is used rather than modifying
unrelated master-baseline files. Also ran pylint directly against all changed
backend files (10.00/10), because the staged-file hook's pylint wrapper
computes its file list from committed history.
Manual verification on a configured deployment:
1. Enable `GLOBAL_ASYNC_QUERIES`; keep websocket disabled for the polling
case.
2. Configure a `QueryManager` subclass that records/delegates `submit` and
`status_changes`; load an uncached chart and verify 202 task UUIDs followed by
the warm-cache response.
3. Initialize the frontend with a custom authenticated reader returning the
same GTF status schema. Verify completion and repeat with websocket enabled to
exercise catch-up.
4. Enable Python deprecation warnings and import each legacy symbol. Verify
the old class-config fallback warns, the new config takes precedence, and
repeated initialization keeps one instance per app.
5. Check the same flow with an authorized guest and a user without access;
custom transport code must preserve the documented authorization contract.
Local HTTP health check was unavailable (no running Superset service), so
live Celery/browser deployment validation was not performed. Unit tests and
middleware tests above are the executed checks; this does not claim full shell
runtime validation.
### ADDITIONAL INFORMATION
- [x] Has associated issue: follow-up to #43407
- [x] Required feature flags: `GLOBAL_ASYNC_QUERIES` for asynchronous chart
execution (auto-enables GTF)
- [ ] Changes UI
- [ ] Includes DB Migration (follow approval process in
[SIP-59](https://github.com/apache/superset/issues/13351))
- [ ] Migration is atomic, supports rollback & is backwards-compatible
- [ ] Confirm DB migration upgrade and downgrade tested
- [ ] Runtime estimates and downtime expectations provided
- [x] Introduces new feature or API
- [ ] Removes existing feature or API
AI-assisted implementation and tests; signed-off commit. Deprecation policy
and supported extension contract are proposed for upstream review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
