villebro opened a new pull request, #43434: URL: https://github.com/apache/superset/pull/43434
### SUMMARY Wires up the realtime WebSocket transport end-to-end (epic step **6b**) on top of the 6a backend foundation (#43431), and in doing so **generalizes `superset-websocket` into a feature-agnostic Task push transport**: the Node server now routes purely by Redis channel name and never interprets a message's contents, so any async task (chart data today; thumbnails / reports / exports later) can ride the same mechanism with its specifics carried in the payload. Targets the `gaq-to-gtf` feature branch (part of the [GAQ→GTF epic](https://github.com/apache/superset/pull/43407)). **Two-tier channel model** (both lossy Redis Pub/Sub; the `status_changes` interval poll from step 4 remains the correctness backstop): - **Tier 1 — public entity-change nudges** (`entity-changes:<type>`): broadcast to every socket, opaque `{entity_type, id}` only. (Shipped in 6a; consumed here.) - **Tier 2 — per-principal messages** (`realtime:<channel_id>`): delivered only to the JWT-bound principal's sockets, may carry task-specific detail (`{task_id, status}`). New in this PR. The browser receives a generic `{channel, payload}` envelope and routes on `channel` exactly as the server does. **Slice C — per-principal (tier-2) publish (backend)** - `superset/websocket/channel.py`: extract `channel_id_for(user_id, guest_key)` as the single source of truth for the principal-channel format (`user:<id>` / `guest-<hmac>`), shared by cookie minting and the task layer. - `TaskDAO.get_subscriber_channels(task_id)`: map a task's subscribers (users + guests) to their per-principal channels. - `TaskManager.publish_task_status(task_uuid, status)`: best-effort fan-out of `{task_id, status}` to `realtime:<channel_id>` for each subscriber channel, called from `publish_completion`. A chart-data client learns each terminal transition over the socket instead of re-polling REST. **Slice D — `superset-websocket` rewrite (generic transport)** - Replace the async-events Redis Streams firehose + `result_url` / per-job-channel model with Redis Pub/Sub. Route by channel name only: `entity-changes:*` broadcast, `realtime:<channel_id>` per-principal by JWT `channel` claim. - Drop stream reconnection / `last_id` replay (Pub/Sub is lossy; poll is the backstop). New config `entityChangesChannelPrefix` / `perPrincipalChannelPrefix`; cookie default `superset-ws-token`. README rewritten. **Slice E — frontend WS client (`asyncEvent.ts`)** - Connect to `WEBSOCKET_URL` (JWT channel cookie rides the handshake) when `WEBSOCKET_ENABLED`; on a tier-2 message for an awaited `task_id`, apply the status immediately (accelerates completion). Reconnect on close; the interval poll stays the backstop. **Guest subscribers in the Task List face pile** - `get_subscribers` now returns embedded guests as anonymized `is_guest` entries with stable `G1`/`G2`… labels (ordered by subscription time) instead of nameless blanks; the face pile renders them as `G1`/`G2` avatars. **Also folded in** (CI fixes for pre-existing failures on the `gaq-to-gtf` branch, per maintainer request): - `tsc`: declare the per-dashboard `async_mode` override (step 5) on `DashboardInfo.metadata`. - `mypy`: widen the chart-data orchestrator's `depends_on` to the `list[CoreTask | UUID | str] | None` union `TaskOptions` accepts. - `UPDATING.md`: document the GAQ→GTF cutover + websocket flow for 7.0, and drop the now-stale `GLOBAL_ASYNC_QUERIES_CACHE_BACKEND` deprecation bullet (the key is fully removed this cycle). ### TESTING INSTRUCTIONS Automated (all green locally): - Backend: `pytest tests/unit_tests/tasks/ tests/unit_tests/daos/test_tasks.py tests/unit_tests/websocket/` — 394 passed. New coverage: `publish_task_status` fan-out, `get_subscriber_channels`, `channel_id_for`, guest schema serialization. - Node: `cd superset-websocket && npm test` — 46 passed. New coverage: pub/sub routing (broadcast + per-principal + malformed/unknown channel). - Frontend: `jest src/middleware/asyncEvent.test.ts src/pages/TaskList/TaskList.test.tsx` — 15 passed. New coverage: WS tier-2 acceleration (settle/reject/ignore-non-realtime). - `mypy` / `ruff` / `pylint` / `tsc` / `eslint` clean on changed files. Manual (with `GLOBAL_ASYNC_QUERIES` on, `DISTRIBUTED_COORDINATION_CONFIG` + `WEBSOCKET_ENABLED` set, `superset-websocket` running on the same host): 1. Load a dashboard with async charts → confirm completion arrives over the socket (faster than the poll interval) and charts render. 2. Kill the websocket server → confirm charts still complete via the interval poll. 3. Open an embedded/guest dashboard that schedules a shared task → confirm it appears in the Task List subscriber face pile as `G1`. ### ADDITIONAL INFORMATION - [ ] Has associated issue: - [x] Required feature flags: `GLOBAL_ASYNC_QUERIES` (async chart data); `WEBSOCKET_ENABLED` (realtime transport, optional) - [x] Changes UI - [ ] Includes DB Migration (follow approval process in [SIP-59](https://github.com/apache/superset/issues/13351)) - [ ] Migration is atomic, supports rollback & is backwards-compatible - [ ] Confirm DB migration upgrade and downgrade tested - [ ] Runtime estimates and downtime expectations provided - [x] Introduces new feature or API - [x] Removes existing feature or API -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
