villebro opened a new pull request, #44998: URL: https://github.com/apache/superset/pull/44998
## SUMMARY This is the long-lived feature branch for **Canvas**, the next generation of Superset dashboards ([SIP #44989](https://github.com/apache/superset/issues/44989)), and the **Widget Framework** it places ([SIP-231, #44875](https://github.com/apache/superset/issues/44875)). The two SIPs share one branch and one `CANVAS` feature flag because neither ships without the other. - **Widget Framework (SIP-231):** a widget is a backend Python class plus a React renderer, delivered by core or an extension. Its properties are a Pydantic model whose JSON Schema drives the property panel, validation and progressive disclosure for agents. A chart is a widget with `QueryMixin`: it builds `SemanticQuery` objects on the backend from its stored props, so data is served by widget instance instead of by caller-shaped query. Instances are persisted (the `widgets` table, with editors and viewers) or inline on a Canvas placement. - **Canvas (#44989):** a document of placements of widget instances, edited live through small, validated operations by people and agents alike, with containers, filter scope through a generic listener bus, permalinks and snapshot export. Each step is reviewed as a focused child PR against `canvas` and squash-merged there. This umbrella PR merges the completed work back to `master`. Its description is the living source of truth for the branch: the progress tracker, a curated summary of what each merged step shipped, how to run the current state, and the open decisions. Everything is behind `CANVAS`; v1 charts and dashboards are untouched. ### How to contribute - Fetch `upstream` and branch child work off `canvas`; open PRs with `canvas` as the base and a Conventional Commits title. - Address review with new commits rather than force-pushes; child PRs are squash-merged. - After a child PR merges, mark it **Merged** in the tracker and add a condensed bullet under "What each step shipped" describing what actually landed. - `master` is merged into `canvas` regularly (merge, not rebase), so open child branches stay valid. - Every step ships MCP tools alongside its REST and UI surface, so agents can create and consume widgets and canvases headlessly. ### Key Features These describe the target state from the two SIPs; the tracker shows what has landed. - **Schema-driven widgets.** One Pydantic model per widget drives the property panel, server-side validation with actionable errors, and agent-facing schemas. - **Backend query building.** Charts turn stored props into `SemanticQuery` objects on the server (`SqlaTable` through a `SemanticView` adapter), so caches can be warmed and reports run without a browser. - **Access by widget instance.** Data requests take only additive runtime filters; the query definition always comes from storage. - **Async and realtime data.** Widget queries run on the Global Task Framework into a required cache, with results delivered as Arrow IPC and completion pushed in realtime (SIP-227). - **Saved and inline widget instances.** Persisted instances can be placed on many canvases, or none; inline instances live on their placement and can be promoted. - **Optional drafts.** Complete instances are written directly; drafts let an instance be built up step by step and previewed before others see it. - **Canvas documents.** Live, validated operations (`add`, `set_props`, `patch_props`, `place`, `move`, `remove`, `set_scope`, `promote`, `inline`) with per-field-group concurrency, containers as widgets, filter scope on placements, permalinks and snapshot export. - **Agent-first.** MCP tools for widget types, saved widgets, drafts and canvases, with progressive schema disclosure that keeps agent context small. ## BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF UI steps add screenshots here as they merge. ## TECHNICAL DETAILS ### Architecture Overview | Layer | Widget Framework (SIP-231) | Canvas (#44989) | | --- | --- | --- | | Contract | `superset_core.widgets`: `Widget`, `QueryMixin`, strict props, `@widget`; `WidgetBehavior` / `WidgetUi` | Container, layout and filter-scope fields on `WidgetBehavior` / `WidgetUi`; `superset_core.canvas` | | Storage | `widgets` (model `Widget`, `widget_type` references the registered class), `widget_editors`, `widget_viewers` | `canvases`, `canvas_ops`, `canvas_editors`, `canvas_viewers`; inline instances on placements | | REST | `/api/v1/widget/{uuid}` (CRUD, `/data`, `/draft`), `/api/v1/widget_type/{widget_type}/…` | `/api/v1/canvas/` (CRUD, `/definition`, `/definition/changes`, `/placement/{id}/data`, `/snapshot`, `/permalink`) | | MCP | `list_widget_types`, `get_widget_type_info`, `get_widget_type_schema`, `get_widget_type_field_options`, `list_widgets`, `get_widget_info`, `*_widget_draft` | `get_canvas`, `apply_canvas_ops` | | Frontend | `widgets` namespace in `@apache-superset/core`: renderer registry, `useWidgetData`, `useRuntimeFilters`, `emitFilter` on a generic event bus | Canvas page, containers, listener bus, `getActiveCanvas` | | Permissions | `Widget` resource (`can_read`, `can_write`) | `Canvas` resource (`can_read`, `can_write`) | ### Operator-Facing Changes | Area | Change | | --- | --- | | Feature flag | `CANVAS` (off by default) gates both features. Convention: REST APIs are always registered so their permissions exist, and answer 404 while the flag is off; MCP tools are removed. | | Database | None merged yet. | | Configuration | None merged yet. | ### Progress Tracker **Widget Framework** (@villebro) | Step | Scope | Status | PR | | --- | --- | --- | --- | | W1 | `Widget` entity: `widgets` table with editors/viewers, `superset_core.widgets` model and DAO, commands with optimistic concurrency (`If-Match` / 409), REST `/api/v1/widget/`, MCP `list_widgets` / `get_widget_info`, `CANVAS` flag | In progress | | | W2 | Widget registry and contract: `Widget`, `QueryMixin`, strict props, `@widget`, `WidgetBehavior` / `WidgetUi` (including the Canvas fields), `/api/v1/widget_type/`, a `markdown` widget, MCP type tools | Planned | | | W3 | Optional drafts: key-value drafts with JSON Patch and revision checks, preview data, MCP draft tools | Planned | | | W4 | Single-widget Canvas view: chat extension with MCP, JSON editor with validation annotations, metadata header (canvases, created/modified by, editors) | Planned | | | W5 | Async-first data serving: GTF tasks, required cache, Arrow IPC, `SqlaTable` `SemanticView` adapter, realtime delivery, widget and placement data routes | Planned | | | W6 | Frontend `widgets` namespace: renderer registry, `useWidgetData`, Arrow decoding | Planned | | | W7 | Templated ECharts widget | Planned | | | W8 | Progressive disclosure engine | Planned | | | W9 | v2 line chart: JSONForms property panel with per-series controls | Planned | | | W10 | Agent comprehension: `describe_widget` | Planned | | **Canvas** (@msyavuz; seeded from the SIP's M1, to be refined) | Step | Scope | Status | PR | | --- | --- | --- | --- | | C1 | Canvas model, storage and access: `canvases` with editors/viewers, REST CRUD, `Canvas` permission | Planned | | | C2 | Canvas document and operations, concurrency and op log, `definition/changes` | Planned | | | C3 | Placement ids, containers and layout as widgets | Planned | | | C4 | Filter scope and the listener bus, wired to SIP-231's hooks | Planned | | | C5 | Canvas page rendering and realtime updates | Planned | | | C6 | Agent access: `get_canvas`, `apply_canvas_ops` | Planned | | | C7 | `promote` / `inline`, permalinks and snapshot export | Planned | | <details> <summary><b>What each step shipped</b></summary> Nothing merged yet. Each merged child PR adds a condensed bullet here. </details> <details> <summary><b>Security and backward compatibility</b></summary> **Security.** Widgets and canvases get their own permission resources (`Widget`, `Canvas`) rather than reusing chart and dashboard permissions, and editors and viewers are subjects (users, roles and groups). Chart data is served by widget instance: the server loads the query definition from storage, and requests may only add runtime filters, so consuming a chart does not let a caller shape arbitrary queries. In M1, reading a chart's data also requires access to its datasource; "chart access without dataset access" needs a new row in the `SECURITY.md` matrix first. Metrics and dimensions are re-resolved against the catalog, server limits clamp requests, and RLS applies at execution. **Backward compatibility.** All tables, routes and permissions are additive and behind `CANVAS`. Charts, dashboards and their APIs are unchanged; migrating `Slice` rows and converting dashboards are later, separately gated milestones in the SIPs. </details> <details> <summary><b>Open decisions</b></summary> - **Grouping edits into commits.** An explicit commit that groups several edits, or every change persisted live with version history; writes can be batched either way. Open in both SIPs. - **Runtime choices.** `x-runtime` / `runtime_props` (for example a dynamic group-by) are deferred until safeguards exist, such as a permission to change runtime choices. M1 uses additive runtime filters only. - **Canvas open questions.** Op log visibility, history and undo, filters across datasources, and conversion from dashboards (#44989). </details> ## TESTING INSTRUCTIONS The branch is `master` plus an empty opening commit, so there is nothing to exercise yet. As steps merge, this section describes how to run the merged state end to end, with targeted test commands. ## ADDITIONAL INFORMATION - [x] Has associated issue: #44875, #44989 - [x] Required feature flags: `CANVAS` - [x] Changes UI - [x] Includes DB Migration (follow approval process in [SIP-59](https://github.com/apache/superset/issues/13351)) - [ ] Migration is atomic, supports rollback & is backwards-compatible - [ ] Confirm DB migration upgrade and downgrade tested - [ ] Runtime estimates and downtime expectations provided - [x] Introduces new feature or API - [ ] Removes existing feature or API -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
