shivamgoel commented on issue #37535:
URL: https://github.com/apache/superset/issues/37535#issuecomment-5417316266
# [SIP] Unified headless query API for Explorable datasources
> Revises an earlier SIP-199 draft. Reviewed with Ville Brofeldt;
## Motivation
Superset has no REST API for querying a datasource by its semantic
definitions. A service, pipeline, or agent
that wants to ask *"give me `sum__sales` by `region` for the last 30 days"*
has exactly one option today:
`POST /api/v1/chart/data`.
That endpoint is designed for the Explore UI. Its payload is a
`query_context` — a structure whose purpose is
to carry frontend state, and whose shape is driven by the needs of chart
controls. Using it headlessly means:
- **Encoding Superset's internal query representation by hand.** Callers
must construct `query_context`,
`queries[]`, `extras`, adhoc column objects with `BASE_AXIS` / `SERIES`
types, `post_processing` pivot and
flatten chains, and `orderby` tuples. That is frontend implementation
detail, not an API.
- **No validation of what you asked for.** A misspelled metric or dimension
surfaces as a SQL error or an
empty result, not as *"unknown metric `sum__sale`, did you mean
`sum__sales`?"*.
- **No stable contract.** `query_context` evolves with the Explore UI. An
external consumer coupling to it is
coupling to frontend internals, and inherits every change made for
visualization reasons.
- **Chart-shaped results.** The response is organised around what a
visualization needs, including
post-processing applied for rendering rather than for the data itself.
In practice every external consumer ends up reverse-engineering a
`query_context`, typically by reading a
saved chart's `form_data` and mutating it. That code is fragile, breaks on
viz-type changes, and duplicates
Superset's own query assembly badly.
Superset already has the right abstraction internally. The `Explorable`
protocol
(`superset/explorables/base.py`) is implemented by both `SqlaTable` and the
semantic-layer `SemanticView`, and
`Explorable.get_query_result` already dispatches to SQL execution or to a
semantic-layer provider. What is
missing is a REST surface that exposes it with a name-based payload.
### Goals
1. A REST endpoint that queries **any** `Explorable` datasource — dataset or
semantic view — from a simple,
name-based payload, with no knowledge of `query_context`.
2. A **stable, versioned public contract** decoupled from Explore's internal
representation.
3. Helpful validation: unknown metrics, dimensions, filter columns, and
order-by names are named back to the
caller with suggestions, before any SQL runs.
4. Ad-hoc metric support for datasets, because real consumers depend on it.
5. **JSON as the primary result format**, with Arrow IPC available as an
explicit per-request opt-in for
throughput-sensitive callers.
6. Capability discovery, so clients can adapt to what a given datasource
supports.
7. A single internal implementation of this path, owned by the REST layer,
that other surfaces consume rather
than reimplement.
### Non-goals
Async execution; `series_limit` / `group_limit`; `post_processing`;
`annotation_layers`; guest-token support;
moving query execution into `superset-core`; convenience URL aliases. Each
is addressed under
*Rejected and deferred*.
## Proposed Change
### The endpoint
Two routes added to the **existing** `DatasourceRestApi`
(`superset/datasource/api.py`), which already serves
this URL shape:
```
existing: POST /api/v1/datasource/<t>/<id>/compatible
GET /api/v1/datasource/<t>/<id>/column/<col>/values/
POST /api/v1/datasource/<t>/<id>/validate_expression
GET /api/v1/datasource/ (combined_list)
new: POST /api/v1/datasource/<t>/<id>/query
GET /api/v1/datasource/<t>/<id> (metadata +
capabilities)
```
`<t>` is a `DatasourceType` value — `table` for datasets, `semantic_view`
for semantic views. Reusing this
class means no new API class and no new permission view menu: the file
already establishes
`DatasourceDAO.get_datasource(DatasourceType(t), id)` + `raise_for_access()`
with `ValueError`→400,
`DatasourceTypeNotSupportedError`→400, `DatasourceNotFound`→404,
`SupersetSecurityException`→403.
### Execution path
No new execution machinery. The endpoint enters the existing pipeline at
`QueryContextFactory`, the boundary
where caching, RLS, and authorization already live:
```
request
|
v
DatasourceQuerySchema.load() <- new marshmallow schema
| invalid type -> 400
| semantic_view + SEMANTIC_LAYERS off -> 404
v
resolve_explorable() <- new shared core
DatasourceDAO.get_datasource() + raise_for_access()
|
v
validate_query_names() <- new shared core
unknown metric/dimension -> 400, named + suggestions
|
v
build_query_dict() <- new shared core
|
v
execute_tabular_query() <- new shared core
QueryContextFactory.create()
set_query_context_form_data() <- Jinja macro support
ChartDataCommand.validate() <- THE security gate (access, RLS)
ChartDataCommand.execute()
|
v
Explorable.get_query_result() <- type dispatch ALREADY lives
here
|-- SqlaTable -> SQL execution
+-- SemanticView -> mapper.get_results() -> provider
```
Two structural facts keep this small:
- **No dispatch layer is needed.** Datasource-type polymorphism already
lives in
`Explorable.get_query_result` (`superset/explorables/base.py:199`). The
endpoint passes `type` as data and
never branches on it to execute.
- **The query engine is untouched.** `QueryObject`, `QueryContext`,
`QueryContextFactory`,
`QueryObjectFactory`, and `ChartDataCommand` need no changes. Result
caching, RLS, post-processing, time
offsets, event logging, and `apply_max_row_limit` clamping all come along
for free precisely because we
enter at `QueryContextFactory` rather than below it.
### The shared core
The resolve → validate → build → execute sequence lands in one new module,
`superset/common/tabular_query.py`:
| Symbol | Responsibility |
| --- | --- |
| `ResolvedExplorable` | Carries the `Explorable`, display name, resolved
time column, and valid metric/dimension name sets |
| `resolve_explorable(...)` | DAO lookup + `raise_for_access()`, then reads
`Explorable.metrics` / `.columns`. Because both datasource types satisfy the
protocol this is **one** code path; only the temporal default differs
(`main_dttm_col` for datasets, first `is_dttm` dimension for views) |
| `validate_query_names(...)` | Validates metrics, dimensions, filter
columns, and order-by names; returns caller-facing messages |
| `build_query_dict(...)` | Translates the request vocabulary into the
`QueryObject` one — `limit`/`offset` → `row_limit`/`row_offset`, `order` →
`orderby`, `time_grain` → a `BASE_AXIS` adhoc column plus
`extras.time_grain_sqla` |
| `execute_tabular_query(...)` | `QueryContextFactory.create()` →
`set_query_context_form_data()` → `ChartDataCommand.validate()` → `.execute()` |
**This core belongs to the REST layer.** The REST endpoint is the stable,
versioned, publicly documented
contract, so it owns the canonical implementation. Other surfaces consume it.
That has an immediate consequence. The MCP service has two tools —
`superset/mcp_service/semantic_layer/tool/get_table.py` and
`superset/mcp_service/dataset/tool/query_dataset.py` — that each implement
this sequence inline, and they have
already drifted apart:
| | `get_table` | `query_dataset` |
| --- | --- | --- |
| `set_query_context_form_data` (Jinja support) | no | yes |
| `custom_cache_timeout` honoured | no | yes |
| Dimension field name | `dimensions` | `columns` |
| `extras.time_grain_sqla` | no | no |
Row 1 is a live bug: the same query against a Jinja-templated virtual
dataset returns **different results**
through the two tools, because only one makes the query context visible to
`{{ current_username() }}` and
`{{ filter_values() }}`. This SIP refactors both onto the shared core,
fixing it structurally.
```
BEFORE AFTER
get_table.py query_dataset.py common/tabular_query.py
resolve/validate resolve/validate resolve_explorable
build/execute build/execute validate_query_names
(drifted) build_query_dict
execute_tabular_query
| | |
v v v
/query get_table query_dataset
canonical (MCP) (MCP)
```
The MCP tools keep what is genuinely theirs: `ctx.report_progress`
choreography, error-schema mapping,
LLM-oriented `summary` prose, and `@requires_data_model_metadata_access`.
They gain Jinja and `cache_timeout`
correctness as a side effect.
**Acceptance bar: the existing MCP unit tests pass unmodified.** That is the
proof the refactor is
behaviour-preserving rather than a rewrite.
## New or Changed Public Interfaces
### `POST /api/v1/datasource/<t>/<id>/query`
Request — new `DatasourceQuerySchema` in the existing
`superset/datasource/schemas.py`:
```
{
"metrics": ["sum__sales", {"expressionType": "SQL",
"sqlExpression": "SUM(a)/SUM(b)",
"label": "Ratio"}],
"dimensions": ["region", "product_category"],
"filters": [{"col": "region", "op": "IN", "val": ["EMEA", "APAC"]}],
"time_range": "Last 30 days",
"time_column": "order_date",
"time_grain": "P1D",
"limit": 1000,
"offset": 0,
"order": [{"column": "sum__sales", "descending": true}],
"use_cache": true,
"force": false,
"cache_timeout": 300,
"result_format": "json"
}
```
Field decisions, each reusing existing types rather than inventing parallel
ones:
| Field | Type | Basis |
| --- | --- | --- |
| `metrics` | `List(Raw)` | `Metric: TypeAlias = AdhocMetric \| str`
(`superset/superset_typing.py:161`), so saved metric names **and** ad-hoc
metrics are accepted at no extra cost — identical to
`ChartDataQueryObjectSchema.metrics` |
| `dimensions` | `List(Raw)` | Maps to `QueryObject.columns`; `Column:
TypeAlias = AdhocColumn \| str` |
| `filters` | `List(Nested(ChartDataFilterSchema))` | Reuses the existing
filter schema, which already validates `op` against the full `FilterOperator`
enum |
| `time_range` | `String` | Resolved by the existing
`get_since_until_from_time_range` |
| `time_column` | `String` | → `QueryObject.granularity` |
| `time_grain` | `String` (ISO 8601, e.g. `P1D`) | →
`extras.time_grain_sqla`. Validated against `get_time_grains()` for datasets
and per-dimension `Grain` variants for views |
| `limit` / `offset` | `Integer` | `SemanticQuery`'s names, translated to
`row_limit` / `row_offset` internally. Server-clamped by `apply_max_row_limit` |
| `order` | `List(Nested)` | `[{column, descending}]`, mirroring
`SemanticQuery`'s `OrderTuple`. Applied in sequence, each with its own
direction |
| `result_format` | `Enum(ChartDataResultFormat)` | **Defaults to `json`.**
`arrow` is an explicit opt-in (see below); `csv` / `xlsx` follow in phase 2 |
Field names follow `SemanticQuery` throughout — `dimensions` rather than
Explore's `columns`, `limit` /
`offset` rather than `row_limit` / `row_offset`, and an `order` array with
per-column direction rather than a
single `order_desc` flag. Translation to the `QueryObject` vocabulary
happens once, in `build_query_dict`.
This is deliberate: it keeps the public contract independent of the
execution model, so the payload would
survive a later move to a `SemanticQuery`-first execution path (see
*Rejected and deferred*).
Response, `result_format: "json"` — **the default and the primary
contract.** Callers that omit
`result_format` get this, and it is the format every client is expected to
support:
```
{"result": [{"data": [...], "colnames": [...], "coltypes": [...],
"rowcount": 100, "is_cached": false, "cache_key": "...",
"applied_filters": [...], "rejected_filters": [...]}]}
```
Response, `result_format: "arrow"` — **an explicit opt-in**, for callers
where serialization cost matters.
`Content-Type: application/vnd.apache.arrow.stream`, body is an Arrow IPC
stream. Requires:
- `ARROW = "arrow"` added to `ChartDataResultFormat`
(`superset/common/chart_data.py`)
- a serializer branch in `QueryContextProcessor.get_data`, alongside the
existing `csv` / `xlsx` / records
branches
Deliberate choices, each keeping the opt-in cheap and JSON unaffected:
- **No feature flag.** Arrow is selected per request, so no operator
configuration is introduced and no
deployment has to enable anything. A caller that never sends
`result_format: "arrow"` is unaffected by its
existence, and a caller that does — such as a future chart v2 framework —
can rely on it being present.
- **Caching is unaffected.** The data cache continues to store the
DataFrame, and Arrow serialization happens
at response time. No cache-format change, no cache-key versioning, and
JSON and Arrow requests share cache
entries.
- **Arrow is deliberately *not* added to
`ChartDataResultFormat.table_like()`**, so it is not gated behind
`can_export_data` / `can_csv`. Arrow is a programmatic transport, not a
human file export. Flagged for
review — a reviewer could reasonably argue the opposite.
Note a useful asymmetry: the semantic-view path already materializes a
`pa.Table` in
`SemanticResult.results` and converts it to pandas in
`map_semantic_result_to_query_result`. A future
optimization can pass Arrow through without the pandas round-trip; this SIP
does not attempt that.
### `GET /api/v1/datasource/<t>/<id>`
Returns `datasource.data` — already implemented by both `SqlaTable` and
`SemanticView` — plus a `capabilities`
block assembled from properties that already exist on `Explorable`:
```
{"result": {
"...": "existing datasource.data payload",
"capabilities": {
"is_rls_supported": true,
"query_language": "sql",
"supports_samples": true,
"supports_drill_to_detail": true,
"supports_adhoc_metrics": true,
"time_grains": [{"name": "Day", "duration": "P1D"}],
"supported_operators": ["==", "!=", "IN", "..."],
"features": []
}}}
```
`features` carries `SemanticViewFeature` values for views and `[]` for
datasets. Compatible metric/dimension
discovery is **not** duplicated here — the existing `POST
/api/v1/datasource/<t>/<id>/compatible` provides it
with an RLS-aware cache.
*Open for PR-time discussion:* whether this per-datasource route is the
right home for capabilities, or
whether `combined_list` should carry them instead.
### Ad-hoc metrics
Ad-hoc metrics are where the two datasource types genuinely differ, and this
is not hypothetical: consumers
routinely define metrics like `sum(error_count)*100.0/sum(request_count)`
per request rather than as saved
metrics. A names-only API would serve none of them.
- **Datasets:** supported, at no additional cost, via the `fields.Raw()`
typing above.
- **Semantic views:** rejected by design — `mapper._validate_metrics` raises
because the provider owns metric
definitions. Surfaced as a **400 naming the offending metric**, never a
500.
Clients needing portability across both types should prefer saved metrics,
which is also the SIP-185
direction.
### `ILIKE` on semantic views
Today `mapper._convert_query_object_filter` **raises** on `ILIKE` /
`NOT_ILIKE`, and core `Operator`
(`superset_core/semantic_layers/types.py`) has no member for them. This
appears to be an oversight rather than
a deliberate restriction, so this SIP proposes fixing it:
- add `ILIKE` / `NOT_ILIKE` to core `Operator`
- implement the mapping in `mapper._convert_query_object_filter` instead of
raising
*Open question for the semantic-layer author:* whether every provider
(Snowflake, dbt, Cube) can express
case-insensitive matching. If any cannot, the fallback is a new
`SemanticViewFeature` gate — the mechanism
that already exists for precisely this situation.
### Permissions
The new routes use a single `can_query on Datasource` permission, granted to
Gamma by adding `"can_query"` to
`READ_ONLY_PERMISSION` (`superset/security/manager.py:1767`). Both the `POST
…/query` and the metadata `GET`
share it.
Three notes for reviewers:
1. **Precedent exists.** `READ_ONLY_PERMISSION` already contains
`can_get_drill_info`, a specific method
permission added for the same reason. Without this, `_is_alpha_only`
(line 3058) withholds the permission
from Gamma, because `Datasource` is in `GAMMA_READ_ONLY_MODEL_VIEWS`
(line 1697).
2. **No collision.** The only existing `can_query` is `("can_query",
"Api")`, which lives in
`PUBLIC_ROLE_PERMISSIONS` (line 1850) as an explicit tuple and is
unaffected by `READ_ONLY_PERMISSION`.
3. **The metadata route is deliberately not named `get`.**
`PUBLIC_ROLE_PERMISSIONS` already contains
`("can_get", "Datasource")` for chart rendering, so a method named `get`
would inherit a permission the
Public role already holds — silently exposing datasource metadata to
unauthenticated users wherever
`PUBLIC_ROLE_LIKE = "Public"`. Sharing `can_query` avoids this.
A related pre-existing issue is **out of scope here and will be filed
separately**:
`can_get_column_values` and `can_compatible` on this same class are subject
to the same `_is_alpha_only`
exclusion and are therefore unreachable by Gamma today. Consolidating them
onto `can_query` is a breaking
permission change requiring a migration, so it belongs in its own PR.
### Security
- Route level: `@protect()`, plus `raise_for_access()` in
`resolve_explorable`.
- Real gate: `ChartDataCommand.validate()`, which applies query-context
authorization and RLS. No second
authorization path is written.
- Error output: `query` and `stacktrace` stripped and messages passed
through `sanitize_error_message` for
low-privilege principals, mirroring existing chart-data behaviour.
- Row limits: schema cap plus the existing server-side `apply_max_row_limit`.
- Guest tokens are **out of scope** for this SIP.
Per `SECURITY.md`, this endpoint grants no capability beyond what datasource
read access +
`raise_for_access` + RLS already permit through `/api/v1/chart/data`. It is
a validated, stable spelling of an
existing entitlement.
### Feature flag behaviour
The route is **always registered**. `SEMANTIC_LAYERS` (default `False`)
gates only the semantic-view branch:
| `datasource_type` | Flag off | Flag on |
| --- | --- | --- |
| `table` | works | works |
| `semantic_view` | **404** | works |
Gating the whole route would make the dataset case unreachable in nearly
every deployment. The 404-when-off
behaviour matches `superset/semantic_layers/api.py` and the conditional half
of `combined_list`.
## Disruption analysis
**New files: 2** — `superset/common/tabular_query.py` and its unit test.
**Modified files: 10**
| File | Change |
| --- | --- |
| `superset/datasource/api.py` | +2 methods |
| `superset/datasource/schemas.py` | +1 schema |
| `superset/mcp_service/semantic_layer/tool/get_table.py` | call shared core
|
| `superset/mcp_service/dataset/tool/query_dataset.py` | call shared core |
| `superset/security/manager.py` | +`can_query` in `READ_ONLY_PERMISSION` |
| `superset-core/.../semantic_layers/types.py` | +`ILIKE` / `NOT_ILIKE` on
`Operator` |
| `superset/semantic_layers/mapper.py` | implement `ILIKE` instead of
raising |
| `superset/common/chart_data.py` | +`ARROW` on `ChartDataResultFormat` |
| `superset/common/query_context_processor.py` | +Arrow serializer branch in
`get_data` |
| `superset/explorables/base.py` | declare `raise_for_access` on the
protocol |
The last row was not in the original plan and is worth calling out: every
explorable implements
`raise_for_access` and callers depend on it before building a query, but the
protocol never declared it, so
`mypy` rejects the call. Declaring it is a one-line fix that also benefits
the existing
`BaseDatasource | Explorable` unions in `security/manager.py`. Safe at
runtime — nothing does
`isinstance(..., Explorable)`, so widening the `@runtime_checkable` protocol
changes no behaviour.
**Untouched:** `QueryObject`, `QueryContext`, `QueryContextFactory`,
`QueryObjectFactory`,
`ChartDataCommand`, `DatasourceDAO`, `DatasourceType`, frontend,
`/api/v1/chart/data`.
**Zero:** database migrations, new feature flags, new API classes, new view
menus, new dependencies
(`pyarrow>=24.0.0,<26` is already a direct dependency —
`pyproject.toml:104`), breaking changes to existing
endpoints.
**One new FAB permission:** `can_query on Datasource`, deliberately.
**One behavioural change to an existing surface:** the `ILIKE` rejection in
`mapper.py` was deliberate and
unit-tested (`test_convert_query_object_filter_ilike_rejected`), with a
comment arguing that collapsing
case-sensitivity onto the backend's collation diverges from the operator the
author chose. Implementing
`ILIKE` replaces that test. The original rationale still deserves an answer
from the semantic-layer author —
see the `ILIKE` section.
## New dependencies
None. `pyarrow` is already a direct dependency of the main package, so Arrow
support adds nothing.
## Migration Plan and Compatibility
- Purely additive at the HTTP layer. No database migrations, no config
changes required.
- `POST /api/v1/chart/data` is unchanged and remains the endpoint for
chart-shaped queries,
`post_processing`, `annotation_layers`, and async.
- Two internal changes are disclosed rather than hidden: `validate_names`
relocates out of
`superset/mcp_service/utils/` (with a one-release re-export at the old
path), and the two MCP tools lose
their inline implementations. Their public tool schemas are unchanged;
`get_table` **gains** Jinja and
`cache_timeout` correctness. That is a bug fix, but it changes results for
anyone depending on the broken
behaviour, so it warrants an `UPDATING.md` note.
- Deployments that grant permissions explicitly rather than via role bundles
will need to grant
`can_query on Datasource`. Noted in `UPDATING.md`.
### PR sequence
```
PR 1 (optional, independent — consistency only, NOT a packaging bug)
superset-core: add semantic_layers/__init__.py
PR 2 (this SIP)
shared core + POST /query (json + arrow) + GET /<t>/<id> capabilities
+ refactor both MCP tools onto the core
+ can_query in READ_ONLY_PERMISSION
+ ILIKE on semantic views
PR 3 (follow-up, separate)
consolidate can_get_column_values / can_compatible onto can_query
+ migration + UPDATING.md
```
On PR 1: `superset_core/semantic_layers/` is the only subpackage without an
`__init__.py`, so adding one is a
consistency tidy-up. It is **not** a packaging fix.
`[tool.setuptools.packages.find]` defaults to
`namespaces = true`, so setuptools already discovers the directory and ships
every module in it — verified by
building a wheel without the file, installing it into a clean environment,
and importing
`superset_core.semantic_layers.types` successfully. The only practical
differences are that a namespace
package cannot carry re-exports or a docstring. Drop this PR if a reviewer
would rather not churn the file.
## Rejected Alternatives
**Continuing to use `POST /api/v1/chart/data`.** It is built for the Explore
UI. Its `query_context` payload
exists to carry frontend state; it requires callers to construct adhoc
column objects with `BASE_AXIS` /
`SERIES` types, `post_processing` pivot and flatten chains, and `extras`
internals; it performs no name
validation; its result shaping serves visualization rather than data
consumption; and its shape legitimately
changes as Explore evolves. Coupling external consumers to it means coupling
them to frontend internals and to
a contract that was never intended to be stable for them. Making it serve
both audiences well would mean
freezing frontend internals as a public API — the wrong trade for both.
**Defining a parallel query model.** The semantic-layer vocabulary
(`metrics`, `dimensions`) is industry
standard and already used internally. This SIP adopts it rather than
inventing a third vocabulary.
**`SemanticQuery` as the wire format.** `SemanticQuery` holds
`pyarrow.DataType` values and *resolved*
`Dimension` / `Metric` objects rather than names, with `set[Filter]` of
frozen dataclasses. It is not
JSON-serializable, and it deliberately omits anything a driver cannot
express (no `time_range` string, no
`granularity` column). It is a driver ABI, constructed only by
`mapper.map_query_object()`. We adopt its
terminology, not the type.
**`SemanticQuery` as the single execution IR — deferred, and this SIP's main
self-imposed limitation.**
A more ambitious design is worth naming because it is plausibly the end
state: rather than translating the
payload into a `QueryObject` and letting `Explorable.get_query_result` fork,
the endpoint could build a
`SemanticQuery` and hand it to a provider for **both** datasource types.
`SemanticQuery` would become the one
internal representation, `superset/semantic_layers/mapper.py` would no
longer need to translate *from*
chart-oriented types, and the `QueryObject` vocabulary would stop leaking
into new surfaces.
Two things block it today, and neither is cheap:
1. **No dataset driver exists.** The core `SemanticView` ABC
(`superset_core/semantic_layers/view.py`) has no in-tree implementations
— the registry is populated
exclusively by extensions (`superset/core/api/core_api_injection.py`).
Executing a dataset as a
`SemanticQuery` therefore requires a new `SqlaTable`-backed driver
implementing `get_table`,
`get_row_count`, `get_dimensions`, `get_metrics` and the compatibility
methods — effectively a second SQL
generator alongside the existing one, and the place where every dialect
quirk currently handled by
`SqlaTable` would have to be re-proven.
2. **The cross-cutting concerns live above the driver.** Result caching, RLS
injection, post-processing,
time-offset joins, event logging and `apply_max_row_limit` are all
applied by
`QueryContextProcessor` / `ChartDataCommand`. A `SemanticQuery`-first
path either re-implements them or
re-enters the same pipeline, at which point the `QueryObject` translation
is back.
So this SIP takes the smaller step deliberately: it standardises the
*request* vocabulary on semantic-layer
terms while leaving execution on the proven pipeline. Nothing here
forecloses the larger move — the shared
core in `superset/common/tabular_query.py` is the seam where a
`SemanticQuery`-first execution path would be
substituted, and the wire format is already expressed in `SemanticQuery`'s
vocabulary rather than
`QueryObject`'s, so the payload would not change.
Flagged explicitly because "based on the simpler `SemanticQuery` interface"
can reasonably be read as asking
for this, and it deserves an answer rather than silence. If the preference
is to pursue it now, it should be
its own SIP with the dataset-driver question settled first.
**Separate endpoints per datasource type.** One schema with `<t>` in the
path is simpler for consumers and
avoids API proliferation. Type dispatch already exists below the API layer.
**`DatasourceQueryCommand` in `superset-core`.** `superset-core` contains no
query execution: no Flask-free
`QueryContext`, no cache manager, no `security_manager`. The chain depends
on `flask.current_app` and
`superset.extensions`. Making execution Flask-free is a substantial refactor
belonging to the SIP-187
workstream; binding this SIP to it would block on unscoped work.
**Building on `QueryObjectFactory` directly.** It is invoked *inside*
`QueryContextFactory`. Entering below
that boundary forfeits the security gate, result caching, RLS,
post-processing, time offsets, event logging,
async compatibility, and row-limit clamping — all of which would need
reimplementing.
**A separate `/capabilities` endpoint.** Every field either already exists
on `Explorable` or is served by
`/compatible`. A dedicated route returning a few booleans and a short enum
list does not justify its own path,
cache key, permission, and OpenAPI entry.
**Convenience aliases (`/dataset/<id>/query`, `/semantic_view/<id>/query`) —
deferred.** The type is one path
segment, and `combined_list` returns `kind` per row, so any client that
listed datasources already knows the
type. Aliases add routes, OpenAPI entries, and a second URL for clients to
disagree about. If ergonomics is
the real need, accepting a UUID is the better answer —
`DatasourceDAO.get_datasource` already handles UUID
strings.
**`series_limit` / `group_limit` — deferred.** These are not a rename of
`limit` / `offset`, which map to
`QueryObject.row_limit` / `row_offset`. `series_limit` / `series_columns` /
`series_limit_metric` map to
`SemanticQuery.group_limit`, a top-N-per-series concept gated on
`SemanticViewFeature.GROUP_LIMIT`. Needs
three more wire fields and capability negotiation.
**Async execution — deferred.** `GLOBAL_ASYNC_QUERIES` returns 202 plus a
result URL. Supporting it requires
deciding whether to reuse `GET /api/v1/chart/data/<cache_key>` or add a
datasource-scoped equivalent. The
payload above is forward-compatible with either.
## Verification
- Existing `tests/unit_tests/mcp_service/**` tests for `get_table` /
`query_dataset` pass **unmodified** —
the proof the refactor is behaviour-preserving.
- Jinja bug, test-first: a virtual dataset using `{{ current_username() }}`
queried through all three surfaces
must agree. It does not today; that failing test is the regression guard.
- Permissions: after `superset init`, assert `can_query on Datasource`
exists, that a Gamma user with dataset
access can call `/query`, and that the **Public role does not** gain
either new route.
- `ILIKE` against a semantic view returns rows rather than raising.
- Ad-hoc metric against a semantic view → 400 naming the metric.
- Omitting `result_format` returns JSON — the default is asserted
explicitly, not assumed.
- Arrow: `result_format: "arrow"` returns
`application/vnd.apache.arrow.stream`, and the bytes round-trip
through `pyarrow.ipc.open_stream` to the same rows the JSON format
returns. A JSON request and an Arrow
request for the same query hit the same cache entry.
- Flag behaviour: `semantic_view` → 404 with `SEMANTIC_LAYERS` off; `table`
unaffected.
- Clamping: `row_limit` above the schema cap → 400; above `ROW_LIMIT` → 200
with the clamped count echoed.
- Caching: repeat query → `is_cached: true`; `"force": true` → `is_cached:
false`.
Integration tests model on the existing `test_get_column_values_*` cases in
`tests/integration_tests/datasource/api_tests.py`, which establish the
fixture and permission-mocking idiom
for this API class.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
