yashmayya opened a new pull request, #19804:
URL: https://github.com/apache/pinot/pull/19804
Phase 3 of IdSet support in the multi-stage engine (MSE). Phases 1 (#19757)
and 2 (#19759) are merged.
## Problem
- MSE rejects `IN_SUBQUERY(col, 'subquery')` with "No match found for
function signature IN_SUBQUERY".
- `IN_ID_SET(col, (SELECT IDSET(...) ...))` has worked since #19757, but
slowly. The planner joins the subquery result to the rows, so every row leaves
the leaf stage before the IdSet test runs.
- To filter by a large set, a client therefore runs two queries: one gets
the IdSet, and the other sends it back as a literal. Over HTTP, a literal can
have at most 20M chars, which is about 12M ids at 1% false positives.
## Changes
- Before it compiles a query, the broker runs each IdSet subquery as a query
of its own and puts the IdSet in its place. The planner and the servers then
see `IN_ID_SET(col, '<IdSet>')`, so the test runs in the leaf stage. Servers
and the wire format do not change. Two forms run first:
- `IN_SUBQUERY(expr, 'subquery')`, anywhere in the query: SELECT, WHERE,
HAVING, JOIN ON, nested queries, CTEs and set operations.
- `IN_ID_SET(expr, (subquery))`, when the subquery validates on its own
(i.e. is not correlated) and names no WITH item of the query. Other subqueries
run as part of the query, as before.
- Each subquery:
- runs on MSE with a new request id, once even if it appears more than
once.
- gets the time left until the query's deadline.
- inherits the query's options. `SET` statements in an `IN_SUBQUERY`
string override them, and `sqlOptionsMode` applies to them. `dropResults` is
not inherited.
- runs like any other query: it is checked for table access, quota and
throttling, it is listed as running, and the query event listener sees it.
- can be cancelled with the query's client query id. The query then fails.
- must return one STRING column and at most one row. No row, or NULL,
gives the empty IdSet: `= 1` is then false and `= 0` true for every row.
- fails the query if its result is partial, e.g. `numGroupsLimitReached`,
because the IdSet would miss ids.
- As in the single-stage engine, subqueries run before the query is checked
for table access and quota.
- EXPLAIN runs no subqueries. The plan uses the empty IdSet instead, and a
new `ID_SET_SUBQUERIES_NOT_RUN` column lists the subqueries as a JSON array.
- `IN_SUBQUERY` is registered in the MSE operator table. This lets paths
that only compile queries accept it, such as the controller routing a query to
a broker. As in the single-stage engine, those paths do not see the tables
named in the subquery string.
- New `QueryEnvironment.validate(SqlNodeAndOptions)`: it validates without
converting or optimizing. The broker uses it to check whether a scalar subquery
stands on its own.
- New `RequestUtils.setOptions(SqlNodeAndOptions, Map)` overload: it takes
the request options as a map.
## Behavior changes
- The `singleStageQueriesInvalidMultiStage` meter no longer counts
single-stage `IN_SUBQUERY` queries.
- EXPLAIN of an uncorrelated `IN_ID_SET(col, (subquery))` now shows the
IdSet test in the leaf stage instead of a join, plus the new column. #19757 is
in no release yet.
## Scale
In the integration test, a STRING IdSet sized for 20M ids (a 32M-char
literal) filters the table in 0.5 s through `IN_SUBQUERY`. The set never
reaches the client or the parser, so the 20M-char HTTP limit does not apply.
## Tests
- `IdSetSubqueryRewriterTest`: every SQL position, nested queries, name
variants, the text of scalar subqueries, correlated subqueries, WITH names,
repeated subqueries, EXPLAIN and invalid calls.
- `MultiStageIdSetSubqueryTest`: inherited options, identity and headers;
time left; result checks, including partial results; errors; access checks on
the subquery's tables; query events; and EXPLAIN.
- `QueryCompilationTest`: queries with `IN_SUBQUERY` compile.
- `IdSetIntegrationTest`: `IN_SUBQUERY` in both engines, outside filters,
and nested; scalar subqueries, both uncorrelated and correlated; a partial
subquery result; and a 20M-id set.
- `BaseClusterIntegrationTestSet`: the `IN_ID_SET` and `IN_SUBQUERY`
hard-coded queries now run on both engines. The hybrid, realtime, logical
table, cursor and MSE suites pass locally.
- `JmxMetricsIntegrationTest`: now expects 2 invalid queries instead of 3.
## Next
- Optional phase 4, measure first: one shared Bloom filter per query and
server while a set is built.
- Single-stage `IN_SUBQUERY` throws an NPE when its subquery runs with null
handling and matches no rows. A separate fix can also match the result checks
above.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
