aho135 opened a new pull request, #20314:
URL: https://github.com/apache/druid/pull/20314
### Description
An extremely complex SQL query — for example, one with a very large `IN`
filter — can spend tens of seconds in the Calcite planning phase on the Broker
(the phase measured by the `sqlQuery/planningTimeMs` metric). A flood of such
queries can occupy all of the Broker's request threads and effectively freeze
the process, degrading the whole cluster. Today there is no way to bound the
time spent planning a query; the request thread stays busy until planning
finishes on its own.
This PR adds a wall-clock timeout on the SQL planning phase.
#### Added `druid.sql.planner.maxPlanningTimeMs`
A new planner config (default `0`, meaning no timeout) that bounds the
wall-clock time allowed for planning a SQL query. It is also overridable per
query via the `maxPlanningTimeMs` query context key, so operators can tune it
for specific workloads without a restart. When the value is not positive,
behavior is unchanged.
#### Wired a Calcite `CancelFlag` into the planner
`PlannerContext` now owns a per-query Calcite `CancelFlag`, and
`PlannerFactory#buildFrameworkConfig` exposes it through the framework
`Context`. Calcite's planner reads this flag from the context and calls
`checkCancel()` at its cancellation checkpoints (e.g. during Volcano/Hep rule
application), so tripping the flag makes in-progress planning abort promptly.
This is the mechanism Calcite provides for aborting long-running planning, so
it is preferred over trying to forcibly stop the thread.
#### Added the `SqlPlanningTimeout` watchdog
`DirectStatement#plan()` arms an `SqlPlanningTimeout` around the validate →
authorize → plan window (the same window measured by `planningTimeMs`). When
the deadline is exceeded, a shared, single daemon-threaded scheduler trips the
`CancelFlag` and interrupts the planning thread. The resulting failure is
surfaced as a `QueryTimeoutException` (HTTP 504) with a message that names the
query id and the effective budget. On close, the watchdog clears any interrupt
it set so it cannot leak to a pooled request thread. Planning runs on the
request thread as before; the watchdog only fires on timeout and is otherwise
cancelled with negligible overhead.
#### Behavioral notes
- The timeout covers the planning phase only. Query execution has its own
`timeout` mechanism.
- The `CancelFlag`/interrupt approach aborts the Druid-convention planning
path (Volcano/Hep rule application), which is where planning time for a large
`IN` on a datasource blows up. Phases that do not check the flag (and are not
interruptible) cannot be aborted mid-operation, but in practice the
pathological planning cost is in cancellable rule application.
#### Release note
You can now bound the time the Broker spends planning a SQL query with the
new `druid.sql.planner.maxPlanningTimeMs` runtime property (also settable per
query via the `maxPlanningTimeMs` context parameter). When planning exceeds the
configured budget, the query fails with an HTTP 504 `Query timeout` error
instead of tying up a Broker thread. The timeout is disabled by default (`0`).
<hr>
##### Key changed/added classes in this PR
* `SqlPlanningTimeout` (new)
* `DirectStatement`
* `PlannerConfig`
* `PlannerContext`
* `PlannerFactory`
<hr>
This PR has:
- [x] been self-reviewed.
- [x] added documentation for new or modified features or behaviors.
- [x] a release note entry in the PR description.
- [x] added Javadocs for most classes and all non-trivial methods. Linked
related entities via Javadoc links.
- [x] added comments explaining the "why" and the intent of the code
wherever would not be obvious for an unfamiliar reader.
- [x] added unit tests or modified existing tests to cover new code paths,
ensuring the threshold for code coverage is met.
- [x] been tested in a test Druid cluster.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
