wenzhenghu opened a new pull request, #68727:
URL: https://github.com/apache/doris/pull/68727
### What problem does this PR solve?
Issue Number: N/A
Related PR: #68694
Problem Summary:
A query that stalls or fails during FE planning can leave little evidence
about its active phase, optimizer job, lock wait, or external metadata
operation. This PR adds bounded Nereids planning diagnostics and numeric
audit/Profile timing details, allowing operators to identify where planning
time is spent, including before a query reaches BE execution.
#### Planning progress and failure diagnostics
- A scoped context records each `planWithLock` pass and restores the outer
diagnostic context on normal returns, failures, cache paths, EXPLAIN, and
nested planning. Internal MV planning can inherit the outer query identity and
summary ownership across different `ConnectContext` instances.
- The existing connection timeout checker emits rate-limited WARN events for
registered queries that are still planning. Events identify the active nested
pass, phase, optimizer job, operation/target, elapsed time, timeout budgets,
and held locks. `root_pass_id` and `root_elapsed_ms` retain the outer pass's
identity and elapsed time. The checker reads published diagnostic state without
acquiring the planner's table-lock monitor; no additional monitoring thread is
created.
- Slow or failed passes emit a completion summary with phase status, elapsed
time, and the failed step. Fast successful passes produce no diagnostic log
events.
- Read-lock wait and hold times are tracked separately. Table resolution,
schema, snapshot, and partition metadata operations include qualified targets.
Targets are resolved only during an active diagnostic scope, once on the
planning thread, and captured for background checks.
- Completed operation events are deferred until the outermost planning
resources are released. Each pass buffers at most 16 events; child events are
merged into the bounded parent buffer, with omitted events counted. Nested
passes share the running-report rate limit and the outer elapsed-time threshold.
Each diagnostic JSON event carries an explicit `query_id` and excludes raw
SQL. Event emission temporarily binds that identity in Log4j MDC and restores
the caller's previous value, so an existing MDC-aware layout cannot attribute
the event to another query. This PR is independently based on master: it does
not depend on the `QueryLogContext` class or `sys_log_enable_query_id` from
#68694, and it changes no BE code or common log-prefix configuration. The two
PRs touch `NereidsPlanner` and `ConnectContext`; when merging both, retain the
planner diagnostic scope together with the query-log scope and preserve both
timeout-checker hooks.
The following FE settings are dynamically configurable:
| Configuration | Default | Behavior |
| --- | ---: | --- |
| `nereids_planning_log_threshold_ms` | `5000` | Slow-pass threshold;
non-positive values disable slow-pass reports. |
| `nereids_planning_operation_log_threshold_ms` | `1000` | Slow
lock/metadata operation threshold; non-positive values disable slow-operation
reports. |
| `nereids_planning_log_interval_ms` | `30000` | Minimum interval between
running reports; nested passes share it, and values below 1000 use 1000 ms. |
Failed passes and failed operations remain eligible for reporting when slow
thresholds are disabled; the event-buffer limit still applies. Running reports
follow the existing connection-checker cadence. Metadata instrumentation covers
the common FE entry points rather than every connector-internal operation.
Existing lock ordering, lock timeouts, and exception propagation are preserved.
#### Audit and Profile details
- Audit `plan_times_ms` adds `preload_external_metadata`, `pre_rewrite_mv`,
and flattened numeric `planning_*` entries for phases, pass counts, and
failures. Its type remains `MAP<STRING, INT>`; new phase totals use `-1` when
not run, preserve elapsed time on failure, and saturate at the INT limit.
- Profile adds `Nereids Planning Details`, accumulating repeated
CTAS/retry/nested passes and recording the last pass's `completed`, `failed`,
or `not_run` phase states. Audit timing collection also works with query
Profile disabled.
- Existing timing fields keep their meaning, including the broad legacy
`translate` field. Nested phase/pass timings are inclusive and must not be
added as disjoint wall-clock durations.
### Release note
FE reports slow or failed Nereids planning and slow lock/metadata
operations, with bounded deferred events and configurable thresholds. Audit
`plan_times_ms` gains numeric planning keys, and Profile gains accumulated
planning details. The audit map type remains unchanged. Disabling slow-report
thresholds does not suppress failure diagnostics.
### Check List (For Author)
- Test:
- Post-split FE validation: **103/103 tests across 10 classes passed**,
with no failures/errors/skips, through `./run-fe-ut.sh --run ...`. Includes all
27 migrated diagnostic tests plus 2 new identity cases, as well as
SummaryProfile, StatementContext/MVCC, NereidsPlanner, external
schema/row-count, and nested MV suites.
- FE common/core class outputs were removed before compilation. The
branch builds and tests without the query-prefix implementation or its
configuration from #68694.
- FE Checkstyle (0 violations), source whitespace checks, and static
cross-review passed. The BE diff is empty. The generated audit result retains
its original trailing separator line.
- The audit regression suite and its runner-generated expected output
are unchanged from the previously validated combined branch (#68694,
`a07bed545df`). They cover successful/failed audit records with Profile
disabled. This regression and live-instance checks were not rerun after the
split; the current split was validated with the targeted FE suites above.
- Full suites, refreshed community coverage, multi-node fault injection,
and production performance benchmarks were not run locally.
- Behavior changed:
- [x] Yes. FE adds slow/failure planning events and audit/Profile
planning fields. All three diagnostic settings are dynamic.
- Does this need documentation?
- [x] Yes. Planning events, thresholds, audit/Profile fields, and
inclusive timing semantics; documentation PR pending.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
