echobeijingecho commented on issue #66401:
URL: https://github.com/apache/doris/issues/66401#issuecomment-5186147987
**Follow-up: the trigger is fully deterministic -- a "cold-query synchronous
chain" -- plus a config amplifier that explains why most deployments never see
this**
**1. Millisecond-level causality (audit log vs fe.log):** Round 1: business
query arrived 17:07:18.339 -> row-count executor opened the ugi=root
connections at 17:07:18.426 (**+87ms**). Round 2: query arrived 18:27:37.077 ->
creations at 18:27:37.192 (**+115ms**). Mechanism (4.0.7 code): planner ->
`ExternalTable.getRowCount()` -> row-count cache miss -> `getCachedRowCount`
runs `supplyAsync(() -> loadRowCount(...), executor)` and the query thread
**blocks waiting** (session var `fetchHiveRowCountSync` defaults to true). The
"background" executor is actually driven synchronously by every cold query; it
only leaves a trace when it also has to open connections.
**2. Why "cache expired" and "pool evicted" always coincide:** both
thresholds are the same number with the same idle-based semantics:
`external_cache_expire_time_seconds_after_access` (row-count expireAfterAccess)
= 300s in our fe.conf, and paimon `client-pool-cache.eviction-interval-ms` = 5
min. A single physical condition ("table idle > 5 min") triggers both: the
row-count load MUST run, and it MUST create connections. Not a coincidence of
independent events.
**3. Deterministic "middle idle window" rule (matches every data point of
the incident day):**
- idle < 5 min: pool warm -> safe
- - idle 5 min to tens of minutes: row-count cache + pool expired but schema
caches still warm -> the query thread itself needs no HMS connection -> the
FIRST creator is the row-count executor thread -> process-user connection.
(Both rounds: idle 12 min and 31 min.)
- - - idle >= hours (our nightly 4-hourly batches): everything expired ->
the query thread performs table load first (wrapped -> correct user) -> pool
warmed -> row-count borrows existing connections -> safe. All nightly batches
were safe, every day.
**4. Config amplifier -- likely why few deployments see this:**
`external_cache_expire_time_seconds_after_access` defaults to 86400 (1 day);
our cluster had tuned it to 300s. With the default, row-count cold misses on
the query path are rare, so the unwrapped creation path is almost never
exercised. A small value multiplies the trigger frequency by about 2 orders of
magnitude. This also gives an easy repro recipe: set it to <=300s and issue
queries with 6-30 min idle gaps.
**5. Remaining gap unchanged:** creation frames provably lack doAs (the
logged ugi reflects the creating thread's current user), while every Doris-side
paimon call site we can find IS wrapped and the authenticator's doAs is real
(`getUGI().doAs`). By elimination, the creations must be issued by paimon-core
lazy HMS access outside the wrapped frames (most likely during
`newScan().plan()` or lazy Table init) -- consistent with your assessment.
Instrumentation offer stands; with the repro recipe above, catching the
creation-site stack should be quick.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
