palashc opened a new pull request, #2597: URL: https://github.com/apache/phoenix/pull/2597
### What changes were proposed in this pull request? When an eventually consistent index consumer polls and finds no new CDC events, it now sleeps a fixed interval instead of walking HBase's `RETRY_BACKOFF` array. - New config `phoenix.index.cdc.consumer.idle.poll.interval.ms`, default `4000`, with per-consumer jitter so consumers don't poll in lock-step after a RegionServer restart. - `retryCount` is reset on any non-throwing iteration, so it now counts only consecutive exceptions. Error paths and the data-visibility retry loop keep their exponential backoff. ### Why are the changes needed? Idle backoff caps at a 200x multiplier — 40s at the default 200ms retry pause, reached after ~96s of inactivity. A write to a quiet region therefore waits up to `timestampBufferMs + 40s` (~45s) before the consumer looks for it, versus ~5.6s measured on a region under steady write load. That backoff is borrowed from RPC retry logic meant to relieve a failing server. An empty poll indicates nothing is wrong, so the escalation buys nothing and costs a 7x worse tail on exactly the low-traffic tables where a single write is most likely to be noticed. It also makes `cdcIndexUpdateLag` unusable as an SLI: idle consumers emit a 5s → 45s sawtooth, so percentiles describe the backoff schedule rather than index freshness. ### Does this PR introduce _any_ user-facing change? Yes. Worst-case index visibility on an idle region drops from `timestampBufferMs + 40s` to `timestampBufferMs + ~4.8s`. Steady-state latency under load is unchanged. Idle CDC query load rises from one poll per region per 40s to one per ~4s. The new config is optional and defaults to the behavior above. ### How was this patch tested? `IndexCDCConsumerLagIT` passes unchanged. No new test was added. `sleepWithLagSampling` emits a sample every `lagSampleIntervalMs` regardless of total sleep length, so the existing count-based assertion is invariant to sleep duration and cannot observe this change either way. A meaningful assertion needs the lag *value* rather than the count — `MetricHistogram` exposes only `add()` and `getCount()`, and cold-start samples floored at `now - consumerStartTime` would have to be excluded first. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Cursor -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
