Hello Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24975
to look at the new patch set (#2).
Change subject: IMPALA-15451: Make hierarchical event executors event-driven
......................................................................
IMPALA-15451: Make hierarchical event executors event-driven
When hierarchical event processing is enabled, catalogd consumes
noticeable CPU on an idle cluster. Each DbEventExecutor and
TableEventExecutor thread was scheduled with scheduleAtFixedRate()
every 10 ms, regardless of whether it had any work. With the default
configuration of 5 db executors, each having 5 table executors, 30
threads wake up 100 times per second each, i.e. ~3000 wakeups per
second. Although each pass finds no events and does little work, the
cost of scheduling and context switches adds up.
This patch replaces the fixed-rate polling with event-driven
wakeups. Each executor thread now runs a process loop and blocks in
a new EventExecutorWaiter when it has nothing to do. It is woken up
in the following cases:
- DbEventExecutor is signalled when:
1. An event is enqueued to one of its DbProcessors.
2. All the TableProcessors waiting on a DbBarrierEvent have
reached it.
- TableEventExecutor is signalled when:
1. An event is enqueued to one of its TableProcessors.
2. A DbBarrierEvent its TableProcessor waits on is processed.
3. The pseudo drop table event of a RenameTableBarrierEvent is
processed, making its pseudo create table event processable.
EventExecutorWaiter records a signal raised while the executor
thread is busy, so the next wait returns immediately and no wakeup
is lost.
TableEventExecutor always waits until it is signalled, since every
event that cannot be processed yet is signalled once it becomes
processable. DbEventExecutor wakes up every 1 second while it has
DbProcessors, to remove the TableProcessors that have been idle for
min_event_processor_idle_ms. Otherwise, it also waits until it is
signalled.
Since events are processed as soon as they are enqueued or become
processable, instead of waiting up to 10 ms for the next tick, this
also removes the scheduling latency from event processing.
Testing:
- Ran existing tests.
- Verified with top -H that DbEventExecutor and TableEventExecutor
threads do not consume CPU on an idle catalogd.
Assisted-by: Claude Opus 4.8 (Claude Code)
Change-Id: I97107e99ead7907023a936d0017f33fcdba23185
---
M fe/src/main/java/org/apache/impala/catalog/events/DbBarrierEvent.java
M fe/src/main/java/org/apache/impala/catalog/events/DbEventExecutor.java
M fe/src/main/java/org/apache/impala/catalog/events/EventExecutorService.java
A fe/src/main/java/org/apache/impala/catalog/events/EventExecutorWaiter.java
M
fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java
M fe/src/main/java/org/apache/impala/catalog/events/RenameTableBarrierEvent.java
M fe/src/main/java/org/apache/impala/catalog/events/TableEventExecutor.java
M
fe/src/test/java/org/apache/impala/catalog/events/EventExecutorServiceTest.java
8 files changed, 269 insertions(+), 57 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/75/24975/2
--
To view, visit http://gerrit.cloudera.org:8080/24975
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I97107e99ead7907023a936d0017f33fcdba23185
Gerrit-Change-Number: 24975
Gerrit-PatchSet: 2
Gerrit-Owner: Anonymous Coward <[email protected]>
Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
