Hello Tianyi Wang, Dimitris Tsirogiannis, Alex Behm, Bikramjeet Vig, I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/9123 to look at the new patch set (#11). Change subject: IMPALA-6437: separate AC/scheduler from catalog topic updates ...................................................................... IMPALA-6437: separate AC/scheduler from catalog topic updates This adds a set of "prioritized" statestore topics that are small but are important to deliver in a timely manner. These are delivered more frequently by a separate thread pool to reduce the window for stale admission control and scheduling information. The contract between statestore and subscriber is changed so that the statestore can send concurrent Update() RPCs for disjoint sets of topics. This required changes to the subscriber implementation, which assumed that only one Update RPC would arrive at a time. It also changes the locking in the statestore so that the prioritized update threads don't get stuck behind the catalog threads holding 'topic_lock_'. Specifically, it uses a reader-writer lock to protect modification of the set of topics and a reader-writer lock per topic to allow the topic data to be read by multiple threads concurrently. Added metrics to monitor the per-topic update interval. Testing: Ran core tests. Inspected metrics on Impala daemons, saw that membership and request queue processing times had more samples recorded than the catalog topic, reflecting the increased frequency. Ran under thread sanitizer, made sure no data races were reported in Statestore or StatestoreSubscriber. Change-Id: Ifc49c2d0f2a5bfad822545616b8c62b4b95dc210 --- M be/src/common/atomic.h M be/src/scheduling/admission-controller.cc M be/src/scheduling/admission-controller.h M be/src/scheduling/scheduler-test-util.cc M be/src/scheduling/scheduler.cc M be/src/scheduling/scheduler.h M be/src/service/impala-server.cc M be/src/statestore/statestore-subscriber.cc M be/src/statestore/statestore-subscriber.h M be/src/statestore/statestore.cc M be/src/statestore/statestore.h M common/thrift/metrics.json M tests/custom_cluster/test_admission_controller.py M tests/statestore/test_statestore.py M www/statestore_subscribers.tmpl M www/statestore_topics.tmpl 16 files changed, 836 insertions(+), 481 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/23/9123/11 -- To view, visit http://gerrit.cloudera.org:8080/9123 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ifc49c2d0f2a5bfad822545616b8c62b4b95dc210 Gerrit-Change-Number: 9123 Gerrit-PatchSet: 11 Gerrit-Owner: Tim Armstrong <tarmstr...@cloudera.com> Gerrit-Reviewer: Alex Behm <alex.b...@cloudera.com> Gerrit-Reviewer: Bikramjeet Vig <bikramjeet....@cloudera.com> Gerrit-Reviewer: Dimitris Tsirogiannis <dtsirogian...@cloudera.com> Gerrit-Reviewer: Tianyi Wang <tw...@cloudera.com> Gerrit-Reviewer: Tim Armstrong <tarmstr...@cloudera.com>