Hello Michael Ho, Lars Volker, Tim Armstrong, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/13061
to look at the new patch set (#7).
Change subject: IMPALA-7665: Fix unwarranted query cancellation on statestore
restart
......................................................................
IMPALA-7665: Fix unwarranted query cancellation on statestore restart
Currently, if the statestore restarts and disseminates an inconsistent
view of cluster membership to the coordinators, then they might believe
that the backends no longer in the membership update are down and would
start canceling queries that are running or scheduled to run on those
allegedly failed backends. This patch adds a grace period after
statestore recovery/successful registration that give it enough time
to gather a consistent state of the cluster.
Testing:
- Added an e2e test.
- Did manual stress testing using concurrent_select.py with
statestore_subscriber_timeout_seconds set to 2 secs and
failed_backends_query_cancellation_grace_period_ms set to 5 seconds,
and the statestore being restarted every 15 seconds. To avoid other
effects of statestore restarts cropping up, I used a local catalog
(catalog v2) and ignored query errors caused due to scheduler having
an incomplete view of the cluster(no backends).
Change-Id: I30b68bd8bde4bf589d58d42d6f683afb166de959
---
M be/src/service/impala-server.cc
M be/src/statestore/statestore-subscriber.cc
M be/src/statestore/statestore-subscriber.h
M tests/custom_cluster/test_restart_services.py
4 files changed, 91 insertions(+), 2 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/61/13061/7
--
To view, visit http://gerrit.cloudera.org:8080/13061
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I30b68bd8bde4bf589d58d42d6f683afb166de959
Gerrit-Change-Number: 13061
Gerrit-PatchSet: 7
Gerrit-Owner: Bikramjeet Vig <bikramjeet....@cloudera.com>
Gerrit-Reviewer: Bikramjeet Vig <bikramjeet....@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com>
Gerrit-Reviewer: Lars Volker <l...@cloudera.com>
Gerrit-Reviewer: Michael Ho <k...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <tarmstr...@cloudera.com>
