Hello David Ribeiro Alves, Mike Percy, I'd like you to do a code review. Please visit
http://gerrit.cloudera.org:8080/6443 to review the following change. Change subject: KUDU-1330: Add a tool to unsafely recover from loss of majority replicas ...................................................................... KUDU-1330: Add a tool to unsafely recover from loss of majority replicas This patch adds an API to allow unsafe config change via an external recovery tool 'kudu remote_replica replace_config'. This tool lets us replace a 3-node config on a tablet server with a 1-node config. This is particularly useful when we have 2 out of 3 replicas down and we want to bring the tablet back to operational state. We can use this tool to force a new config on the surviving node providing all the details of the new config from the tool. As a result of the forced config change, the automatic leader election kicks in via raft mechanisms and the re-replication is triggered from master to bring the replica count back upto 3-node config. The lonely survivor being operated by the tool tends to become the leader in new config in majority of the situations because: a) The tool acts as a fake leader and generates the consensus update with a bumped up term. The surviving node accepts the request and holds a pre-election upon recognizing the leader heartbeat failure. This results in the surviving node electing himself as the leader and once the new config update is reported to master, master can re-replicate the tablet to other healthy nodes in the cluster bringing the tablet back to online state. b) Assumption is that, the dead nodes are not coming back during this recovery, hence the leader elected in step a) will still be the leader when we see the replication factor restored back to 3. Also, the ReplaceConfig() API adds a flag to bypass the 'allow-max-one-pending-config-change' rule to append another change_config op while there is one pending on the log. This patch is a first in series for unsafe config changes, and assumes that the dead servers are not coming back while the new config change is taking effect. Tests associated with this patch: - Unsafe config change when there is one follower survivor in the cluster. - Unsafe config change when there is one leader survivor in the cluster. - Unsafe config change when the unsafe config contains 2 nodes. - Unsafe config change on a 5-replica config with 2 nodes in the new config. - Unsafe config change when there is a pending config on a surviving leader. - Unsafe config change when there is a pending config on a surviving leader. - Unsafe config change when there are back to back pending configs on logs. TODO: 1) Test exercising all the error cases in the UnsafeChangeConfig API. 2) Test the UnsafeChangeConfig RPC directly without going via external tool. Change-Id: I908d8c981df74d56dbd034e72001d379fb314700 ver 2 Change-Id: Icdf7b962d404ab4a9e838695d60ac2cd9cf3df1a --- M src/kudu/consensus/consensus.h M src/kudu/consensus/consensus.proto M src/kudu/consensus/consensus_queue.cc M src/kudu/consensus/metadata.proto M src/kudu/consensus/raft_consensus.cc M src/kudu/consensus/raft_consensus.h M src/kudu/consensus/raft_consensus_state.cc M src/kudu/integration-tests/cluster_itest_util.cc M src/kudu/integration-tests/cluster_itest_util.h M src/kudu/integration-tests/raft_consensus-itest.cc M src/kudu/tools/kudu-admin-test.cc M src/kudu/tools/kudu-tool-test.cc M src/kudu/tools/tool_action_common.cc M src/kudu/tools/tool_action_common.h M src/kudu/tools/tool_action_local_replica.cc M src/kudu/tools/tool_action_remote_replica.cc M src/kudu/tserver/tablet_service.cc M src/kudu/tserver/tablet_service.h 18 files changed, 1,149 insertions(+), 151 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/43/6443/1 -- To view, visit http://gerrit.cloudera.org:8080/6443 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: newchange Gerrit-Change-Id: Icdf7b962d404ab4a9e838695d60ac2cd9cf3df1a Gerrit-PatchSet: 1 Gerrit-Project: kudu Gerrit-Branch: master Gerrit-Owner: Dinesh Bhat <din...@cloudera.com> Gerrit-Reviewer: David Ribeiro Alves <dral...@apache.org> Gerrit-Reviewer: Mike Percy <mpe...@apache.org>