[ https://issues.apache.org/jira/browse/KUDU-3098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
YifanZhang updated KUDU-3098: ----------------------------- Summary: leadership change during tablet_copy process may lead to an isolate replica (was: leader change during 'add_peer' process for a tablet may lead to an isolate replica) > leadership change during tablet_copy process may lead to an isolate replica > --------------------------------------------------------------------------- > > Key: KUDU-3098 > URL: https://issues.apache.org/jira/browse/KUDU-3098 > Project: Kudu > Issue Type: Bug > Components: consensus, master > Affects Versions: 1.10.1 > Reporter: YifanZhang > Priority: Major > > Lately we found some tablets in a cluster with a very large > "time_since_last_leader_heartbeat" metric, they are LEARNER/NON_VOTER and > seems couldn't become VOTER for a long time. > These replicas created during the rebalance/tablet_copy process. After > beginning a new copy session from leader to the new added NON_VOTER peer, > leadership changed, old leader aborted uncommited CHANGE_CONFIG_OP operation. > Finally the tablet_copy session ended but new leader knew nothing about the > new peer. > Master didn't delete this new added replica because it has a larger > opid_index than the latest reported committed config. See the comments in > CatalogManager::ProcessTabletReport > {code:java} > // 5. Tombstone a replica that is no longer part of the Raft config (and > // not already tombstoned or deleted outright). > // > // If the report includes a committed raft config, we only tombstone if > // the opid_index is strictly less than the latest reported committed > // config. This prevents us from spuriously deleting replicas that have > // just been added to the committed config and are in the process of copying. > {code} > Maybe we shouldn't use opid_index to determine if replicas are in the process > of copying. -- This message was sent by Atlassian Jira (v8.3.4#803005)