Ashu Pachauri created HBASE-18192:
-------------------------------------
Summary: Replication drops recovered queues on region server
shutdown
Key: HBASE-18192
URL: https://issues.apache.org/jira/browse/HBASE-18192
Project: HBase
Issue Type: Bug
Components: Replication
Affects Versions: 1.2.6, 1.3.1, 2.0.0, 1.4.0
Reporter: Ashu Pachauri
Assignee: Ashu Pachauri
Priority: Blocker
Fix For: 2.0.0, 1.4.0, 1.3.2, 1.2.7
When a recovered queue has only one active ReplicationWorkerThread, the
recovered queue is completely dropped on a region server shutdown. This will
happen in situation when
1. DefaultWALProvider is used.
2. RegionGroupingProvider provider is used but replication is stuck on one WAL
group for some reason (for example HBASE-18137)
3. All other replication workers have died due to unhandled exception, and the
only one finishes. This will cause the recovered queue to get deleted without a
regionserver shutdown. This can happen on deployments without fix for
HBASE-17381.
The problematic piece of code is:
{Code}
while (isWorkerActive()){
// The worker thread run loop...
}
if (replicationQueueInfo.isQueueRecovered()) {
// use synchronize to make sure one last thread will clean the queue
synchronized (workerThreads) {
Threads.sleep(100);// wait a short while for other worker thread to
fully exit
boolean allOtherTaskDone = true;
for (ReplicationSourceWorkerThread worker : workerThreads.values()) {
if (!worker.equals(this) && worker.isAlive()) {
allOtherTaskDone = false;
break;
}
}
if (allOtherTaskDone) {
manager.closeRecoveredQueue(this.source);
LOG.info("Finished recovering queue " + peerClusterZnode
+ " with the following stats: " + getStats());
}
}
{Code}
The conceptual issue is that isWorkerActive() tells whether a worker is
currently running or not and it's being used as a proxy for whether a worker
has finished it's work. But, in fact, "Should a worker should exit?" and "Has
completed it's work?" are two different questions.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
