[
https://issues.apache.org/jira/browse/HBASE-3890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jeffrey Zhong updated HBASE-3890:
---------------------------------
Attachment: hbase-3890.patch
The fix is trivial. In the patch, I also increased the log level for double
deletion case because the logs are important when debugging issues like this.
It won't cause overwhelming log info as double delete doesn't happen frequently.
Existing implementation is using a single thread to schedule split log task. If
it shows two scheduled task, it must have two files for recovery. While there
could be a potential hole when updating the task complete status as follows:
1) There are left over zk nodes when HMaster starts
2) function SplitLogManager#lookForOrphans picks left paths from zk and try to
recover them from previous runs
3)function createTaskIfAbsent inside SplitLogManager#enqueueSplitTask tries to
attach batch instance to the orphan task created by LookForOrphans as below:
{code}
if (oldtask.status == IN_PROGRESS) {
oldtask.batch = batch;
batch.installed++;
LOG.debug("Previously orphan task " + path + " is now being waited
upon");
return null;
}
{code}
Since Task.batch is NOT declared as volatile, when the orphan task completes
and setDone potentially doesn't see the task.batch is set. The following lines
of code are skipped. In the end, we have the inconsistant behavior documented
in the JIRA.
{code}
if (task.batch != null) {
synchronized (task.batch) {
if (status == SUCCESS) {
task.batch.done++;
} else {
task.batch.error++;
}
task.batch.notify();
}
}
{code}
Based on the above analysis, the bug should happen very rarely.
Lars,
Could you pls confirm the issue isn't easy to repro otherwise there are other
reasons?
Thanks,
-Jeffrey
> Scheduled tasks in distributed log splitting not in sync with ZK
> ----------------------------------------------------------------
>
> Key: HBASE-3890
> URL: https://issues.apache.org/jira/browse/HBASE-3890
> Project: HBase
> Issue Type: Bug
> Components: regionserver
> Affects Versions: 0.92.0
> Reporter: Lars George
> Fix For: 0.96.0
>
> Attachments: hbase-3890.patch
>
>
> This is in continuation to HBASE-3889:
> Note that there must be more slightly off here. Although the splitlogs znode
> is now empty the master is still stuck here:
> {noformat}
> Doing distributed log split in
> hdfs://localhost:8020/hbase/.logs/10.0.0.65,60020,1305406356765
> - Waiting for distributed tasks to finish. scheduled=2 done=1 error=0 4380s
> Master startup
> - Splitting logs after master startup 4388s
> {noformat}
> There seems to be an issue with what is in ZK and what the TaskBatch holds.
> In my case it could be related to the fact that the task was already in ZK
> after many faulty restarts because of the NPE. Maybe it was added once (since
> that is keyed by path, and that is unique on my machine), but the reference
> count upped twice? Now that the real one is done, the done counter has been
> increased, but will never match the scheduled.
> The code could also check if ZK is actually depleted, and therefore treat the
> scheduled task as bogus? This of course only treats the symptom, not the root
> cause of this condition.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
