[
https://issues.apache.org/jira/browse/HBASE-21358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16684364#comment-16684364
]
Tak Lon (Stephen) Wu edited comment on HBASE-21358 at 11/12/18 9:55 PM:
------------------------------------------------------------------------
@[~apurtell] may I try this one?
In addition, do you know if there is any way to test this problem on a live
cluster ? or would it be fine if I write a test to cover this case ?
updates, also I found that there is two situation the procedure cannot be found
from {{ProcedureCoordinator#getProcedure}} :
1. Running procedure completed but {{TakeSnapshotHandler#process}} is not yet
completed. here TakeSnapshotHandler will call {{completeSnapshot}} and then
{{handler.isFinished()} will get notified. (seems like there is a race
condition)
2. Failed procedure, {{TakeSnapshotHandler}} is never finished and it will
takes a while to be failed
I assumed the case 2) in case is what we're trying to fail fast, but I'm still
wondered if the race condition between client of
{{SnapshotManager#isSnapshotDone}} and {{TakeSnapshotHandler#process}} is the
originally thought of checking the completeness of {{*SnapshotHandler}}
periodically.
was (Author: taklwu):
@[~apurtell] may I try this one?
In addition, do you know if there is any way to test this problem on a live
cluster ? or would it be fine if I write a test to cover this case ?
updates, also I found that there is two situation the procedure cannot be found
from {{ProcedureCoordinator#getProcedure}} :
1. Running procedure completed but {{TakeSnapshotHandler#process}} is not yet
completed. here TakeSnapshotHandler will call {{completeSnapshot}} and then
{{handler.isFinished()} will get notified. (seems like there is a race
condition)
2. Failed procedure, {{TakeSnapshotHandler}} is never finished and it will
takes a while to be failed
I assumed the case 2) in case is what we're trying to fail fast, but I'm still
wondered if the race condition between client of
{{SnapshotManager#isSnapshotDone}} and {{TakeSnapshotHandler#process}} is the
originally thought of checking periodically.
> Snapshot procedure fails but SnapshotManager thinks it is still running
> -----------------------------------------------------------------------
>
> Key: HBASE-21358
> URL: https://issues.apache.org/jira/browse/HBASE-21358
> Project: HBase
> Issue Type: Bug
> Components: snapshots
> Affects Versions: 1.3.2
> Reporter: Andrew Purtell
> Priority: Major
>
> A snapshot procedure fails due to chaotic test action but the snapshot
> manager still thinks it is running. The test client spins needlessly checking
> for something that will never actually complete. We give up eventually but we
> could be failing this a lot faster.
> On the integration client we are checking and re-checking:
> 2018-10-20 01:06:11,718 DEBUG [ChaosMonkeyThread] client.HBaseAdmin: Getting
> current status of snapshot from master...
> 2018-10-20 01:06:11,719 DEBUG [ChaosMonkeyThread] client.HBaseAdmin: (#40)
> Sleeping: 8571ms while waiting for snapshot completion.
> This is what it looks like on the master side each time the client checks in:
> 2018-10-20 01:04:54,565 DEBUG
> [RpcServer.FifoWFPBQ.default.handler=29,queue=2,port=8100]
> master.MasterRpcServices: Checking to see if snapshot from request:{
> ss=IntegrationTestBigLinkedList-it-1539997289258
> table=IntegrationTestBigLinkedList type=FLUSH } is done
> 2018-10-20 01:04:54,565 DEBUG
> [RpcServer.FifoWFPBQ.default.handler=29,queue=2,port=8100]
> snapshot.SnapshotManager: Snapshoting '{
> ss=IntegrationTestBigLinkedList-it-1539997289258
> table=IntegrationTestBigLinkedList type=FLUSH }' is still in progress!
> There is no running procedure for the snapshot. The procedure has failed. The
> snapshot manager does not take any useful action afterward but believes the
> snapshot to still be in progress.
> I see related complaint from the hfile archiver task afterward, empty
> directories, failure to parse protobuf in descriptor files... Seems like
> there was junk in the filesystem left over from the failed snapshot. The
> master was soon restarted by chaos action, and now I don't see these
> complaints, so that partially complete snapshot may have been cleaned up.
> This is with 1.3.2, but patched to include the multithreaded hfile archiving
> improvements from later versions.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
