[jira] [Updated] (HBASE-8127) Region of a disabling or disabled table could be stuck in transition state when RS dies during Master initialization
[
https://issues.apache.org/jira/browse/HBASE-8127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lars Hofhansl updated HBASE-8127:
-
Fix Version/s: (was: 0.94.8)
0.94.7
Thanks Ram. Will commit tomorrow.
(This is fixed already in 0.95+ I assume)
> Region of a disabling or disabled table could be stuck in transition state
> when RS dies during Master initialization
>
>
> Key: HBASE-8127
> URL: https://issues.apache.org/jira/browse/HBASE-8127
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.94.5
>Reporter: Jeffrey Zhong
>Assignee: rajeshbabu
> Fix For: 0.94.7
>
> Attachments: HBASE-8127_94_2.patch, HBASE-8127_94_3.patch,
> HBASE-8127_94_4.patch, HBASE-8127_feedback.patch, HBASE-8127.patch,
> hbase-8127_v1.patch, reproduce-hang.patch
>
>
> The issue happens when a RS dies during a master starts up. After the RS
> reports open to the new master instance and dies immediately thereafter, the
> RITs of disabling tables(or disabled table) on the died RS will be in RIT
> state forever.
> I attached a patch to simulate the situation and you can run the following
> command to reproduce the issue:
> {code}mvn test -PlocalTests
> -Dtest=TestMasterFailover#testMasterFailoverWithMockedRITOnDeadRS{code}
> Basically, we skip regions of a dead server inside
> AM.processDeadServersAndRecoverLostRegions as the following code and relies
> on SSH to process those skipped regions:
> {code}
> for (Pair deadRegion : deadServer.getValue()) {
> nodes.remove(deadRegion.getFirst().getEncodedName());
> }
> {code}
> While in SSH, we skip regions of disabling(or disabled table) again by
> function processDeadRegion. Finally comes to the issue that RITs of
> disabling(or disabled table) stuck there forever.
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8127) Region of a disabling or disabled table could be stuck in transition state when RS dies during Master initialization
[
https://issues.apache.org/jira/browse/HBASE-8127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lars Hofhansl updated HBASE-8127:
-
Fix Version/s: (was: 0.94.7)
0.94.8
> Region of a disabling or disabled table could be stuck in transition state
> when RS dies during Master initialization
>
>
> Key: HBASE-8127
> URL: https://issues.apache.org/jira/browse/HBASE-8127
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.94.5
>Reporter: Jeffrey Zhong
>Assignee: rajeshbabu
> Fix For: 0.94.8
>
> Attachments: HBASE-8127_94_2.patch, HBASE-8127_94_3.patch,
> HBASE-8127_94_4.patch, HBASE-8127_feedback.patch, HBASE-8127.patch,
> hbase-8127_v1.patch, reproduce-hang.patch
>
>
> The issue happens when a RS dies during a master starts up. After the RS
> reports open to the new master instance and dies immediately thereafter, the
> RITs of disabling tables(or disabled table) on the died RS will be in RIT
> state forever.
> I attached a patch to simulate the situation and you can run the following
> command to reproduce the issue:
> {code}mvn test -PlocalTests
> -Dtest=TestMasterFailover#testMasterFailoverWithMockedRITOnDeadRS{code}
> Basically, we skip regions of a dead server inside
> AM.processDeadServersAndRecoverLostRegions as the following code and relies
> on SSH to process those skipped regions:
> {code}
> for (Pair deadRegion : deadServer.getValue()) {
> nodes.remove(deadRegion.getFirst().getEncodedName());
> }
> {code}
> While in SSH, we skip regions of disabling(or disabled table) again by
> function processDeadRegion. Finally comes to the issue that RITs of
> disabling(or disabled table) stuck there forever.
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8127) Region of a disabling or disabled table could be stuck in transition state when RS dies during Master initialization
[
https://issues.apache.org/jira/browse/HBASE-8127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
rajeshbabu updated HBASE-8127:
--
Attachment: HBASE-8127_94_4.patch
Extracted duplicate code into a method. Functionality wise no change.
> Region of a disabling or disabled table could be stuck in transition state
> when RS dies during Master initialization
>
>
> Key: HBASE-8127
> URL: https://issues.apache.org/jira/browse/HBASE-8127
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.94.5
>Reporter: Jeffrey Zhong
>Assignee: rajeshbabu
> Fix For: 0.94.7
>
> Attachments: HBASE-8127_94_2.patch, HBASE-8127_94_3.patch,
> HBASE-8127_94_4.patch, HBASE-8127_feedback.patch, HBASE-8127.patch,
> hbase-8127_v1.patch, reproduce-hang.patch
>
>
> The issue happens when a RS dies during a master starts up. After the RS
> reports open to the new master instance and dies immediately thereafter, the
> RITs of disabling tables(or disabled table) on the died RS will be in RIT
> state forever.
> I attached a patch to simulate the situation and you can run the following
> command to reproduce the issue:
> {code}mvn test -PlocalTests
> -Dtest=TestMasterFailover#testMasterFailoverWithMockedRITOnDeadRS{code}
> Basically, we skip regions of a dead server inside
> AM.processDeadServersAndRecoverLostRegions as the following code and relies
> on SSH to process those skipped regions:
> {code}
> for (Pair deadRegion : deadServer.getValue()) {
> nodes.remove(deadRegion.getFirst().getEncodedName());
> }
> {code}
> While in SSH, we skip regions of disabling(or disabled table) again by
> function processDeadRegion. Finally comes to the issue that RITs of
> disabling(or disabled table) stuck there forever.
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8127) Region of a disabling or disabled table could be stuck in transition state when RS dies during Master initialization
[
https://issues.apache.org/jira/browse/HBASE-8127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
rajeshbabu updated HBASE-8127:
--
Attachment: HBASE-8127_94_3.patch
Patch Addressing Jeffrey's comments. Please review.
> Region of a disabling or disabled table could be stuck in transition state
> when RS dies during Master initialization
>
>
> Key: HBASE-8127
> URL: https://issues.apache.org/jira/browse/HBASE-8127
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.94.5
>Reporter: Jeffrey Zhong
>Assignee: rajeshbabu
> Fix For: 0.94.7
>
> Attachments: HBASE-8127_94_2.patch, HBASE-8127_94_3.patch,
> HBASE-8127_feedback.patch, HBASE-8127.patch, hbase-8127_v1.patch,
> reproduce-hang.patch
>
>
> The issue happens when a RS dies during a master starts up. After the RS
> reports open to the new master instance and dies immediately thereafter, the
> RITs of disabling tables(or disabled table) on the died RS will be in RIT
> state forever.
> I attached a patch to simulate the situation and you can run the following
> command to reproduce the issue:
> {code}mvn test -PlocalTests
> -Dtest=TestMasterFailover#testMasterFailoverWithMockedRITOnDeadRS{code}
> Basically, we skip regions of a dead server inside
> AM.processDeadServersAndRecoverLostRegions as the following code and relies
> on SSH to process those skipped regions:
> {code}
> for (Pair deadRegion : deadServer.getValue()) {
> nodes.remove(deadRegion.getFirst().getEncodedName());
> }
> {code}
> While in SSH, we skip regions of disabling(or disabled table) again by
> function processDeadRegion. Finally comes to the issue that RITs of
> disabling(or disabled table) stuck there forever.
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8127) Region of a disabling or disabled table could be stuck in transition state when RS dies during Master initialization
[
https://issues.apache.org/jira/browse/HBASE-8127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
rajeshbabu updated HBASE-8127:
--
Attachment: HBASE-8127_94_2.patch
Patch with test cases.
1)TestRSKilledWhenMasterInitializing.testMasterFailoverWhenDisablingTableRegionsInRITOnDeadRS
reproduces exact problem of this JIRA.
2)TestAssignmentManager.testSSHWhenDisablingTableRegionsInOpeningState
reproduces the problem with Opening regions of disabling table(some times there
may be RIT hang or some times unnecessary assignment of partially disabled
table).
Please review.
> Region of a disabling or disabled table could be stuck in transition state
> when RS dies during Master initialization
>
>
> Key: HBASE-8127
> URL: https://issues.apache.org/jira/browse/HBASE-8127
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.94.5
>Reporter: Jeffrey Zhong
>Assignee: rajeshbabu
> Fix For: 0.94.7
>
> Attachments: HBASE-8127_94_2.patch, HBASE-8127_feedback.patch,
> HBASE-8127.patch, hbase-8127_v1.patch, reproduce-hang.patch
>
>
> The issue happens when a RS dies during a master starts up. After the RS
> reports open to the new master instance and dies immediately thereafter, the
> RITs of disabling tables(or disabled table) on the died RS will be in RIT
> state forever.
> I attached a patch to simulate the situation and you can run the following
> command to reproduce the issue:
> {code}mvn test -PlocalTests
> -Dtest=TestMasterFailover#testMasterFailoverWithMockedRITOnDeadRS{code}
> Basically, we skip regions of a dead server inside
> AM.processDeadServersAndRecoverLostRegions as the following code and relies
> on SSH to process those skipped regions:
> {code}
> for (Pair deadRegion : deadServer.getValue()) {
> nodes.remove(deadRegion.getFirst().getEncodedName());
> }
> {code}
> While in SSH, we skip regions of disabling(or disabled table) again by
> function processDeadRegion. Finally comes to the issue that RITs of
> disabling(or disabled table) stuck there forever.
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8127) Region of a disabling or disabled table could be stuck in transition state when RS dies during Master initialization
[
https://issues.apache.org/jira/browse/HBASE-8127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
rajeshbabu updated HBASE-8127:
--
Assignee: rajeshbabu (was: Jeffrey Zhong)
> Region of a disabling or disabled table could be stuck in transition state
> when RS dies during Master initialization
>
>
> Key: HBASE-8127
> URL: https://issues.apache.org/jira/browse/HBASE-8127
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.94.5
>Reporter: Jeffrey Zhong
>Assignee: rajeshbabu
> Fix For: 0.94.7
>
> Attachments: HBASE-8127_feedback.patch, HBASE-8127.patch,
> hbase-8127_v1.patch, reproduce-hang.patch
>
>
> The issue happens when a RS dies during a master starts up. After the RS
> reports open to the new master instance and dies immediately thereafter, the
> RITs of disabling tables(or disabled table) on the died RS will be in RIT
> state forever.
> I attached a patch to simulate the situation and you can run the following
> command to reproduce the issue:
> {code}mvn test -PlocalTests
> -Dtest=TestMasterFailover#testMasterFailoverWithMockedRITOnDeadRS{code}
> Basically, we skip regions of a dead server inside
> AM.processDeadServersAndRecoverLostRegions as the following code and relies
> on SSH to process those skipped regions:
> {code}
> for (Pair deadRegion : deadServer.getValue()) {
> nodes.remove(deadRegion.getFirst().getEncodedName());
> }
> {code}
> While in SSH, we skip regions of disabling(or disabled table) again by
> function processDeadRegion. Finally comes to the issue that RITs of
> disabling(or disabled table) stuck there forever.
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8127) Region of a disabling or disabled table could be stuck in transition state when RS dies during Master initialization
[
https://issues.apache.org/jira/browse/HBASE-8127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jeffrey Zhong updated HBASE-8127:
-
Attachment: HBASE-8127_feedback.patch
{quote}
Looking at reproduce-hang.patch, can we modify TestMasterFailover so that this
scenario can be reproduced without changing master code ?
{quote}
The reproduce-hang patch is just for description purpose only. The issue could
happen in both meta RS or non-meta RS scenarios.
{quote}
We are removing deadserver nodes to avoid double assignments from SSH and
master startup. Even if we remove the code as in patch, same problem can come
with DISABLING.IMO we need to handle in SSH case only.
{quote}
The reproduce-hang.patch is already set the table as
DISABLING(zktable.setDisablingTable(Bytes.toString(disabledTable));) and the
original patch can work for both disabled and disabling case.
{quote}
double assignments concern
{quote}
Assuming the check removed in the original patch does to prevent double
assignment or other issues while right after the check there is another RS(with
similar state as the RS skipped by the check) just died. Therefore, possible
double assignment still could happen. IMHO, the effectiveness of the check is
questionable.
{quote}
s there a way to differentiate a deadServer that was really dead when the
master came up and the one that went down before the master could finish
initialization.
Now when the already dead Servers are added to the deadServers list we get into
this problem.
{quote}
It's possible. It's better not to introduce a new type of dead servers though.
{quote}
we were ensuring the region was removed from RIT but the znode was not getting
delted in the case where the znode for Disabled table was in OPENING state. But
in a normal case i don't think this could happen. Need to check this.
{quote}
I'm thinking if can change the test case to set the disabledTable to disabling
because I'm not sure if a RIT in opening state could exist for a disabled table.
{quote}
Rough patch for 94. I will add some more tests and upload again.
{quote}
I saw that you address the issue in SSH and I'm fine with it. I'm afraid to do
so before is due to its higher impact. I didn't see you handle disabled case so
the test case will fail again if I set disabledTable to disabled in the test
case. In addition, RITs in the test case exists not only for died RS but on the
live RS as well. Once a while I saw the opening RIT of the disabled table
either got assigned or stuck there so I think it's better to add some safe
guards when we deal with RITs during a master node starts up. I put my
feedbacks in the feedback patch for your references.
> Region of a disabling or disabled table could be stuck in transition state
> when RS dies during Master initialization
>
>
> Key: HBASE-8127
> URL: https://issues.apache.org/jira/browse/HBASE-8127
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.94.5
>Reporter: Jeffrey Zhong
>Assignee: Jeffrey Zhong
> Fix For: 0.94.7
>
> Attachments: HBASE-8127_feedback.patch, HBASE-8127.patch,
> hbase-8127_v1.patch, reproduce-hang.patch
>
>
> The issue happens when a RS dies during a master starts up. After the RS
> reports open to the new master instance and dies immediately thereafter, the
> RITs of disabling tables(or disabled table) on the died RS will be in RIT
> state forever.
> I attached a patch to simulate the situation and you can run the following
> command to reproduce the issue:
> {code}mvn test -PlocalTests
> -Dtest=TestMasterFailover#testMasterFailoverWithMockedRITOnDeadRS{code}
> Basically, we skip regions of a dead server inside
> AM.processDeadServersAndRecoverLostRegions as the following code and relies
> on SSH to process those skipped regions:
> {code}
> for (Pair deadRegion : deadServer.getValue()) {
> nodes.remove(deadRegion.getFirst().getEncodedName());
> }
> {code}
> While in SSH, we skip regions of disabling(or disabled table) again by
> function processDeadRegion. Finally comes to the issue that RITs of
> disabling(or disabled table) stuck there forever.
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8127) Region of a disabling or disabled table could be stuck in transition state when RS dies during Master initialization
[
https://issues.apache.org/jira/browse/HBASE-8127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
rajeshbabu updated HBASE-8127:
--
Attachment: HBASE-8127.patch
Rough patch for 94. I will add some more tests and upload again.
I think similar problem exists in trunk as well when the table is disabling and
rit state is other than PENDING_CLOSE or CLOSING.
I will check more in detail and come back.
> Region of a disabling or disabled table could be stuck in transition state
> when RS dies during Master initialization
>
>
> Key: HBASE-8127
> URL: https://issues.apache.org/jira/browse/HBASE-8127
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.94.5
>Reporter: Jeffrey Zhong
>Assignee: Jeffrey Zhong
> Fix For: 0.94.7
>
> Attachments: HBASE-8127.patch, hbase-8127_v1.patch,
> reproduce-hang.patch
>
>
> The issue happens when a RS dies during a master starts up. After the RS
> reports open to the new master instance and dies immediately thereafter, the
> RITs of disabling tables(or disabled table) on the died RS will be in RIT
> state forever.
> I attached a patch to simulate the situation and you can run the following
> command to reproduce the issue:
> {code}mvn test -PlocalTests
> -Dtest=TestMasterFailover#testMasterFailoverWithMockedRITOnDeadRS{code}
> Basically, we skip regions of a dead server inside
> AM.processDeadServersAndRecoverLostRegions as the following code and relies
> on SSH to process those skipped regions:
> {code}
> for (Pair deadRegion : deadServer.getValue()) {
> nodes.remove(deadRegion.getFirst().getEncodedName());
> }
> {code}
> While in SSH, we skip regions of disabling(or disabled table) again by
> function processDeadRegion. Finally comes to the issue that RITs of
> disabling(or disabled table) stuck there forever.
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-8127) Region of a disabling or disabled table could be stuck in transition state when RS dies during Master initialization
[
https://issues.apache.org/jira/browse/HBASE-8127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-8127:
--
Summary: Region of a disabling or disabled table could be stuck in
transition state when RS dies during Master initialization (was: Region of a
disabling or disabled table could be stucked in transition state when RS dies
during Master initialization)
> Region of a disabling or disabled table could be stuck in transition state
> when RS dies during Master initialization
>
>
> Key: HBASE-8127
> URL: https://issues.apache.org/jira/browse/HBASE-8127
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.94.5
>Reporter: Jeffrey Zhong
>Assignee: Jeffrey Zhong
> Fix For: 0.94.7
>
> Attachments: hbase-8127_v1.patch, reproduce-hang.patch
>
>
> The issue happens when a RS dies during a master starts up. After the RS
> reports open to the new master instance and dies immediately thereafter, the
> RITs of disabling tables(or disabled table) on the died RS will be in RIT
> state forever.
> I attached a patch to simulate the situation and you can run the following
> command to reproduce the issue:
> {code}mvn test -PlocalTests
> -Dtest=TestMasterFailover#testMasterFailoverWithMockedRITOnDeadRS{code}
> Basically, we skip regions of a dead server inside
> AM.processDeadServersAndRecoverLostRegions as the following code and relies
> on SSH to process those skipped regions:
> {code}
> for (Pair deadRegion : deadServer.getValue()) {
> nodes.remove(deadRegion.getFirst().getEncodedName());
> }
> {code}
> While in SSH, we skip regions of disabling(or disabled table) again by
> function processDeadRegion. Finally comes to the issue that RITs of
> disabling(or disabled table) stuck there forever.
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
