[jira] [Comment Edited] (HBASE-20566) Creating a system table after enabling rsgroup feature puts in region into RIT
[
https://issues.apache.org/jira/browse/HBASE-20566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16473552#comment-16473552
]
Nihal Jain edited comment on HBASE-20566 at 5/13/18 4:30 PM:
-
I have moved quota enabling to {{setUp()}}. Now, we don't need to re-start
hbase at all, as it will be unnecessary to disable quota at the end of
{{testRSGroupsWithHBaseQuota()}}.
Please review the changes.
was (Author: nihaljain.cs):
I have moved quota enabling to {{setUp()}}. Now, we don't need to re-start
hbase at all, as it does is unnecessary to disable quota at the end of
{{testRSGroupsWithHBaseQuota()}}.
Please review the changes.
> Creating a system table after enabling rsgroup feature puts in region into RIT
> --
>
> Key: HBASE-20566
> URL: https://issues.apache.org/jira/browse/HBASE-20566
> Project: HBase
> Issue Type: Bug
> Components: master
>Reporter: Biju Nair
>Assignee: Nihal Jain
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HBASE-20566.master.001.patch,
> HBASE-20566.master.002.patch
>
>
> *Steps to reproduce*
> - Enable {{rsgroup}} feature
> - Enable {{quota}} feature which created {{hbase::quota}} table
> - quota table region will be marked as RIT since the {{rsgroup}} for the
> table is not known
> {noformat}
> 2018-05-10 14:33:32,392 INFO [ProcedureExecutorThread-0]
> zookeeper.ZKTableStateManager: Moving table hbase:quota state from null to
> ENABLING
> 2018-05-10 14:33:32,397 WARN [ProcedureExecutorThread-0]
> rsgroup.RSGroupBasedLoadBalancer: Group for table hbase:quota is null
> 2018-05-10 14:33:32,398 WARN [ProcedureExecutorThread-0]
> master.RegionStates: Failed to open/close 89490cd5e00ea8948af413a1df65091a on
> null, set to FAILED_OPEN
> 2018-05-10 14:33:32,398 INFO [ProcedureExecutorThread-0]
> master.RegionStates: Transition {89490cd5e00ea8948af413a1df65091a
> state=OFFLINE, ts=1525977212397, server=null} to
> {89490cd5e00ea8948af413a1df65091a state=FAILED_OPEN, ts=1525977212398,
> server=null}
> 2018-05-10 14:33:32,398 INFO [ProcedureExecutorThread-0]
> zookeeper.ZKTableStateManager: Moving table hbase:quota state from ENABLING
> to ENABLED
> {noformat}
> - Reason for this issue: Issue
> - [system table
> creation|https://github.com/apache/hbase/blob/061a31fad1654d9ded96d118e04c14860413fa25/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java#L1793]
> doesn't move the table to the appropriate rs group to which system namespace
> is assigned to. Need to execute logic similar to what is done in the
> RSGroupAdminEndpoint for [post table creation|#L377] for user table creation.
> *Work Around*
> - Assigning the system table to ``default`` rsgroup (or to the rsgroup to
> which the system namespace has been assigned).
> - Manually assigning the region in RIT from the system table
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (HBASE-20566) Creating a system table after enabling rsgroup feature puts in region into RIT
[
https://issues.apache.org/jira/browse/HBASE-20566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16473413#comment-16473413
]
Nihal Jain edited comment on HBASE-20566 at 5/13/18 9:06 AM:
-
{quote}I think the enabling of quota can be done in setUp() method so that we
don't need to restart the cluster twice.
{quote}
I had written the test case in a way that it does not hinder the result of any
other test cases in the given class. Enabling quota in the set up can be done,
but say we run the test class without the fix (i.e. if quota table creation
gets stuck), then none of the test cases would pass as the bug for some reasons
causes dropping of tables to fail and ultimately rolls back. Hence will cause
all test cases to fail.
{code:java}
2018-05-13 14:16:02,171 INFO
[RpcServer.default.FPBQ.Fifo.handler=29,queue=2,port=16000] master.HMaster:
Client=root1//127.0.0.1 delete test
2018-05-13 14:16:02,287 INFO
[RpcServer.default.FPBQ.Fifo.handler=29,queue=2,port=16000]
rsgroup.RSGroupAdminServer: Moving table test to RSGroup null
2018-05-13 14:16:02,292 INFO [PEWorker-16] hbase.MetaTableAccessor: Deleted 1
regions from META
2018-05-13 14:16:02,307 INFO [PEWorker-16] hbase.MetaTableAccessor: Deleted 1
regions from META
2018-05-13 14:16:02,309 INFO [PEWorker-16] hbase.MetaTableAccessor: Deleted
table test state from META
2018-05-13 14:16:02,313 INFO [PEWorker-16] procedure2.ProcedureExecutor:
Finished pid=20, state=SUCCESS; DeleteTableProcedure table=test in 140msec
2018-05-13 14:16:07,045 WARN [ProcExecTimeout] assignment.AssignmentManager:
STUCK Region-In-Transition rit=OFFLINE, location=localhost,1,1,
table=hbase:quota, region=2b9f9ceca3cb8a77ae2220b053c1c4e7
2018-05-13 14:16:12,045 WARN [ProcExecTimeout] assignment.AssignmentManager:
STUCK Region-In-Transition rit=OFFLINE, location=localhost,1,1,
table=hbase:quota, region=2b9f9ceca3cb8a77ae2220b053c1c4e7
2018-05-13 14:16:17,046 WARN [ProcExecTimeout] assignment.AssignmentManager:
STUCK Region-In-Transition rit=OFFLINE, location=localhost,1,1,
table=hbase:quota, region=2b9f9ceca3cb8a77ae2220b053c1c4e7
2018-05-13 14:16:22,046 WARN [ProcExecTimeout] assignment.AssignmentManager:
STUCK Region-In-Transition rit=OFFLINE, location=localhost,1,1,
table=hbase:quota, region=2b9f9ceca3cb8a77ae2220b053c1c4e7
2018-05-13 14:16:27,046 WARN [ProcExecTimeout] assignment.AssignmentManager:
STUCK Region-In-Transition rit=OFFLINE, location=localhost,1,1,
table=hbase:quota, region=2b9f9ceca3cb8a77ae2220b053c1c4e7
2018-05-13 14:16:32,046 WARN [ProcExecTimeout] assignment.AssignmentManager:
STUCK Region-In-Transition rit=OFFLINE, location=localhost,1,1,
table=hbase:quota, region=2b9f9ceca3cb8a77ae2220b053c1c4e7
2018-05-13 14:16:37,047 WARN [ProcExecTimeout] assignment.AssignmentManager:
STUCK Region-In-Transition rit=OFFLINE, location=localhost,1,1,
table=hbase:quota, region=2b9f9ceca3cb8a77ae2220b053c1c4e7
2018-05-13 14:16:42,047 WARN [ProcExecTimeout] assignment.AssignmentManager:
STUCK Region-In-Transition rit=OFFLINE, location=localhost,1,1,
table=hbase:quota, region=2b9f9ceca3cb8a77ae2220b053c1c4e7
2018-05-13 14:16:47,048 WARN [ProcExecTimeout] assignment.AssignmentManager:
STUCK Region-In-Transition rit=OFFLINE, location=localhost,1,1,
table=hbase:quota, region=2b9f9ceca3cb8a77ae2220b053c1c4e7
2018-05-13 14:16:52,049 WARN [ProcExecTimeout] assignment.AssignmentManager:
STUCK Region-In-Transition rit=OFFLINE, location=localhost,1,1,
table=hbase:quota, region=2b9f9ceca3cb8a77ae2220b053c1c4e7
2018-05-13 14:16:57,049 WARN [ProcExecTimeout] assignment.AssignmentManager:
STUCK Region-In-Transition rit=OFFLINE, location=localhost,1,1,
table=hbase:quota, region=2b9f9ceca3cb8a77ae2220b053c1c4e7
2018-05-13 14:17:02,050 WARN [ProcExecTimeout] assignment.AssignmentManager:
STUCK Region-In-Transition rit=OFFLINE, location=localhost,1,1,
table=hbase:quota, region=2b9f9ceca3cb8a77ae2220b053c1c4e7
2018-05-13 14:17:02,282 INFO
[RpcServer.default.FPBQ.Fifo.handler=28,queue=1,port=16000] master.HMaster:
Client=root1//127.0.0.1 delete test
2018-05-13 14:17:02,411 INFO [PEWorker-3] procedure2.ProcedureExecutor: Rolled
back pid=21, state=ROLLEDBACK,
exception=org.apache.hadoop.hbase.TableNotFoundException via
master-delete-table:org.apache.hadoop.hbase.TableNotFoundException: test;
DeleteTableProcedure table=test exec-time=128msec
{code}
Since with the fix the test case will pass, we can surely move enabling of
quota to setUp(), but am not sure about negative scenarios. What do you suggest?
was (Author: nihaljain.cs):
{quote}I think the enabling of quota can be done in setUp() method so that we
don't need to restart the cluster twice.
{quote}
I had written the test case in a way that it does not hinder the result of any
other test cases in the given class. Enabling quota in the set up can be done,
but say we run the test cl
[jira] [Comment Edited] (HBASE-20566) Creating a system table after enabling rsgroup feature puts in region into RIT
[
https://issues.apache.org/jira/browse/HBASE-20566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16473304#comment-16473304
]
Ted Yu edited comment on HBASE-20566 at 5/13/18 12:08 AM:
--
I think the enabling of quota can be done in setUp() method so that we don't
need to restart the cluster twice.
On my Mac, the duration for TestRSGroups went from 182 seconds (with one of the
two restarts in your patch since I moved the enabling of quota to setUp, hence
dropping the first restart) to 166 seconds.
You just need to adjust the assertion in basic test:
{code}
-//3 meta,namespace, group
-assertEquals(3, count);
+//3 meta,namespace, rsgroup, quota
+assertEquals(4, count);
{code}
This also proves that all assertions in previous tests still hold with quota
enabled, except for the above one.
was (Author: yuzhih...@gmail.com):
I think the enabling of quota can be done in setUp() method so that we don't
need to restart the cluster twice.
On my Mac, the duration for TestRSGroups went from 182 seconds (with one of the
two restarts in your patch since I moved the enabling of quota to setUp, hence
dropping the first restart) to 166 seconds.
You just need to adjust the assertion in basic test:
{code}
-//3 meta,namespace, group
-assertEquals(3, count);
+//3 meta,namespace, rsgroup, quota
+assertEquals(4, count);
{code}
This also proves that all assertions in previous tests still hold with quota
enabled.
> Creating a system table after enabling rsgroup feature puts in region into RIT
> --
>
> Key: HBASE-20566
> URL: https://issues.apache.org/jira/browse/HBASE-20566
> Project: HBase
> Issue Type: Bug
> Components: master
>Reporter: Biju Nair
>Assignee: Nihal Jain
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HBASE-20566.master.001.patch
>
>
> *Steps to reproduce*
> - Enable {{rsgroup}} feature
> - Enable {{quota}} feature which created {{hbase::quota}} table
> - quota table region will be marked as RIT since the {{rsgroup}} for the
> table is not known
> {noformat}
> 2018-05-10 14:33:32,392 INFO [ProcedureExecutorThread-0]
> zookeeper.ZKTableStateManager: Moving table hbase:quota state from null to
> ENABLING
> 2018-05-10 14:33:32,397 WARN [ProcedureExecutorThread-0]
> rsgroup.RSGroupBasedLoadBalancer: Group for table hbase:quota is null
> 2018-05-10 14:33:32,398 WARN [ProcedureExecutorThread-0]
> master.RegionStates: Failed to open/close 89490cd5e00ea8948af413a1df65091a on
> null, set to FAILED_OPEN
> 2018-05-10 14:33:32,398 INFO [ProcedureExecutorThread-0]
> master.RegionStates: Transition {89490cd5e00ea8948af413a1df65091a
> state=OFFLINE, ts=1525977212397, server=null} to
> {89490cd5e00ea8948af413a1df65091a state=FAILED_OPEN, ts=1525977212398,
> server=null}
> 2018-05-10 14:33:32,398 INFO [ProcedureExecutorThread-0]
> zookeeper.ZKTableStateManager: Moving table hbase:quota state from ENABLING
> to ENABLED
> {noformat}
> - Reason for this issue: Issue
> - [system table
> creation|https://github.com/apache/hbase/blob/061a31fad1654d9ded96d118e04c14860413fa25/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java#L1793]
> doesn't move the table to the appropriate rs group to which system namespace
> is assigned to. Need to execute logic similar to what is done in the
> RSGroupAdminEndpoint for [post table creation|#L377] for user table creation.
> *Work Around*
> - Assigning the system table to ``default`` rsgroup (or to the rsgroup to
> which the system namespace has been assigned).
> - Manually assigning the region in RIT from the system table
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (HBASE-20566) Creating a system table after enabling rsgroup feature puts in region into RIT
[
https://issues.apache.org/jira/browse/HBASE-20566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16473304#comment-16473304
]
Ted Yu edited comment on HBASE-20566 at 5/12/18 11:59 PM:
--
I think the enabling of quota can be done in setUp() method so that we don't
need to restart the cluster twice.
On my Mac, the duration for TestRSGroups went from 182 seconds (with one of the
two restarts in your patch since I moved the enabling of quota to setUp, hence
dropping the first restart) to 166 seconds.
You just need to adjust the assertion in basic test:
{code}
-//3 meta,namespace, group
-assertEquals(3, count);
+//3 meta,namespace, rsgroup, quota
+assertEquals(4, count);
{code}
This also proves that all assertions in previous tests still hold with quota
enabled.
was (Author: yuzhih...@gmail.com):
I think the enabling of quota can be done in setUp() method so that we don't
need to restart the cluster twice.
On my Mac, the duration for TestRSGroups went from 182 seconds to 166 seconds.
You just need to adjust the assertion in basic test:
{code}
-//3 meta,namespace, group
-assertEquals(3, count);
+//3 meta,namespace, rsgroup, quota
+assertEquals(4, count);
{code}
This also proves that all assertions in previous tests still hold with quota
enabled.
> Creating a system table after enabling rsgroup feature puts in region into RIT
> --
>
> Key: HBASE-20566
> URL: https://issues.apache.org/jira/browse/HBASE-20566
> Project: HBase
> Issue Type: Bug
> Components: master
>Reporter: Biju Nair
>Assignee: Nihal Jain
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HBASE-20566.master.001.patch
>
>
> *Steps to reproduce*
> - Enable {{rsgroup}} feature
> - Enable {{quota}} feature which created {{hbase::quota}} table
> - quota table region will be marked as RIT since the {{rsgroup}} for the
> table is not known
> {noformat}
> 2018-05-10 14:33:32,392 INFO [ProcedureExecutorThread-0]
> zookeeper.ZKTableStateManager: Moving table hbase:quota state from null to
> ENABLING
> 2018-05-10 14:33:32,397 WARN [ProcedureExecutorThread-0]
> rsgroup.RSGroupBasedLoadBalancer: Group for table hbase:quota is null
> 2018-05-10 14:33:32,398 WARN [ProcedureExecutorThread-0]
> master.RegionStates: Failed to open/close 89490cd5e00ea8948af413a1df65091a on
> null, set to FAILED_OPEN
> 2018-05-10 14:33:32,398 INFO [ProcedureExecutorThread-0]
> master.RegionStates: Transition {89490cd5e00ea8948af413a1df65091a
> state=OFFLINE, ts=1525977212397, server=null} to
> {89490cd5e00ea8948af413a1df65091a state=FAILED_OPEN, ts=1525977212398,
> server=null}
> 2018-05-10 14:33:32,398 INFO [ProcedureExecutorThread-0]
> zookeeper.ZKTableStateManager: Moving table hbase:quota state from ENABLING
> to ENABLED
> {noformat}
> - Reason for this issue: Issue
> - [system table
> creation|https://github.com/apache/hbase/blob/061a31fad1654d9ded96d118e04c14860413fa25/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java#L1793]
> doesn't move the table to the appropriate rs group to which system namespace
> is assigned to. Need to execute logic similar to what is done in the
> RSGroupAdminEndpoint for [post table creation|#L377] for user table creation.
> *Work Around*
> - Assigning the system table to ``default`` rsgroup (or to the rsgroup to
> which the system namespace has been assigned).
> - Manually assigning the region in RIT from the system table
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (HBASE-20566) Creating a system table after enabling rsgroup feature puts in region into RIT
[
https://issues.apache.org/jira/browse/HBASE-20566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16473206#comment-16473206
]
Ted Yu edited comment on HBASE-20566 at 5/12/18 11:22 PM:
--
Patch makes sense.
See TestFavoredNodeTableImport.java for example of calling
shutdownMiniHBaseCluster / restartHBaseCluster - you don't need to write your
own method.
was (Author: yuzhih...@gmail.com):
I wonder if adding postCreateTable hook to
hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/MasterSpaceQuotaObserver.java
would help solve the issue.
> Creating a system table after enabling rsgroup feature puts in region into RIT
> --
>
> Key: HBASE-20566
> URL: https://issues.apache.org/jira/browse/HBASE-20566
> Project: HBase
> Issue Type: Bug
> Components: master
>Reporter: Biju Nair
>Assignee: Nihal Jain
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HBASE-20566.master.001.patch
>
>
> *Steps to reproduce*
> - Enable {{rsgroup}} feature
> - Enable {{quota}} feature which created {{hbase::quota}} table
> - quota table region will be marked as RIT since the {{rsgroup}} for the
> table is not known
> {noformat}
> 2018-05-10 14:33:32,392 INFO [ProcedureExecutorThread-0]
> zookeeper.ZKTableStateManager: Moving table hbase:quota state from null to
> ENABLING
> 2018-05-10 14:33:32,397 WARN [ProcedureExecutorThread-0]
> rsgroup.RSGroupBasedLoadBalancer: Group for table hbase:quota is null
> 2018-05-10 14:33:32,398 WARN [ProcedureExecutorThread-0]
> master.RegionStates: Failed to open/close 89490cd5e00ea8948af413a1df65091a on
> null, set to FAILED_OPEN
> 2018-05-10 14:33:32,398 INFO [ProcedureExecutorThread-0]
> master.RegionStates: Transition {89490cd5e00ea8948af413a1df65091a
> state=OFFLINE, ts=1525977212397, server=null} to
> {89490cd5e00ea8948af413a1df65091a state=FAILED_OPEN, ts=1525977212398,
> server=null}
> 2018-05-10 14:33:32,398 INFO [ProcedureExecutorThread-0]
> zookeeper.ZKTableStateManager: Moving table hbase:quota state from ENABLING
> to ENABLED
> {noformat}
> - Reason for this issue: Issue
> - [system table
> creation|https://github.com/apache/hbase/blob/061a31fad1654d9ded96d118e04c14860413fa25/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java#L1793]
> doesn't move the table to the appropriate rs group to which system namespace
> is assigned to. Need to execute logic similar to what is done in the
> RSGroupAdminEndpoint for [post table creation|#L377] for user table creation.
> *Work Around*
> - Assigning the system table to ``default`` rsgroup (or to the rsgroup to
> which the system namespace has been assigned).
> - Manually assigning the region in RIT from the system table
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (HBASE-20566) Creating a system table after enabling rsgroup feature puts in region into RIT
[
https://issues.apache.org/jira/browse/HBASE-20566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16471503#comment-16471503
]
Nihal Jain edited comment on HBASE-20566 at 5/11/18 5:16 AM:
-
We can handle this by adding the logic to add system tables to appropriate
rsgroup in *preCreateTableAction()*
was (Author: nihaljain.cs):
We can handle this by adding the logic to add system tables to appropriate **
rsgroup ** in *preCreateTableAction()*
> Creating a system table after enabling rsgroup feature puts in region into RIT
> --
>
> Key: HBASE-20566
> URL: https://issues.apache.org/jira/browse/HBASE-20566
> Project: HBase
> Issue Type: Bug
> Components: master
>Reporter: Biju Nair
>Priority: Major
>
> *Steps to reproduce*
> - Enable {{rsgroup}} feature
> - Enable {{quota}} feature which created {{hbase::quota}} table
> - quota table region will be marked as RIT since the {{rsgroup}} for the
> table is not known
> {noformat}
> 2018-05-10 14:33:32,392 INFO [ProcedureExecutorThread-0]
> zookeeper.ZKTableStateManager: Moving table hbase:quota state from null to
> ENABLING
> 2018-05-10 14:33:32,397 WARN [ProcedureExecutorThread-0]
> rsgroup.RSGroupBasedLoadBalancer: Group for table hbase:quota is null
> 2018-05-10 14:33:32,398 WARN [ProcedureExecutorThread-0]
> master.RegionStates: Failed to open/close 89490cd5e00ea8948af413a1df65091a on
> null, set to FAILED_OPEN
> 2018-05-10 14:33:32,398 INFO [ProcedureExecutorThread-0]
> master.RegionStates: Transition {89490cd5e00ea8948af413a1df65091a
> state=OFFLINE, ts=1525977212397, server=null} to
> {89490cd5e00ea8948af413a1df65091a state=FAILED_OPEN, ts=1525977212398,
> server=null}
> 2018-05-10 14:33:32,398 INFO [ProcedureExecutorThread-0]
> zookeeper.ZKTableStateManager: Moving table hbase:quota state from ENABLING
> to ENABLED
> {noformat}
> - Reason for this issue: Issue
> - [system table
> creation|https://github.com/apache/hbase/blob/061a31fad1654d9ded96d118e04c14860413fa25/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java#L1793]
> doesn't move the table to the appropriate rs group to which system namespace
> is assigned to. Need to execute logic similar to what is done in the
> RSGroupAdminEndpoint for [post table creation|#L377] for user table creation.
> *Work Around*
> - Assigning the system table to ``default`` rsgroup (or to the rsgroup to
> which the system namespace has been assigned).
> - Manually assigning the region in RIT from the system table
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
