[ https://issues.apache.org/jira/browse/HBASE-19937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16358398#comment-16358398 ]
Hudson commented on HBASE-19937: -------------------------------- FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #4554 (See [https://builds.apache.org/job/HBase-Trunk_matrix/4554/]) HBASE-19937 Ensure createRSGroupTable be called after ProcedureExecutor (tedyu: rev bc720374c33701c13f6227c0edfe3a792e7aa333) * (edit) hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupInfoManager.java * (add) hbase-rsgroup/src/test/java/org/apache/hadoop/hbase/rsgroup/TestEnableRSGroup.java * (edit) hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupBasedLoadBalancer.java * (edit) hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupInfoManagerImpl.java > Ensure createRSGroupTable be called after ProcedureExecutor and LoadBalancer > are initialized > --------------------------------------------------------------------------------------------- > > Key: HBASE-19937 > URL: https://issues.apache.org/jira/browse/HBASE-19937 > Project: HBase > Issue Type: Bug > Components: rsgroup > Affects Versions: 2.0.0-beta-2 > Reporter: Xiaolin Ha > Assignee: Xiaolin Ha > Priority: Major > Fix For: 2.0.0-beta-2 > > Attachments: HBASE-19937.branch-2.001.patch, > HBASE-19937.branch-2.002.patch, HBASE-19937.branch-2.003.patch, > HBASE-19937.branch-2.004.patch, HBASE-19937.branch-2.005.patch, > HBASE-19937.branch-2.006.patch, import-order.png > > > hbase:rsgroup table will be created by calling createRSGroupTable when master > load system coprocessors in > > {code:java} > 844 this.cpHost = new MasterCoprocessorHost(this, this.conf);{code} > when ProcedureExecutor hasn't been initialized before createRSGroupTable, it > will encounter Exception as follows, > > {noformat} > Exception in thread > "org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker-localhost,49715,1518088607130" > java.lang.IllegalArgumentException > at > org.apache.hbase.thirdparty.com.google.common.base.Preconditions.checkArgument(Preconditions.java:120) > at > org.apache.hadoop.hbase.procedure2.ProcedureExecutor.submitProcedure(ProcedureExecutor.java:847) > at > org.apache.hadoop.hbase.procedure2.ProcedureExecutor.submitProcedure(ProcedureExecutor.java:835) > at org.apache.hadoop.hbase.master.HMaster.createSystemTable(HMaster.java:1795) > at > org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker.createRSGroupTable(RSGroupInfoManagerImpl.java:858) > at > org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker.waitForGroupTableOnline(RSGroupInfoManagerImpl.java:823) > at > org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker.run(RSGroupInfoManagerImpl.java:743){noformat} > while ProcedureExecutor initialized by calling > {code:java} > 848 startServiceThreads();{code} > And LoadBalancer is initialized by calling > {code:java} > 868 this.balancer.initialize();{code} > When LoadBalancer hasn't been initialized before createRSGroupTable, it will > encounters Exception as follows, > 2018-02-02,16:12:45,688 ERROR > org.apache.hadoop.hbase.procedure2.ProcedureExecutor: CODE-BUG: Uncaught > runtime exception: pid=7, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS; > CreateTableProcedure table=hbase:rsgroup > java.lang.NullPointerException > at > org.apache.hadoop.hbase.rsgroup.RSGroupBasedLoadBalancer.generateGroupMaps(RSGroupBasedLoadBalancer.java:254) > at > org.apache.hadoop.hbase.rsgroup.RSGroupBasedLoadBalancer.roundRobinAssignment(RSGroupBasedLoadBalancer.java:162) > at > org.apache.hadoop.hbase.master.assignment.AssignmentManager.createRoundRobinAssignProcedures(AssignmentManager.java:603) > at > org.apache.hadoop.hbase.master.procedure.CreateTableProcedure.executeFromState(CreateTableProcedure.java:108) > at > org.apache.hadoop.hbase.master.procedure.CreateTableProcedure.executeFromState(CreateTableProcedure.java:51) > at > org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:182) > at org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:845) > at > org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1458) > at > org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1227) > at > org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$800(ProcedureExecutor.java:78) > at > org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1738) > > As a result of CreateTableProcedure.rollbackState, it may then print logs > warning TableExistsException as follows, > 2018-02-02,16:12:55,503 WARN > org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker: > Failed to perform check > java.io.IOException: Failed to create group table. > org.apache.hadoop.hbase.TableExistsException: hbase:rsgroup > at > org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker.createRSGroupTable(RSGroupInfoManagerImpl.java:877) > > After some auto-retries, it loops running the thread RSGroupStartupWorker, > will print logs as follows, > 2018-02-02,16:23:17,626 INFO > org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker: > RSGroup table=hbase:rsgroup isOnline=true, regionCount=0, assignCount=0, > rootMetaFound=true > 2018-02-02,16:23:17,730 INFO > org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker: > RSGroup table=hbase:rsgroup isOnline=true, regionCount=0, assignCount=0, > rootMetaFound=true > 2018-02-02,16:23:17,834 INFO > org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker: > RSGroup table=hbase:rsgroup isOnline=true, regionCount=0, assignCount=0, > rootMetaFound=true > 2018-02-02,16:23:17,937 INFO > org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker: > RSGroup table=hbase:rsgroup isOnline=true, regionCount=0, assignCount=0, > rootMetaFound=true > > And using shells of rsgroup, it will tips that currently is in "offline mode". > > The reason of this problem is that the order of createRSGroupTable and > initializing of ProcedureExecutor and LoadBalancer is out of control. If the > former is excuted earlier, it will encounter Exception mentioned before. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)