[ https://issues.apache.org/jira/browse/HBASE-24585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17140164#comment-17140164 ]
Michael Stack commented on HBASE-24585: --------------------------------------- Issue is actually something else. Sorry for the confusion. The wal splitter does not cause RS exit. It happens earlier up here: {code} 2020-06-18 19:53:54,175 ERROR [main] master.HMasterCommandLine: Master exiting java.lang.RuntimeException: Master not initialized after 200000ms at org.apache.hadoop.hbase.util.JVMClusterUtil.waitForEvent(JVMClusterUtil.java:232) at org.apache.hadoop.hbase.util.JVMClusterUtil.startup(JVMClusterUtil.java:200) at org.apache.hadoop.hbase.LocalHBaseCluster.startup(LocalHBaseCluster.java:430) at org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:232) at org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:140) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76) at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:149) at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:3059) {code} This is standalone. We can't recover in this situation. We just keep crashing out. Let me see... > If RSProcedureHandler throws exception, it aborts the hosting RS > ---------------------------------------------------------------- > > Key: HBASE-24585 > URL: https://issues.apache.org/jira/browse/HBASE-24585 > Project: HBase > Issue Type: Bug > Reporter: Michael Stack > Priority: Major > > HBASE-24574 proc v2 distributed log splitting is enabled. A remote split > fails because it was interrupted. The InterruptedException became an IOE and > then bubbled up and out of the RSPH below causing a RS abort. > {code} > 2020-06-17 21:20:37,472 ERROR > [RS_LOG_REPLAY_OPS-regionserver/localhost:16020-0] > handler.RSProcedureHandler: Error when call RSProcedureCallable: > java.io.IOException: Failed WAL split, status=RESIGNED, > wal=file:/Users/stack/checkouts/hbase.apache.git/tmp/hbase/WALs/localhost,16020,1592440848604-splitting/localhost%2C16020%2C1592440848604.meta.1592440852959.meta > at > org.apache.hadoop.hbase.regionserver.SplitWALCallable.splitWal(SplitWALCallable.java:106) > at > org.apache.hadoop.hbase.regionserver.SplitWALCallable.call(SplitWALCallable.java:86) > at > org.apache.hadoop.hbase.regionserver.SplitWALCallable.call(SplitWALCallable.java:49) > at > org.apache.hadoop.hbase.regionserver.handler.RSProcedureHandler.process(RSProcedureHandler.java:49) > at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > {code} > The remote-procedure framework needs to be more resilient? Log the exception > unless an ERROR and keep going? Otherwise, makes features like procedurev2 > distributed log splitting brittle. -- This message was sent by Atlassian Jira (v8.3.4#803005)