justice created HBASE-21316: ------------------------------- Summary: All RegionServer Down when RS_LOG_REPLAY_OPS Key: HBASE-21316 URL: https://issues.apache.org/jira/browse/HBASE-21316 Project: HBase Issue Type: Bug Components: regionserver Affects Versions: 2.0.0 Reporter: justice Attachments: log.tgz
1. One RegionServer die as unknow reason, log as follow: {code:java} 2018-10-14 20:31:47,423 INFO [main-SendThread(11.3.20.101:2181)] zookeeper.ClientCnxn: Socket connection established to 11.3.20.101/11.3.20.101:2181, initiating session 2018-10-14 20:31:47,433 INFO [main-SendThread(11.3.20.101:2181)] zookeeper.ClientCnxn: Session establishment complete on server 11.3.20.101/11.3.20.101:2181, sessionid = 0x6500073f944a8e79, negotiated timeout = 30000 2018-10-14日 Sunday 21:03:05 CST Starting regionserver on 11-3-19-199.JD.LOCAL core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited {code} 2. Master receive zk deletenode event, and start ServerCrashProcedure Task {code:java} 2018-10-14 20:31:47,437 INFO [main-EventThread] master.RegionServerTracker: RegionServer ephemeral node deleted, processing expiration [11-3-19-199.jd.local,16020,1539492869470] 2018-10-14 20:31:47,539 INFO [PEWorker-1] procedure.ServerCrashProcedure: Start pid=25053, state=RUNNABLE:SERVER_CRASH_START; ServerCrashProcedure server=11-3-19-199.jd.local,16020,1539492869470, splitWal=true, meta=false 2018-10-14 20:31:47,550 INFO [PEWorker-1] master.SplitLogManager: Started splitting 63 logs in [hdfs://11-3-18-67.JD.LOCAL:9000/hbase/WALs/11-3-19-199.jd.local,16020,1539492869470-splitting] for [11-3-19-199.jd.local,16020,1539492869470] ... 2018-10-14 20:31:48,592 INFO [main-EventThread] coordination.SplitLogManagerCoordination: Task /hbase/splitWAL/WALs%2F11-3-19-199.jd.local%2C16020%2C1539492869470-splitting%2F11-3-19-199.jd.local%252C16020%252C1539492869470.1539520250598 acquired by 11-3-18-71.jd.local,16020,1539492869409 {code} 3. One alive RegionServer Node get SplitLogWorker, has an error and stop {code:java} 2018-10-14 20:31:48,602 INFO [SplitLogWorker-11-3-18-71:16020] coordination.ZkSplitLogWorkerCoordination: worker 11-3-18-71.jd.local,16020,1539492869409 acquired task /hbase/splitWAL/WALs%2F11-3-19-199.jd.local%2C16020%2C1539492869470-splitting%2F11-3-19-199.jd.local%252C16020%252C1539492869470.1539520250598 ... 2018-10-14 21:03:26,219 ERROR [RS_LOG_REPLAY_OPS-regionserver/11-3-18-71:16020-1] executor.EventHandler: Caught throwable while processing event RS_LOG_REPLAY java.lang.ArrayIndexOutOfBoundsException: 8811 at org.apache.hadoop.hbase.KeyValue.getFamilyLength(KeyValue.java:1365) at org.apache.hadoop.hbase.KeyValue.getFamilyLength(KeyValue.java:1358) at org.apache.hadoop.hbase.PrivateCellUtil.matchingFamily(PrivateCellUtil.java:735) at org.apache.hadoop.hbase.CellUtil.matchingFamily(CellUtil.java:816) at org.apache.hadoop.hbase.wal.WALEdit.isMetaEditFamily(WALEdit.java:102) at org.apache.hadoop.hbase.wal.WALEdit.isMetaEdit(WALEdit.java:107) at org.apache.hadoop.hbase.wal.WALSplitter.splitLogFile(WALSplitter.java:296) at org.apache.hadoop.hbase.wal.WALSplitter.splitLogFile(WALSplitter.java:194) at org.apache.hadoop.hbase.regionserver.SplitLogWorker$1.exec(SplitLogWorker.java:99) at org.apache.hadoop.hbase.regionserver.handler.WALSplitterHandler.process(WALSplitterHandler.java:70) at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2018-10-14 21:03:26,227 ERROR [RS_LOG_REPLAY_OPS-regionserver/11-3-18-71:16020-1] regionserver.HRegionServer: ***** ABORTING region server 11-3-18-71.jd.local,16020,1539522186368: Caught throwable while processing event RS_LOG_REPLAY ***** .... 2018-10-14 20:31:48,780 INFO [RS_LOG_REPLAY_OPS-regionserver/11-3-18-71:16020-0] regionserver.HRegionServer: ***** STOPPING region server '11-3-18-71.jd.local,16020,1539492869409' ***** {code} 4. other alive regionserver node die one by one, at last, all regionserver node die -- This message was sent by Atlassian JIRA (v7.6.3#76005)