Dear all， For help, many many thanks to help。
Some days ago，our production hbase cluster experience a trouble，about Apr 23 21:52, a machine suddenly crash down，the regionserver and datanode also down in the machine, then about six regionserver handler be filled, hbase cluster request rapidly go down. Until we stop the six regionserver the hbase cluster recover. we guess the hbase numCallsInGeneralQueue overstocke, but we increase numCallsInGeneralQueue and decrease handler and cut off network of a machine, we can not reappear the scenario. So , I want to ask for help how to solve the scenario gracefully and how we can reappear the scenario for test. Error log ： 2018-04-23 21:57:11,472 INFO [StoreFileOpenerThread-info-1] regionserver.StoreFile$Reader: Loaded Delete Family Bloom (CompoundBloom Filter) metadata for 1bade4c9adb6444baebf9e797a037f78 2018-04-23 21:57:14,151 WARN [StoreFileOpenerThread-info-1] hdfs.BlockReaderFactory: I/O error constructing remote block reader. java.net.NoRouteToHostException: No route to host at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:529) at org.apache.hadoop.hdfs.DFSClient.newConnectedPeer(DFSClient.java:2884) at org.apache.hadoop.hdfs.BlockReaderFactory.nextTcpPeer(BlockReaderFactory.java:747) at org.apache.hadoop.hdfs.BlockReaderFactory.getRemoteBlockReaderFromTcp(BlockReaderFactory.java:662) at org.apache.hadoop.hdfs.BlockReaderFactory.build(BlockReaderFactory.java:326) at org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(DFSInputStream.java:570) at org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:793) at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:840) at java.io.DataInputStream.readFully(DataInputStream.java:195) at org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:390) at org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:482) at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:525) at org.apache.hadoop.hbase.regionserver.StoreFile$Reader.<init>(StoreFile.java:1164) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.open(StoreFileInfo.java:259) at org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:427) at org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:528) at org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:518) at org.apache.hadoop.hbase.regionserver.HStore.createStoreFileAndReader(HStore.java:667) at org.apache.hadoop.hbase.regionserver.HStore.access$000(HStore.java:119) at org.apache.hadoop.hbase.regionserver.HStore$1.call(HStore.java:534) at org.apache.hadoop.hbase.regionserver.HStore$1.call(HStore.java:531) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 2018-04-23 21:57:14,151 WARN [StoreFileOpenerThread-info-1] hdfs.BlockReaderFactory: I/O error constructing remote block reader. java.net.NoRouteToHostException: No route to host
