Yes, It happened in my cluster.
[RegionServer LOG] 2017-03-20 11:02:21,466 WARN org.apache.hadoop.hbase.regionserver.wal.FSHLog: Couldn't find oldest seqNum for the region we are about to flush: [] 2017-03-20 11:02:21,466 INFO org.apache.hadoop.hbase.regionserver.HRegion: Finished memstore flush of ~0/0, currentsize=/ for region . in 0ms, sequenceid=, compaction requested=false 2017-03-20 11:02:21,466 FATAL org.apache.hadoop.hbase.regionserver.HRegionServer: ABORTING region server : Unrecoverable exception while closing region , still finishing close org.apache.hadoop.hbase.DroppedSnapshotException: Failed clearing memory after 6 attempts on region: . at org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1108) at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1046) at org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:147) at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:128) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 2017-03-20 11:02:21,467 FATAL org.apache.hadoop.hbase.regionserver.HRegionServer: RegionServer abort: loaded coprocessors are: [] 2017-03-20 11:02:21,528 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: STOPPED: Unrecoverable exception while closing region , still finishing close 2017-03-20 11:02:21,528 ERROR org.apache.hadoop.hbase.executor.EventHandler: Caught throwable while processing event M_RS_CLOSE_REGION java.lang.RuntimeException: org.apache.hadoop.hbase.DroppedSnapshotException: Failed clearing memory after 6 attempts on region: . at org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:161) at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:128) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hbase.DroppedSnapshotException: Failed clearing memory after 6 attempts on region: . at org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1108) at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1046) at org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:147) ... 4 more 2017-03-20 11:02:21,528 INFO org.apache.hadoop.ipc.RpcServer: Stopping server on 2017-03-20 11:02:21,531 INFO org.apache.hadoop.ipc.RpcServer: RpcServer.listener,port=: stopping 2017-03-20 11:02:21,531 INFO org.apache.hadoop.ipc.RpcServer: Priority.RpcServer.handler=0,port=: exiting 2017-03-20 11:02:21,531 INFO org.apache.hadoop.hbase.regionserver.SplitLogWorker: Sending interrupt to stop the worker thread 2017-03-20 11:02:21,531 INFO org.apache.hadoop.ipc.RpcServer: Priority.RpcServer.handler=1,port=: exiting ... 2017-03-20 11:02:30,556 INFO org.apache.zookeeper.ZooKeeper: Session: closed 2017-03-20 11:02:30,556 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down 2017-03-20 11:02:30,556 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: stopping server ; zookeeper connection closed. 2017-03-20 11:02:30,556 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: exiting 2017-03-20 11:02:30,556 ERROR org.apache.hadoop.hbase.regionserver.HRegionServerCommandLine: Region server exiting java.lang.RuntimeException: HRegionServer Aborted at org.apache.hadoop.hbase.regionserver.HRegionServerCommandLine.start(HRegionServerCommandLine.java:66) at org.apache.hadoop.hbase.regionserver.HRegionServerCommandLine.run(HRegionServerCommandLine.java:85) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:126) at org.apache.hadoop.hbase.regionserver.HRegionServer.main(HRegionServer.java:2340) 2017-03-20 11:02:30,593 INFO org.apache.hadoop.hbase.regionserver.ShutdownHook: Shutdown hook starting; hbase.shutdown.hook=true; fsShutdownHook=org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer@56ddd32a 2017-03-20 11:02:30,593 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: STOPPED: Shutdown hook 2017-03-20 11:02:30,593 INFO org.apache.hadoop.hbase.regionserver.ShutdownHook: Starting fs shutdown hook thread. 2017-03-20 11:02:30,593 ERROR org.apache.hadoop.hdfs.DFSClient: Failed to close file java.net.SocketTimeoutException: 20000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local= remote=] at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:164) at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:118) at java.io.FilterInputStream.read(FilterInputStream.java:83) at java.io.FilterInputStream.read(FilterInputStream.java:83) at org.apache.hadoop.hdfs.protocolPB.PBHelper.vintPrefixed(PBHelper.java:1984) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.transfer(DFSOutputStream.java:1064) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1031) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1175) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:924) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:486) 2017-03-20 11:02:30,594 INFO org.apache.hadoop.hbase.regionserver.ShutdownHook: Shutdown hook finished. [HMaster LOG] 2017-03-20 11:02:21,525 ERROR org.apache.hadoop.hbase.master.HMaster: Region server reported a fatal error: ABORTING region server : Unrecoverable exception while closing region , still finishing close 2017-03-20 11:02:22,261 INFO org.apache.hadoop.hbase.master.RegionStates: Offlined from 2017-03-20 11:02:22,723 INFO org.apache.hadoop.hbase.master.RegionStates: Offlined from 2017-03-20 11:02:30,535 INFO org.apache.hadoop.hbase.zookeeper.RegionServerTracker: RegionServer ephemeral node deleted, processing expiration [] 2017-03-20 11:02:31,165 INFO org.apache.hadoop.hbase.master.handler.ServerShutdownHandler: Splitting logs for before assignment. Thanks, Minwoo. ________________________________ 보낸 사람: Ted Yu <yuzhih...@gmail.com> 보낸 날짜: 2017년 3월 20일 월요일 오후 12:10:46 받는 사람: user@hbase.apache.org 제목: Re: Why IOException occur when region server is closing (CloseRegionHandler.java#L110)? See HBASE-4270 Did you see this happen in your cluster ? If so, mind sharing related log snippets ? Cheers On Sun, Mar 19, 2017 at 7:50 PM, Kang Minwoo <minwoo.k...@outlook.com> wrote: > Hello! > > In this code (https://github.com/apache/hbase/blob/master/hbase- > server/src/main/java/org/apache/hadoop/hbase/regionserver/handler/ > CloseRegionHandler.java#L110), > Region server can occur IOException, When they are closing. > Why IOException occur here? > If I want to know specific reason, Where I should check? > > Thanks, > Minwoo. > >