Seems reasonable, but having trouble making sense of the GC logs I had turned on. Basically since there was a full GC a minute before this happens on that server that lasts less than a second.
Example: So, here is what the last of the GC logs say for that Regionserver (04.hadoop on 10.100.154.10) (note the last full GC only 1 minutes before @ 5:03:29.850+0000): 2010-12-09T05:02:19.472+0000: 12170.437: [GC [PSYoungGen: 278229K->5024K(287360K)] 694269K->421693K(721856K), 0.0274840 secs] [Times: user=0.04 sys=0.00, real=0.03 secs] 2010-12-09T05:02:55.479+0000: 12206.444: [GC [PSYoungGen: 281450K->7968K(273344K)] 698120K->429658K(707840K), 0.0503000 secs] [Times: user=0.10 sys=0.00, real=0.05 secs] 2010-12-09T05:03:29.800+0000: 12240.766: [GC [PSYoungGen: 273295K->7712K(262464K)] 694986K->437306K(696960K), 0.0490770 secs] [Times: user=0.12 sys=0.01, real=0.05 secs] 2010-12-09T05:03:29.850+0000: 12240.815: [Full GC [PSYoungGen: 7712K->0K(262464K)] [PSOldGen: 429594K->353152K(429824K)] 437306K->353152K(692288K) [PSPermGen: 18016K->18016K(21248K)], 0.8960760 secs] [Times: user=0.89 sys=0.00, real=0.90 secs] 2010-12-09T05:04:03.345+0000: 12274.310: [GC [PSYoungGen: 254720K->6944K(251584K)] 607872K->360528K(681408K), 0.0331060 secs] [Times: user=0.03 sys=0.00, real=0.04 secs] 2010-12-09T05:04:33.875+0000: 12304.840: [GC [PSYoungGen: 251552K->7200K(242240K)] 605136K->367800K(672064K), 0.0336420 secs] [Times: user=0.09 sys=0.01, real=0.03 secs] Heap PSYoungGen total 242240K, used 223390K [0x00007fcc640b0000, 0x00007fcc77a90000, 0x00007fcc78e00000) eden space 235008K, 91% used [0x00007fcc640b0000,0x00007fcc713cfa30,0x00007fcc72630000) from space 7232K, 99% used [0x00007fcc769b0000,0x00007fcc770b8090,0x00007fcc770c0000) to space 10048K, 0% used [0x00007fcc770c0000,0x00007fcc770c0000,0x00007fcc77a90000) PSOldGen total 429824K, used 360600K [0x00007fcc3a600000, 0x00007fcc549c0000, 0x00007fcc640b0000) object space 429824K, 83% used [0x00007fcc3a600000,0x00007fcc50626060,0x00007fcc549c0000) PSPermGen total 21248K, used 18131K [0x00007fcc35200000, 0x00007fcc366c0000, 0x00007fcc3a600000) object space 21248K, 85% used [0x00007fcc35200000,0x00007fcc363b4d60,0x00007fcc366c0000) Now Dead / End of log That said, I will continue down the path you pointed me on . Thanks, Lance On Dec 9, 2010, at 11:40 AM, Jean-Daniel Cryans wrote: > Lance, > > Both those lines indicate the problem: > > IPC Server handler 13 on 60020 took 182416ms > Client session timed out, have not heard from server in 182936ms > > It's very clear that your region servers are suffering from > pause-of-the-world garbage collection issues. Basically this one GC'ed > for 3 minutes, which is over the 1 minute timeout to consider a region > server dead. The rest is just the side effect of that. > > This subject comes often on the mailing list, for example: > http://search-hadoop.com/m/t97q027tr7f2 > > Make sure you don't swap, give plenty of RAM to HBase, use LZO > compression, don't underprovision your machines, etc > > J-D > > On Thu, Dec 9, 2010 at 11:27 AM, Lance Riedel <lancerie...@gmail.com> wrote: >> >> We have a 6 node cluster, 5 with region serves. 2 of the region servers >> have been stable for days, but 3 of them keep crashing. Here are the logs >> around around when the crash occurs. (btw, we are shoving approximately the >> twitter firehose into hbase via flume) I'm an hbase newbie, but I have been >> reading. Not sure what else is needed to help debug. When the problem >> occurs, it is concurrent with issues that appear all the way down in the >> data node. Also, interesting to note, all 3 servers seem to fail >> differently somewhat: >> >> Servers (6): >> 00.hadoop is the master >> 01.hadoop, 03.hadoop (there is no 02.hadoop - flaky machine that got taken >> out) - are stable Region Servers >> 04-06.hadoop crash at different times, very different logs, Region Servers >> >> >> Master Attributes: >> HBase Version 0.89.20100924+28, r >> Hadoop Version 0.20.2+737, r98c55c28258aa6f42250569bd7fa431ac657bdbd >> >> HBase Root Directory hdfs://00.hadoop.****:54310/hbase >> >> Load average >> 103.0 >> Average number of regions per regionserver. Naive computation. >> Regions On FS >> 577 >> >> >> (This is after recently restarting 04-06) >> >> >> Address Start Code Load >> 01.hadoop:60030 1291268910638 requests=223, regions=118, usedHeap=519, >> maxHeap=987 >> 03.hadoop:60030 1291269219610 requests=23, regions=111, usedHeap=862, >> maxHeap=987 >> 04.hadoop:60030 1291910665912 requests=169, regions=82, usedHeap=194, >> maxHeap=888 >> 05.hadoop:60030 1291909584060 requests=232, regions=110, usedHeap=477, >> maxHeap=888 >> 06.hadoop60030 1291909723787 requests=99, regions=94, usedHeap=394, >> maxHeap=888 >> Total: >> servers: 5 >> requests=746, regions=515 >> >> >> >> Following are various logs around the time of the failures for 04-06 >> >> ************************************************************************************ >> 04.hadoop >> ************************************************************************************ >> REGION SERVER -> >> 2010-12-09 05:01:58,640 INFO org.apache.hadoop.hbase.regionserver.HRegion: >> compaction completed on region >> article,a83858a08f2270d319f75a7b43c756c2453988e7,1291811929868.58f6d9fc80 >> c78f3ca490b0280b4f1226. in 0sec >> 2010-12-09 05:04:30,898 DEBUG >> org.apache.hadoop.hbase.io.hfile.LruBlockCache: LRU Stats: total=1.46 MB, >> free=176.33 MB, max=177.79 MB, blocks=0, accesses=200583, hits=0, hitRatio= >> 0.00%%, evictions=0, evicted=0, evictedPerRun=NaN >> 2010-12-09 05:04:59,612 INFO >> org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter: Using syncFs >> -- HDFS-200 >> 2010-12-09 05:04:59,618 INFO org.apache.hadoop.hbase.regionserver.wal.HLog: >> Roll >> /hbase/.logs/04.hadoop.****,60020,1291858770125/10.100.154.103%3A60020.1291870585253, >> entries=5318 >> 6, filesize=63776311. New hlog >> /hbase/.logs/04.hadoop.****,60020,1291858770125/10.100.154.103%3A60020.1291871099582 >> 2010-12-09 05:08:02,033 DEBUG org.apache.hadoop.hbase.regionserver.wal.HLog: >> Found 1 hlogs to remove out of total 6; oldest outstanding sequenceid is >> 32891801 from region article >> ,4b2039b791e894dd479b90661ca97087f61645d3,1291813201039.7dff47c63bb5648bb1ee6670c60553a5. >> 2010-12-09 05:08:02,034 WARN org.apache.hadoop.hbase.regionserver.wal.HLog: >> IPC Server handler 13 on 60020 took 182416ms appending an edit to hlog; >> editcount=0 >> 2010-12-09 05:08:02,066 INFO org.apache.hadoop.hbase.regionserver.wal.HLog: >> moving old hlog file >> /hbase/.logs/04.hadoop.****,60020,1291858770125/10.100.154.103%3A60020.12918679476 >> 65 whose highest sequenceid is 32880227 to >> /hbase/.oldlogs/10.100.154.103%3A60020.1291867947665 >> 2010-12-09 05:08:02,066 INFO org.apache.zookeeper.ClientCnxn: Client session >> timed out, have not heard from server in 182936ms for sessionid >> 0x12ca58c08ff0048, closing socket conn >> ection and attempting reconnect >> 2010-12-09 05:08:02,089 FATAL >> org.apache.hadoop.hbase.regionserver.HRegionServer: Aborting region server >> serverName=04.hadoop.****,60020,1291858770125, load=(requests=24, regions= >> 64, usedHeap=548, maxHeap=888): Unhandled exception >> org.apache.hadoop.hbase.YouAreDeadException: >> org.apache.hadoop.hbase.YouAreDeadException: Server REPORT rejected; >> currently processing 04.hadoop.****,60020,1291858770125 as dead s >> erver >> at >> org.apache.hadoop.hbase.master.ServerManager.checkIsDead(ServerManager.java:217) >> at >> org.apache.hadoop.hbase.master.ServerManager.regionServerReport(ServerManager.java:271) >> at >> org.apache.hadoop.hbase.master.HMaster.regionServerReport(HMaster.java:744) >> at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source) >> at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) >> at java.lang.reflect.Method.invoke(Method.java:597) >> at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:561) >> at >> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1039) >> >> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native >> Method) >> at >> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) >> at >> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) >> at java.lang.reflect.Constructor.newInstance(Constructor.java:513) >> at >> org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:94) >> at >> org.apache.hadoop.hbase.RemoteExceptionHandler.checkThrowable(RemoteExceptionHandler.java:48) >> at >> org.apache.hadoop.hbase.RemoteExceptionHandler.checkIOException(RemoteExceptionHandler.java:66) >> at >> org.apache.hadoop.hbase.regionserver.HRegionServer.doRun(HRegionServer.java:635) >> at >> org.apache.hadoop.hbase.regionserver.HRegionServer.access$000(HRegionServer.java:126) >> at >> org.apache.hadoop.hbase.regionserver.HRegionServer$1.run(HRegionServer.java:518) >> at >> org.apache.hadoop.hbase.regionserver.HRegionServer$1.run(HRegionServer.java:516) >> at java.security.AccessController.doPrivileged(Native Method) >> at javax.security.auth.Subject.doAs(Subject.java:337) >> at >> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1046) >> at >> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:516) >> at java.lang.Thread.run(Thread.java:662) >> 2010-12-09 05:08:02,090 INFO >> org.apache.hadoop.hbase.regionserver.HRegionServer: Dump of metrics: >> request=0.0, regions=64, stores=64, storefiles=136, storefileIndexSize=27, >> memsto >> reSize=353, compactionQueueSize=0, usedHeap=549, maxHeap=888, >> blockCacheSize=1530552, blockCacheFree=184893160, blockCacheCount=0, >> blockCacheHitRatio=0 >> 2010-12-09 05:08:02,090 INFO org.apache.hadoop.ipc.HBaseServer: Stopping >> server on 60020 >> 2010-12-09 05:08:02,090 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC >> Server handler 0 on 60020: exiting2010-12-09 05:08:02,090 INFO >> org.apache.hadoop.ipc.HBaseServer: Stopping IPC Server listener on 60020 >> >> >> MASTER HBASE -> >> >> 2010-12-09 05:05:06,072 INFO org.apache.hadoop.hbase.master.BaseScanner: All >> 1 .META. region(s) scanned >> 2010-12-09 05:05:40,004 INFO org.apache.hadoop.hbase.master.ServerManager: >> 04.hadoop.jive,60020,1291858770125 znode expired >> 2010-12-09 05:05:40,005 DEBUG org.apache.hadoop.hbase.master.ServerManager: >> Added=04.hadoop.jive,60020,1291858770125 to dead servers, added shutdown >> processing operation >> 2010-12-09 05:05:40,005 DEBUG >> org.apache.hadoop.hbase.master.RegionServerOperationQueue: Processing todo: >> ProcessServerShutdown of 04.hadoop.jive,60020,12918587701252010-12-09 >> 05:05:40,005 INFO org.apache.hadoop.hbase.master.RegionServerOperation: >> Process shutdown of server 04.hadoop.jive,60020,1291858770125: logSplit: >> false, rootRescanned: f >> alse, numberOfMetaRegions: 1, onlineMetaRegions.size(): 1 >> 2010-12-09 05:05:40,008 INFO org.apache.hadoop.hbase.regionserver.wal.HLog: >> Splitting 7 hlog(s) in >> hdfs://00.hadoop.jive:54310/hbase/.logs/04.hadoop.jive,60020,12918587701252010-12-09 >> 05:05:40,008 DEBUG org.apache.hadoop.hbase.regionserver.wal.HLog: Splitting >> hlog 1 of 7: >> hdfs://00.hadoop.jive:54310/hbase/.logs/04.hadoop.jive,60020,1291858770125/10.1 >> 00.154.103%3A60020.1291867947665, length=637577092010-12-09 05:05:40,008 >> INFO org.apache.hadoop.hbase.util.FSUtils: Recovering >> filehdfs://00.hadoop.jive:54310/hbase/.logs/04.hadoop.jive,60020,1291858770125/10.100.154.103%3A60020 >> .1291867947665 >> >> >> NAMENODE -> >> >> 2010-12-09 05:08:02,471 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* >> NameSystem.addStoredBlock: blockMap updated: 10.100.159.13:50010 is added to >> blk_1531008743226086399_251615 size 63757709 >> 2010-12-09 05:08:02,473 INFO org.apache.hadoop.ipc.Server: IPC Server >> handler 9 on 54310, call >> complete(/hbase/.logs/04.hadoop.jive,60020,1291858770125/10.100.154.103%3A60020.1291867947665, >> DFSClient_hb_m_10.194.194.79:60000_1291788452343) from 10.194.194.79:44117: >> error: org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: No >> lease on >> /hbase/.logs/04.hadoop.jive,60020,1291858770125/10.100.154.103%3A60020.1291867947665 >> File does not exist. [Lease. Holder: >> DFSClient_hb_m_10.194.194.79:60000_1291788452343, pendingcreates: 1] >> org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: No lease on >> /hbase/.logs/04.hadoop.jive,60020,1291858770125/10.100.154.103%3A60020.1291867947665 >> File does not exist. [Lease. Holder: >> DFSClient_hb_m_10.194.194.79:60000_1291788452343, pendingcreates: 1] >> at >> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:1488) >> at >> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:1479) >> at >> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.completeFileInternal(FSNamesystem.java:1534) >> at >> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.completeFile(FSNamesystem.java:1522) >> at >> org.apache.hadoop.hdfs.server.namenode.NameNode.complete(NameNode.java:610) >> at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source) >> at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) >> at java.lang.reflect.Method.invoke(Method.java:597) >> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:528) >> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1319) >> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1315) >> at java.security.AccessController.doPrivileged(Native Method) >> at javax.security.auth.Subject.doAs(Subject.java:396) >> at >> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1063) >> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1313) >> 2010-12-09 05:08:04,206 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* >> NameSystem.addStoredBlock: blockMap updated: 10.99.97.106:50010 is added to >> blk_-734280257049179934_251614 size 7330 >> >> DATANODE -> >> >> 2010-12-09 05:08:02,212 WARN >> org.apache.hadoop.hdfs.server.datanode.DataNode: IOException in >> BlockReceiver.lastNodeRun: java.io.IOException: Broken pipe >> at sun.nio.ch.FileDispatcher.write0(Native Method) >> at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29) >> at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:100) at >> sun.nio.ch.IOUtil.write(IOUtil.java:71) >> at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334) >> at >> org.apache.hadoop.net.SocketOutputStream$Writer.performIO(SocketOutputStream.java:55) >> at >> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142) >> at >> org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:146) >> at >> org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:107) >> at java.io.DataOutputStream.writeLong(DataOutputStream.java:207) >> at >> org.apache.hadoop.hdfs.protocol.DataTransferProtocol$PipelineAck.write(DataTransferProtocol.java:133) >> at >> org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.lastDataNodeRun(BlockReceiver.java:840) >> at >> org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run(BlockReceiver.java:870) >> at java.lang.Thread.run(Thread.java:662) >> 2010-12-09 05:08:02,213 WARN >> org.apache.hadoop.hdfs.server.datanode.DataNode: checkDiskError: exception: >> 2010-12-09 05:08:02,213 WARN >> org.apache.hadoop.hdfs.server.datanode.DataNode: checkDiskError: exception: >> java.io.IOException: Broken pipe >> at sun.nio.ch.FileDispatcher.write0(Native Method) >> at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29) >> at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:100) >> at sun.nio.ch.IOUtil.write(IOUtil.java:71) >> at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334) >> at >> org.apache.hadoop.net.SocketOutputStream$Writer.performIO(SocketOutputStream.java:55) >> at >> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142) >> at >> org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:146) >> at >> org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:107) >> at java.io.DataOutputStream.writeLong(DataOutputStream.java:207) >> at >> org.apache.hadoop.hdfs.protocol.DataTransferProtocol$PipelineAck.write(DataTransferProtocol.java:133) >> at >> org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.lastDataNodeRun(BlockReceiver.java:840) >> at >> org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run(BlockReceiver.java:870) >> at java.lang.Thread.run(Thread.java:662) >> 2010-12-09 05:08:02,221 INFO >> org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder >> blk_-8817504198034990390_251613 0 Exception java.io.IOException: Broken pipe >> at sun.nio.ch.FileDispatcher.write0(Native Method) >> at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29) >> at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:100) >> at sun.nio.ch.IOUtil.write(IOUtil.java:71) >> at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334) >> at >> org.apache.hadoop.net.SocketOutputStream$Writer.performIO(SocketOutputStream.java:55) >> at >> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142) >> at >> org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:146) >> at >> org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:107) >> at java.io.DataOutputStream.writeLong(DataOutputStream.java:207) >> at >> org.apache.hadoop.hdfs.protocol.DataTransferProtocol$PipelineAck.write(DataTransferProtocol.java:133) >> at >> org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.lastDataNodeRun(BlockReceiver.java:840) >> at >> org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run(BlockReceiver.java:870) >> at java.lang.Thread.run(Thread.java:662) >> >> 2010-12-09 05:08:02,221 INFO >> org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder 0 for block >> blk_-8817504198034990390_251613 terminating >> 2010-12-09 05:08:02,224 INFO >> org.apache.hadoop.hdfs.server.datanode.DataNode: writeBlock >> blk_-8817504198034990390_251613 received exception java.io.IOException: >> Connection reset by peer >> 2010-12-09 05:08:02,224 INFO >> org.apache.hadoop.hdfs.server.datanode.DataNode: Scheduling block >> blk_4505306716668305567_250752 file >> /dist1/data/hadoop-data/current/subdir19/blk_4505306716668305567 for deletion >> 2010-12-09 05:08:02,226 ERROR >> org.apache.hadoop.hdfs.server.datanode.DataNode: >> DatanodeRegistration(10.100.154.103:50010, >> storageID=DS-866555524-10.100.154.103-50010-1291262739506, infoPort=50075, >> ipcPort=50020):DataXceiver >> java.io.IOException: Connection reset by peer >> at sun.nio.ch.FileDispatcher.read0(Native Method) >> at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21) >> at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:237) >> at sun.nio.ch.IOUtil.read(IOUtil.java:210) >> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:236) >> at >> org.apache.hadoop.net.SocketInputStream$Reader.performIO(SocketInputStream.java:55) >> at >> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142) >> at >> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155) >> at >> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128) >> at java.io.BufferedInputStream.read1(BufferedInputStream.java:256) >> at java.io.BufferedInputStream.read(BufferedInputStream.java:317) >> at java.io.DataInputStream.read(DataInputStream.java:132) >> at >> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.readToBuf(BlockReceiver.java:267) >> at >> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.readNextPacket(BlockReceiver.java:357) >> at >> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:378) >> at >> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:534) >> at >> org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:417) >> at >> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:122) >> 2010-12-09 05:08:02,226 INFO >> org.apache.hadoop.hdfs.server.datanode.DataNode: Deleted block >> blk_1422641301942366074_250732 at file >> /dist1/data/hadoop-data/current/subdir19/blk_1422641301942366074 >> 2010-12-09 05:08:02,227 INFO >> org.apache.hadoop.hdfs.server.datanode.DataNode: Deleted block >> blk_4505306716668305567_250752 at file >> /dist1/data/hadoop-data/current/subdir19/blk_4505306716668305567 >> >> >> >> ************************************************************************************ >> 05.hadoop >> ************************************************************************************ >> REGION SERVER -> >> 2010-12-09 04:29:25,047 DEBUG org.apache.hadoop.hbase.regionserver.Store: >> Compaction size of data: 250.5m; Skipped 1 file(s), size: 206896367 >> 2010-12-09 04:29:25,047 INFO org.apache.hadoop.hbase.regionserver.Store: >> Started compaction of 2 file(s) in data of >> article,060c4b75067378c045eead9edec2d4fb7d8f3e5e,1291812563166.95e92e97823e11e57850720a63433a62. >> into >> hdfs://00.hadoop.****:54310/hbase/article/95e92e97823e11e57850720a63433a62/.tmp, >> sequenceid=33050221 >> 2010-12-09 04:29:26,328 INFO org.apache.hadoop.hbase.regionserver.Store: >> Completed compaction of 2 file(s) in data of >> article,060c4b75067378c045eead9edec2d4fb7d8f3e5e,1291812563166.95e92e97823e11e57850720a63433a62.; >> new storefile is >> hdfs://00.hadoop.****:54310/hbase/article/95e92e97823e11e57850720a63433a62/data/2787768957794924068; >> store size is 250.5m >> 2010-12-09 04:29:26,328 INFO org.apache.hadoop.hbase.regionserver.HRegion: >> compaction completed on region >> article,060c4b75067378c045eead9edec2d4fb7d8f3e5e,1291812563166.95e92e97823e11e57850720a63433a62. >> in 1sec >> 2010-12-09 04:29:26,328 INFO org.apache.hadoop.hbase.regionserver.HRegion: >> Starting compaction on region >> article,ed0cdc7ae6db73432e2ca47c973d545bee11ddd8,1291813400028.85a8d31d099f409dd36f3bcc4be1eba6. >> 2010-12-09 04:29:26,330 INFO org.apache.hadoop.hbase.regionserver.HRegion: >> compaction completed on region >> article,ed0cdc7ae6db73432e2ca47c973d545bee11ddd8,1291813400028.85a8d31d099f409dd36f3bcc4be1eba6. >> in 0sec >> 2010-12-09 04:32:33,081 INFO >> org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGIONSERVER_STOP >> 2010-12-09 04:32:33,081 INFO org.apache.hadoop.ipc.HBaseServer: Stopping >> server on 60020 >> 2010-12-09 04:32:33,089 WARN org.apache.hadoop.hdfs.DFSClient: >> DFSOutputStream ResponseProcessor exception for block >> blk_2012253533858803341_250248java.io.EOFException >> at java.io.DataInputStream.readFully(DataInputStream.java:180) >> at java.io.DataInputStream.readLong(DataInputStream.java:399) >> at >> org.apache.hadoop.hdfs.protocol.DataTransferProtocol$PipelineAck.readFields(DataTransferProtocol.java:120) >> at >> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$ResponseProcessor.run(DFSClient.java:2621) >> >> 2010-12-09 04:32:33,089 INFO org.apache.zookeeper.ClientCnxn: Unable to read >> additional data from server sessionid 0x12ca58c08ff0049, likely server has >> closed socket, closing sock >> et connection and attempting reconnect >> 2010-12-09 04:32:33,089 WARN org.apache.hadoop.hbase.regionserver.wal.HLog: >> IPC Server handler 5 on 60020 took 75446ms appending an edit to hlog; >> editcount=50505 >> 2010-12-09 04:32:33,090 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server >> handler 18 on 60020: exiting >> 2010-12-09 04:32:33,139 INFO org.apache.hadoop.hdfs.DFSClient: Error >> Recovery for block blk_2012253533858803341_250248 waiting for responder to >> exit. >> 2010-12-09 04:32:33,139 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server >> handler 8 on 60020: exiting >> 2010-12-09 04:32:33,139 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC >> Server handler 1 on 60020: exiting >> 2010-12-09 04:32:33,139 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server >> handler 10 on 60020: exiting >> 2010-12-09 04:32:33,140 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC >> Server handler 0 on 60020: exiting2010-12-09 04:32:33,140 INFO >> org.apache.hadoop.ipc.HBaseServer: PRI IPC Server handler 5 on 60020: exiting >> 2010-12-09 04:32:33,139 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC >> Server handler 8 on 60020: exiting2010-12-09 04:32:33,140 INFO >> org.apache.hadoop.ipc.HBaseServer: PRI IPC Server handler 4 on 60020: exiting >> 2010-12-09 04:32:33,140 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC >> Server handler 3 on 60020: exiting2010-12-09 04:32:33,142 INFO >> org.apache.hadoop.ipc.HBaseServer: PRI IPC Server handler 7 on 60020: exiting >> 2010-12-09 04:32:33,142 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC >> Server handler 9 on 60020: exiting2010-12-09 04:32:33,143 INFO >> org.apache.hadoop.hbase.regionserver.HRegionServer: Stopping infoServer >> 2010-12-09 04:32:33,143 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC >> Server handler 2 on 60020: exiting >> 2010-12-09 04:32:33,143 INFO org.apache.hadoop.ipc.HBaseServer: Stopping IPC >> Server Responder >> 2010-12-09 04:32:33,143 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC >> Server handler 6 on 60020: exiting >> 2010-12-09 04:32:33,144 INFO org.mortbay.log: Stopped >> selectchannelconnec...@0.0.0.0:60030 >> >> >> >> >> NAMENODE -> >> 2010-12-09 04:32:08,408 WARN org.apache.hadoop.hdfs.StateChange: DIR* >> NameSystem.startFile: failed to create file >> /hbase/.logs/05.hadoop.jive,60020,1291862294517/10.100.159.13%3A60020.1291868481555 >> for DFSClient_hb_m_10.194.194.79:60000_1291788452343 on client >> 10.194.194.79, because this file is already being created by NN_Recovery on >> 10.100.159.13 >> 2010-12-09 04:32:08,408 INFO org.apache.hadoop.ipc.Server: IPC Server >> handler 6 on 54310, call >> append(/hbase/.logs/05.hadoop.jive,60020,1291862294517/10.100.159.13%3A60020.1291868481555, >> DFSClient_hb_m_10.194.194.79:60000_1291788452343) from 10.194.194.79:59118: >> error: org.apache.hadoop.hdfs.protocol.AlreadyBeingCreatedException: failed >> to create file /hba >> se/.logs/05.hadoop.jive,60020,1291862294517/10.100.159.13%3A60020.1291868481555 >> for DFSClient_hb_m_10.194.194.79:60000_1291788452343 on client >> 10.194.194.79, because this file is already being created by NN_Recovery on >> 10.100.159.13 >> org.apache.hadoop.hdfs.protocol.AlreadyBeingCreatedException: failed to >> create file >> /hbase/.logs/05.hadoop.jive,60020,1291862294517/10.100.159.13%3A60020.1291868481555 >> for DFSClient_hb_m_10.194.194.79:60000_1291788452343 on client >> 10.194.194.79, because this file is already being created by NN_Recovery on >> 10.100.159.13 >> at >> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInternal(FSNamesystem.java:1194) >> at >> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.appendFile(FSNamesystem.java:1282) >> at >> org.apache.hadoop.hdfs.server.namenode.NameNode.append(NameNode.java:541) >> at sun.reflect.GeneratedMethodAccessor37.invoke(Unknown Source) >> at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) >> at java.lang.reflect.Method.invoke(Method.java:597) >> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:528) at >> org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1319) >> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1315) >> at java.security.AccessController.doPrivileged(Native Method) >> at javax.security.auth.Subject.doAs(Subject.java:396) at >> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1063) >> at >> org.apache.hadoop.ipc.Server$Handler.run(Server.java:1313)2010-12-09 >> 04:32:09,411 WARN org.apache.hadoop.hdfs.StateChange: DIR* >> NameSystem.startFile: failed to create file >> /hbase/.logs/05.hadoop.jive,60020,1291862294517/10.100.159.13%3A6 >> 0020.1291868481555 for DFSClient_hb_m_10.194.194.79:60000_1291788452343 on >> client 10.194.194.79, because this file is already being created by >> NN_Recovery on 10.100.159.132010-12-09 04:32:09,411 INFO >> org.apache.hadoop.ipc.Server: IPC Server handler 9 on 54310, call >> append(/hbase/.logs/05.hadoop.jive,60020,1291862294517/10.100.159.13%3A60020.1291868 >> 481555, DFSClient_hb_m_10.194.194.79:60000_1291788452343) from >> 10.194.194.79:59118: error: >> org.apache.hadoop.hdfs.protocol.AlreadyBeingCreatedException: failed to >> create file >> /hbase/.logs/05.hadoop.jive,60020,1291862294517/10.100.159.13%3A60020.1291868481555 >> for DFSClient_hb_m_10.194.194.79:60000_1291788452343 on client >> 10.194.194.79, because this file is >> already being created by NN_Recovery on >> 10.100.159.13org.apache.hadoop.hdfs.protocol.AlreadyBeingCreatedException: >> failed to create file >> /hbase/.logs/05.hadoop.jive,60020,1291862294517/10.100.159.13%3A60020.1291868481555 >> for DFSClie >> nt_hb_m_10.194.194.79:60000_1291788452343 on client 10.194.194.79, because >> this file is already being created by NN_Recovery on 10.100.159.13 at >> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInternal(FSNamesystem.java:1194) >> at >> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.appendFile(FSNamesystem.java:1282) >> at >> org.apache.hadoop.hdfs.server.namenode.NameNode.append(NameNode.java:541) >> at sun.reflect.GeneratedMethodAccessor37.invoke(Unknown Source) >> at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) >> at java.lang.reflect.Method.invoke(Method.java:597) at >> org.apache.hadoop.ipc.RPC$Server.call(RPC.java:528) >> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1319) >> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1315) >> >> >> >> DATANODE >> >> 2010-12-09 04:32:09,534 INFO >> org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder >> blk_2012253533858803341_250248 1 : Thread is interrupted. >> 2010-12-09 04:32:09,534 ERROR >> org.apache.hadoop.hdfs.server.datanode.DataNode: >> DatanodeRegistration(10.100.159.13:50010, >> storageID=DS-145997007-10.100.159.13-50010-1291262743182, infoPort=50075, >> ipcPort=50020):DataXceiver >> java.io.IOException: Interrupted receiveBlock >> at >> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:579) >> at >> org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:417) >> at >> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:122) >> 2010-12-09 04:32:09,534 INFO >> org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder 1 for block >> blk_2012253533858803341_250248 terminating >> >> >> ************************************************************************************ >> 06.hadoop >> ************************************************************************************ >> REGION SERVER -> >> >> 2010-12-09 06:17:47,215 DEBUG >> org.apache.hadoop.hbase.regionserver.HRegionServer: >> NotServingRegionException; >> article,8d36892f14dfa6d17d08900743960d0b0973fc96,1291812431639.cce2b650859704eb184df8fb5c5eee6f. >> 2010-12-09 06:17:47,430 DEBUG >> org.apache.hadoop.hbase.regionserver.HRegionServer: >> NotServingRegionException; >> article,8d36892f14dfa6d17d08900743960d0b0973fc96,1291812431639.cce2b650859704eb184df8fb5c5eee6f. >> 2010-12-09 06:17:47,549 DEBUG >> org.apache.hadoop.hbase.regionserver.HRegionServer: >> NotServingRegionException; >> article,8d36892f14dfa6d17d08900743960d0b0973fc96,1291812431639.cce2b650859704eb184df8fb5c5eee6f. >> 2010-12-09 06:17:47,557 DEBUG >> org.apache.hadoop.hbase.regionserver.HRegionServer: >> NotServingRegionException; >> article,8d36892f14dfa6d17d08900743960d0b0973fc96,1291812431639.cce2b650859704eb184df8fb5c5eee6f. >> 2010-12-09 06:17:47,563 DEBUG >> org.apache.hadoop.hbase.regionserver.HRegionServer: >> NotServingRegionException; >> article,8d36892f14dfa6d17d08900743960d0b0973fc96,1291812431639.cce2b650859704eb184df8fb5c5eee6f. >> 2010-12-09 06:17:48,619 DEBUG >> org.apache.hadoop.hbase.regionserver.HRegionServer: >> NotServingRegionException; >> article,8d36892f14dfa6d17d08900743960d0b0973fc96,1291812431639.cce2b650859704eb184df8fb5c5eee6f. >> 2010-12-09 06:17:48,619 DEBUG >> org.apache.hadoop.hbase.regionserver.HRegionServer: >> NotServingRegionException; >> article,8d36892f14dfa6d17d08900743960d0b0973fc96,1291812431639.cce2b650859704eb184df8fb5c5eee6f. >> 2010-12-09 06:17:48,619 DEBUG >> org.apache.hadoop.hbase.regionserver.HRegionServer: >> NotServingRegionException; >> article,8d36892f14dfa6d17d08900743960d0b0973fc96,1291812431639.cce2b650859704eb184df8fb5c5eee6f. >> 2010-12-09 06:17:48,619 DEBUG >> org.apache.hadoop.hbase.regionserver.HRegionServer: >> NotServingRegionException; >> article,8d36892f14dfa6d17d08900743960d0b0973fc96,1291812431639.cce2b650859704eb184df8fb5c5eee6f. >> 2010-12-09 06:17:48,619 DEBUG >> org.apache.hadoop.hbase.regionserver.HRegionServer: >> NotServingRegionException; >> article,8d36892f14dfa6d17d08900743960d0b0973fc96,1291812431639.cce2b650859704eb184df8fb5c5eee6f. >> 2010-12-09 06:17:49,703 DEBUG >> org.apache.hadoop.hbase.regionserver.HRegionServer: >> NotServingRegionException; >> article,8d36892f14dfa6d17d08900743960d0b0973fc96,1291812431639.cce2b650859704eb184df8fb5c5eee6f. >> 2010-12-09 06:17:49,703 DEBUG >> org.apache.hadoop.hbase.regionserver.HRegionServer: >> NotServingRegionException; >> article,8d36892f14dfa6d17d08900743960d0b0973fc96,1291812431639.cce2b650859704eb184df8fb5c5eee6f. >> 2010-12-09 06:17:49,703 DEBUG >> org.apache.hadoop.hbase.regionserver.HRegionServer: >> NotServingRegionException; >> article,8d36892f14dfa6d17d08900743960d0b0973fc96,1291812431639.cce2b650859704eb184df8fb5c5eee6f. >> 2010-12-09 06:17:49,705 DEBUG >> org.apache.hadoop.hbase.regionserver.HRegionServer: >> NotServingRegionException; >> article,8d36892f14dfa6d17d08900743960d0b0973fc96,1291812431639.cce2b650859704eb184df8fb5c5eee6f. >> 2010-12-09 06:17:49,729 DEBUG >> org.apache.hadoop.hbase.regionserver.HRegionServer: >> NotServingRegionException; >> article,8d36892f14dfa6d17d08900743960d0b0973fc96,1291812431639.cce2b650859704eb184df8fb5c5eee6f. >> >> A bunch of these ^ >> >> Then a little later:: >> .... >> >> 2010-12-09 06:23:23,561 DEBUG >> org.apache.hadoop.hbase.regionserver.CompactSplitThread: Compaction >> requested for region >> article,993143fca3613014990cc6975cacdd62f4147e03,12918133247 >> 53.19413f33fc660536d9d4f315a1283440. because: regionserver60020.cacheFlusher >> 2010-12-09 06:23:27,058 INFO >> org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter: Using syncFs >> -- HDFS-200 >> 2010-12-09 06:23:27,080 INFO org.apache.hadoop.hbase.regionserver.wal.HLog: >> Roll >> /hbase/.logs/06.hadoop.****,60020,1291864620851/10.99.97.106%3A60020.1291875047947, >> entries=53315, >> filesize=63781925. New hlog >> /hbase/.logs/06.hadoop.****,60020,1291864620851/10.99.97.106%3A60020.1291875807006 >> 2010-12-09 06:23:27,080 DEBUG org.apache.hadoop.hbase.regionserver.wal.HLog: >> Found 1 hlogs to remove out of total 8; oldest outstanding sequenceid is >> 33526881 from region article >> ,1221b0013c61fd7446cb822f698d0bcafa487342,1291872610788.bb2aa6ae8c694a096a00e7bb91fc46c3. >> 2010-12-09 06:23:27,080 INFO org.apache.hadoop.hbase.regionserver.wal.HLog: >> moving old hlog file >> /hbase/.logs/06.hadoop.****,60020,1291864620851/10.99.97.106%3A60020.1291872064424 >> whose highest sequenceid is 33506639 to >> /hbase/.oldlogs/10.99.97.106%3A60020.1291872064424 >> 2010-12-09 06:25:39,162 WARN org.apache.hadoop.hbase.regionserver.wal.HLog: >> IPC Server handler 12 on 60020 took 119034ms appending an edit to hlog; >> editcount=3683 >> 2010-12-09 06:25:39,162 WARN org.apache.hadoop.hdfs.DFSClient: DataStreamer >> Exception: java.io.IOException: Broken pipe >> at sun.nio.ch.FileDispatcher.write0(Native Method) >> at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29) >> at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:100) >> at sun.nio.ch.IOUtil.write(IOUtil.java:71) >> at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334) >> at >> org.apache.hadoop.net.SocketOutputStream$Writer.performIO(SocketOutputStream.java:55) >> at >> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142) >> at >> org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:146) >> at >> org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:107) >> at java.io.BufferedOutputStream.write(BufferedOutputStream.java:105) >> at java.io.DataOutputStream.write(DataOutputStream.java:90) >> at >> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2509) >> 2010-12-09 06:25:39,163 WARN org.apache.hadoop.hdfs.DFSClient: >> DFSOutputStream ResponseProcessor exception for block >> blk_1141147913164105086_258331java.io.EOFException >> at java.io.DataInputStream.readFully(DataInputStream.java:180) >> at java.io.DataInputStream.readLong(DataInputStream.java:399) >> at >> org.apache.hadoop.hdfs.protocol.DataTransferProtocol$PipelineAck.readFields(DataTransferProtocol.java:120) >> at >> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$ResponseProcessor.run(DFSClient.java:2621) >> >> 2010-12-09 06:25:39,163 INFO org.apache.hadoop.hdfs.DFSClient: Error >> Recovery for block blk_1141147913164105086_258331 waiting for responder to >> exit. >> 2010-12-09 06:25:39,164 WARN org.apache.hadoop.hdfs.DFSClient: >> DFSOutputStream ResponseProcessor exception for block >> blk_-7076574401277381592_258324java.io.EOFException >> at java.io.DataInputStream.readFully(DataInputStream.java:180) >> at java.io.DataInputStream.readLong(DataInputStream.java:399) >> at >> org.apache.hadoop.hdfs.protocol.DataTransferProtocol$PipelineAck.readFields(DataTransferProtocol.java:120) >> at >> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$ResponseProcessor.run(DFSClient.java:2621) >> >> 2010-12-09 06:25:39,164 WARN org.apache.hadoop.hdfs.DFSClient: Error >> Recovery for block blk_-7076574401277381592_258324 bad datanode[0] >> 10.99.97.106:50010 >> 2010-12-09 06:25:39,164 INFO org.apache.zookeeper.ClientCnxn: Unable to read >> additional data from server sessionid 0x12ca58c08ff004b, likely server has >> closed socket, closing socket connection and attempting reconnect >> 2010-12-09 06:25:39,164 WARN org.apache.hadoop.hdfs.DFSClient: Error >> Recovery for block blk_-7076574401277381592_258324 in pipeline >> 10.99.97.106:50010, 10.100.154.103:50010: bad datanode 10.99.97.106:50010 >> 2010-12-09 06:25:39,165 WARN org.apache.hadoop.hdfs.DFSClient: Error >> Recovery for block blk_1141147913164105086_258331 bad datanode[0] >> 10.99.97.106:50010 >> 2010-12-09 06:25:39,167 INFO org.apache.zookeeper.ClientCnxn: Unable to read >> additional data from server sessionid 0x12ca58c08ff004c, likely server has >> closed socket, closing socket connection and attempting reconnect >> 2010-12-09 06:25:39,169 INFO org.apache.hadoop.ipc.Client: Retrying connect >> to server: /10.100.154.103:50020. Already tried 0 time(s). >> 2010-12-09 06:25:39,169 WARN org.apache.hadoop.hdfs.DFSClient: Error >> Recovery for block blk_1141147913164105086_258331 in pipeline >> 10.99.97.106:50010, 10.100.159.13:50010: bad datanode 10.99.97.106:50010 >> 2010-12-09 06:25:39,209 INFO >> org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGIONSERVER_STOP >> 2010-12-09 06:25:39,210 INFO org.apache.hadoop.ipc.HBaseServer: Stopping >> server on 60020 >> 2010-12-09 06:25:39,210 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server >> handler 3 on 60020: exiting >> 2010-12-09 06:25:39,210 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server >> handler 9 on 60020: exiting >> 2010-12-09 06:25:39,210 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server >> handler 24 on 60020: exiting >> 2010-12-09 06:25:39,210 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC >> Server handler 1 on 60020: exiting >> 2010-12-09 06:25:39,210 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server >> handler 15 on 60020: exiting >> 2010-12-09 06:25:39,215 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC >> Server handler 0 on 60020: exiting >> 2010-12-09 06:25:39,214 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC >> Server handler 3 on 60020: exiting >> 2010-12-09 06:25:39,214 INFO org.apache.hadoop.ipc.HBaseServer: Stopping IPC >> Server listener on 60020 >> 2010-12-09 06:25:39,215 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC >> Server handler 2 on 60020: exiting >> 2010-12-09 06:25:39,215 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC >> Server handler 7 on 60020: exiting >> 2010-12-09 06:25:39,213 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server >> handler 21 on 60020: exiting >> 2010-12-09 06:25:39,215 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC >> Server handler 4 on 60020: exiting >> 2010-12-09 06:25:39,215 INFO org.apache.hadoop.ipc.HBaseServer: Stopping IPC >> Server Responder >> 2010-12-09 06:25:39,215 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC >> Server handler 5 on 60020: exiting >> 2010-12-09 06:25:39,216 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC >> Server handler 8 on 60020: exiting >> 2010-12-09 06:25:39,216 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC >> Server handler 9 on 60020: exiting >> 2010-12-09 06:25:39,216 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC >> Server handler 6 on 60020: exiting >> 2010-12-09 06:25:39,216 INFO >> org.apache.hadoop.hbase.regionserver.HRegionServer: Stopping infoServer >> 2010-12-09 06:25:39,223 WARN org.apache.hadoop.hdfs.DFSClient: Failed >> recovery attempt #0 from primary datanode 10.100.159.13:50010 >> >> >> >>
