Hello,
I'm doing performance test on hbase 0.20.6. My cluster contains 7
regionservers. After lodding 4TB data, one regionserver shutdown.
Exception info in the log is below. Can someone tell me what's the matter?
Regionserver log:
2010-12-16 09:42:09,724 WARN org.apache.hadoop.hdfs.DFSClient: DataStreamer
Exception: java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcher.write0(Native Method)
at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29)
at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:100)
at sun.nio.ch.IOUtil.write(IOUtil.java:71)
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334)
at
org.apache.hadoop.net.SocketOutputStream$Writer.performIO(SocketOutputStream
.java:55)
at
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:143)
at
org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:146)
at
org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:107)
at java.io.BufferedOutputStream.write(BufferedOutputStream.java:105)
at java.io.DataOutputStream.write(DataOutputStream.java:90)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.
java:2904)
namenode log:
2010-12-16 09:39:04,859 INFO FSNamesystem.audit
(FSNamesystem.java:logAuditEvent(188)) - ugi=root,root,sfcb ip=/167.6.5.13
cmd=create
src=/hbase/ufdr/compaction.dir/811978903/4122372667779754772
perm=root:supergroup:rw-r--r--
2010-12-16 09:39:04,865 INFO hdfs.StateChange
(FSNamesystem.java:allocateBlock(1627)) - BLOCK* NameSystem.allocateBlock:
/hbase/ufdr/compaction.dir/811978903/4122372667779754772.
blk_1292432378799_352605
2010-12-16 09:39:05,084 INFO FSNamesystem.audit
(FSNamesystem.java:logAuditEvent(202)) - ugi=root,root,sfcb ip=/167.6.5.14
cmd=delete
src=/hbase/.logs/c3s5.site,60020,1292432509342/hlog.dat.1292463618818
2010-12-16 09:39:05,098 INFO namenode.NameNode
(NameNode.java:errorReport(811)) - Error report from 167.6.5.14:50010:
DataNode failed volumes:/hdfsdata/1/current;
2010-12-16 09:39:05,099 WARN namenode.NameNode
(NameNode.java:errorReport(817)) - Volume failed on 167.6.5.14:50010
2010-12-16 09:39:05,464 INFO hdfs.StateChange
(FSNamesystem.java:allocateBlock(1627)) - BLOCK* NameSystem.allocateBlock:
/hbase/ufdr/565226849/value/9091457825705166400. blk_1292432378800_352605
2010-12-16 09:39:05,481 INFO namenode.FSNamesystem
(FSNamesystem.java:nextGenerationStampForBlock(5058)) -
blk_1292432378630_352428 is already commited, storedBlock == null.
2010-12-16 09:39:05,481 INFO ipc.Server (Server.java:run(1000)) - IPC
Server handler 1 on 9000, call nextGenerationStamp(blk_1292432378630_352428)
from 167.6.5.11:37392: error:
java.io.IOException: blk_1292432378630_352428 is already commited,
storedBlock == null.
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.nextGenerationStampForBl
ock(FSNamesystem.java:5059)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.nextGenerationStamp(NameNode
.java:540)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39
)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl
.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:514)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:990)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:986)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:984)
2010-12-16 09:39:05,801 INFO hdfs.StateChange
(FSNamesystem.java:addStoredBlock(3326)) - BLOCK* NameSystem.addStoredBlock:
blockMap updated: 167.6.5.12:50010 is added to blk_1292432378796_352603 size
42162794
2010-12-16 09:39:05,802 INFO hdfs.StateChange
(FSNamesystem.java:addStoredBlock(3326)) - BLOCK* NameSystem.addStoredBlock:
blockMap updated: 167.6.5.11:50010 is added to blk_1292432378796_352603 size
42162794
2010-12-16 09:39:05,804 INFO hdfs.StateChange
(FSNamesystem.java:addStoredBlock(3326)) - BLOCK* NameSystem.addStoredBlock:
blockMap updated: 167.6.5.17:50010 is added to blk_1292432378796_352603 size
42162794
2010-12-16 09:39:05,804 INFO hdfs.StateChange
(FSNamesystem.java:completeFileInternal(1576)) - DIR*
NameSystem.completeFile: file
/hbase/ufdr/326058079/value/1061306473103433726 is closed by
DFSClient_-1318892215
2010-12-16 09:39:05,809 INFO FSNamesystem.audit
(FSNamesystem.java:logAuditEvent(188)) - ugi=root,root,sfcb ip=/167.6.5.17
cmd=open src=/hbase/ufdr/326058079/value/1061306473103433726
perm=root:supergroup:rw-r--r--
2010-12-16 09:39:05,813 INFO hdfs.StateChange
(FSNamesystem.java:addStoredBlock(3326)) - BLOCK* NameSystem.addStoredBlock:
blockMap updated: 167.6.5.15:50010 is added to blk_1292432378788_352600 size
5756003
2010-12-16 09:39:05,815 INFO hdfs.StateChange
(FSNamesystem.java:addStoredBlock(3326)) - BLOCK* NameSystem.addStoredBlock:
blockMap updated: 167.6.5.12:50010 is added to blk_1292432378788_352600 size
5756003
2010-12-16 09:39:05,815 INFO hdfs.StateChange
(FSNamesystem.java:completeFileInternal(1576)) - DIR*
NameSystem.completeFile: file /hbase/ufdr/73697944/value/278352731434344857
is closed by DFSClient_-961611556
2010-12-16 09:39:05,825 INFO FSNamesystem.audit
(FSNamesystem.java:logAuditEvent(188)) - ugi=root,root,sfcb ip=/167.6.5.12
cmd=open src=/hbase/ufdr/73697944/value/278352731434344857
perm=root:supergroup:rw-r--r--
2010-12-16 09:39:05,861 INFO namenode.FSNamesystem
(FSNamesystem.java:nextGenerationStampForBlock(5058)) -
blk_1292432378630_352428 is already commited, storedBlock == null.
2010-12-16 09:39:05,861 INFO ipc.Server (Server.java:run(1000)) - IPC
Server handler 6 on 9000, call nextGenerationStamp(blk_1292432378630_352428)
from 167.6.5.11:37392: error:
java.io.IOException: blk_1292432378630_352428 is already commited,
storedBlock == null.
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.nextGenerationStampForBl
ock(FSNamesystem.java:5059)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.nextGenerationStamp(NameNode
.java:540)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39
)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl
.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:514)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:990)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:986)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:984)
2010-12-16 09:39:05,883 INFO FSNamesystem.audit
(FSNamesystem.java:logAuditEvent(188)) - ugi=root,root,sfcb ip=/167.6.5.17
cmd=create src=/hbase/ufdr/238795490/value/543132150737023964
perm=root:supergroup:rw-r--r--
2010-12-16 09:39:05,893 INFO hdfs.StateChange
(FSNamesystem.java:allocateBlock(1627)) - BLOCK* NameSystem.allocateBlock:
/hbase/ufdr/238795490/value/543132150737023964. blk_1292432378801_352606
2010-12-16 09:39:05,903 INFO hdfs.StateChange
(FSNamesystem.java:addStoredBlock(3326)) - BLOCK* NameSystem.addStoredBlock:
blockMap updated: 167.6.5.11:50010 is added to blk_1292432378789_352600 size
6049189
2010-12-16 09:39:05,904 INFO hdfs.StateChange
(FSNamesystem.java:addStoredBlock(3326)) - BLOCK* NameSystem.addStoredBlock:
blockMap updated: 167.6.5.12:50010 is added to blk_1292432378789_352600 size
6049189
Data node log:
2010-12-16 08:25:32,242 INFO DataNode.clienttrace
(BlockSender.java:sendBlock(487)) - src: /167.6.5.14:50010, dest:
/167.6.5.14:36758, bytes: 3138, op: HDFS_READ, cliID: DFSClient_708626937,
srvID: DS-814382105-167.6.5.14-50010-1291837985732, blockid:
blk_1292432361424_333765
2010-12-16 08:25:32,426 WARN datanode.DataNode
(DataXceiver.java:readBlock(276)) - DatanodeRegistration(167.6.5.14:50010,
storageID=DS-814382105-167.6.5.14-50010-1291837985732, infoPort=50075,
ipcPort=50020):Got exception while serving blk_1292432361541_333880 to
/167.6.5.14:
java.io.IOException: Block blk_1292432361541_333880 is not valid.
at
org.apache.hadoop.hdfs.server.datanode.FSDataset.getBlockFile(FSDataset.java
:962)
at
org.apache.hadoop.hdfs.server.datanode.FSDataset.getLength(FSDataset.java:94
9)
at
org.apache.hadoop.hdfs.server.datanode.BlockSender.<init>(BlockSender.java:1
02)
at
org.apache.hadoop.hdfs.server.datanode.DataXceiver.readBlock(DataXceiver.jav
a:240)
at
org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:132)
at java.lang.Thread.run(Thread.java:662)
2010-12-16 08:25:32,427 ERROR datanode.DataNode (DataXceiver.java:run(185))
- org.apache.hadoop.hdfs.server.datanode.DataNode (167.6.5.14:50010,
storageID=DS-814382105-167.6.5.14-50010-1291837985732, infoPort=50075,
ipcPort=50020):DataXceiver
java.io.IOException: Block blk_1292432361541_333880 is not valid.
at
org.apache.hadoop.hdfs.server.datanode.FSDataset.getBlockFile(FSDataset.java
:962)
at
org.apache.hadoop.hdfs.server.datanode.FSDataset.getLength(FSDataset.java:94
9)
at
org.apache.hadoop.hdfs.server.datanode.BlockSender.<init>(BlockSender.java:1
02)
at
org.apache.hadoop.hdfs.server.datanode.DataXceiver.readBlock(DataXceiver.jav
a:240)
at
org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:132)
at java.lang.Thread.run(Thread.java:662)
2010-12-16 08:25:33,419 INFO datanode.DataNode
(DataXceiver.java:writeBlock(307)) - Receiving block
blk_1292432365462_338068 src: /167.6.5.14:36774 dest: /167.6.5.14:50010
2010-12-16 08:25:33,427 INFO datanode.DataNode
(FSDataset.java:checkDirs(754)) - Completed FSVolumeSet.checkDirs.
Removed=0volumes. List of current volumes:
/hdfsdata/0/current,/hdfsdata/1/current,/hdfsdata/2/current,/hdfsdata/3/curr
ent,/hdfsdata/4/current,/hdfsdata/5/current,/hdfsdata/6/current
2010-12-16 08:25:33,489 INFO datanode.DataNode
(DataXceiver.java:writeBlock(307)) - Receiving block
blk_1292432365464_338069 src: /167.6.5.17:51536 dest: /167.6.5.14:50010
----------------------------------------------------------------------------
---------------------------------------------------------
This e-mail and its attachments contain confidential information from
HUAWEI, which
is intended only for the person or entity whose address is listed above. Any
use of the
information contained herein in any way (including, but not limited to,
total or partial
disclosure, reproduction, or dissemination) by persons other than the
intended
recipient(s) is prohibited. If you receive this e-mail in error, please
notify the sender by
phone or email immediately and delete it!
