Hi Subroto, Anything in the DN logs? Maybe the one DN is out of disk space? You need to have 5*BLOCK_SIZE free.
-Todd On Tue, Jun 19, 2012 at 1:29 AM, Subroto <ssan...@datameer.com> wrote: > Hi, > > When DFS cluster; > The datanodes keep on dumping the logs: > 2012-06-19 04:05:04,842 INFO > org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Adding > block pool BP-1233167859-10.10.35.8-1340093005960 > 2012-06-19 04:05:04,861 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: > Block pool BP-1233167859-10.10.35.8-1340093005960 (storage id > DS-1255092928-10.244.15.192-50010-1340093104449) service to > ec2-107-22-39-220.compute-1.amazonaws.com/10.10.35.8:9000 beginning handshake > with NN > 2012-06-19 04:05:04,979 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: > Block pool Block pool BP-1233167859-10.10.35.8-1340093005960 (storage id > DS-1255092928-10.244.15.192-50010-1340093104449) service to > ec2-107-22-39-220.compute-1.amazonaws.com/10.10.35.8:9000 successfully > registered with NN > 2012-06-19 04:05:04,980 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: > For namenode ec2-107-22-39-220.compute-1.amazonaws.com/10.10.35.8:9000 using > DELETEREPORT_INTERVAL of 300000 msec BLOCKREPORT_INTERVAL of 21600000msec > Initial delay: 0msec; heartBeatInterval=3000 > 2012-06-19 04:05:05,258 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: > Namenode Block pool BP-1233167859-10.10.35.8-1340093005960 (storage id > DS-1255092928-10.244.15.192-50010-1340093104449) service to > ec2-107-22-39-220.compute-1.amazonaws.com/10.10.35.8:9000 trying to claim > ACTIVE state with txid=24 > 2012-06-19 04:05:05,258 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: > Acknowledging ACTIVE Namenode Block pool > BP-1233167859-10.10.35.8-1340093005960 (storage id > DS-1255092928-10.244.15.192-50010-1340093104449) service to > ec2-107-22-39-220.compute-1.amazonaws.com/10.10.35.8:9000 > 2012-06-19 04:05:05,394 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: > BlockReport of 0 blocks took 1 msec to generate and 135 msecs for RPC and NN > processing > 2012-06-19 04:05:05,394 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: > sent block report, processed > command:org.apache.hadoop.hdfs.server.protocol.FinalizeCommand@49d31859 > 2012-06-19 04:05:05,396 INFO > org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Periodic Block > Verification Scanner initialized with interval 504 hours for block pool > BP-1233167859-10.10.35.8-1340093005960. > 2012-06-19 04:05:05,400 INFO > org.apache.hadoop.hdfs.server.datanode.DataBlockScanner: Added > bpid=BP-1233167859-10.10.35.8-1340093005960 to blockPoolScannerMap, new size=1 > 2012-06-19 04:05:09,845 INFO > org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Starting a new > period : work left in prev period : 0.00% > 2012-06-19 04:05:14,848 INFO > org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Starting a new > period : work left in prev period : 0.00% > 2012-06-19 04:05:19,849 INFO > org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Starting a new > period : work left in prev period : 0.00% > 2012-06-19 04:05:24,850 INFO > org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Starting a new > period : work left in prev period : 0.00% > 2012-06-19 04:05:29,851 INFO > org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Starting a new > period : work left in prev period : 0.00% > 2012-06-19 04:05:34,852 INFO > org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Starting a new > period : work left in prev period : 0.00% > 2012-06-19 04:05:39,853 INFO > org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Starting a new > period : work left in prev period : 0.00% > 2012-06-19 04:05:44,854 INFO > org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Starting a new > period : work left in prev period : 0.00% > 2012-06-19 04:05:49,855 INFO > org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Starting a new > period : work left in prev period : 0.00% > > The DFSClient operations are also not taking place properly: > 2012-06-19 04:04:56,653 WARN org.apache.hadoop.hdfs.DFSClient: DataStreamer > Exception > java.io.IOException: File /hbase/hbase.version could only be replicated to 0 > nodes instead of minReplication (=1). There are 1 datanode(s) running and no > node(s) are excluded in this operation. > at > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget(BlockManager.java:1256) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1977) > at > org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:470) > at > org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:292) > at > org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java:42602) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:427) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:916) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1692) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1688) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:396) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232) > > Please let me know if there is any configuration which needs to be set….. > The hadoop version being used is: 2.0.0 > > > > Cheers, > Subroto Sanyal -- Todd Lipcon Software Engineer, Cloudera
