AvatarDataNode Error
Hi,all. When start a AvatarDataNode,show error below: 2012-02-14 17:33:50,719 ERROR org.apache.hadoop.hdfs.server.datanode.AvatarDataNode: java.lang.IllegalArgumentException: not a proxy instance at java.lang.reflect.Proxy.getInvocationHandler(Proxy.java:637) at org.apache.hadoop.ipc.RPC.stopProxy(RPC.java:393) at org.apache.hadoop.hdfs.server.datanode.DataNode.shutdown(DataNode.java:603) at org.apache.hadoop.hdfs.server.datanode.AvatarDataNode.shutdown(AvatarDataNode.java:576) at org.apache.hadoop.hdfs.server.datanode.DataNode.(DataNode.java:218) at org.apache.hadoop.hdfs.server.datanode.AvatarDataNode.(AvatarDataNode.java:119) at org.apache.hadoop.hdfs.server.datanode.AvatarDataNode.makeInstance(AvatarDataNode.java:691) at org.apache.hadoop.hdfs.server.datanode.AvatarDataNode.instantiateDataNode(AvatarDataNode.java:715) at org.apache.hadoop.hdfs.server.datanode.AvatarDataNode.createDataNode(AvatarDataNode.java:720) at org.apache.hadoop.hdfs.server.datanode.AvatarDataNode.main(AvatarDataNode.java:728) Does anybody know the reason? Thank you, Bourne
Re: Re: DN limit
Hi, The replica of block is 1. Threre is 150million block in NN web UI. Bourne 发件人: Harsh J 发送时间: 2011年12月24日(星期六) 下午2:09 收件人: common-user 主题: Re: Re: DN limit Bourne, You have 14 million files, each taking up a single block or are these files multi-blocked? What does the block count come up as in the live nodes list of the NN web UI? 2011/12/23 bourne1900 : > Sorry, a detailed description: > I wanna know how many files a datanode can hold, so there is only one > datanode in my cluster. > When the datanode save 14million files, the cluster can't work, and the > datanode has used all of it's MEM(32G), the namenode's MEM is OK. > > > > > Bourne > > Sender: Adrian Liu > Date: 2011年12月23日(星期五) 上午10:47 > To: common-user@hadoop.apache.org > Subject: Re: DN limit > In my understanding, the max number of files stored in the HDFS should be > /sizeof(inode struct). This max number of HDFS files > should be no smaller than max files a datanode can hold. > > Please feel free to correct me because I'm just beginning learning hadoop. > > On Dec 23, 2011, at 10:35 AM, bourne1900 wrote: > >> Hi all, >> How many files a datanode can hold? >> In my test case, when a datanode save 14million files, the cluster can't >> work. >> >> >> >> >> Bourne > > Adrian Liu > adri...@yahoo-inc.com -- Harsh J
Re: Re: DN limit
Sorry, a detailed description: I wanna know how many files a datanode can hold, so there is only one datanode in my cluster. When the datanode save 14million files, the cluster can't work, and the datanode has used all of it's MEM(32G), the namenode's MEM is OK. Bourne Sender: Adrian Liu Date: 2011年12月23日(星期五) 上午10:47 To: common-user@hadoop.apache.org Subject: Re: DN limit In my understanding, the max number of files stored in the HDFS should be /sizeof(inode struct). This max number of HDFS files should be no smaller than max files a datanode can hold. Please feel free to correct me because I'm just beginning learning hadoop. On Dec 23, 2011, at 10:35 AM, bourne1900 wrote: > Hi all, > How many files a datanode can hold? > In my test case, when a datanode save 14million files, the cluster can't work. > > > > > Bourne Adrian Liu adri...@yahoo-inc.com
Re: Re: could not complete file...
Thank you for your reply. There is "PIPE ERROR" in datanode log, and nothing else. Client only shows "Could not complete file" ceaselessly. From "namonodeIP:50070/dfshealth.jsp ", I found the datanode is hang-up, and there is only a datanode in my cluster :) BTW, the retry times is unlimit I think, my hadoop version is 0.20.2, the DataNode.java is while (!fileComplete) { fileComplete = namenode.complete(src, clientName); if (!fileComplete) { try { Thread.sleep(400); if (System.currentTimeMillis() - localstart > 5000) { LOG.info("Could not complete file " + src + " retrying..."); } } catch (InterruptedException ie) { } } } bourne1900 Sender: Uma Maheswara Rao G 72686 Date: 2011年10月18日(星期二) 下午6:00 To: common-user CC: common-user Subject: Re: could not complete file... - Original Message - From: bourne1900 Date: Tuesday, October 18, 2011 3:21 pm Subject: could not complete file... To: common-user > Hi, > > There are 20 threads which put file into HDFS ceaseless, every > file is 2k. > When 1 million files have finished, client begin throw "coulod not > complete file" exception ceaseless. Could not complete file log is actually info log. This will be logged from client when closing the file. It will retry for some time (i remember 100 times) to ensure the suuceefull writes. Did you observe any write failures here? > At that time, datanode is hang-up. > > I think maybe heart beat is lost, so namenode does not know the > state of datanode. But I do not know why heart beat have lost. Is > there any info can be found from log when datanode can not send > heart beat? Can you check the NN UI to verify the number of live nodes. By this we can decide whether DN stopped sending heartbeats or not. > > Thanks and regards! > bourne Regards, Uma
could not complete file...
Hi, There are 20 threads which put file into HDFS ceaseless, every file is 2k. When 1 million files have finished, client begin throw "coulod not complete file" exception ceaseless. At that time, datanode is hang-up. I think maybe heart beat is lost, so namenode does not know the state of datanode. But I do not know why heart beat have lost. Is there any info can be found from log when datanode can not send heart beat? Thanks and regards! bourne