Re: problems with hadoop-0.20.205.0
Hi Harsh, Here is the dtatnode log file - 2011-11-28 12:33:53,584 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: STARTUP_MSG: / STARTUP_MSG: Starting DataNode STARTUP_MSG: host = ubuntu/127.0.1.1 STARTUP_MSG: args = [] STARTUP_MSG: version = 0.20.205.0 STARTUP_MSG: build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20-security-205 -r 1179940; compiled by 'hortonfo' on Fri Oct 7 06:20:32 UTC 2011 / 2011-11-28 12:33:53,685 INFO org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from hadoop-metrics2.properties 2011-11-28 12:33:53,693 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source MetricsSystem,sub=Stats registered. 2011-11-28 12:33:53,693 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s). 2011-11-28 12:33:53,693 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: DataNode metrics system started 2011-11-28 12:33:53,792 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source ugi registered. 2011-11-28 12:33:53,795 WARN org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Source name ugi already exists! 2011-11-28 12:33:59,659 INFO org.apache.hadoop.hdfs.server.common.Storage: Storage directory /home/solr/hadoop-0.20.205.0/file:/home/solr/hdfs/data does not exist. 2011-11-28 12:33:59,660 INFO org.apache.hadoop.hdfs.server.common.Storage: Storage directory file:/home/solr/hdfs/data does not exist. 2011-11-28 12:33:59,821 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: java.io.IOException: All specified directories are not accessible or do not exist. at org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:139) at org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:367) at org.apache.hadoop.hdfs.server.datanode.DataNode.(DataNode.java:281) at org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1545) at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1484) at org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:1502) at org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:1628) at org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:1645) 2011-11-28 12:33:59,822 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: SHUTDOWN_MSG: / SHUTDOWN_MSG: Shutting down DataNode at ubuntu/127.0.1.1 / The namenode is working fine now, but the datanode problem still persists. Regards, Mohammad Tariq On Sun, Nov 27, 2011 at 7:53 PM, Harsh J wrote: > Perhaps you need to carry out an upgrade of DFS before you start it > normally. Could you share what error you're seeing when you start your > namenode (from log files, on cli, etc.)? > > On Sun, Nov 27, 2011 at 6:40 PM, Mohammad Tariq wrote: >> I am able to work properly when using 0.20.203.0, but facing problem >> with 0.20.205.0..The namenode and datanode are not getting started.Is >> there any change in the command syntax or in the configuration??? >> >> Regards, >> Mohammad Tariq >> > > > > -- > Harsh J >
how to create my own applications to test HDFS after fault injection
Hi, all: I am a graduate student, now working on HDFS fault injection course project. I found FI framework a good tool to use, but I don't know how to create my own tests after "ant injectfaults" and "ant jar-fault-inject". It seems that only default tests can be used after fault injection. I wish to know whether I can develop my own applications to test the injected HDFS. For instance, if I create data corruption in datanodes and write a related BlockReceiver.aj, how can I create my own test (e.g one client writes files to the server and the files get corrupted. if the client fetch the file again it will be noticed)? Thank you very much for your time. Best regards Hao Yang
RE: Clarification on federated HDFS
Hey Shesha, In Fedatated HDFS, same DataNodes can work with multiple NameNode. Where as in your setup, complete cluster itself is different. I would suggest you to take a look at HDFS-2471, Suresh has explained very neet and breifly here. Regards, Uma From: Sesha Kumar [sesha...@gmail.com] Sent: Monday, November 28, 2011 9:05 AM To: hdfs-user@hadoop.apache.org Subject: Clarification on federated HDFS Hi guys, Is Federated HDFS same as having a set of individual Hadoop Clusters, each managing its own namespace and knowing nothing about the existence of other clusters (but having separate set of data nodes instead of a common pool)? Assuming we have something similar to ViewFS which provides a single global namespace combining the namespaces of each namenode and it also provides some facility to add new namenodes to the setup. What are the differences between Federated HDFS and the above given setup?
Re: Analyzing the performance of HDFS
this is a good reference http://www.michael-noll.com/blog/2011/04/09/benchmarking-and-stress-testing-an-hadoop-cluster-with-terasort-testdfsio-nnbench-mrbench/#namenode-benchmark-nnbench On Sun, Nov 27, 2011 at 9:39 PM, Sesha Kumar wrote: > Hi Guys, > > How should the performance of HDFS be analyzed? Are there any tools > available? What should be the minimum size of the cluster to get a reliable > estimate of the performance? I am currently working on a three machine > cluster. >
Analyzing the performance of HDFS
Hi Guys, How should the performance of HDFS be analyzed? Are there any tools available? What should be the minimum size of the cluster to get a reliable estimate of the performance? I am currently working on a three machine cluster.
Clarification on federated HDFS
Hi guys, Is Federated HDFS same as having a set of individual Hadoop Clusters, each managing its own namespace and knowing nothing about the existence of other clusters (but having separate set of data nodes instead of a common pool)? Assuming we have something similar to ViewFS which provides a single global namespace combining the namespaces of each namenode and it also provides some facility to add new namenodes to the setup. What are the differences between Federated HDFS and the above given setup?
Re: problems with hadoop-0.20.205.0
Perhaps you need to carry out an upgrade of DFS before you start it normally. Could you share what error you're seeing when you start your namenode (from log files, on cli, etc.)? On Sun, Nov 27, 2011 at 6:40 PM, Mohammad Tariq wrote: > I am able to work properly when using 0.20.203.0, but facing problem > with 0.20.205.0..The namenode and datanode are not getting started.Is > there any change in the command syntax or in the configuration??? > > Regards, > Mohammad Tariq > -- Harsh J
problems with hadoop-0.20.205.0
I am able to work properly when using 0.20.203.0, but facing problem with 0.20.205.0..The namenode and datanode are not getting started.Is there any change in the command syntax or in the configuration??? Regards, Mohammad Tariq
