RE: Error while configuring HDFS fedration
Hi, Thanks for your inputs. I fixed the issue. Thanks, Manickam P From: el...@mellanox.com To: user@hadoop.apache.org Subject: RE: Error while configuring HDFS fedration Date: Mon, 23 Sep 2013 14:05:47 + Ports in use may result from actual processes using them, or just ghost processes. The second error may be caused by inconsistent permissions on different nodes, and/or a format is needed on DFS. I suggest the following: 1. sbin/stop-dfs.sh && sbin/stop-yarn.sh 2. sudo killall java (on all nodes) 3. sudo chmod –R 755 /home/lab/hadoop-2.1.0-beta/tmp/dfs (on all nodes) 4. sudo rm –rf /home/lab/hadoop-2.1.0-beta/tmp/dfs/* (on all nodes) 5. bin/hdfs namenode –format –force 6. sbin/start-dfs.sh && sbin/start-yarn.sh Then see if you get that error again. From: Manickam P [mailto:manicka...@outlook.com] Sent: Monday, September 23, 2013 4:44 PM To: user@hadoop.apache.org Subject: Error while configuring HDFS fedration Guys, I'm trying to configure HDFS federation with 2.1.0 beta version. I am having 3 machines in that i want to have two name nodes and one data node. I have done the other thing like password less ssh and host entries properly. when i start the cluster i'm getting the below error. In node one i'm getting this error. java.net.BindException: Port in use: lab-hadoop.eng.com:50070 In another node i'm getting this error. org.apache.hadoop.hdfs.server.common.InconsistentFSStateException: Directory /home/lab/hadoop-2.1.0-beta/tmp/dfs/name is in an inconsistent state: storage directory does not exist or is not accessible. My core-site xml has the below. fs.default.name hdfs://10.101.89.68:9000 hadoop.tmp.dir /home/lab/hadoop-2.1.0-beta/tmp My hdfs-site xml has the below. dfs.replication 2 dfs.permissions false dfs.federation.nameservices ns1,ns2 dfs.namenode.rpc-address.ns1 10.101.89.68:9001 dfs.namenode.http-address.ns1 10.101.89.68:50070 dfs.namenode.secondary.http-address.ns1 10.101.89.68:50090 dfs.namenode.rpc-address.ns2 10.101.89.69:9001 dfs.namenode.http-address.ns2 10.101.89.69:50070 dfs.namenode.secondary.http-address.ns2 10.101.89.69:50090 Please help me to fix this error. Thanks, Manickam P
RE: Error while configuring HDFS fedration
Hi, I followed your steps. That bind error got resolved but still i'm getting the second exception. I've given the complete stack below. 2013-09-23 10:26:01,887 INFO org.mortbay.log: Stopped selectchannelconnec...@lab2-hadoop2-vm1.eng.com:50070 2013-09-23 10:26:01,988 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode metrics system... 2013-09-23 10:26:01,989 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system stopped. 2013-09-23 10:26:01,990 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system shutdown complete. 2013-09-23 10:26:01,991 FATAL org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode join org.apache.hadoop.hdfs.server.common.InconsistentFSStateException: Directory /home/lab/hadoop-2.1.0-beta/tmp/dfs/name is in an inconsistent state: storage directory does not exist or is not accessible. at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverStorageDirs(FSImage.java:292) at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:200) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:777) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:558) at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:418) at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:466) at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:659) at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:644) at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1221) at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1287) 2013-09-23 10:26:02,001 INFO org.apache.hadoop.util.ExitUtil: Exiting with status 1 2013-09-23 10:26:02,018 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG: Thanks, Manickam P From: el...@mellanox.com To: user@hadoop.apache.org Subject: RE: Error while configuring HDFS fedration Date: Mon, 23 Sep 2013 14:05:47 + Ports in use may result from actual processes using them, or just ghost processes. The second error may be caused by inconsistent permissions on different nodes, and/or a format is needed on DFS. I suggest the following: 1. sbin/stop-dfs.sh && sbin/stop-yarn.sh 2. sudo killall java (on all nodes) 3. sudo chmod –R 755 /home/lab/hadoop-2.1.0-beta/tmp/dfs (on all nodes) 4. sudo rm –rf /home/lab/hadoop-2.1.0-beta/tmp/dfs/* (on all nodes) 5. bin/hdfs namenode –format –force 6. sbin/start-dfs.sh && sbin/start-yarn.sh Then see if you get that error again. From: Manickam P [mailto:manicka...@outlook.com] Sent: Monday, September 23, 2013 4:44 PM To: user@hadoop.apache.org Subject: Error while configuring HDFS fedration Guys, I'm trying to configure HDFS federation with 2.1.0 beta version. I am having 3 machines in that i want to have two name nodes and one data node. I have done the other thing like password less ssh and host entries properly. when i start the cluster i'm getting the below error. In node one i'm getting this error. java.net.BindException: Port in use: lab-hadoop.eng.com:50070 In another node i'm getting this error. org.apache.hadoop.hdfs.server.common.InconsistentFSStateException: Directory /home/lab/hadoop-2.1.0-beta/tmp/dfs/name is in an inconsistent state: storage directory does not exist or is not accessible. My core-site xml has the below. fs.default.name hdfs://10.101.89.68:9000 hadoop.tmp.dir /home/lab/hadoop-2.1.0-beta/tmp My hdfs-site xml has the below. dfs.replication 2 dfs.permissions false dfs.federation.nameservices ns1,ns2 dfs.namenode.rpc-address.ns1 10.101.89.68:9001 dfs.namenode.http-address.ns1 10.101.89.68:50070 dfs.namenode.secondary.http-address.ns1 10.101.89.68:50090 dfs.namenode.rpc-address.ns2 10.101.89.69:9001 dfs.namenode.http-address.ns2 10.101.89.69:50070 dfs.namenode.secondary.http-address.ns2 10.101.89.69:50090 Please help me to fix this error. Thanks, Manickam P
RE: Error while configuring HDFS fedration
Ports in use may result from actual processes using them, or just ghost processes. The second error may be caused by inconsistent permissions on different nodes, and/or a format is needed on DFS. I suggest the following: 1. sbin/stop-dfs.sh && sbin/stop-yarn.sh 2. sudo killall java (on all nodes) 3. sudo chmod -R 755 /home/lab/hadoop-2.1.0-beta/tmp/dfs (on all nodes) 4. sudo rm -rf /home/lab/hadoop-2.1.0-beta/tmp/dfs/* (on all nodes) 5. bin/hdfs namenode -format -force 6. sbin/start-dfs.sh && sbin/start-yarn.sh Then see if you get that error again. From: Manickam P [mailto:manicka...@outlook.com] Sent: Monday, September 23, 2013 4:44 PM To: user@hadoop.apache.org Subject: Error while configuring HDFS fedration Guys, I'm trying to configure HDFS federation with 2.1.0 beta version. I am having 3 machines in that i want to have two name nodes and one data node. I have done the other thing like password less ssh and host entries properly. when i start the cluster i'm getting the below error. In node one i'm getting this error. java.net.BindException: Port in use: lab-hadoop.eng.com:50070 In another node i'm getting this error. org.apache.hadoop.hdfs.server.common.InconsistentFSStateException: Directory /home/lab/hadoop-2.1.0-beta/tmp/dfs/name is in an inconsistent state: storage directory does not exist or is not accessible. My core-site xml has the below. fs.default.name hdfs://10.101.89.68:9000 hadoop.tmp.dir /home/lab/hadoop-2.1.0-beta/tmp My hdfs-site xml has the below. dfs.replication 2 dfs.permissions false dfs.federation.nameservices ns1,ns2 dfs.namenode.rpc-address.ns1 10.101.89.68:9001 dfs.namenode.http-address.ns1 10.101.89.68:50070 dfs.namenode.secondary.http-address.ns1 10.101.89.68:50090 dfs.namenode.rpc-address.ns2 10.101.89.69:9001 dfs.namenode.http-address.ns2 10.101.89.69:50070 dfs.namenode.secondary.http-address.ns2 10.101.89.69:50090 Please help me to fix this error. Thanks, Manickam P
Error while configuring HDFS fedration
Guys, I'm trying to configure HDFS federation with 2.1.0 beta version. I am having 3 machines in that i want to have two name nodes and one data node. I have done the other thing like password less ssh and host entries properly. when i start the cluster i'm getting the below error. In node one i'm getting this error. java.net.BindException: Port in use: lab-hadoop.eng.com:50070 In another node i'm getting this error. org.apache.hadoop.hdfs.server.common.InconsistentFSStateException: Directory /home/lab/hadoop-2.1.0-beta/tmp/dfs/name is in an inconsistent state: storage directory does not exist or is not accessible. My core-site xml has the below. fs.default.name hdfs://10.101.89.68:9000 hadoop.tmp.dir /home/lab/hadoop-2.1.0-beta/tmp My hdfs-site xml has the below. dfs.replication 2 dfs.permissions false dfs.federation.nameservices ns1,ns2 dfs.namenode.rpc-address.ns1 10.101.89.68:9001 dfs.namenode.http-address.ns1 10.101.89.68:50070 dfs.namenode.secondary.http-address.ns1 10.101.89.68:50090 dfs.namenode.rpc-address.ns2 10.101.89.69:9001 dfs.namenode.http-address.ns2 10.101.89.69:50070 dfs.namenode.secondary.http-address.ns2 10.101.89.69:50090 Please help me to fix this error. Thanks, Manickam P
