@ Ravi. Not able to do that.
On Thu, Jun 4, 2009 at 5:38 PM, Raghu Angadi <[email protected]> wrote: > > Did you try 'telnet 198.55.35.229 54310' from this datanode? The log show > that it is not able to connect to "master:54310". ssh from datanode does not > matter. > > Raghu. > > asif md wrote: > >> I can SSH both ways .i.e. From master to slave and slave to master. >> >> the datanode is getting intialized at master but the log at slave looks >> like >> this >> >> ************************************************************/ >> 2009-06-04 15:20:06,066 INFO org.apache.hadoop.dfs.DataNode: STARTUP_MSG: >> /************************************************************ >> STARTUP_MSG: Starting DataNode >> STARTUP_MSG: host = ************ >> STARTUP_MSG: args = [] >> STARTUP_MSG: version = 0.18.3 >> STARTUP_MSG: build = >> https://svn.apache.org/repos/asf/hadoop/core/branches/branch-0.18 -r >> 736250; >> compiled by 'ndaley' on Thu Jan 22 23:12:08 UTC 2009 >> ************************************************************/ >> 2009-06-04 15:20:08,826 INFO org.apache.hadoop.ipc.Client: Retrying >> connect >> to server: master/198.55.35.229:54310. Already tried 0 time(s). >> 2009-06-04 15:20:09,829 INFO org.apache.hadoop.ipc.Client: Retrying >> connect >> to server: master/198.55.35.229:54310. Already tried 1 time(s). >> 2009-06-04 15:20:10,831 INFO org.apache.hadoop.ipc.Client: Retrying >> connect >> to server: master/198.55.35.229:54310. Already tried 2 time(s). >> 2009-06-04 15:20:11,832 INFO org.apache.hadoop.ipc.Client: Retrying >> connect >> to server: master/198.55.35.229:54310. Already tried 3 time(s). >> 2009-06-04 15:20:12,834 INFO org.apache.hadoop.ipc.Client: Retrying >> connect >> to server: master/198.55.35.229:54310. Already tried 4 time(s). >> 2009-06-04 15:20:13,837 INFO org.apache.hadoop.ipc.Client: Retrying >> connect >> to server: master/198.55.35.229:54310. Already tried 5 time(s). >> 2009-06-04 15:20:14,840 INFO org.apache.hadoop.ipc.Client: Retrying >> connect >> to server: master/198.55.35.229:54310. Already tried 6 time(s). >> 2009-06-04 15:20:15,841 INFO org.apache.hadoop.ipc.Client: Retrying >> connect >> to server: master/198.55.35.229:54310. Already tried 7 time(s). >> 2009-06-04 15:20:16,844 INFO org.apache.hadoop.ipc.Client: Retrying >> connect >> to server: master/198.55.35.229:54310. Already tried 8 time(s). >> 2009-06-04 15:20:17,847 INFO org.apache.hadoop.ipc.Client: Retrying >> connect >> to server: master/198.55.35.229:54310. Already tried 9 time(s). >> 2009-06-04 15:20:17,873 ERROR org.apache.hadoop.dfs.DataNode: >> java.io.IOException: Call to master/198.55.35.229:54310 failed on local >> exception: java.net.NoRouteToHostException: No route to host >> at org.apache.hadoop.ipc.Client.wrapException(Client.java:751) >> at org.apache.hadoop.ipc.Client.call(Client.java:719) >> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:216) >> at org.apache.hadoop.dfs.$Proxy4.getProtocolVersion(Unknown Source) >> at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:348) >> at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:335) >> at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:372) >> at org.apache.hadoop.ipc.RPC.waitForProxy(RPC.java:309) >> at org.apache.hadoop.ipc.RPC.waitForProxy(RPC.java:286) >> at org.apache.hadoop.dfs.DataNode.startDataNode(DataNode.java:277) >> at org.apache.hadoop.dfs.DataNode.<init>(DataNode.java:223) >> at org.apache.hadoop.dfs.DataNode.makeInstance(DataNode.java:3071) >> at >> org.apache.hadoop.dfs.DataNode.instantiateDataNode(DataNode.java:3026) >> at org.apache.hadoop.dfs.DataNode.createDataNode(DataNode.java:3034) >> at org.apache.hadoop.dfs.DataNode.main(DataNode.java:3156) >> Caused by: java.net.NoRouteToHostException: No route to host >> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) >> at >> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574) >> at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:100) >> at >> org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:301) >> at org.apache.hadoop.ipc.Client$Connection.access$1700(Client.java:178) >> at org.apache.hadoop.ipc.Client.getConnection(Client.java:820) >> at org.apache.hadoop.ipc.Client.call(Client.java:705) >> ... 13 more >> >> 2009-06-04 15:20:17,874 INFO org.apache.hadoop.dfs.DataNode: SHUTDOWN_MSG: >> /************************************************************ >> SHUTDOWN_MSG: Shutting down DataNode at ******* >> >> **********************************************************************************88 >> >> Please suggest. >> >> Asif. >> >> >> On Thu, Jun 4, 2009 at 4:15 PM, asif md <[email protected]> wrote: >> >> @Ravi >>> >>> thanx ravi .. i'm now using my a definded tmp dir so the second issue is >>> resolved. >>> >>> But i have ssh keys tht have passwords. But i am able to ssh to the slave >>> and master from the master. >>> >>> should i be able to do tht from the slave as well. >>> >>> @ALL >>> >>> Any suggestions. >>> >>> Thanx >>> >>> Asif. >>> >>> >>> On Thu, Jun 4, 2009 at 3:17 PM, Ravi Phulari <[email protected] >>> >wrote: >>> >>> From logs looks like your Hadoop cluster is facing two different issues >>>> . >>>> >>>> At Slave >>>> >>>> 1. exception: java.net.NoRouteToHostException: No route to host in >>>> >>>> your logs >>>> >>>> Diagnosis - One of your nodes cannot be reached correctly. Make sure you >>>> can ssh to your master and slave and passwordless ssh keys are set . >>>> >>>> At master >>>> 2. java.io.IOException: Incompatible namespaceIDs in >>>> Diagnosis - Your Hadoop namespaceID became corrupted. Unfortunately the >>>> easiest thing to do reformat the HDFS >>>> >>>> As you have not configured hadoop.tmp.dir or data.dir by default Hadoop >>>> will you /tmp as directory for temporary files , log files and for data >>>> dir >>>> which is not good practice . >>>> I would suggest you using some tmpForHadoop dir somewhere else than >>>> /tmp >>>> . >>>> >>>> - >>>> Ravi >>>> >>>> On 6/4/09 12:39 PM, "asif md" <[email protected]> wrote: >>>> >>>> Hello all, >>>> >>>> I'm trying to setup a two node cluster < remote > using the following >>>> tutorials >>>> { NOTE : i'm ignoring the tmp directory property in hadoop-site.xml >>>> suggested by Michael } >>>> >>>> Running Hadoop On Ubuntu Linux (Single-Node Cluster) - Michael G. >>>> Noll< >>>> >>>> http://www.michael-noll.com/wiki/Running_Hadoop_On_Ubuntu_Linux_%28Single-Node_Cluster%29 >>>> Running Hadoop On Ubuntu Linux (Multi-Node Cluster) - Michael G. >>>> Noll< >>>> >>>> http://www.michael-noll.com/wiki/Running_Hadoop_On_Ubuntu_Linux_%28Multi-Node_Cluster%29 >>>> >>>> I get the following logs when try to run $HADOOP_HOME/bin/start-dfs.sh >>>> at >>>> master. >>>> >>>> >>>> >>>> *************************************************************************************************** >>>> AT MASTER : >>>> ----------------- >>>> 2009-06-04 12:16:30,864 INFO org.apache.hadoop.dfs.DataNode: >>>> STARTUP_MSG: >>>> /************************************************************ >>>> STARTUP_MSG: Starting DataNode >>>> STARTUP_MSG: host = ******* >>>> STARTUP_MSG: args = [] >>>> STARTUP_MSG: version = 0.18.3 >>>> STARTUP_MSG: build = >>>> https://svn.apache.org/repos/asf/hadoop/core/branches/branch-0.18 -r >>>> 736250; >>>> compiled by 'ndaley' on Thu Jan 22 23:12:08 UTC 2009 >>>> ************************************************************/ >>>> 2009-06-04 12:16:31,071 ERROR org.apache.hadoop.dfs.DataNode: >>>> java.io.IOException: Incompatible namespaceIDs in /tmp/*****/dfs/data: >>>> namenode namespaceID = 34351921; datanode namespaceID = 539590337 >>>> at >>>> org.apache.hadoop.dfs.DataStorage.doTransition(DataStorage.java:226) >>>> at >>>> >>>> >>>> org.apache.hadoop.dfs.DataStorage.recoverTransitionRead(DataStorage.java:141) >>>> at >>>> org.apache.hadoop.dfs.DataNode.startDataNode(DataNode.java:306) >>>> at org.apache.hadoop.dfs.DataNode.<init>(DataNode.java:223) >>>> at >>>> org.apache.hadoop.dfs.DataNode.makeInstance(DataNode.java:3071) >>>> at >>>> org.apache.hadoop.dfs.DataNode.instantiateDataNode(DataNode.java:3026) >>>> at >>>> org.apache.hadoop.dfs.DataNode.createDataNode(DataNode.java:3034) >>>> at org.apache.hadoop.dfs.DataNode.main(DataNode.java:3156) >>>> >>>> 2009-06-04 12:16:31,071 INFO org.apache.hadoop.dfs.DataNode: >>>> SHUTDOWN_MSG: >>>> /************************************************************ >>>> >>>> >>>> ******************************************************************************************************* >>>> AT SLAVE : >>>> ---------------- >>>> 2009-06-04 12:16:28,203 INFO org.apache.hadoop.dfs.DataNode: >>>> STARTUP_MSG: >>>> /************************************************************ >>>> STARTUP_MSG: Starting DataNode >>>> STARTUP_MSG: host = **************** >>>> STARTUP_MSG: args = [] >>>> STARTUP_MSG: version = 0.18.3 >>>> STARTUP_MSG: build = >>>> https://svn.apache.org/repos/asf/hadoop/core/branches/branch-0.18 -r >>>> 736250; >>>> compiled by 'ndaley' on Thu Jan 22 23:12:08 UTC 2009 >>>> ************************************************************/ >>>> 2009-06-04 12:16:32,175 INFO org.apache.hadoop.ipc.Client: Retrying >>>> connect >>>> to server: master/198.55.35.229:54310. Already tried 0 time(s). >>>> 2009-06-04 12:16:33,178 INFO org.apache.hadoop.ipc.Client: Retrying >>>> connect >>>> to server: master/198.55.35.229:54310. Already tried 1 time(s). >>>> 2009-06-04 12:16:34,181 INFO org.apache.hadoop.ipc.Client: Retrying >>>> connect >>>> to server: master/198.55.35.229:54310. Already tried 2 time(s). >>>> 2009-06-04 12:16:35,184 INFO org.apache.hadoop.ipc.Client: Retrying >>>> connect >>>> to server: master/198.55.35.229:54310. Already tried 3 time(s). >>>> 2009-06-04 12:16:36,187 INFO org.apache.hadoop.ipc.Client: Retrying >>>> connect >>>> to server: master/198.55.35.229:54310. Already tried 4 time(s). >>>> 2009-06-04 12:16:37,190 INFO org.apache.hadoop.ipc.Client: Retrying >>>> connect >>>> to server: master/198.55.35.229:54310. Already tried 5 time(s). >>>> 2009-06-04 12:16:38,193 INFO org.apache.hadoop.ipc.Client: Retrying >>>> connect >>>> to server: master/198.55.35.229:54310. Already tried 6 time(s). >>>> 2009-06-04 12:16:39,196 INFO org.apache.hadoop.ipc.Client: Retrying >>>> connect >>>> to server: master/198.55.35.229:54310. Already tried 7 time(s). >>>> 2009-06-04 12:16:40,198 INFO org.apache.hadoop.ipc.Client: Retrying >>>> connect >>>> to server: master/198.55.35.229:54310. Already tried 8 time(s). >>>> 2009-06-04 12:16:41,200 INFO org.apache.hadoop.ipc.Client: Retrying >>>> connect >>>> to server: master/198.55.35.229:54310. Already tried 9 time(s). >>>> 2009-06-04 12:16:41,222 ERROR org.apache.hadoop.dfs.DataNode: >>>> java.io.IOException: Call to master/198.55.35.229:54310 failed on local >>>> exception: java.net.NoRouteToHostException: No route to host >>>> at org.apache.hadoop.ipc.Client.wrapException(Client.java:751) >>>> at org.apache.hadoop.ipc.Client.call(Client.java:719) >>>> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:216) >>>> at org.apache.hadoop.dfs.$Proxy4.getProtocolVersion(Unknown Source) >>>> at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:348) >>>> at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:335) >>>> at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:372) >>>> at org.apache.hadoop.ipc.RPC.waitForProxy(RPC.java:309) >>>> at org.apache.hadoop.ipc.RPC.waitForProxy(RPC.java:286) >>>> at org.apache.hadoop.dfs.DataNode.startDataNode(DataNode.java:277) >>>> at org.apache.hadoop.dfs.DataNode.<init>(DataNode.java:223) >>>> at org.apache.hadoop.dfs.DataNode.makeInstance(DataNode.java:3071) >>>> at >>>> org.apache.hadoop.dfs.DataNode.instantiateDataNode(DataNode.java:3026) >>>> at org.apache.hadoop.dfs.DataNode.createDataNode(DataNode.java:3034) >>>> at org.apache.hadoop.dfs.DataNode.main(DataNode.java:3156) >>>> Caused by: java.net.NoRouteToHostException: No route to host >>>> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) >>>> at >>>> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574) >>>> at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:100) >>>> at >>>> org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:301) >>>> at >>>> org.apache.hadoop.ipc.Client$Connection.access$1700(Client.java:178) >>>> at org.apache.hadoop.ipc.Client.getConnection(Client.java:820) >>>> at org.apache.hadoop.ipc.Client.call(Client.java:705) >>>> ... 13 more >>>> >>>> 2009-06-04 12:16:41,222 INFO org.apache.hadoop.dfs.DataNode: >>>> SHUTDOWN_MSG: >>>> /************************************************************ >>>> SHUTDOWN_MSG: Shutting down DataNode at >>>> opencirrus-1262.hpl.hp.com/198.55.36.243 >>>> ************************************************************/ >>>> >>>> >>>> >>>> PLEASE COMMENT. >>>> >>>> Thanks. >>>> >>>> Asif. >>>> >>>> >>>> Ravi >>>> -- >>>> >>>> >>>> >> >
