Hi, The dfs.http.address is for human use, not program interoperability. You can visit http://whatever.address.your.namenode.has:50070 in a web browser and see statistics about your filesystem.
The address of cluster 2 is in its fs.default.name. This should be set to something like hdfs://cluster2.master.name:9000/ The file:// "protocol" only refers to paths on the current machine in its real (non-DFS) filesystem. - Aaron On Wed, Oct 22, 2008 at 3:47 PM, bzheng <[EMAIL PROTECTED]> wrote: > > Thanks. The fs.default.name is "file:///" and dfs.http.address is > "0.0.0.0:50070". I tried: > > hadoop dfs -ls /path/file to make sure file exists on cluster1 > hadoop distcp file:///cluster1_master_node_ip:50070/path/file > file:///cluster2_master_node_ip:50070/path/file > > It gives this error message: > 08/10/22 15:43:47 INFO util.CopyFiles: > srcPaths=[file:/cluster1_master_node_ip:50070/path/file] > 08/10/22 15:43:47 INFO util.CopyFiles: > destPath=file:/cluster2_master_node_ip:50070/path/file > With failures, global counters are inaccurate; consider running with -i > Copy failed: org.apache.hadoop.mapred.InvalidInputException: Input source > file:/cluster1_master_node_ip:50070/path/file does not exist. > at org.apache.hadoop.util.CopyFiles.checkSrcPath(CopyFiles.java:578) > at org.apache.hadoop.util.CopyFiles.copy(CopyFiles.java:594) > at org.apache.hadoop.util.CopyFiles.run(CopyFiles.java:743) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79) > at org.apache.hadoop.util.CopyFiles.main(CopyFiles.java:763) > > > If I use hdfs:// instead of file:///, I get: > Copy failed: java.net.SocketTimeoutException: timed out waiting for rpc > response > at org.apache.hadoop.ipc.Client.call(Client.java:559) > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212) > at org.apache.hadoop.dfs.$Proxy0.getProtocolVersion(Unknown Source) > at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:313) > at > org.apache.hadoop.dfs.DFSClient.createRPCNamenode(DFSClient.java:102) > at org.apache.hadoop.dfs.DFSClient.<init>(DFSClient.java:178) > at > > org.apache.hadoop.dfs.DistributedFileSystem.initialize(DistributedFileSystem.java:68) > at > org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1280) > at org.apache.hadoop.fs.FileSystem.access$300(FileSystem.java:56) > at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1291) > at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:203) > at org.apache.hadoop.fs.Path.getFileSystem(Path.java:175) > at org.apache.hadoop.util.CopyFiles.checkSrcPath(CopyFiles.java:572) > at org.apache.hadoop.util.CopyFiles.copy(CopyFiles.java:594) > at org.apache.hadoop.util.CopyFiles.run(CopyFiles.java:743) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79) > at org.apache.hadoop.util.CopyFiles.main(CopyFiles.java:763) > > > > s29752-hadoopuser wrote: > > > > Hi, > > > > There is no such thing called distcp port. distcp uses (generic) file > > system API and so it does not care about the file system implementation > > details like port number. > > > > It is common to use distcp with HDFS or HFTP. The urls will look like > > hdfs://namenode:port/path and hftp://namenode:port/path for HDFS and > HFTP, > > respectively. The HDFS and HFTP ports are specified by fs.default.name > > and dfs.http.address, respectively. > > > > Nicholas Sze > > > > > > > > > > ----- Original Message ---- > >> From: bzheng <[EMAIL PROTECTED]> > >> To: [email protected] > >> Sent: Wednesday, October 22, 2008 11:57:43 AM > >> Subject: distcp port for 0.17.2 > >> > >> > >> What's the port number for distcp in 0.17.2? I can't find any > >> documentation > >> on distcp for version 0.17.2. For version 0.18, the documentation says > >> it's > >> 8020. > >> > >> I'm using a standard install and the only open ports associated with > >> hadoop > >> are 50030, 50070, and 50090. None of them work with distcp. So, how do > >> you > >> use distcp in 0.17.2? are there any extra setup/configuration needed? > >> > >> Thanks in advance for your help. > >> -- > >> View this message in context: > >> http://www.nabble.com/distcp-port-for-0.17.2-tp20117463p20117463.html > >> Sent from the Hadoop core-user mailing list archive at Nabble.com. > > > > > > > > -- > View this message in context: > http://www.nabble.com/distcp-port-for-0.17.2-tp20117463p20121246.html > Sent from the Hadoop core-user mailing list archive at Nabble.com. > >
