It's working for me now. Turns out the cluster have multiple network interfaces and I was using the wrong one. Thanks.
Aaron Kimball-3 wrote: > > Hi, > > The dfs.http.address is for human use, not program interoperability. You > can > visit http://whatever.address.your.namenode.has:50070 in a web browser and > see statistics about your filesystem. > > The address of cluster 2 is in its fs.default.name. This should be set to > something like hdfs://cluster2.master.name:9000/ > > The file:// "protocol" only refers to paths on the current machine in its > real (non-DFS) filesystem. > - Aaron > > On Wed, Oct 22, 2008 at 3:47 PM, bzheng <[EMAIL PROTECTED]> wrote: > >> >> Thanks. The fs.default.name is "file:///" and dfs.http.address is >> "0.0.0.0:50070". I tried: >> >> hadoop dfs -ls /path/file to make sure file exists on cluster1 >> hadoop distcp file:///cluster1_master_node_ip:50070/path/file >> file:///cluster2_master_node_ip:50070/path/file >> >> It gives this error message: >> 08/10/22 15:43:47 INFO util.CopyFiles: >> srcPaths=[file:/cluster1_master_node_ip:50070/path/file] >> 08/10/22 15:43:47 INFO util.CopyFiles: >> destPath=file:/cluster2_master_node_ip:50070/path/file >> With failures, global counters are inaccurate; consider running with -i >> Copy failed: org.apache.hadoop.mapred.InvalidInputException: Input source >> file:/cluster1_master_node_ip:50070/path/file does not exist. >> at >> org.apache.hadoop.util.CopyFiles.checkSrcPath(CopyFiles.java:578) >> at org.apache.hadoop.util.CopyFiles.copy(CopyFiles.java:594) >> at org.apache.hadoop.util.CopyFiles.run(CopyFiles.java:743) >> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) >> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79) >> at org.apache.hadoop.util.CopyFiles.main(CopyFiles.java:763) >> >> >> If I use hdfs:// instead of file:///, I get: >> Copy failed: java.net.SocketTimeoutException: timed out waiting for rpc >> response >> at org.apache.hadoop.ipc.Client.call(Client.java:559) >> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212) >> at org.apache.hadoop.dfs.$Proxy0.getProtocolVersion(Unknown >> Source) >> at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:313) >> at >> org.apache.hadoop.dfs.DFSClient.createRPCNamenode(DFSClient.java:102) >> at org.apache.hadoop.dfs.DFSClient.<init>(DFSClient.java:178) >> at >> >> org.apache.hadoop.dfs.DistributedFileSystem.initialize(DistributedFileSystem.java:68) >> at >> org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1280) >> at org.apache.hadoop.fs.FileSystem.access$300(FileSystem.java:56) >> at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1291) >> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:203) >> at org.apache.hadoop.fs.Path.getFileSystem(Path.java:175) >> at >> org.apache.hadoop.util.CopyFiles.checkSrcPath(CopyFiles.java:572) >> at org.apache.hadoop.util.CopyFiles.copy(CopyFiles.java:594) >> at org.apache.hadoop.util.CopyFiles.run(CopyFiles.java:743) >> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) >> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79) >> at org.apache.hadoop.util.CopyFiles.main(CopyFiles.java:763) >> >> >> >> s29752-hadoopuser wrote: >> > >> > Hi, >> > >> > There is no such thing called distcp port. distcp uses (generic) file >> > system API and so it does not care about the file system implementation >> > details like port number. >> > >> > It is common to use distcp with HDFS or HFTP. The urls will look like >> > hdfs://namenode:port/path and hftp://namenode:port/path for HDFS and >> HFTP, >> > respectively. The HDFS and HFTP ports are specified by >> fs.default.name >> > and dfs.http.address, respectively. >> > >> > Nicholas Sze >> > >> > >> > >> > >> > ----- Original Message ---- >> >> From: bzheng <[EMAIL PROTECTED]> >> >> To: core-user@hadoop.apache.org >> >> Sent: Wednesday, October 22, 2008 11:57:43 AM >> >> Subject: distcp port for 0.17.2 >> >> >> >> >> >> What's the port number for distcp in 0.17.2? I can't find any >> >> documentation >> >> on distcp for version 0.17.2. For version 0.18, the documentation >> says >> >> it's >> >> 8020. >> >> >> >> I'm using a standard install and the only open ports associated with >> >> hadoop >> >> are 50030, 50070, and 50090. None of them work with distcp. So, how >> do >> >> you >> >> use distcp in 0.17.2? are there any extra setup/configuration needed? >> >> >> >> Thanks in advance for your help. >> >> -- >> >> View this message in context: >> >> http://www.nabble.com/distcp-port-for-0.17.2-tp20117463p20117463.html >> >> Sent from the Hadoop core-user mailing list archive at Nabble.com. >> > >> > >> > >> >> -- >> View this message in context: >> http://www.nabble.com/distcp-port-for-0.17.2-tp20117463p20121246.html >> Sent from the Hadoop core-user mailing list archive at Nabble.com. >> >> > > -- View this message in context: http://www.nabble.com/distcp-port-for-0.17.2-tp20117463p20137577.html Sent from the Hadoop core-user mailing list archive at Nabble.com.
