transferring between HDFS which reside in different subnet
Hi, I have a question to the hadoop experts: I have two HDFS, in different subnet. HDFS1 : 192.168.*.* HDFS2: 10.10.*.* the namenode of HDFS2 has two NIC. One connected to 192.168.*.* and another to 10.10.*.*. So, is it possible to transfer data from HDFS1 to HDFS2 and vice versa. Regards, Arindam
Re: transferring between HDFS which reside in different subnet
If you could cross-access HDFS from both name nodes, then it should be transferable using /distcp /command. Shi * * On 5/11/2012 8:45 AM, Arindam Choudhury wrote: Hi, I have a question to the hadoop experts: I have two HDFS, in different subnet. HDFS1 : 192.168.*.* HDFS2: 10.10.*.* the namenode of HDFS2 has two NIC. One connected to 192.168.*.* and another to 10.10.*.*. So, is it possible to transfer data from HDFS1 to HDFS2 and vice versa. Regards, Arindam
Re: transferring between HDFS which reside in different subnet
I can not cross access HDFS. Though HDFS2 has two NIC the HDFS is running on the other subnet. On Fri, May 11, 2012 at 3:57 PM, Shi Yu wrote: > If you could cross-access HDFS from both name nodes, then it should be > transferable using /distcp /command. > > Shi * > * > > On 5/11/2012 8:45 AM, Arindam Choudhury wrote: > >> Hi, >> >> I have a question to the hadoop experts: >> >> I have two HDFS, in different subnet. >> >> HDFS1 : 192.168.*.* >> HDFS2: 10.10.*.* >> >> the namenode of HDFS2 has two NIC. One connected to 192.168.*.* and >> another >> to 10.10.*.*. >> >> So, is it possible to transfer data from HDFS1 to HDFS2 and vice versa. >> >> Regards, >> Arindam >> >> >
Re: transferring between HDFS which reside in different subnet
It seems in your case HDFS2 could access HDFS, so you should be able to transfer HDFS data to HDFS2. If you want to cross-transfer, you don't need to do distcp on cluster nodes, if any client node (not necessary to be namenode, datanode, secondary node, etc.) could access to both HDFSs, then run transfer command on that client node. On 5/11/2012 9:03 AM, Arindam Choudhury wrote: I can not cross access HDFS. Though HDFS2 has two NIC the HDFS is running on the other subnet. On Fri, May 11, 2012 at 3:57 PM, Shi Yu wrote: If you could cross-access HDFS from both name nodes, then it should be transferable using /distcp /command. Shi * * On 5/11/2012 8:45 AM, Arindam Choudhury wrote: Hi, I have a question to the hadoop experts: I have two HDFS, in different subnet. HDFS1 : 192.168.*.* HDFS2: 10.10.*.* the namenode of HDFS2 has two NIC. One connected to 192.168.*.* and another to 10.10.*.*. So, is it possible to transfer data from HDFS1 to HDFS2 and vice versa. Regards, Arindam
Re: transferring between HDFS which reside in different subnet
Looks like both are private subnets, so you got to route via a public default gateway. Try adding route using route command if your in linux(windows i have no idea). Just a thought i havent tried it though. Thanks, Rajesh Typed from mobile, please bear with typos. On May 11, 2012 10:03 AM, "Arindam Choudhury" wrote: > I can not cross access HDFS. Though HDFS2 has two NIC the HDFS is running > on the other subnet. > > On Fri, May 11, 2012 at 3:57 PM, Shi Yu wrote: > > > If you could cross-access HDFS from both name nodes, then it should be > > transferable using /distcp /command. > > > > Shi * > > * > > > > On 5/11/2012 8:45 AM, Arindam Choudhury wrote: > > > >> Hi, > >> > >> I have a question to the hadoop experts: > >> > >> I have two HDFS, in different subnet. > >> > >> HDFS1 : 192.168.*.* > >> HDFS2: 10.10.*.* > >> > >> the namenode of HDFS2 has two NIC. One connected to 192.168.*.* and > >> another > >> to 10.10.*.*. > >> > >> So, is it possible to transfer data from HDFS1 to HDFS2 and vice versa. > >> > >> Regards, > >> Arindam > >> > >> > > >
Re: transferring between HDFS which reside in different subnet
So, hadoop dfs -cp hdfs:// hdfs://... this will work. On Fri, May 11, 2012 at 4:14 PM, Rajesh Sai T wrote: > Looks like both are private subnets, so you got to route via a public > default gateway. Try adding route using route command if your in > linux(windows i have no idea). Just a thought i havent tried it though. > > Thanks, > Rajesh > > Typed from mobile, please bear with typos. > On May 11, 2012 10:03 AM, "Arindam Choudhury" > > wrote: > > > I can not cross access HDFS. Though HDFS2 has two NIC the HDFS is running > > on the other subnet. > > > > On Fri, May 11, 2012 at 3:57 PM, Shi Yu wrote: > > > > > If you could cross-access HDFS from both name nodes, then it should be > > > transferable using /distcp /command. > > > > > > Shi * > > > * > > > > > > On 5/11/2012 8:45 AM, Arindam Choudhury wrote: > > > > > >> Hi, > > >> > > >> I have a question to the hadoop experts: > > >> > > >> I have two HDFS, in different subnet. > > >> > > >> HDFS1 : 192.168.*.* > > >> HDFS2: 10.10.*.* > > >> > > >> the namenode of HDFS2 has two NIC. One connected to 192.168.*.* and > > >> another > > >> to 10.10.*.*. > > >> > > >> So, is it possible to transfer data from HDFS1 to HDFS2 and vice > versa. > > >> > > >> Regards, > > >> Arindam > > >> > > >> > > > > > >
