Re: Any other way to copy to HDFS ?
When you start the NameNode in Linux Machine, it will listen on one address.You
can configure that address in NameNode by using fs.default.name.
>From the clients, you can give this address to connect to your NameNode.
initialize API will take URI and configuration.
Assume if your NameNode is running on hdfs://10.18.52.63:9000
Then you can caonnect to your NameNode like below.
FileSystem fs =new DistributedFileSystem();
fs.initialize(new URI("hdfs://10.18.52.63:9000/"), new Configuration());
Please go through the below mentioned docs, you will more understanding.
>if I want to
> copy data from windows machine to namenode machine ?
In DFS namenode will be responsible for only nameSpace.
in simple words to understand quickly the flow:
Clients will ask NameNode to give some DNs to copy the data. Then NN will
create file entry in NameSpace and also will return the block entries based on
client request. Then clients directly will connect to the DNs and copy the data.
Reading data back also will the sameway.
I hope you will understand better now :-)
Regards,
Uma
- Original Message -
From: praveenesh kumar
Date: Wednesday, September 21, 2011 3:11 pm
Subject: Re: Any other way to copy to HDFS ?
To: common-user@hadoop.apache.org
> So I want to copy the file from windows machine to linux namenode.
> How can I define NAMENODE_URI in the code you mention, if I want to
> copy data from windows machine to namenode machine ?
>
> Thanks,
> Praveenesh
>
> On Wed, Sep 21, 2011 at 2:37 PM, Uma Maheswara Rao G 72686 <
> mahesw...@huawei.com> wrote:
>
> > For more understanding the flows, i would recommend you to go
> through once
> > below docs
> >
> >
> http://hadoop.apache.org/common/docs/r0.16.4/hdfs_design.html#The+File+System+Namespace>
> > Regards,
> > Uma
> >
> > - Original Message -
> > From: Uma Maheswara Rao G 72686
> > Date: Wednesday, September 21, 2011 2:36 pm
> > Subject: Re: Any other way to copy to HDFS ?
> > To: common-user@hadoop.apache.org
> >
> > >
> > > Hi,
> > >
> > > You need not copy the files to NameNode.
> > >
> > > Hadoop provide Client code as well to copy the files.
> > > To copy the files from other node ( non dfs), you need to put the
> > > hadoop**.jar's into classpath and use the below code snippet.
> > >
> > > FileSystem fs =new DistributedFileSystem();
> > > fs.initialize("NAMENODE_URI", configuration);
> > >
> > > fs.copyFromLocal(srcPath, dstPath);
> > >
> > > using this API, you can copy the files from any machine.
> > >
> > > Regards,
> > > Uma
> > >
> > >
> > >
> > >
> > >
> > > - Original Message -
> > > From: praveenesh kumar
> > > Date: Wednesday, September 21, 2011 2:14 pm
> > > Subject: Any other way to copy to HDFS ?
> > > To: common-user@hadoop.apache.org
> > >
> > > > Guys,
> > > >
> > > > As far as I know hadoop, I think, to copy the files to HDFS,
> > > first
> > > > it needs
> > > > to be copied to the NameNode's local filesystem. Is it right ??
> > > > So does it mean that even if I have a hadoop cluster of 10 nodes
> > > with> overall capacity of 6TB, but if my NameNode's hard disk
> > > capacity
> > > > is 500 GB,
> > > > I can not copy any file to HDFS greater than 500 GB ?
> > > >
> > > > Is there any other way to directly copy to HDFS without copy the
> > > > file to
> > > > namenode's local filesystem ?
> > > > What can be other ways to copy large files greater than
> > > namenode's
> > > > diskcapacity ?
> > > >
> > > > Thanks,
> > > > Praveenesh.
> > > >
> > >
> >
>
Re: Any other way to copy to HDFS ?
So I want to copy the file from windows machine to linux namenode.
How can I define NAMENODE_URI in the code you mention, if I want to
copy data from windows machine to namenode machine ?
Thanks,
Praveenesh
On Wed, Sep 21, 2011 at 2:37 PM, Uma Maheswara Rao G 72686 <
mahesw...@huawei.com> wrote:
> For more understanding the flows, i would recommend you to go through once
> below docs
>
> http://hadoop.apache.org/common/docs/r0.16.4/hdfs_design.html#The+File+System+Namespace
>
> Regards,
> Uma
>
> - Original Message -
> From: Uma Maheswara Rao G 72686
> Date: Wednesday, September 21, 2011 2:36 pm
> Subject: Re: Any other way to copy to HDFS ?
> To: common-user@hadoop.apache.org
>
> >
> > Hi,
> >
> > You need not copy the files to NameNode.
> >
> > Hadoop provide Client code as well to copy the files.
> > To copy the files from other node ( non dfs), you need to put the
> > hadoop**.jar's into classpath and use the below code snippet.
> >
> > FileSystem fs =new DistributedFileSystem();
> > fs.initialize("NAMENODE_URI", configuration);
> >
> > fs.copyFromLocal(srcPath, dstPath);
> >
> > using this API, you can copy the files from any machine.
> >
> > Regards,
> > Uma
> >
> >
> >
> >
> >
> > - Original Message -
> > From: praveenesh kumar
> > Date: Wednesday, September 21, 2011 2:14 pm
> > Subject: Any other way to copy to HDFS ?
> > To: common-user@hadoop.apache.org
> >
> > > Guys,
> > >
> > > As far as I know hadoop, I think, to copy the files to HDFS,
> > first
> > > it needs
> > > to be copied to the NameNode's local filesystem. Is it right ??
> > > So does it mean that even if I have a hadoop cluster of 10 nodes
> > with> overall capacity of 6TB, but if my NameNode's hard disk
> > capacity
> > > is 500 GB,
> > > I can not copy any file to HDFS greater than 500 GB ?
> > >
> > > Is there any other way to directly copy to HDFS without copy the
> > > file to
> > > namenode's local filesystem ?
> > > What can be other ways to copy large files greater than
> > namenode's
> > > diskcapacity ?
> > >
> > > Thanks,
> > > Praveenesh.
> > >
> >
>
Re: Any other way to copy to HDFS ?
For more understanding the flows, i would recommend you to go through once
below docs
http://hadoop.apache.org/common/docs/r0.16.4/hdfs_design.html#The+File+System+Namespace
Regards,
Uma
- Original Message -
From: Uma Maheswara Rao G 72686
Date: Wednesday, September 21, 2011 2:36 pm
Subject: Re: Any other way to copy to HDFS ?
To: common-user@hadoop.apache.org
>
> Hi,
>
> You need not copy the files to NameNode.
>
> Hadoop provide Client code as well to copy the files.
> To copy the files from other node ( non dfs), you need to put the
> hadoop**.jar's into classpath and use the below code snippet.
>
> FileSystem fs =new DistributedFileSystem();
> fs.initialize("NAMENODE_URI", configuration);
>
> fs.copyFromLocal(srcPath, dstPath);
>
> using this API, you can copy the files from any machine.
>
> Regards,
> Uma
>
>
>
>
>
> - Original Message -
> From: praveenesh kumar
> Date: Wednesday, September 21, 2011 2:14 pm
> Subject: Any other way to copy to HDFS ?
> To: common-user@hadoop.apache.org
>
> > Guys,
> >
> > As far as I know hadoop, I think, to copy the files to HDFS,
> first
> > it needs
> > to be copied to the NameNode's local filesystem. Is it right ??
> > So does it mean that even if I have a hadoop cluster of 10 nodes
> with> overall capacity of 6TB, but if my NameNode's hard disk
> capacity
> > is 500 GB,
> > I can not copy any file to HDFS greater than 500 GB ?
> >
> > Is there any other way to directly copy to HDFS without copy the
> > file to
> > namenode's local filesystem ?
> > What can be other ways to copy large files greater than
> namenode's
> > diskcapacity ?
> >
> > Thanks,
> > Praveenesh.
> >
>
Re: Any other way to copy to HDFS ?
Hi,
You need not copy the files to NameNode.
Hadoop provide Client code as well to copy the files.
To copy the files from other node ( non dfs), you need to put the
hadoop**.jar's into classpath and use the below code snippet.
FileSystem fs =new DistributedFileSystem();
fs.initialize("NAMENODE_URI", configuration);
fs.copyFromLocal(srcPath, dstPath);
using this API, you can copy the files from any machine.
Regards,
Uma
- Original Message -
From: praveenesh kumar
Date: Wednesday, September 21, 2011 2:14 pm
Subject: Any other way to copy to HDFS ?
To: common-user@hadoop.apache.org
> Guys,
>
> As far as I know hadoop, I think, to copy the files to HDFS, first
> it needs
> to be copied to the NameNode's local filesystem. Is it right ??
> So does it mean that even if I have a hadoop cluster of 10 nodes with
> overall capacity of 6TB, but if my NameNode's hard disk capacity
> is 500 GB,
> I can not copy any file to HDFS greater than 500 GB ?
>
> Is there any other way to directly copy to HDFS without copy the
> file to
> namenode's local filesystem ?
> What can be other ways to copy large files greater than namenode's
> diskcapacity ?
>
> Thanks,
> Praveenesh.
>
