Yes Sadak. The API would do the splitting for you, no need of MR for that.
It'd be better keeping the file sizes atleast same as an hdfs block size.
Sequence file is definitely a good choice. If you are looking out for a
process and then archival of input then look into HAR (hadoop archives as
well).
Thanks and Regards
Bejoy.K.S
On Wed, Oct 5, 2011 at 1:10 AM, Bejoy KS <bejoy.had...@gmail.com> wrote:
> Hi Sadak
> You really don't need to fire a map reduce job to copy files from
> a local file system to hdfs. You can do it in two easy ways
>
> *Using linux CLI* - if you are going in with a shell script. The most
> convenient option and handy.
> hadoop fs -copyFromLocal <file/dir in lfs> <destination dir in hdfs>
>
> *Using JAVA API*
> //load the hadoop configuration
> Configuration hadoopConf=new Configuration();
> //get the default associated file system
> FileSystem fileSystem=FileSystem.get(hadoopConf) ;
> //copy from lfs to hdfs
> fileSystem.copyFromLocalFile(new Path("source file/dir in lfs"), new
> Path("Destn dir in hdfs"));
>
> Please read the API before implementation. There are variants of the method
> copyFromLocalFile as well lot of other methods which you'd find useful if
> you are choosing java API path.
>
> You can do the reversal operation as
> hadoop fs -copyToLocal
>
> fileSystem.copyToLocalFile(new Path("dir/file dir in hdfs"), new
> Path("Destn dir in lfs"));
>
> Hope it helps and gives you a kick start into hadoop.
>
> Thanks and Regards
> Bejoy.K.S
>
>
>
> On Wed, Oct 5, 2011 at 12:23 AM, visioner sadak
> <visioner.sa...@gmail.com>wrote:
>
>> Hello guys,
>>
>> I would like to know how to do file uploads in HDFS using
>> java,is it to be done using map reduce what if i have a large number of
>> small files should i use sequence file along with map reduce???,It will be
>> great if you can provide some sort of information...
>
>
>
