Hello guys,
Thanks a lot again for your previous guidance guys,i tried out
java api to do file uploads its wrking fine,now i need to modify the code
using sequence files so that i can handle large number of small files in
hadoop.. for that i encountered 2 links
1. http://stuartsierra.com/2008/04/24/a-million-little-files (tar to
sequence)
2. http://www.jointhegrid.com/hadoop_filecrush/index.jsp (file crush)
could you pls tell me which approach is better to follow or should i follow
HAR(hadoop archive) approach,i came to know that in sequence file we can
combine smaller files in to one big one but dunt know how to split and
retrieve the small files again while reading files,,, as well..
Thanks and Gratitude
On Wed, Oct 5, 2011 at 1:27 AM, visioner sadak <visioner.sa...@gmail.com>wrote:
> Thanks a lot wellington and bejoy for your inputs will try out this api and
> sequence file....
>
>
> On Wed, Oct 5, 2011 at 1:17 AM, Wellington Chevreuil <
> wellington.chevre...@gmail.com> wrote:
>
>> Yes, Sadak,
>>
>> Within this API, you'll copy your files into Hadoop HDFS as you do
>> when writing to an OutputStream. It will be replicated in your
>> cluster's HDFS then.
>>
>> Cheers.
>>
>> 2011/10/4 visioner sadak <visioner.sa...@gmail.com>:
>> > Hey thanks wellington just a thought will my data be replicated as
>> well coz
>> > i thought tht mapper does the job of breaking data in to pieces and
>> > distribution and reducer will do the joining and combining while
>> fetching
>> > data back thts why was confused to use a MR..can i use this API for
>> > uploading a large number of small files as well thru my application or
>> > should i use sequence file class for that...because i saw the small file
>> > problem in hadoop as well as mentioned in below link
>> >
>> > http://www.cloudera.com/blog/2009/02/the-small-files-problem/
>> >
>> > On Wed, Oct 5, 2011 at 12:54 AM, Wellington Chevreuil
>> > <wellington.chevre...@gmail.com> wrote:
>> >>
>> >> Hey Sadak,
>> >>
>> >> you don't need to write a MR job for that. You can make your java
>> >> program use Hadoop Java API for that. You would need to use FileSystem
>> >>
>> >> (
>> http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/fs/FileSystem.html
>> )
>> >> and Path
>> >> (
>> http://hadoop.apache.org/common/docs/current/api/index.html?org/apache/hadoop/fs/Path.html
>> )
>> >> classes for that.
>> >>
>> >> Cheers,
>> >> Wellington.
>> >>
>> >> 2011/10/4 visioner sadak <visioner.sa...@gmail.com>:
>> >> > Hello guys,
>> >> >
>> >> > I would like to know how to do file uploads in HDFS using
>> >> > java,is it to be done using map reduce what if i have a large number
>> of
>> >> > small files should i use sequence file along with map reduce???,It
>> will
>> >> > be
>> >> > great if you can provide some sort of information...
>> >
>> >
>>
>
>
