I guess this is exactly the problem is! 

Is there any way I could do "Greping archives" inside the MP program ? Or some 
hadoop command that could combine all small pieces files into a big one ?



Thanks,

-Kun


--- On Mon, 10/19/09, Ashutosh Chauhan <[email protected]> wrote:

> From: Ashutosh Chauhan <[email protected]>
> Subject: Re: Hadoop dfs can't allocate memory with enough hard disk space 
> when  data gets huge
> To: [email protected]
> Date: Monday, October 19, 2009, 3:30 PM
> You might be hitting into the problem
> of "small-files". This has been
> discussed multiple times on the list. Greping through
> archives will help.
> Also http://www.cloudera.com/blog/2009/02/02/the-small-files-problem/
> 
> Ashutosh
> 
> On Sun, Oct 18, 2009 at 22:57, Kunsheng Chen <[email protected]>
> wrote:
> 
> > I and running a hadoop program to perform MapReduce
> work on files inside a
> > folder.
> >
> > My program is basically doing Map and Reduce work,
> each line of any file is
> > a pair of string, and the result is a string associate
> with occurence inside
> > all files.
> >
> > The program works fine until the number of files grow
> to about 80,000,then
> > the 'cannot allocate memory' error occur for some
> reason.
> >
> > Each of the file contains around 50 lines, but the
> total size of all files
> > is no more than 1.5 GB. There are 3 datanodes
> performing calculation,each of
> > them have more than 10GB hd left.
> >
> > I am wondering if that is normal for Hadoop because
> the data is too large ?
> > Or it might be my programs problem ?
> >
> > It is really not supposed to be since Hadoop was
> developed for processing
> > large data sets.
> >
> >
> > Any idea is well appreciated
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> 

__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com 

Reply via email to