It's possible to write your own class to take better encapsulate writing of
side-effect files, but as people have said, you can run into unanticipated
issues if the number of files you try to write at once becomes high.
On Sat, Feb 6, 2010 at 3:47 AM, Udaya Lakshmi wrote:
> Hi Amareshwari,
>
>
Hi Amareshwari,
But this feature is not available in Hadoop 0.18.3. Is there any work
around for this version.
Thanks,
Udaya.
On Fri, Feb 5, 2010 at 10:49 AM, Amareshwari Sri Ramadasu <
amar...@yahoo-inc.com> wrote:
> See MultipleOutputs at
> http://hadoop.apache.org/common/docs/r0.20.0/api/o
Hi,
You should not in general write many small files for namenode to perform well:
http://www.cloudera.com/blog/2009/02/the-small-files-problem/
To answer your question, you can write them as task side effect files, which
will get propagated to your output directory by hadoop upon successful
com
See MultipleOutputs at
http://hadoop.apache.org/common/docs/r0.20.0/api/org/apache/hadoop/mapred/lib/MultipleOutputs.html
-Amareshwari
On 2/5/10 10:41 AM, "Udaya Lakshmi" wrote:
Hi,
I was wondering if it is possible to write each key-value pair produced by
the reduce function to a different
Hi,
I was wondering if it is possible to write each key-value pair produced by
the reduce function to a different file. How could I open a new file in the
reduce function of the reducer? I know its possible in configure function
but it will write all the output that reducer to that file.
Thanks,
