Hi, You should not in general write many small files for namenode to perform well: http://www.cloudera.com/blog/2009/02/the-small-files-problem/
To answer your question, you can write them as task side effect files, which will get propagated to your output directory by hadoop upon successful completion, so you wont have to worry about speculative execution etc. Amogh On 2/5/10 10:41 AM, "Udaya Lakshmi" <udaya...@gmail.com> wrote: Hi, I was wondering if it is possible to write each key-value pair produced by the reduce function to a different file. How could I open a new file in the reduce function of the reducer? I know its possible in configure function but it will write all the output that reducer to that file. Thanks, Udaya.
