If your intention is to move the data to a different file system, you can 
always concatenate the output with a wildcard, ie hadoop fs -cat /outdir/p*. If 
you need the keys to be sorted however, you will need to run a separate job 
with one reducer. Or you can use the total order partitioner with the original 
job. 

Sent from my iPad

On Oct 23, 2010, at 3:32 PM, "M. C. Srivas" <[email protected]> wrote:

> Not with HDFS, since only one process may write to a single file (and its
> not visible until the file is closed). In fact, its worse than that ... the
> same process that's writing that file cannot see it or read it until after
> its done.
> 
> If you have multiple reducers, you are simply out of luck and will have to
> run a separate "job" to copy the data out.
> 
> 
> On Sat, Oct 23, 2010 at 3:08 PM, Steve Lewis <[email protected]> wrote:
> 
>> Once I run a map-reduce job I get output in the form of
>> part-r-00000 part-r-00001 ...
>> 
>> In many cases the output is significantly smaller than the original input -
>> take the classic word count
>> 
>> In most cases I want to combine the output into a single file that may well
>> not live on HDFS but on a more accessible file system
>> 
>> Are there standard libraries or approaches for consolidating reducer
>> output.
>> 
>> A second Map-Reduce job taking the output directory as an input is an OK
>> start but as output there needs to be a single reducer that
>> writes a real file and not reduce output -
>> 
>> Are there standard libraries or approaches to this?????
>> 
>> --
>> Steven M. Lewis PhD
>> 4221 105th Ave Ne
>> Kirkland, WA 98033
>> 206-384-1340 (cell)
>> Institute for Systems Biology
>> Seattle WA
>> 

Reply via email to