Hello ,
I finally moved to per task write and reducers gather them all and write
them into the file
Thanks for the help
regards
rab
On Fri, Jul 11, 2014 at 10:50 AM, Bertrand Dechoux
wrote:
> And beside with a single file, if that were possible, how do you handle
> error? Let' say task 1 r
And beside with a single file, if that were possible, how do you handle
error? Let' say task 1 ran 3 times : 1 error, 1 speculative and 1
success... A per-task file has been a standard to easily solve that problem.
Bertrand Dechoux
On Thu, Jul 10, 2014 at 10:00 PM, Vinod Kumar Vavilapalli <
vino
Current writes to a single file in HDFS is not possible today. You may
want to write a per-task file and use that entire directory as your output.
+Vinod
Hortonworks Inc.
http://hortonworks.com/
On Wed, Jul 9, 2014 at 10:42 PM, rab ra wrote:
>
> hello
>
>
>
> I have one use-case that spans mu
hello
I have one use-case that spans multiple map tasks in hadoop environment. I
use hadoop 1.2.1 and with 6 task nodes. Each map task writes their output
into a file stored in hdfs. This file is shared across all the map tasks.
Though, they all computes thier output but some of them are missing
