Thanks, I was unaware of mapred.max.map.failures.percent
-Håvard
On Sun, Jan 6, 2013 at 3:46 PM, Harsh J wrote:
> You can use the mapred.max.map.failures.percent and
> mapred.max.reduce.failures.percent features to control the percentage
> of allowed failures of tasks in a single job (despite wh
You can use the mapred.max.map.failures.percent and
mapred.max.reduce.failures.percent features to control the percentage
of allowed failures of tasks in a single job (despite which the job is
marked successful).
On Sun, Jan 6, 2013 at 8:04 PM, Håvard Wahl Kongsgård
wrote:
>> Are tasks being exec
> Are tasks being executed multiple times due to failures? Sorry, it was not
> very clear from your question.
yes, and I simply want to skip them if they fail more than x
times(after all this is big data :) ).
-Håvard
On Sun, Jan 6, 2013 at 3:01 PM, Hemanth Yamijala
wrote:
> Hi,
>
> Are tasks b
Hi,
Are tasks being executed multiple times due to failures? Sorry, it was not
very clear from your question.
Thanks
hemanth
On Sat, Jan 5, 2013 at 7:44 PM, David Parks wrote:
> Thinking here... if you submitted the task programmatically you should be
> able to capture the failure of the task
yes, but I use pydoop not the native java library. The problem is that
the same task times, so a solution is not that straightforward. And
Pydoop does not seem to have any methods to inform the task how many
times it has failed. So if there is no native method in hadoop, I
could use a database or s
Thinking here... if you submitted the task programmatically you should be
able to capture the failure of the task and gracefully move past it to your
next tasks.
To say it in a long-winded way: Let's say you submit a job to Hadoop, a
java jar, and your main class implements Tool. That code has th
