Had a similar situation and landed on this question.
Finally I was able to make it do what I needed by cheating the spark driver
:)
i.e By setting a very high value for "--conf spark.task.maxFailures=800".
I made it 800 deliberately which typically is 4. So by the time 800
attempts for failed
Hi,
I know a task can fail 2 times and only the 3rd breaks the entire job.
I am good with this number of attempts.
I would like that after trying a task 3 times, it continues with the other
tasks.
The job can be "failed", but I want all tasks run.
Please see my use case.
I read a hadoop
I searched the code base and looked at:
https://spark.apache.org/docs/latest/running-on-yarn.html
I didn't find mapred.max.map.failures.percent or its counterpart.
FYI
On Fri, Nov 13, 2015 at 9:05 AM, Nicolae Marasoiu <
nicolae.maras...@adswizz.com> wrote:
> Hi,
>
>
> I know a task can fail 2
