Re: How to Set Retry Policy in Spark

Additional Info: I am running Spark on YARN. 2015-10-01 15:42 GMT-07:00 Renxia Wang : > Hi guys, > > I know there is a way to set the number of retry of failed tasks, using > spark.task.maxFailures. what is the default policy for the failed tasks > retry? Is it exponential

How to Set Retry Policy in Spark

Hi guys, I know there is a way to set the number of retry of failed tasks, using spark.task.maxFailures. what is the default policy for the failed tasks retry? Is it exponential backoff? My tasks sometimes failed because of Socket connection timeout/reset, even with retry, some of the tasks will