Found a setting that seems to fix this problem, but it does not seems to be available until Spark 1.3. See https://issues.apache.org/jira/browse/SPARK-2165
However, glad to see a work is being done with the issue. On Tue, Jan 13, 2015 at 8:00 PM, Anders Arpteg <arp...@spotify.com> wrote: > Yes Andrew, I am. Tried setting spark.yarn.applicationMaster.waitTries to > 1 (thanks Sean), but with no luck. Any ideas? > > On Tue, Jan 13, 2015 at 7:58 PM, Andrew Or <and...@databricks.com> wrote: > >> Hi Anders, are you using YARN by any chance? >> >> 2015-01-13 0:32 GMT-08:00 Anders Arpteg <arp...@spotify.com>: >> >> Since starting using Spark 1.2, I've experienced an annoying issue with >>> failing apps that gets executed twice. I'm not talking about tasks inside a >>> job, that should be executed multiple times before failing the whole app. >>> I'm talking about the whole app, that seems to close the previous Spark >>> context, start a new, and rerun the app again. >>> >>> This is annoying since it overwrite the log files as well and it becomes >>> hard to troubleshoot the failing app. Does anyone know how to turn this >>> "feature" off? >>> >>> Thanks, >>> Anders >>> >> >> >
