[ https://issues.apache.org/jira/browse/SPARK-2324?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aaron Davidson resolved SPARK-2324. ----------------------------------- Resolution: Fixed Resolved by https://github.com/apache/spark/pull/1274 > SparkContext should not exit directly when spark.local.dir is a list of > multiple paths and one of them has error > ---------------------------------------------------------------------------------------------------------------- > > Key: SPARK-2324 > URL: https://issues.apache.org/jira/browse/SPARK-2324 > Project: Spark > Issue Type: Bug > Components: Spark Core > Reporter: YanTang Zhai > > The spark.local.dir is configured as a list of multiple paths as follows > /data1/sparkenv/local,/data2/sparkenv/local. If the disk data2 of the driver > node has error, the application will exit since DiskBlockManager exits > directly at createLocalDirs. If the disk data2 of the worker node has error, > the executor will exit either. > DiskBlockManager should not exit directly at createLocalDirs if one of > spark.local.dir has error. Since spark.local.dir has multiple paths, a > problem should not affect the overall situation. > I think DiskBlockManager could ignore the bad directory at createLocalDirs. -- This message was sent by Atlassian JIRA (v6.2#6252)