Yazhi Wang created SPARK-44469: ---------------------------------- Summary: Utils.getOrCreateLocalRootDirs will never take effect after the first call fails, even if the exception is recovered Key: SPARK-44469 URL: https://issues.apache.org/jira/browse/SPARK-44469 Project: Spark Issue Type: Improvement Components: Spark Core Affects Versions: 3.4.1 Reporter: Yazhi Wang
{code:java} private[spark] def getOrCreateLocalRootDirs(conf: SparkConf): Array[String] = { if (localRootDirs == null || localRootDirs.isEmpty) { this.synchronized { if (localRootDirs == null) { localRootDirs = getOrCreateLocalRootDirsImpl(conf) } } } localRootDirs }{code} localRootDirs is only initialized once in the Executor/Driver life cycle. If it fails due to a FileSystem exception (such as a full disk) during the first initialization, localRootDirs will be assigned a value of None instead of null. Even if the FileSystem exception recovered, the localRootDirs won't re-apply from FileSystem, causing the task on the Executor to continue to fail (tasks relay on fetchFiles locally) -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org