[
https://issues.apache.org/jira/browse/SPARK-40469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Apache Spark reassigned SPARK-40469:
------------------------------------
Assignee: (was: Apache Spark)
> Avoid creating directory failures
> ---------------------------------
>
> Key: SPARK-40469
> URL: https://issues.apache.org/jira/browse/SPARK-40469
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Affects Versions: 3.4.0
> Reporter: Yuming Wang
> Priority: Major
>
> {noformat}
> java.nio.file.NoSuchFileException:
> /hadoop/3/yarn/local/usercache/b_carmel/appcache/application_1654776504115_37917/blockmgr-e18b484f-8c49-4c7d-b649-710439b0e4c3/3c
> at
> sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
> at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
> at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
> at
> sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:384)
> at java.nio.file.Files.createDirectory(Files.java:674)
> at
> org.apache.spark.storage.DiskBlockManager.getFile(DiskBlockManager.scala:123)
> at
> org.apache.spark.storage.DiskBlockManager.getFile(DiskBlockManager.scala:146)
> at org.apache.spark.storage.DiskStore.contains(DiskStore.scala:147)
> at
> org.apache.spark.storage.BlockManager.getLocalValues(BlockManager.scala:853)
> at
> org.apache.spark.broadcast.TorrentBroadcast.$anonfun$readBroadcastBlock$4(TorrentBroadcast.scala:253)
> at scala.Option.getOrElse(Option.scala:189)
> at
> org.apache.spark.broadcast.TorrentBroadcast.$anonfun$readBroadcastBlock$2(TorrentBroadcast.scala:250)
> at org.apache.spark.util.KeyLock.withLock(KeyLock.scala:64)
> at
> org.apache.spark.broadcast.TorrentBroadcast.$anonfun$readBroadcastBlock$1(TorrentBroadcast.scala:245)
> at org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:1383)
> at
> org.apache.spark.broadcast.TorrentBroadcast.readBroadcastBlock(TorrentBroadcast.scala:245)
> at
> org.apache.spark.broadcast.TorrentBroadcast.getValue(TorrentBroadcast.scala:109)
> at org.apache.spark.broadcast.Broadcast.value(Broadcast.scala:70)
> at
> org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:86)
> at
> org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:52)
> at org.apache.spark.scheduler.Task.run(Task.scala:132)
> at
> org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:487)
> at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1417)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:490)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
