[
https://issues.apache.org/jira/browse/STORM-3769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17335791#comment-17335791
]
Aaron Gresch commented on STORM-3769:
-------------------------------------
In this case, a worker for topology A was no longer assigned on a supervisor.
At some point the blob cache exceeded the size limit. When cleanup was called,
1 or 2 of the topology A's blobs were deleted, and this was enough to get under
the cache size limit. Because some of the topology blobs remain, the code
currently assumes the blobs all remain downloaded. Then when a worker for
topology A gets reassigned back to the node, this exception occurs and workers
will be unable to start.
> Failed adding references to blobs: FileNotFoundException
> --------------------------------------------------------
>
> Key: STORM-3769
> URL: https://issues.apache.org/jira/browse/STORM-3769
> Project: Apache Storm
> Issue Type: Bug
> Reporter: Aaron Gresch
> Assignee: Aaron Gresch
> Priority: Major
>
> We hit a file not found exception with AsyncLocalizer:
> {code:java}
> 2021-04-23 17:39:13.380 o.a.s.l.AsyncLocalizer
> ForkJoinPool.commonPool-worker-23 [ERROR] Failed adding references to blobs
> for TimePortAndAssignment{xxx-1-15-1616201755 on 6708}
> java.io.FileNotFoundException: File
> '/home/y/var/storm/supervisor/stormdist/xxx-1-15-1616201755/stormconf.ser'
> does not exist
> at
> org.apache.storm.shade.org.apache.commons.io.FileUtils.openInputStream(FileUtils.java:297)
> ~[storm-shaded-deps-2.3.0.y.jar:2.3.0.y]
> at
> org.apache.storm.shade.org.apache.commons.io.FileUtils.readFileToByteArray(FileUtils.java:1851)
> ~[storm-shaded-deps-2.3.0.y.jar:2.3.0.y]
> at
> org.apache.storm.utils.ConfigUtils.readSupervisorStormConfGivenPath(ConfigUtils.java:311)
> ~[storm-client-2.3.0.y.jar:2.3.0.y]
> at
> org.apache.storm.utils.ConfigUtils.readSupervisorStormConfImpl(ConfigUtils.java:472)
> ~[storm-client-2.3.0.y.jar:2.3.0.y]
> at
> org.apache.storm.utils.ConfigUtils.readSupervisorStormConf(ConfigUtils.java:306)
> ~[storm-client-2.3.0.y.jar:2.3.0.y]
> at
> org.apache.storm.localizer.AsyncLocalizer.getLocalResources(AsyncLocalizer.java:368)
> ~[storm-server-2.3.0.y.jar:2.3.0.y]
> at
> org.apache.storm.localizer.AsyncLocalizer.addReferencesToBlobs(AsyncLocalizer.java:398)
> ~[storm-server-2.3.0.y.jar:2.3.0.y]
> at
> org.apache.storm.localizer.AsyncLocalizer.lambda$null$7(AsyncLocalizer.java:235)
> ~[storm-server-2.3.0.y.jar:2.3.0.y]
> at
> java.util.concurrent.ConcurrentHashMap.compute(ConcurrentHashMap.java:1877)
> ~[?:1.8.0_262]
> at
> org.apache.storm.localizer.AsyncLocalizer.lambda$requestDownloadTopologyBlobs$8(AsyncLocalizer.java:229)
> ~[storm-server-2.3.0.y.jar:2.3.0.y]
> at
> java.util.concurrent.CompletableFuture.uniCompose(CompletableFuture.java:966)
> [?:1.8.0_262]
> at
> java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:940)
> [?:1.8.0_262]
> at
> java.util.concurrent.CompletableFuture$Completion.exec(CompletableFuture.java:457)
> [?:1.8.0_262]
> at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
> [?:1.8.0_262]
> at
> java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
> [?:1.8.0_262]
> at
> java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
> [?:1.8.0_262]
> at
> java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:163)
> [?:1.8.0_262]
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
