[ https://issues.apache.org/jira/browse/STORM-3733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aaron Gresch resolved STORM-3733. --------------------------------- Fix Version/s: 2.3.0 Resolution: Fixed > AsyncLocalizer stuck looking for missing topology > ------------------------------------------------- > > Key: STORM-3733 > URL: https://issues.apache.org/jira/browse/STORM-3733 > Project: Apache Storm > Issue Type: Bug > Reporter: Aaron Gresch > Assignee: Aaron Gresch > Priority: Major > Fix For: 2.3.0 > > Time Spent: 50m > Remaining Estimate: 0h > > {code:java} > 2020-11-09 20:18:12.325 o.a.s.l.AsyncLocalizer AsyncLocalizer Task Executor - > 2 [ERROR] Could not update blob, will retry again later > 2020-11-09 20:18:43.744 o.a.s.l.AsyncLocalizer AsyncLocalizer Task Executor - > 2 [ERROR] Could not update blob, will retry again later > 2020-11-09 20:19:14.726 o.a.s.l.AsyncLocalizer AsyncLocalizer Task Executor - > 2 [ERROR] Could not update blob, will retry again later > 2020-11-09 20:19:46.148 o.a.s.l.AsyncLocalizer AsyncLocalizer Task Executor - > 2 [ERROR] Could not update blob, will retry again later > 2020-11-09 20:20:16.560 o.a.s.l.AsyncLocalizer AsyncLocalizer Task Executor - > 0 [ERROR] Could not update blob, will retry again later > 2020-11-09 20:20:47.990 o.a.s.l.AsyncLocalizer AsyncLocalizer Task Executor - > 0 [ERROR] Could not update blob, will retry again later > 2020-11-09 20:21:19.403 o.a.s.l.AsyncLocalizer AsyncLocalizer Task Executor - > 0 [ERROR] Could not update blob, will retry again later > 2020-11-09 20:21:50.818 o.a.s.l.AsyncLocalizer AsyncLocalizer Task Executor - > 0 [ERROR] Could not update blob, will retry again later > 2020-11-09 20:22:21.257 o.a.s.l.AsyncLocalizer AsyncLocalizer Task Executor - > 1 [ERROR] Could not update blob, will retry again later > 2020-11-09 20:22:52.668 o.a.s.l.AsyncLocalizer AsyncLocalizer Task Executor - > 1 [ERROR] Could not update blob, will retry again later > 2020-11-09 20:23:24.082 o.a.s.l.AsyncLocalizer AsyncLocalizer Task Executor - > 1 [ERROR] Could not update blob, will retry again later > 2020-11-09 20:23:55.512 o.a.s.l.AsyncLocalizer AsyncLocalizer Task Executor - > 1 [ERROR] Could not update blob, will retry again later > 2020-11-09 20:24:25.919 o.a.s.l.AsyncLocalizer AsyncLocalizer Task Executor - > 2 [ERROR] Could not update blob, will retry again later > 2020-11-09 20:24:57.343 o.a.s.l.AsyncLocalizer AsyncLocalizer Task Executor - > 2 [ERROR] Could not update blob, will retry again later > 2020-11-09 20:25:28.773 o.a.s.l.AsyncLocalizer AsyncLocalizer Task Executor - > 2 [ERROR] Could not update blob, will retry again later > 2020-11-09 20:16:09.659 o.a.s.l.AsyncLocalizer AsyncLocalizer Task Executor - > 1 [ERROR] Could not update blob, will retry again later > java.util.concurrent.ExecutionException: java.lang.RuntimeException: Could > not download... > at > java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) > ~[?:1.8.0_262] > at > java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908) > ~[?:1.8.0_262] > at > org.apache.storm.localizer.AsyncLocalizer.updateBlobs(AsyncLocalizer.java:333) > ~[storm-server-2.3.0.y.jar:2.3.0.y] > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > [?:1.8.0_262] > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) > [?:1.8.0_262] > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) > [?:1.8.0_262] > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) > [?:1.8.0_262] > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > [?:1.8.0_262] > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > [?:1.8.0_262] > at java.lang.Thread.run(Thread.java:748) [?:1.8.0_262] > Caused by: java.lang.RuntimeException: Could not download... > at > org.apache.storm.localizer.AsyncLocalizer.lambda$downloadOrUpdate$10(AsyncLocalizer.java:297) > ~[storm-server-2.3.0.y.jar:2.3.0.y] > at > java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1640) > ~[?:1.8.0_262] > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > ~[?:1.8.0_262] > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > ~[?:1.8.0_262] > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) > ~[?:1.8.0_262] > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > ~[?:1.8.0_262] > ... 3 more > Caused by: org.apache.storm.utils.WrappedKeyNotFoundException: > testTopology-743-1603821880-stormconf.ser > at > org.apache.storm.hdfs.blobstore.HdfsBlobStore.getStoredBlobMeta(HdfsBlobStore.java:192) > ~[storm-hdfs-blobstore-2.3.0.y.jar:2.3.0.y] > at > org.apache.storm.hdfs.blobstore.HdfsBlobStore.getBlobMeta(HdfsBlobStore.java:221) > ~[storm-hdfs-blobstore-2.3.0.y.jar:2.3.0.y] > at > org.apache.storm.hdfs.blobstore.HdfsClientBlobStore.getBlobMeta(HdfsClientBlobStore.java:73) > ~[storm-hdfs-blobstore-2.3.0.y.jar:2.3.0.y] > at > org.apache.storm.localizer.LocallyCachedTopologyBlob.getRemoteVersion(LocallyCachedTopologyBlob.java:129) > ~[storm-server-2.3.0.y.jar:2.3.0.y] > at > org.apache.storm.localizer.AsyncLocalizer.lambda$downloadOrUpdate$10(AsyncLocalizer.java:273) > ~[storm-server-2.3.0.y.jar:2.3.0.y] > at > java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1640) > ~[?:1.8.0_262] > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > ~[?:1.8.0_262] > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > ~[?:1.8.0_262] > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) > ~[?:1.8.0_262] > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > ~[?:1.8.0_262] > ... 3 more > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005)