-1, I think this issue is a blocker. It's caused by https://github.com/apache/storm/pull/3093/files, which prevents the localizer from deleting blobs that are deleted from Nimbus, even if those blobs are only scheduled for download.
Have opened https://github.com/apache/storm/pull/3119 Den tir. 27. aug. 2019 kl. 19.54 skrev Stig Rohde Døssing < [email protected]>: > I think there may be an issue with topology cleanup > I ran WordCountTopology for a bit, then killed it. The supervisor seems to > keep trying to download the topology jar indefinitely. > > 2019-08-27 19:49:53.459 o.a.s.l.AsyncLocalizer AsyncLocalizer Executor - 1 > [WARN] Failed to download blob LOCAL TOPO BLOB TOPO_JAR > word-count-1-1566926844 will try again in 100 ms > org.apache.storm.generated.KeyNotFoundException: null > at > org.apache.storm.generated.Nimbus$getBlobMeta_result$getBlobMeta_resultStandardScheme.read(Nimbus.java:25919) > ~[storm-client-2.1.0.jar:2.1.0] > at > org.apache.storm.generated.Nimbus$getBlobMeta_result$getBlobMeta_resultStandardScheme.read(Nimbus.java:25887) > ~[storm-client-2.1.0.jar:2.1.0] > at > org.apache.storm.generated.Nimbus$getBlobMeta_result.read(Nimbus.java:25818) > ~[storm-client-2.1.0.jar:2.1.0] > at > org.apache.storm.thrift.TServiceClient.receiveBase(TServiceClient.java:88) > ~[storm-shaded-deps-2.1.0.jar:2.1.0] > at > org.apache.storm.generated.Nimbus$Client.recv_getBlobMeta(Nimbus.java:794) > ~[storm-client-2.1.0.jar:2.1.0] > at org.apache.storm.generated.Nimbus$Client.getBlobMeta(Nimbus.java:781) > ~[storm-client-2.1.0.jar:2.1.0] > at > org.apache.storm.blobstore.NimbusBlobStore.getBlobMeta(NimbusBlobStore.java:85) > ~[storm-client-2.1.0.jar:2.1.0] > at > org.apache.storm.localizer.LocallyCachedTopologyBlob.getRemoteVersion(LocallyCachedTopologyBlob.java:127) > ~[storm-server-2.1.0.jar:2.1.0] > at > org.apache.storm.localizer.AsyncLocalizer.lambda$downloadOrUpdate$10(AsyncLocalizer.java:264) > ~[storm-server-2.1.0.jar:2.1.0] > at > java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1626) > [?:1.8.0_144] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > [?:1.8.0_144] > at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_144] > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) > [?:1.8.0_144] > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > [?:1.8.0_144] > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > [?:1.8.0_144] > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > [?:1.8.0_144] > at java.lang.Thread.run(Thread.java:748) [?:1.8.0_144] > > These repeat, intermittently broken up by > > Caused by: java.lang.RuntimeException: Could not download... > at > org.apache.storm.localizer.AsyncLocalizer.lambda$downloadOrUpdate$10(AsyncLocalizer.java:285) > ~[storm-server-2.1.0.jar:2.1.0] > at > java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1626) > ~[?:1.8.0_144] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > ~[?:1.8.0_144] > at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_144] > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) > ~[?:1.8.0_144] > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > ~[?:1.8.0_144] > ... 3 more > Caused by: org.apache.storm.generated.KeyNotFoundException > at > org.apache.storm.generated.Nimbus$getBlobMeta_result$getBlobMeta_resultStandardScheme.read(Nimbus.java:25919) > ~[storm-client-2.1.0.jar:2.1.0] > at > org.apache.storm.generated.Nimbus$getBlobMeta_result$getBlobMeta_resultStandardScheme.read(Nimbus.java:25887) > ~[storm-client-2.1.0.jar:2.1.0] > at > org.apache.storm.generated.Nimbus$getBlobMeta_result.read(Nimbus.java:25818) > ~[storm-client-2.1.0.jar:2.1.0] > at > org.apache.storm.thrift.TServiceClient.receiveBase(TServiceClient.java:88) > ~[storm-shaded-deps-2.1.0.jar:2.1.0] > at > org.apache.storm.generated.Nimbus$Client.recv_getBlobMeta(Nimbus.java:794) > ~[storm-client-2.1.0.jar:2.1.0] > at org.apache.storm.generated.Nimbus$Client.getBlobMeta(Nimbus.java:781) > ~[storm-client-2.1.0.jar:2.1.0] > at > org.apache.storm.blobstore.NimbusBlobStore.getBlobMeta(NimbusBlobStore.java:85) > ~[storm-client-2.1.0.jar:2.1.0] > at > org.apache.storm.localizer.LocallyCachedTopologyBlob.getRemoteVersion(LocallyCachedTopologyBlob.java:127) > ~[storm-server-2.1.0.jar:2.1.0] > at > org.apache.storm.localizer.AsyncLocalizer.lambda$downloadOrUpdate$10(AsyncLocalizer.java:264) > ~[storm-server-2.1.0.jar:2.1.0] > at > java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1626) > ~[?:1.8.0_144] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > ~[?:1.8.0_144] > at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_144] > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) > ~[?:1.8.0_144] > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > ~[?:1.8.0_144] > ... 3 more > > Restarting the supervisor seems to fix it. > > Here's what I did before hitting this issue > > Verified asc and sha512 files. > Built Storm from the zipped source. > Extracted zipped binary and started dev-zookeeper, nimbus, supervisor, > logviewer, ui. > Ran WordCountTopology for a while. > Checked rebalance, deactivate/activate, worker log viewing, kill from the > UI. > > Minor things we might want to handle: > It might be good to put your key up on a keyserver e.g. > http://pgp.mit.edu/, as that is where the > https://www.apache.org/info/verification.html page suggests users go to > find keys. > We should add a note to the Storm downloads page that links to the > verification page. Currently users aren't informed how to check signatures > and checksums. > > Den tir. 20. aug. 2019 kl. 23.05 skrev Ethan Li <[email protected]>: > >> This is a call to vote on releasing Apache Storm 2.1.0 (rc3) >> >> Full list of changes in this release: >> >> >> https://dist.apache.org/repos/dist/dev/storm/apache-storm-2.1.0-rc3/RELEASE_NOTES.html >> >> The tag/commit to be voted upon is v2.1.0: >> >> >> https://gitbox.apache.org/repos/asf?p=storm.git;a=tree;h=4288646ef99b97f9737dd9c21b7b6b62c8b17afa;hb=6562d6ee438c2858d92a51bd30ca0a1f2f623c77 >> >> The source archive being voted upon can be found here: >> >> >> https://dist.apache.org/repos/dist/dev/storm/apache-storm-2.1.0-rc3/apache-storm-2.1.0-src.tar.gz >> >> Other release files, signatures and digests can be found here: >> >> https://dist.apache.org/repos/dist/dev/storm/apache-storm-2.1.0-rc3 >> >> The release artifacts are signed with the following key: >> >> https://www.apache.org/dist/storm/KEYS >> >> The Nexus staging repository for this release is: >> >> https://repository.apache.org/content/repositories/orgapachestorm-1087/ >> >> Please vote on releasing this package as Apache Storm 2.1.0. >> >> When voting, please list the actions taken to verify the release. >> >> This vote will be open for at least 72 hours. >> >> [ ] +1 Release this package as Apache Storm 2.1.0 >> [ ] 0 No opinion >> [ ] -1 Do not release this package because... >> >> Thanks to everyone who contributed to this release. >> >> Ethan >> >
