Hello Devender, The zookeeper entry is expected to be removed when the topology is killed. It appears that you are using local file store blob. So the blobs are (expected to be) on the nimbus. Can you check the mailbox directory and confirm whether or not the blob for this topology made it that far? It is possible that there is some issue with disk space on the nimbus host? —Bipin
Sent from Yahoo Mail for iPhone On Thursday, January 18, 2024, 11:39 AM, Devendar Rao <devendar.gu...@gmail.com> wrote: Thanks Bipin for the response. To add more details: Whenever a new topology is deployed nimbus doesn't respond and supervisor(s) go down. We have to restart the services to bring the cluster back to normal. Another error we see: o.a.s.b.BlobStoreUtils BLOB-STORE-TIMER [ERROR] Could not download the blob with key: topology_B-7-1704896101-stormjar.jar There is a stale entry in the zk path: /storm/blobstore/topology_B-7-1704896101-stormjar.jar. Not sure why it was not getting cleared off. This is pretty consistent. This error goes away after manually deleting stale entry from zk path: rmr /storm/blobstore/topology_B-7-1704896101-stormjar.jar Thanks,Devendar On Thu, Jan 18, 2024 at 11:28 AM Devendar Rao <devendar.gu...@gmail.com> wrote: Full stack trace: 2024-01-18 18:27:12.889 o.a.s.d.n.Nimbus pool-29-thread-22 [WARN] get blob meta exception.org.apache.storm.utils.WrappedKeyNotFoundException: topology-A-7-1704896101-stormjar.jar at org.apache.storm.blobstore.LocalFsBlobStore.getStoredBlobMeta(LocalFsBlobStore.java:258) ~[storm-server-2.3.0.jar:2.3.0] at org.apache.storm.blobstore.LocalFsBlobStore.getBlobMeta(LocalFsBlobStore.java:288) ~[storm-server-2.3.0.jar:2.3.0] at org.apache.storm.daemon.nimbus.Nimbus.getBlobMeta(Nimbus.java:3815) [storm-server-2.3.0.jar:2.3.0] at org.apache.storm.generated.Nimbus$Processor$getBlobMeta.getResult(Nimbus.java:4278) [storm-client-2.3.0.jar:2.3.0] at org.apache.storm.generated.Nimbus$Processor$getBlobMeta.getResult(Nimbus.java:4257) [storm-client-2.3.0.jar:2.3.0] at org.apache.storm.thrift.ProcessFunction.process(ProcessFunction.java:38) [storm-shaded-deps-2.3.0.jar:2.3.0] at org.apache.storm.thrift.TBaseProcessor.process(TBaseProcessor.java:38) [storm-shaded-deps-2.3.0.jar:2.3.0] at org.apache.storm.security.auth.SimpleTransportPlugin$SimpleWrapProcessor.process(SimpleTransportPlugin.java:172) [storm-client-2.3.0.jar:2.3.0] at org.apache.storm.thrift.server.AbstractNonblockingServer$FrameBuffer.invoke(AbstractNonblockingServer.java:524) [storm-shaded-deps-2.3.0.jar:2.3.0] at org.apache.storm.thrift.server.Invocation.run(Invocation.java:18) [storm-shaded-deps-2.3.0.jar:2.3.0] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at java.lang.Thread.run(Thread.java:834) [?:?] On Thu, Jan 18, 2024 at 11:26 AM Bipin Prasad <bipin_pra...@yahoo.com.invalid> wrote: Can you post the full stack trace? I want to confirm that this is logged while trying to obtain the heartbeat.This message is a warning message and is not expected to shutdown nimbus. Sent from Yahoo Mail for iPhone On Thursday, January 18, 2024, 11:19 AM, Devendar Rao <devendar.gu...@gmail.com> wrote: Hi, We're constantly seeing issues in storm 2.3.0 with blobs with each topology deployment. Supervisor/nimbus dies after seeing the below exceptions. Is this a known issue? Are we hitting any blob cache size limits? Exceptions: "Could not download the blob with key:" o.a.s.d.n.Nimbus pool-29-thread-22 [WARN] .org.apache.storm.utils.WrappedKeyNotFoundException Can someone please shed some light on this? Thanks
