Thanks Bipin for the response. To add more details: Whenever a new topology is deployed nimbus doesn't respond and supervisor(s) go down. We have to restart the services to bring the cluster back to normal.
Another error we see: o.a.s.b.BlobStoreUtils BLOB-STORE-TIMER [ERROR] Could not download the blob with key: topology_B-7-1704896101-stormjar.jar There is a stale entry in the zk path: /storm/blobstore/topology_B-7- 1704896101-stormjar.jar. Not sure why it was not getting cleared off. This is pretty consistent. This error goes away after manually deleting stale entry from zk path: rmr /storm/blobstore/topology_B-7-1704896101-stormjar.jar Thanks, Devendar On Thu, Jan 18, 2024 at 11:28 AM Devendar Rao <devendar.gu...@gmail.com> wrote: > Full stack trace: > > 2024-01-18 18:27:12.889 o.a.s.d.n.Nimbus pool-29-thread-22 [WARN] get blob > meta exception. org.apache.storm.utils.WrappedKeyNotFoundException: > topology-A-7-1704896101-stormjar.jar at org.apache.storm.blobstore. > LocalFsBlobStore.getStoredBlobMeta(LocalFsBlobStore.java:258) ~[storm- > server-2.3.0.jar:2.3.0] at org.apache.storm.blobstore.LocalFsBlobStore. > getBlobMeta(LocalFsBlobStore.java:288) ~[storm-server-2.3.0.jar:2.3.0] at > org.apache.storm.daemon.nimbus.Nimbus.getBlobMeta(Nimbus.java:3815) [storm > -server-2.3.0.jar:2.3.0] at org.apache.storm.generated.Nimbus$Processor$ > getBlobMeta.getResult(Nimbus.java:4278) [storm-client-2.3.0.jar:2.3.0] at > org.apache.storm.generated.Nimbus$Processor$getBlobMeta.getResult(Nimbus. > java:4257) [storm-client-2.3.0.jar:2.3.0] at org.apache.storm.thrift. > ProcessFunction.process(ProcessFunction.java:38) [storm-shaded-deps-2.3.0. > jar:2.3.0] at org.apache.storm.thrift.TBaseProcessor.process( > TBaseProcessor.java:38) [storm-shaded-deps-2.3.0.jar:2.3.0] at org.apache. > storm.security.auth.SimpleTransportPlugin$SimpleWrapProcessor.process( > SimpleTransportPlugin.java:172) [storm-client-2.3.0.jar:2.3.0] at org. > apache.storm.thrift.server.AbstractNonblockingServer$FrameBuffer.invoke( > AbstractNonblockingServer.java:524) [storm-shaded-deps-2.3.0.jar:2.3.0] at > org.apache.storm.thrift.server.Invocation.run(Invocation.java:18) [storm- > shaded-deps-2.3.0.jar:2.3.0] at java.util.concurrent.ThreadPoolExecutor. > runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.util.concurrent. > ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at java. > lang.Thread.run(Thread.java:834) [?:?] > > On Thu, Jan 18, 2024 at 11:26 AM Bipin Prasad > <bipin_pra...@yahoo.com.invalid> wrote: > >> Can you post the full stack trace? I want to confirm that this is logged >> while trying to obtain the heartbeat.This message is a warning message and >> is not expected to shutdown nimbus. >> >> >> Sent from Yahoo Mail for iPhone >> >> >> On Thursday, January 18, 2024, 11:19 AM, Devendar Rao < >> devendar.gu...@gmail.com> wrote: >> >> Hi, >> We're constantly seeing issues in storm 2.3.0 with blobs with each >> topology deployment. Supervisor/nimbus dies after seeing the below >> exceptions. >> Is this a known issue? Are we hitting any blob cache size limits? >> Exceptions: >> "Could not download the blob with key:" >> o.a.s.d.n.Nimbus pool-29-thread-22 [WARN] >> .org.apache.storm.utils.WrappedKeyNotFoundException >> >> >> Can someone please shed some light on this? >> Thanks >> >> >> >>
