Since migrating from Storm 1.06 to 2.1.0, I am seeing some strange warnings in
nimbus log.
I am getting a lot of warnings such as:
2020-04-25 20:22:05.604 o.a.s.d.n.Nimbus timer [WARN] Negative fragmented CPU
on 9971b56c-3549-445a-9132-c5f2185cb2f8-10.4.0.5
2020-04-25 20:22:05.605 o.a.s.d.n.Nimbus timer [WARN] Negative fragmented Mem
on 9971b56c-3549-445a-9132-c5f2185cb2f8-10.4.0.5
I understand if the message was not enough memory, but what does negative
memory mean??
Also, during deployment of topologies (after undeploying the previous versions
of these topologies) I am getting ~10 warnings per topology saying:
2020-04-25 20:20:01.653 o.a.s.d.n.Nimbus pool-29-thread-50 [WARN] get blob meta
exception.
org.apache.storm.utils.WrappedKeyNotFoundException:
summary-generator-17-1587642602-stormcode.ser
at
org.apache.storm.blobstore.LocalFsBlobStore.getStoredBlobMeta(LocalFsBlobStore.java:257)
~[storm-server-2.1.0.jar:2.1.0]
at
org.apache.storm.blobstore.LocalFsBlobStore.getBlobMeta(LocalFsBlobStore.java:287)
~[storm-server-2.1.0.jar:2.1.0]
at
org.apache.storm.daemon.nimbus.Nimbus.getBlobMeta(Nimbus.java:3651)
[storm-server-2.1.0.jar:2.1.0]
at
org.apache.storm.generated.Nimbus$Processor$getBlobMeta.getResult(Nimbus.java:3951)
[storm-client-2.1.0.jar:2.1.0]
at
org.apache.storm.generated.Nimbus$Processor$getBlobMeta.getResult(Nimbus.java:3930)
[storm-client-2.1.0.jar:2.1.0]
at
org.apache.storm.thrift.ProcessFunction.process(ProcessFunction.java:38)
[storm-shaded-deps-2.1.0.jar:2.1.0]
at
org.apache.storm.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
[storm-shaded-deps-2.1.0.jar:2.1.0]
at
org.apache.storm.security.auth.SimpleTransportPlugin$SimpleWrapProcessor.process(SimpleTransportPlugin.java:174)
[storm-client-2.1.0.jar:2.1.0]
at
org.apache.storm.thrift.server.AbstractNonblockingServer$FrameBuffer.invoke(AbstractNonblockingServer.java:518)
[storm-shaded-deps-2.1.0.jar:2.1.0]
at
org.apache.storm.thrift.server.Invocation.run(Invocation.java:18)
[storm-shaded-deps-2.1.0.jar:2.1.0]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[?:1.8.0_252]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[?:1.8.0_252]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_252]
The strange thing is that the reported version
“summary-generator-17-1587642602” is the instance that was just undeployed, so
it really should not be in the blob store.
Not sure at this point if this is some misconfiguration at my side, or
something else.
Yuval Lieberman
Team Leader, 2bPrecise
IL: +972.52.5311.790 | M
2bPreciseHealth.com
It’s Time 2bPrecise
