These are EC2 instances? Or is Storm running in AWS EKS? Socket timeout is
generally 60 seconds, but can you verify from your log timestamps to infer what
your socket timeout might be.
Sent from Yahoo Mail for iPhone
On Friday, January 19, 2024, 12:45 PM, Devendar Rao
wrote:
The size of th
The size of the jar is 270M. We've 19 topologies each with 270M jar size.
Yes, we use AWS EBS volumes for supervisors and Nimbus uses the local
Nvme(instance store).
Do you think increasing this topology.max.replication.wait.time.sec would
help?
On Fri, Jan 19, 2024 at 12:29 PM Bipin Prasad
wrot
What is the size of the blob (the larger is probably the jar). The download
time would depend on the network speed. Also noticed the name was f the mounted
file system as ephemeral. These are not persistent storage? How about on the
nimbus?
Sent from Yahoo Mail for iPhone
On Friday, January
We also see these warnings during the issue. Are there any settings which
controls the availability of these files.
Is there a setting which controls the wokrer to to keep in wait till the
required files are available?
Do we need to increase the value for this setting -
topology.max.replication.w
Hi Bipin,
There are no disk space issues on the nimbus host. We also see this
exception. I think this is the effect not the cause. Not completely sure
though.
2024-01-18 11:47:16.018 o.a.s.t.ProcessFunction pool-29-thread-60 [ERROR]
Internal error processing beginBlobDownload
java.lang.RuntimeExc
Hi,
We're constantly seeing issues in storm 2.3.0 with blobs with each topology
deployment. Supervisor/nimbus dies after seeing the below exceptions.
Is this a known issue? Are we hitting any blob cache size limits?
Exceptions:
"Could not download the blob with key:"
o.a.s.d.n.Nimbus pool-29-t
Hello Devender, The zookeeper entry is expected to be removed when the
topology is killed. It appears that you are using local file store blob. So the
blobs are (expected to be) on the nimbus. Can you check the mailbox directory
and confirm whether or not the blob for this topology made it tha
: rmr
/storm/blobstore/topology_B-7-1704896101-stormjar.jar
Thanks,
Devendar
On Thu, Jan 18, 2024 at 11:28 AM Devendar Rao
wrote:
> Full stack trace:
>
> 2024-01-18 18:27:12.889 o.a.s.d.n.Nimbus pool-29-thread-22 [WARN] get blob
> meta exception. org.apache.storm.utils.WrappedKey
Full stack trace:
2024-01-18 18:27:12.889 o.a.s.d.n.Nimbus pool-29-thread-22 [WARN] get blob
meta exception. org.apache.storm.utils.WrappedKeyNotFoundException:
topology-A-7-1704896101-stormjar.jar at org.apache.storm.blobstore.
LocalFsBlobStore.getStoredBlobMeta(LocalFsBlobStore.java:258
Can you post the full stack trace? I want to confirm that this is logged while
trying to obtain the heartbeat.This message is a warning message and is not
expected to shutdown nimbus.
Sent from Yahoo Mail for iPhone
On Thursday, January 18, 2024, 11:19 AM, Devendar Rao
wrote:
Hi,
We're con
18 18:27:12.889 o.a.s.d.n.Nimbus pool-29-thread-22 [WARN] get blob
meta exception. org.apache.storm.utils.WrappedKeyNotFoundException:
Can someone please shed some light on this?
Thanks,
Devendar
11 matches
Mail list logo
