Have you checked your OS / JVM level resources consumption when the failure happens? Is there any kind of outlier?
On Wed, 20 Apr 2022 at 17:20, Le, Binh T. <binh.t...@accenture.com> wrote: > Yes, I have updated tried both. I stated that in my other post. Maybe you > might have not seen it. > > > > Also, to be clear, I didn’t think there was a way to disable backpressure. > You can only extend the backpressure check time, which is what I did. > > > > *From:* Rui Abreu <rui.ab...@gmail.com> > *Sent:* Wednesday, April 20, 2022 12:13 PM > *To:* user@storm.apache.org > *Subject:* Re: [External] Re: Storm 2 Topology Stopped Processing > > > > As Bipin said, it could be a number of things. Have you tried disabling > the backpressure mechanism and limiting the in flight messages with > topology.max.spout.pending ? > > > > https://storm.apache.org/releases/current/Performance.html > <https://urldefense.proofpoint.com/v2/url?u=https-3A__storm.apache.org_releases_current_Performance.html&d=DwMFaQ&c=eIGjsITfXP_y-DLLX0uEHXJvU8nOHrUK8IrwNKOtkVU&r=BxRQQZWnE3D8_17HHQE0-XVUNXeS_tqliMsO6ITEvoM&m=s47ZhwnlsJfgRfqqRdG5OTTKWEhdHkgLkGtF_SjROaMshzIe6FWZkzm5zgkrjsUm&s=LZzB9UAuLr_8bW2hG700FHwj68JSQHEtSCVG4Rd4lis&e=> > > > > On Wed, 20 Apr 2022 at 17:04, Le, Binh T. <binh.t...@accenture.com> wrote: > > Hi Bipin, > > Thanks for your response. Unfortunately, this problem is not easily > reproducible. It only manifests itself when we're running our topologies > with production volume. It works fine, so far, with low volume. So it would > be difficult for me to provide an example topology and all that, to > reproduce this. > > Before storm 2.4.0 was released. We did try upgrading to 2.3.0, but > unfortunately encountered tons of errors we've not seen before with 2.2.0. > So we reverted back to 2.2.0. At this point, I'm not sure if we can upgrade > to 2.4.0. We'll see. > > Thanks, > Binh > > -----Original Message----- > From: Bipin Prasad <bipinpra...@apache.org> > Sent: Wednesday, April 20, 2022 10:32 AM > To: user@storm.apache.org > Subject: [External] Re: Storm 2 Topology Stopped Processing > > This message is from an EXTERNAL SENDER - be CAUTIOUS, particularly with > links and attachments. > > Hello Binh, > > Can you can provide a small example that can be duplicated - for example > with a storm config file (storm.yaml) and default, and running > WordCountTopology (for instance). > > Please check the CLASSPATH and java version, and any Exception in the log > files as well (before the NPE). > > For debugging locally, you can set_log_level for a specific class for a > specified duration (using storm.py), and/or start in java debug mode and > attach a debugger to it. > > From your earlier description of the stack trace, it seems there is a > storm internal error when the message is retrieved from the queue. > > Is it possible for you to switch to the latest released version of storm > (2.4.0). You might get better support for it. If the problem still > persists, please raise a storm-jira at > https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_projects_STORM_issues&d=DwIFaQ&c=eIGjsITfXP_y-DLLX0uEHXJvU8nOHrUK8IrwNKOtkVU&r=BxRQQZWnE3D8_17HHQE0-XVUNXeS_tqliMsO6ITEvoM&m=HLhVczPJ4MPWrYsR8Lap37z-NJEPIiIKCwq_thIXP-rlAYKZZpv1VjC7H4Of-1HC&s=5ZSfHRDiQM7q6Z22OTpYVihJx1X1508wu1RadoWQ72k&e= > > Thanks > --Bipin > > On 2022/04/18 17:17:04 "Le, Binh T." wrote: > > Hi, > > > > We are upgrading our Storm 1 cluster to 2.2.0 and noticed a strange > behavior. The topology is processing millions of messages fine. However, > all of a sudden, processing just stopped. Capacity for all bolts got > reduced to zero and stays at zero. Does anyone know what could possibly be > going on here and how we can troubleshoot this issue? > > > > Thanks, > > Binh > > > > ________________________________ > > > > This message is for the designated recipient only and may contain > privileged, proprietary, or otherwise confidential information. If you have > received it in error, please notify the sender immediately and delete the > original. Any other use of the e-mail by you is prohibited. Where allowed > by local law, electronic communications with Accenture and its affiliates, > including e-mail and instant messaging (including content), may be scanned > by our systems for the purposes of information security and assessment of > internal compliance with Accenture policy. Your privacy is important to us. > Accenture uses your personal data only in compliance with data protection > laws. For further information on how Accenture processes your personal > data, please see our privacy statement at > https://www.accenture.com/us-en/privacy-policy. > > > ______________________________________________________________________________________ > > > > http://www.accenture.com > > > >
