The diagnostic data is consistent with a slow or stuck consumer. However,
without a thread dump from the consumer it's impossible to say for sure.
Is the consumer running in WildFly or on a remote machine?
Justin
On Wed, Nov 9, 2022 at 4:00 PM Thomas, Patrick R <
patrick.r.tho...@questdiagnostics.com> wrote:
> This problem is occurring again today after more than a month without
> issues. I ran the request again, and this time I am seeing the
> delivering-count rise and fall. It's not completely stuck. It's oddly slow
> to process very small messages. If I check during normal times, the number
> is usually 0 or 1.
>
> I haven't figured out how to get the thread dump yet. I'm having
> difficulty figuring out how to connect it to my WildFly instance.
>
> {
> "outcome" => "success",
> "result" => {
> "consumer-count" => 1,
> "dead-letter-address" => "jms.queue.DLQ",
> "delivering-count" => 17,
> "durable" => true,
> "entries" => ["java:jboss/exported/jms/queue/MyQueue"],
> "expiry-address" => "jms.queue.ExpiryQueue",
> "legacy-entries" => undefined,
> "message-count" => 17L,
> "messages-added" => 17481L,
> "paused" => false,
> "queue-address" => "jms.queue.MyQueue",
> "scheduled-count" => 0L,
> "selector" => undefined,
> "temporary" => false
> }
> }
>
> Thank you,
>
> Patrick R. Thomas
> Software Engineer
>
> Quest Diagnostics | Action from Insight | 14225 Newbrook Drive |
> Chantilly, VA 20151 USA | phone 703.802.6900 x67351 | fax 703.802.7107
> | patrick.r.tho...@questdiagnostics.com | QuestDiagnostics.com
>
> -----Original Message-----
> From: Justin Bertram <jbert...@apache.org>
> Sent: Monday, September 26, 2022 4:27 PM
> To: users@activemq.apache.org
> Subject: Re: ActiveMQ Strange Delays (old version)
>
> CAUTION! This email originated outside of Quest Diagnostics. DO NOT click
> links or open attachments unless you recognize the sender and know the
> content is safe. Please report suspicious emails to:
> ph...@questdiagnostics.com
>
>
> You may need to use the WildFly CLI to get all the information you need.
> I'm not terribly familiar with the WildFly admin console. For example, if
> you execute this command in the WildFly CLI:
>
>
>
> /subsystem=messaging-activemq/server=default/jms-queue=myQueue:read-resource(include-runtime=true)
>
> You'll get something like this:
>
> {
> "outcome" => "success",
> "result" => {
> "consumer-count" => 0,
> "dead-letter-address" => "jms.queue.myQueue",
> "delivering-count" => 0,
> "durable" => true,
> "entries" => ["java:/jms/queue/myQueue"],
> "expiry-address" => "jms.queue.ExpiryQueue",
> "legacy-entries" => undefined,
> "message-count" => 0L,
> "messages-added" => 0L,
> "paused" => false,
> "queue-address" => "jms.queue.myQueue",
> "scheduled-count" => 0L,
> "selector" => undefined,
> "temporary" => false
> }
> }
>
> That will give you all the metrics I mentioned previously. The
> "delivery-count" is a key metric here so it's important to get that if you
> can.
>
> Regarding the metrics you did provide, it looks like there's nothing stuck
> at that point since there are no messages on the queue. It's important to
> capture the metrics when the problem actually happens.
>
> Regarding the thread dump, check out this info [1]. Again, you need to get
> a thread dump from the consumer when the problem actually happens.
>
> I recommend you set up scripts to gather this data at regular intervals so
> that the next time the problem arises you can go back and look at what was
> happening on the server and the client leading up to the issue.
>
>
> Justin
>
> [1]
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.baeldung.com%2Fjava-thread-dump&data=05%7C01%7CPatrick.R.Thomas%40questdiagnostics.com%7Ce1e7c55cc6fa4fc0e47708da9ffd7eb2%7Cb68c6481b22b46b38c4c0024bb9b9b1f%7C1%7C0%7C637998208343868967%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=DQPdPE3e7bRb9sVWx92U2oidj1bv4X4lm8OxN0BhlWs%3D&reserved=0
>
> On Mon, Sep 26, 2022 at 2:40 PM Thomas, Patrick R <
> patrick.r.tho...@questdiagnostics.com> wrote:
>
> > Thank you for the speedy response. Yes, it is very difficult to
> > upgrade systems in my particular area for a variety of reasons. I
> > would not have expected issues with the messaging system to be the
> > area that caused me problems. Our system only sends a few thousand
> > messages per day. That seems like a tiny amount compared to what
> ActiveMQ can handle.
> >
> > I am checking this through WildFly's admin console, and this is what I
> > can
> > see:
> >
> > Consumer Count: 1
> > Message Count: 0
> > Messages Added: 1061 (going up gradually)
> > Scheduled Count: 0
> >
> > I'm not sure how to check the thread dump. I've never found any errors
> > in any of my application's or WildFly's logs.
> >
> > Thank you,
> >
> > Patrick R. Thomas
> >
> > -----Original Message-----
> > From: Justin Bertram <jbert...@apache.org>
> > Sent: Monday, September 26, 2022 3:20 PM
> > To: users@activemq.apache.org
> > Subject: Re: ActiveMQ Strange Delays (old version)
> >
> > CAUTION! This email originated outside of Quest Diagnostics. DO NOT
> > click links or open attachments unless you recognize the sender and
> > know the content is safe. Please report suspicious emails to:
> > ph...@questdiagnostics.com
> >
> >
> > You weren't kidding about the problem being on an "old version."
> > WildFly
> > 10 was released in 2016 and shipped with ActiveMQ Artemis 1.1.0.
> > There's been almost 50 releases of ActiveMQ Artemis and around 30
> > releases of WildFly in the last 6 years since then.
> >
> > Things I would check:
> > - On the queue:
> > - Number of consumers
> > - Number of messages
> > - Number of messages in delivery (this indicates how many messages
> > have been dispatched to consumers but have not yet been acknowledged)
> > - On the consumer:
> > - Thread dump to see if the consumer is stuck for any reason
> >
> >
> > Justin
> >
> > On Mon, Sep 26, 2022 at 1:38 PM Thomas, Patrick R <
> > patrick.r.tho...@questdiagnostics.com> wrote:
> >
> > > Hello,
> > >
> > > I am new to this mailing list. I built at application years ago
> > > using WildFly 10 and the ActiveMQ that is embedded. Recently, on
> > > extremely rare occasions, one of our queues slow down. My log files
> > > of my application show that messages are delayed by anywhere from 5
> > > to 30 minutes. The queue is on the same server running under the
> > > same WildFly instance. One web app is sending another web app a
> > > simple
> > message that a file has been received.
> > > We're using Spring to create and read the messages. There is very
> > > little code on our part, and it has not changed since the beginning.
> > >
> > > To add to the mystery, we have 2 other queues that seem unaffected.
> > > One queue is virtually the same message being sent back to the
> > > previous web app. The third queue is a message being sent from the
> > > middle app to another app, and those messages are much larger but
> > > are
> > never delayed.
> > >
> > > Usually a reboot clears the issue, but today I did a regular reboot
> > > and the delay started happening right away, although a smaller delay
> > > of only a few minutes. Does anyone have any idea how I can
> > > troubleshoot this? Could there be something wrong with my queue,
> > > like corruption? Is there any log I can check to be sure the issue
> > > is with the queue itself? Any help would be greatly appreciated. It
> > > may be a bug that was fixed long ago. I won't be able to update this
> > > application
> > any time soon.
> > >
> > > Thank you,
> > >
> > > Patrick
> > >
> > > ____________________________________________________________________
> > > __ The contents of this message, together with any attachments, are
> > > intended only for the use of the person(s) to which they are
> > > addressed and may contain confidential and/or privileged
> > > information. Further, any medical information herein is confidential
> and protected by law.
> > > It is unlawful for unauthorized persons to use, review, copy,
> > > disclose, or disseminate confidential medical information. If you
> > > are not the intended recipient, immediately advise the sender and
> > > delete
> > this message and any attachments.
> > > Any distribution, or copying of this message, or any attachment, is
> > > prohibited.
> > >
> > >
> >
> > ______________________________________________________________________
> > The contents of this message, together with any attachments, are
> > intended only for the use of the person(s) to which they are addressed
> > and may contain confidential and/or privileged information. Further,
> > any medical information herein is confidential and protected by law.
> > It is unlawful for unauthorized persons to use, review, copy,
> > disclose, or disseminate confidential medical information. If you are
> > not the intended recipient, immediately advise the sender and delete
> this message and any attachments.
> > Any distribution, or copying of this message, or any attachment, is
> > prohibited.
> >
>
> ______________________________________________________________________
> The contents of this message, together with any attachments, are intended
> only for the use of the person(s) to which they are addressed and may
> contain confidential and/or privileged information. Further, any medical
> information herein is confidential and protected by law. It is unlawful for
> unauthorized persons to use, review, copy, disclose, or disseminate
> confidential medical information. If you are not the intended recipient,
> immediately advise the sender and delete this message and any attachments.
> Any distribution, or copying of this message, or any attachment, is
> prohibited.
>
>
