Both senders and consumers are running in the same WildFly instance. It was
built to run on different servers, but we never went that far. The queues
basically send simple messages that say "here is a new file" and "I am done
with the file". The message of "here is a new file" is the one that randomly
slows down. The "I am done with the file" message has no issues. There is a
third queue that gets much larger messages, and it has no issues either.
Thank you,
Patrick R. Thomas
-----Original Message-----
From: Justin Bertram <jbert...@apache.org>
Sent: Wednesday, November 16, 2022 3:56 PM
To: users@activemq.apache.org
Subject: Re: ActiveMQ Strange Delays (old version)
CAUTION! This email originated outside of Quest Diagnostics. DO NOT click links
or open attachments unless you recognize the sender and know the content is
safe. Please report suspicious emails to: ph...@questdiagnostics.com
The diagnostic data is consistent with a slow or stuck consumer. However,
without a thread dump from the consumer it's impossible to say for sure.
Is the consumer running in WildFly or on a remote machine?
Justin
On Wed, Nov 9, 2022 at 4:00 PM Thomas, Patrick R <
patrick.r.tho...@questdiagnostics.com> wrote:
> This problem is occurring again today after more than a month without
> issues. I ran the request again, and this time I am seeing the
> delivering-count rise and fall. It's not completely stuck. It's oddly
> slow to process very small messages. If I check during normal times,
> the number is usually 0 or 1.
>
> I haven't figured out how to get the thread dump yet. I'm having
> difficulty figuring out how to connect it to my WildFly instance.
>
> {
> "outcome" => "success",
> "result" => {
> "consumer-count" => 1,
> "dead-letter-address" => "jms.queue.DLQ",
> "delivering-count" => 17,
> "durable" => true,
> "entries" => ["java:jboss/exported/jms/queue/MyQueue"],
> "expiry-address" => "jms.queue.ExpiryQueue",
> "legacy-entries" => undefined,
> "message-count" => 17L,
> "messages-added" => 17481L,
> "paused" => false,
> "queue-address" => "jms.queue.MyQueue",
> "scheduled-count" => 0L,
> "selector" => undefined,
> "temporary" => false
> }
> }
>
> Thank you,
>
> Patrick R. Thomas
> Software Engineer
>
> -----Original Message-----
> From: Justin Bertram <jbert...@apache.org>
> Sent: Monday, September 26, 2022 4:27 PM
> To: users@activemq.apache.org
> Subject: Re: ActiveMQ Strange Delays (old version)
>
> CAUTION! This email originated outside of Quest Diagnostics. DO NOT
> click links or open attachments unless you recognize the sender and
> know the content is safe. Please report suspicious emails to:
> ph...@questdiagnostics.com
>
>
> You may need to use the WildFly CLI to get all the information you need.
> I'm not terribly familiar with the WildFly admin console. For example,
> if you execute this command in the WildFly CLI:
>
>
>
> /subsystem=messaging-activemq/server=default/jms-queue=myQueue:read-re
> source(include-runtime=true)
>
> You'll get something like this:
>
> {
> "outcome" => "success",
> "result" => {
> "consumer-count" => 0,
> "dead-letter-address" => "jms.queue.myQueue",
> "delivering-count" => 0,
> "durable" => true,
> "entries" => ["java:/jms/queue/myQueue"],
> "expiry-address" => "jms.queue.ExpiryQueue",
> "legacy-entries" => undefined,
> "message-count" => 0L,
> "messages-added" => 0L,
> "paused" => false,
> "queue-address" => "jms.queue.myQueue",
> "scheduled-count" => 0L,
> "selector" => undefined,
> "temporary" => false
> }
> }
>
> That will give you all the metrics I mentioned previously. The
> "delivery-count" is a key metric here so it's important to get that if
> you can.
>
> Regarding the metrics you did provide, it looks like there's nothing
> stuck at that point since there are no messages on the queue. It's
> important to capture the metrics when the problem actually happens.
>
> Regarding the thread dump, check out this info [1]. Again, you need to
> get a thread dump from the consumer when the problem actually happens.
>
> I recommend you set up scripts to gather this data at regular
> intervals so that the next time the problem arises you can go back and
> look at what was happening on the server and the client leading up to the
> issue.
>
>
> Justin
>
> [1]
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.
> baeldung.com%2Fjava-thread-dump&data=05%7C01%7CPatrick.R.Thomas%40
> questdiagnostics.com%7Cd9d96c16fbaf4ccf209008dac81510d4%7Cb68c6481b22b
> 46b38c4c0024bb9b9b1f%7C1%7C0%7C638042290056052255%7CUnknown%7CTWFpbGZs
> b3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D
> %7C3000%7C%7C%7C&sdata=QKnZizer23F08TvL4%2BM6zkp6HAHoDV%2F5dRRYfSp
> K0cc%3D&reserved=0
>
> On Mon, Sep 26, 2022 at 2:40 PM Thomas, Patrick R <
> patrick.r.tho...@questdiagnostics.com> wrote:
>
> > Thank you for the speedy response. Yes, it is very difficult to
> > upgrade systems in my particular area for a variety of reasons. I
> > would not have expected issues with the messaging system to be the
> > area that caused me problems. Our system only sends a few thousand
> > messages per day. That seems like a tiny amount compared to what
> ActiveMQ can handle.
> >
> > I am checking this through WildFly's admin console, and this is what
> > I can
> > see:
> >
> > Consumer Count: 1
> > Message Count: 0
> > Messages Added: 1061 (going up gradually)
> > Scheduled Count: 0
> >
> > I'm not sure how to check the thread dump. I've never found any
> > errors in any of my application's or WildFly's logs.
> >
> > Thank you,
> >
> > Patrick R. Thomas
> >
> > -----Original Message-----
> > From: Justin Bertram <jbert...@apache.org>
> > Sent: Monday, September 26, 2022 3:20 PM
> > To: users@activemq.apache.org
> > Subject: Re: ActiveMQ Strange Delays (old version)
> >
> > CAUTION! This email originated outside of Quest Diagnostics. DO NOT
> > click links or open attachments unless you recognize the sender and
> > know the content is safe. Please report suspicious emails to:
> > ph...@questdiagnostics.com
> >
> >
> > You weren't kidding about the problem being on an "old version."
> > WildFly
> > 10 was released in 2016 and shipped with ActiveMQ Artemis 1.1.0.
> > There's been almost 50 releases of ActiveMQ Artemis and around 30
> > releases of WildFly in the last 6 years since then.
> >
> > Things I would check:
> > - On the queue:
> > - Number of consumers
> > - Number of messages
> > - Number of messages in delivery (this indicates how many
> > messages have been dispatched to consumers but have not yet been
> > acknowledged)
> > - On the consumer:
> > - Thread dump to see if the consumer is stuck for any reason
> >
> >
> > Justin
> >
> > On Mon, Sep 26, 2022 at 1:38 PM Thomas, Patrick R <
> > patrick.r.tho...@questdiagnostics.com> wrote:
> >
> > > Hello,
> > >
> > > I am new to this mailing list. I built at application years ago
> > > using WildFly 10 and the ActiveMQ that is embedded. Recently, on
> > > extremely rare occasions, one of our queues slow down. My log
> > > files of my application show that messages are delayed by anywhere
> > > from 5 to 30 minutes. The queue is on the same server running
> > > under the same WildFly instance. One web app is sending another
> > > web app a simple
> > message that a file has been received.
> > > We're using Spring to create and read the messages. There is very
> > > little code on our part, and it has not changed since the beginning.
> > >
> > > To add to the mystery, we have 2 other queues that seem unaffected.
> > > One queue is virtually the same message being sent back to the
> > > previous web app. The third queue is a message being sent from the
> > > middle app to another app, and those messages are much larger but
> > > are
> > never delayed.
> > >
> > > Usually a reboot clears the issue, but today I did a regular
> > > reboot and the delay started happening right away, although a
> > > smaller delay of only a few minutes. Does anyone have any idea how
> > > I can troubleshoot this? Could there be something wrong with my
> > > queue, like corruption? Is there any log I can check to be sure
> > > the issue is with the queue itself? Any help would be greatly
> > > appreciated. It may be a bug that was fixed long ago. I won't be
> > > able to update this application
> > any time soon.
> > >
> > > Thank you,
> > >
> > > Patrick
> > >
> > > __________________________________________________________________
> > > __ __ The contents of this message, together with any attachments,
> > > are intended only for the use of the person(s) to which they are
> > > addressed and may contain confidential and/or privileged
> > > information. Further, any medical information herein is
> > > confidential
> and protected by law.
> > > It is unlawful for unauthorized persons to use, review, copy,
> > > disclose, or disseminate confidential medical information. If you
> > > are not the intended recipient, immediately advise the sender and
> > > delete
> > this message and any attachments.
> > > Any distribution, or copying of this message, or any attachment,
> > > is prohibited.
> > >
> > >
> >
> > ____________________________________________________________________
> > __ The contents of this message, together with any attachments, are
> > intended only for the use of the person(s) to which they are
> > addressed and may contain confidential and/or privileged
> > information. Further, any medical information herein is confidential and
> > protected by law.
> > It is unlawful for unauthorized persons to use, review, copy,
> > disclose, or disseminate confidential medical information. If you
> > are not the intended recipient, immediately advise the sender and
> > delete
> this message and any attachments.
> > Any distribution, or copying of this message, or any attachment, is
> > prohibited.
> >
>
> ______________________________________________________________________
> The contents of this message, together with any attachments, are
> intended only for the use of the person(s) to which they are addressed
> and may contain confidential and/or privileged information. Further,
> any medical information herein is confidential and protected by law.
> It is unlawful for unauthorized persons to use, review, copy,
> disclose, or disseminate confidential medical information. If you are
> not the intended recipient, immediately advise the sender and delete this
> message and any attachments.
> Any distribution, or copying of this message, or any attachment, is
> prohibited.
>
>
______________________________________________________________________
The contents of this message, together with any attachments, are intended only
for the use of the person(s) to which they are addressed and may contain
confidential and/or privileged information. Further, any medical information
herein is confidential and protected by law. It is unlawful for unauthorized
persons to use, review, copy, disclose, or disseminate confidential medical
information. If you are not the intended recipient, immediately advise the
sender and delete this message and any attachments. Any distribution, or
copying of this message, or any attachment, is prohibited.
