Please provide a thread dump when you are eventually able to acquire one.
Justin
On Thu, Nov 17, 2022 at 6:50 AM Thomas, Patrick R <
patrick.r.tho...@questdiagnostics.com> wrote:
> Both senders and consumers are running in the same WildFly instance. It
> was built to run on different servers, but we never went that far. The
> queues basically send simple messages that say "here is a new file" and "I
> am done with the file". The message of "here is a new file" is the one that
> randomly slows down. The "I am done with the file" message has no issues.
> There is a third queue that gets much larger messages, and it has no issues
> either.
>
> Thank you,
>
> Patrick R. Thomas
>
> -----Original Message-----
> From: Justin Bertram <jbert...@apache.org>
> Sent: Wednesday, November 16, 2022 3:56 PM
> To: users@activemq.apache.org
> Subject: Re: ActiveMQ Strange Delays (old version)
>
> CAUTION! This email originated outside of Quest Diagnostics. DO NOT click
> links or open attachments unless you recognize the sender and know the
> content is safe. Please report suspicious emails to:
> ph...@questdiagnostics.com
>
>
> The diagnostic data is consistent with a slow or stuck consumer. However,
> without a thread dump from the consumer it's impossible to say for sure.
>
> Is the consumer running in WildFly or on a remote machine?
>
>
> Justin
>
> On Wed, Nov 9, 2022 at 4:00 PM Thomas, Patrick R <
> patrick.r.tho...@questdiagnostics.com> wrote:
>
> > This problem is occurring again today after more than a month without
> > issues. I ran the request again, and this time I am seeing the
> > delivering-count rise and fall. It's not completely stuck. It's oddly
> > slow to process very small messages. If I check during normal times,
> > the number is usually 0 or 1.
> >
> > I haven't figured out how to get the thread dump yet. I'm having
> > difficulty figuring out how to connect it to my WildFly instance.
> >
> > {
> > "outcome" => "success",
> > "result" => {
> > "consumer-count" => 1,
> > "dead-letter-address" => "jms.queue.DLQ",
> > "delivering-count" => 17,
> > "durable" => true,
> > "entries" => ["java:jboss/exported/jms/queue/MyQueue"],
> > "expiry-address" => "jms.queue.ExpiryQueue",
> > "legacy-entries" => undefined,
> > "message-count" => 17L,
> > "messages-added" => 17481L,
> > "paused" => false,
> > "queue-address" => "jms.queue.MyQueue",
> > "scheduled-count" => 0L,
> > "selector" => undefined,
> > "temporary" => false
> > }
> > }
> >
> > Thank you,
> >
> > Patrick R. Thomas
> > Software Engineer
> >
> > -----Original Message-----
> > From: Justin Bertram <jbert...@apache.org>
> > Sent: Monday, September 26, 2022 4:27 PM
> > To: users@activemq.apache.org
> > Subject: Re: ActiveMQ Strange Delays (old version)
> >
> > CAUTION! This email originated outside of Quest Diagnostics. DO NOT
> > click links or open attachments unless you recognize the sender and
> > know the content is safe. Please report suspicious emails to:
> > ph...@questdiagnostics.com
> >
> >
> > You may need to use the WildFly CLI to get all the information you need.
> > I'm not terribly familiar with the WildFly admin console. For example,
> > if you execute this command in the WildFly CLI:
> >
> >
> >
> > /subsystem=messaging-activemq/server=default/jms-queue=myQueue:read-re
> > source(include-runtime=true)
> >
> > You'll get something like this:
> >
> > {
> > "outcome" => "success",
> > "result" => {
> > "consumer-count" => 0,
> > "dead-letter-address" => "jms.queue.myQueue",
> > "delivering-count" => 0,
> > "durable" => true,
> > "entries" => ["java:/jms/queue/myQueue"],
> > "expiry-address" => "jms.queue.ExpiryQueue",
> > "legacy-entries" => undefined,
> > "message-count" => 0L,
> > "messages-added" => 0L,
> > "paused" => false,
> > "queue-address" => "jms.queue.myQueue",
> > "scheduled-count" => 0L,
> > "selector" => undefined,
> > "temporary" => false
> > }
> > }
> >
> > That will give you all the metrics I mentioned previously. The
> > "delivery-count" is a key metric here so it's important to get that if
> > you can.
> >
> > Regarding the metrics you did provide, it looks like there's nothing
> > stuck at that point since there are no messages on the queue. It's
> > important to capture the metrics when the problem actually happens.
> >
> > Regarding the thread dump, check out this info [1]. Again, you need to
> > get a thread dump from the consumer when the problem actually happens.
> >
> > I recommend you set up scripts to gather this data at regular
> > intervals so that the next time the problem arises you can go back and
> > look at what was happening on the server and the client leading up to
> the issue.
> >
> >
> > Justin
> >
> > [1]
> > https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.
> > baeldung.com%2Fjava-thread-dump&data=05%7C01%7CPatrick.R.Thomas%40
> > questdiagnostics.com%7Cd9d96c16fbaf4ccf209008dac81510d4%7Cb68c6481b22b
> > 46b38c4c0024bb9b9b1f%7C1%7C0%7C638042290056052255%7CUnknown%7CTWFpbGZs
> > b3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D
> > %7C3000%7C%7C%7C&sdata=QKnZizer23F08TvL4%2BM6zkp6HAHoDV%2F5dRRYfSp
> > K0cc%3D&reserved=0
> >
> > On Mon, Sep 26, 2022 at 2:40 PM Thomas, Patrick R <
> > patrick.r.tho...@questdiagnostics.com> wrote:
> >
> > > Thank you for the speedy response. Yes, it is very difficult to
> > > upgrade systems in my particular area for a variety of reasons. I
> > > would not have expected issues with the messaging system to be the
> > > area that caused me problems. Our system only sends a few thousand
> > > messages per day. That seems like a tiny amount compared to what
> > ActiveMQ can handle.
> > >
> > > I am checking this through WildFly's admin console, and this is what
> > > I can
> > > see:
> > >
> > > Consumer Count: 1
> > > Message Count: 0
> > > Messages Added: 1061 (going up gradually)
> > > Scheduled Count: 0
> > >
> > > I'm not sure how to check the thread dump. I've never found any
> > > errors in any of my application's or WildFly's logs.
> > >
> > > Thank you,
> > >
> > > Patrick R. Thomas
> > >
> > > -----Original Message-----
> > > From: Justin Bertram <jbert...@apache.org>
> > > Sent: Monday, September 26, 2022 3:20 PM
> > > To: users@activemq.apache.org
> > > Subject: Re: ActiveMQ Strange Delays (old version)
> > >
> > > CAUTION! This email originated outside of Quest Diagnostics. DO NOT
> > > click links or open attachments unless you recognize the sender and
> > > know the content is safe. Please report suspicious emails to:
> > > ph...@questdiagnostics.com
> > >
> > >
> > > You weren't kidding about the problem being on an "old version."
> > > WildFly
> > > 10 was released in 2016 and shipped with ActiveMQ Artemis 1.1.0.
> > > There's been almost 50 releases of ActiveMQ Artemis and around 30
> > > releases of WildFly in the last 6 years since then.
> > >
> > > Things I would check:
> > > - On the queue:
> > > - Number of consumers
> > > - Number of messages
> > > - Number of messages in delivery (this indicates how many
> > > messages have been dispatched to consumers but have not yet been
> > > acknowledged)
> > > - On the consumer:
> > > - Thread dump to see if the consumer is stuck for any reason
> > >
> > >
> > > Justin
> > >
> > > On Mon, Sep 26, 2022 at 1:38 PM Thomas, Patrick R <
> > > patrick.r.tho...@questdiagnostics.com> wrote:
> > >
> > > > Hello,
> > > >
> > > > I am new to this mailing list. I built at application years ago
> > > > using WildFly 10 and the ActiveMQ that is embedded. Recently, on
> > > > extremely rare occasions, one of our queues slow down. My log
> > > > files of my application show that messages are delayed by anywhere
> > > > from 5 to 30 minutes. The queue is on the same server running
> > > > under the same WildFly instance. One web app is sending another
> > > > web app a simple
> > > message that a file has been received.
> > > > We're using Spring to create and read the messages. There is very
> > > > little code on our part, and it has not changed since the beginning.
> > > >
> > > > To add to the mystery, we have 2 other queues that seem unaffected.
> > > > One queue is virtually the same message being sent back to the
> > > > previous web app. The third queue is a message being sent from the
> > > > middle app to another app, and those messages are much larger but
> > > > are
> > > never delayed.
> > > >
> > > > Usually a reboot clears the issue, but today I did a regular
> > > > reboot and the delay started happening right away, although a
> > > > smaller delay of only a few minutes. Does anyone have any idea how
> > > > I can troubleshoot this? Could there be something wrong with my
> > > > queue, like corruption? Is there any log I can check to be sure
> > > > the issue is with the queue itself? Any help would be greatly
> > > > appreciated. It may be a bug that was fixed long ago. I won't be
> > > > able to update this application
> > > any time soon.
> > > >
> > > > Thank you,
> > > >
> > > > Patrick
> > > >
> > > > __________________________________________________________________
> > > > __ __ The contents of this message, together with any attachments,
> > > > are intended only for the use of the person(s) to which they are
> > > > addressed and may contain confidential and/or privileged
> > > > information. Further, any medical information herein is
> > > > confidential
> > and protected by law.
> > > > It is unlawful for unauthorized persons to use, review, copy,
> > > > disclose, or disseminate confidential medical information. If you
> > > > are not the intended recipient, immediately advise the sender and
> > > > delete
> > > this message and any attachments.
> > > > Any distribution, or copying of this message, or any attachment,
> > > > is prohibited.
> > > >
> > > >
> > >
> > > ____________________________________________________________________
> > > __ The contents of this message, together with any attachments, are
> > > intended only for the use of the person(s) to which they are
> > > addressed and may contain confidential and/or privileged
> > > information. Further, any medical information herein is confidential
> and protected by law.
> > > It is unlawful for unauthorized persons to use, review, copy,
> > > disclose, or disseminate confidential medical information. If you
> > > are not the intended recipient, immediately advise the sender and
> > > delete
> > this message and any attachments.
> > > Any distribution, or copying of this message, or any attachment, is
> > > prohibited.
> > >
> >
> > ______________________________________________________________________
> > The contents of this message, together with any attachments, are
> > intended only for the use of the person(s) to which they are addressed
> > and may contain confidential and/or privileged information. Further,
> > any medical information herein is confidential and protected by law.
> > It is unlawful for unauthorized persons to use, review, copy,
> > disclose, or disseminate confidential medical information. If you are
> > not the intended recipient, immediately advise the sender and delete
> this message and any attachments.
> > Any distribution, or copying of this message, or any attachment, is
> > prohibited.
> >
> >
>
> ______________________________________________________________________
> The contents of this message, together with any attachments, are intended
> only for the use of the person(s) to which they are addressed and may
> contain confidential and/or privileged information. Further, any medical
> information herein is confidential and protected by law. It is unlawful for
> unauthorized persons to use, review, copy, disclose, or disseminate
> confidential medical information. If you are not the intended recipient,
> immediately advise the sender and delete this message and any attachments.
> Any distribution, or copying of this message, or any attachment, is
> prohibited.
>
>
