You may need to use the WildFly CLI to get all the information you need.
I'm not terribly familiar with the WildFly admin console. For example, if
you execute this command in the WildFly CLI:
/subsystem=messaging-activemq/server=default/jms-queue=myQueue:read-resource(include-runtime=true)
You'll get something like this:
{
"outcome" => "success",
"result" => {
"consumer-count" => 0,
"dead-letter-address" => "jms.queue.myQueue",
"delivering-count" => 0,
"durable" => true,
"entries" => ["java:/jms/queue/myQueue"],
"expiry-address" => "jms.queue.ExpiryQueue",
"legacy-entries" => undefined,
"message-count" => 0L,
"messages-added" => 0L,
"paused" => false,
"queue-address" => "jms.queue.myQueue",
"scheduled-count" => 0L,
"selector" => undefined,
"temporary" => false
}
}
That will give you all the metrics I mentioned previously. The
"delivery-count" is a key metric here so it's important to get that if you
can.
Regarding the metrics you did provide, it looks like there's nothing stuck
at that point since there are no messages on the queue. It's important to
capture the metrics when the problem actually happens.
Regarding the thread dump, check out this info [1]. Again, you need to get
a thread dump from the consumer when the problem actually happens.
I recommend you set up scripts to gather this data at regular intervals so
that the next time the problem arises you can go back and look at what was
happening on the server and the client leading up to the issue.
Justin
[1] https://www.baeldung.com/java-thread-dump
On Mon, Sep 26, 2022 at 2:40 PM Thomas, Patrick R <
patrick.r.tho...@questdiagnostics.com> wrote:
> Thank you for the speedy response. Yes, it is very difficult to upgrade
> systems in my particular area for a variety of reasons. I would not have
> expected issues with the messaging system to be the area that caused me
> problems. Our system only sends a few thousand messages per day. That seems
> like a tiny amount compared to what ActiveMQ can handle.
>
> I am checking this through WildFly's admin console, and this is what I can
> see:
>
> Consumer Count: 1
> Message Count: 0
> Messages Added: 1061 (going up gradually)
> Scheduled Count: 0
>
> I'm not sure how to check the thread dump. I've never found any errors in
> any of my application's or WildFly's logs.
>
> Thank you,
>
> Patrick R. Thomas
>
> -----Original Message-----
> From: Justin Bertram <jbert...@apache.org>
> Sent: Monday, September 26, 2022 3:20 PM
> To: users@activemq.apache.org
> Subject: Re: ActiveMQ Strange Delays (old version)
>
> CAUTION! This email originated outside of Quest Diagnostics. DO NOT click
> links or open attachments unless you recognize the sender and know the
> content is safe. Please report suspicious emails to:
> ph...@questdiagnostics.com
>
>
> You weren't kidding about the problem being on an "old version." WildFly
> 10 was released in 2016 and shipped with ActiveMQ Artemis 1.1.0. There's
> been almost 50 releases of ActiveMQ Artemis and around 30 releases of
> WildFly in the last 6 years since then.
>
> Things I would check:
> - On the queue:
> - Number of consumers
> - Number of messages
> - Number of messages in delivery (this indicates how many messages
> have been dispatched to consumers but have not yet been acknowledged)
> - On the consumer:
> - Thread dump to see if the consumer is stuck for any reason
>
>
> Justin
>
> On Mon, Sep 26, 2022 at 1:38 PM Thomas, Patrick R <
> patrick.r.tho...@questdiagnostics.com> wrote:
>
> > Hello,
> >
> > I am new to this mailing list. I built at application years ago using
> > WildFly 10 and the ActiveMQ that is embedded. Recently, on extremely
> > rare occasions, one of our queues slow down. My log files of my
> > application show that messages are delayed by anywhere from 5 to 30
> > minutes. The queue is on the same server running under the same
> > WildFly instance. One web app is sending another web app a simple
> message that a file has been received.
> > We're using Spring to create and read the messages. There is very
> > little code on our part, and it has not changed since the beginning.
> >
> > To add to the mystery, we have 2 other queues that seem unaffected.
> > One queue is virtually the same message being sent back to the
> > previous web app. The third queue is a message being sent from the
> > middle app to another app, and those messages are much larger but are
> never delayed.
> >
> > Usually a reboot clears the issue, but today I did a regular reboot
> > and the delay started happening right away, although a smaller delay
> > of only a few minutes. Does anyone have any idea how I can
> > troubleshoot this? Could there be something wrong with my queue, like
> > corruption? Is there any log I can check to be sure the issue is with
> > the queue itself? Any help would be greatly appreciated. It may be a
> > bug that was fixed long ago. I won't be able to update this application
> any time soon.
> >
> > Thank you,
> >
> > Patrick
> >
> > ______________________________________________________________________
> > The contents of this message, together with any attachments, are
> > intended only for the use of the person(s) to which they are addressed
> > and may contain confidential and/or privileged information. Further,
> > any medical information herein is confidential and protected by law.
> > It is unlawful for unauthorized persons to use, review, copy,
> > disclose, or disseminate confidential medical information. If you are
> > not the intended recipient, immediately advise the sender and delete
> this message and any attachments.
> > Any distribution, or copying of this message, or any attachment, is
> > prohibited.
> >
> >
>
> ______________________________________________________________________
> The contents of this message, together with any attachments, are intended
> only for the use of the person(s) to which they are addressed and may
> contain confidential and/or privileged information. Further, any medical
> information herein is confidential and protected by law. It is unlawful for
> unauthorized persons to use, review, copy, disclose, or disseminate
> confidential medical information. If you are not the intended recipient,
> immediately advise the sender and delete this message and any attachments.
> Any distribution, or copying of this message, or any attachment, is
> prohibited.
>
