This problem is occurring again today after more than a month without issues. I 
ran the request again, and this time I am seeing the delivering-count rise and 
fall. It's not completely stuck. It's oddly slow to process very small 
messages. If I check during normal times, the number is usually 0 or 1.

I haven't figured out how to get the thread dump yet. I'm having difficulty 
figuring out how to connect it to my WildFly instance.

{
    "outcome" => "success",
    "result" => {
        "consumer-count" => 1,
        "dead-letter-address" => "jms.queue.DLQ",
        "delivering-count" => 17,
        "durable" => true,
        "entries" => ["java:jboss/exported/jms/queue/MyQueue"],
        "expiry-address" => "jms.queue.ExpiryQueue",
        "legacy-entries" => undefined,
        "message-count" => 17L,
        "messages-added" => 17481L,
        "paused" => false,
        "queue-address" => "jms.queue.MyQueue",
        "scheduled-count" => 0L,
        "selector" => undefined,
        "temporary" => false
    }
}

Thank you,

Patrick R. Thomas
Software Engineer
 
Quest Diagnostics | Action from Insight | 14225 Newbrook Drive | Chantilly, VA 
20151 USA | phone 703.802.6900 x67351 | fax 703.802.7107 | 
patrick.r.tho...@questdiagnostics.com | QuestDiagnostics.com

-----Original Message-----
From: Justin Bertram <jbert...@apache.org> 
Sent: Monday, September 26, 2022 4:27 PM
To: users@activemq.apache.org
Subject: Re: ActiveMQ Strange Delays (old version)

CAUTION! This email originated outside of Quest Diagnostics. DO NOT click links 
or open attachments unless you recognize the sender and know the content is 
safe. Please report suspicious emails to: ph...@questdiagnostics.com


You may need to use the WildFly CLI to get all the information you need.
I'm not terribly familiar with the WildFly admin console. For example, if you 
execute this command in the WildFly CLI:


/subsystem=messaging-activemq/server=default/jms-queue=myQueue:read-resource(include-runtime=true)

You'll get something like this:

  {
      "outcome" => "success",
      "result" => {
          "consumer-count" => 0,
          "dead-letter-address" => "jms.queue.myQueue",
          "delivering-count" => 0,
          "durable" => true,
          "entries" => ["java:/jms/queue/myQueue"],
          "expiry-address" => "jms.queue.ExpiryQueue",
          "legacy-entries" => undefined,
          "message-count" => 0L,
          "messages-added" => 0L,
          "paused" => false,
          "queue-address" => "jms.queue.myQueue",
          "scheduled-count" => 0L,
          "selector" => undefined,
          "temporary" => false
      }
  }

That will give you all the metrics I mentioned previously. The "delivery-count" 
is a key metric here so it's important to get that if you can.

Regarding the metrics you did provide, it looks like there's nothing stuck at 
that point since there are no messages on the queue. It's important to capture 
the metrics when the problem actually happens.

Regarding the thread dump, check out this info [1]. Again, you need to get a 
thread dump from the consumer when the problem actually happens.

I recommend you set up scripts to gather this data at regular intervals so that 
the next time the problem arises you can go back and look at what was happening 
on the server and the client leading up to the issue.


Justin

[1] 
https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.baeldung.com%2Fjava-thread-dump&amp;data=05%7C01%7CPatrick.R.Thomas%40questdiagnostics.com%7Ce1e7c55cc6fa4fc0e47708da9ffd7eb2%7Cb68c6481b22b46b38c4c0024bb9b9b1f%7C1%7C0%7C637998208343868967%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=DQPdPE3e7bRb9sVWx92U2oidj1bv4X4lm8OxN0BhlWs%3D&amp;reserved=0

On Mon, Sep 26, 2022 at 2:40 PM Thomas, Patrick R < 
patrick.r.tho...@questdiagnostics.com> wrote:

> Thank you for the speedy response. Yes, it is very difficult to 
> upgrade systems in my particular area for a variety of reasons. I 
> would not have expected issues with the messaging system to be the 
> area that caused me problems. Our system only sends a few thousand 
> messages per day. That seems like a tiny amount compared to what ActiveMQ can 
> handle.
>
> I am checking this through WildFly's admin console, and this is what I 
> can
> see:
>
> Consumer Count: 1
> Message Count:          0
> Messages Added: 1061 (going up gradually)
> Scheduled Count:        0
>
> I'm not sure how to check the thread dump. I've never found any errors 
> in any of my application's or WildFly's logs.
>
> Thank you,
>
> Patrick R. Thomas
>
> -----Original Message-----
> From: Justin Bertram <jbert...@apache.org>
> Sent: Monday, September 26, 2022 3:20 PM
> To: users@activemq.apache.org
> Subject: Re: ActiveMQ Strange Delays (old version)
>
> CAUTION! This email originated outside of Quest Diagnostics. DO NOT 
> click links or open attachments unless you recognize the sender and 
> know the content is safe. Please report suspicious emails to:
> ph...@questdiagnostics.com
>
>
> You weren't kidding about the problem being on an "old version." 
> WildFly
> 10 was released in 2016 and shipped with ActiveMQ Artemis 1.1.0. 
> There's been almost 50 releases of ActiveMQ Artemis and around 30 
> releases of WildFly in the last 6 years since then.
>
> Things I would check:
>  - On the queue:
>     - Number of consumers
>     - Number of messages
>     - Number of messages in delivery (this indicates how many messages 
> have been dispatched to consumers but have not yet been acknowledged)
>  - On the consumer:
>     - Thread dump to see if the consumer is stuck for any reason
>
>
> Justin
>
> On Mon, Sep 26, 2022 at 1:38 PM Thomas, Patrick R < 
> patrick.r.tho...@questdiagnostics.com> wrote:
>
> > Hello,
> >
> > I am new to this mailing list. I built at application years ago 
> > using WildFly 10 and the ActiveMQ that is embedded. Recently, on 
> > extremely rare occasions, one of our queues slow down. My log files 
> > of my application show that messages are delayed by anywhere from 5 
> > to 30 minutes. The queue is on the same server running under the 
> > same WildFly instance. One web app is sending another web app a 
> > simple
> message that a file has been received.
> > We're using Spring to create and read the messages. There is very 
> > little code on our part, and it has not changed since the beginning.
> >
> > To add to the mystery, we have 2 other queues that seem unaffected.
> > One queue is virtually the same message being sent back to the 
> > previous web app. The third queue is a message being sent from the 
> > middle app to another app, and those messages are much larger but 
> > are
> never delayed.
> >
> > Usually a reboot clears the issue, but today I did a regular reboot 
> > and the delay started happening right away, although a smaller delay 
> > of only a few minutes. Does anyone have any idea how I can 
> > troubleshoot this? Could there be something wrong with my queue, 
> > like corruption? Is there any log I can check to be sure the issue 
> > is with the queue itself? Any help would be greatly appreciated. It 
> > may be a bug that was fixed long ago. I won't be able to update this 
> > application
> any time soon.
> >
> > Thank you,
> >
> > Patrick
> >
> > ____________________________________________________________________
> > __ The contents of this message, together with any attachments, are 
> > intended only for the use of the person(s) to which they are 
> > addressed and may contain confidential and/or privileged 
> > information. Further, any medical information herein is confidential and 
> > protected by law.
> > It is unlawful for unauthorized persons to use, review, copy, 
> > disclose, or disseminate confidential medical information. If you 
> > are not the intended recipient, immediately advise the sender and 
> > delete
> this message and any attachments.
> > Any distribution, or copying of this message, or any attachment, is 
> > prohibited.
> >
> >
>
> ______________________________________________________________________
> The contents of this message, together with any attachments, are 
> intended only for the use of the person(s) to which they are addressed 
> and may contain confidential and/or privileged information. Further, 
> any medical information herein is confidential and protected by law. 
> It is unlawful for unauthorized persons to use, review, copy, 
> disclose, or disseminate confidential medical information. If you are 
> not the intended recipient, immediately advise the sender and delete this 
> message and any attachments.
> Any distribution, or copying of this message, or any attachment, is 
> prohibited.
>

______________________________________________________________________
The contents of this message, together with any attachments, are intended only 
for the use of the person(s) to which they are addressed and may contain 
confidential and/or privileged information. Further, any medical information 
herein is confidential and protected by law. It is unlawful for unauthorized 
persons to use, review, copy, disclose, or disseminate confidential medical 
information. If you are not the intended recipient, immediately advise the 
sender and delete this message and any attachments. Any distribution, or 
copying of this message, or any attachment, is prohibited.

Reply via email to