Thanks for the project. I was able to reproduce the problem and identify the root cause of the message accumulation. In short, your application is not acknowledging the messages it receives when invoking ClientRequestor.request().
When using the core protocol messages must always be explicitly acknowledged. Whether or not those acknowledgements are committed automatically depends on how the session is created. In your case you're creating a JMS session using the AUTO_ACKNOWLEDGE mode which means the underlying core session has autoCommitAcks = true. This isn't obvious since you're starting with a JMS implementation object and acquiring the core object from it. It would probably be more clear if you just used the core API for everything or the JMS API for everything. Lastly, if performance really isn't important then it may be simpler to eliminate the lock and the object caching and just create the session, etc. anytime you need to make a request. Simpler code is easier to write initially as well as debug and maintain in the long run. Developer productivity is worth a lot so it's worth considering. Justin On Mon, Jan 16, 2023 at 11:57 AM John Lilley <john.lil...@redpointglobal.com.invalid> wrote: > Justin, > > > > Link to sample project: > > > https://drive.google.com/file/d/1--vMXNJ3NIUVlRQ9liaMwa-VN3X5uEOE/view?usp=sharing > > > > Any of the three amqUtils.* calls in the main() loop will induce the > leakage. I have not figured out where the second queue in our production > comes from, but that isn’t critical right now. > > > > Everything serializes through a single lock object, so I don’t think I > have any thread-safety issues. Performance is not important. > > > > Thanks > > John > > > > > > > [image: rg] <https://www.redpointglobal.com/> > > John Lilley > > Data Management Chief Architect, Redpoint Global Inc. > > 888 Worcester Street, Suite 200 Wellesley, MA 02482 > > *M: *+1 7209385761 <+1%207209385761> | john.lil...@redpointglobal.com > > *From:* Justin Bertram <jbert...@apache.org> > *Sent:* Monday, January 16, 2023 10:31 AM > *To:* users@activemq.apache.org > *Subject:* Re: Messages accumulating in management queue > > > > **** [Caution] This email is from an external source. Please use caution > responding, opening attachments or clicking embedded links. **** > > > > > Is ClientRequestor cheap to make each time? > > > > It's relatively cheap, but of course just about anything can be done > enough to become expensive over time. Also, a ClientRequestor is not > thread-safe so a single instance may be a bottleneck in your application. I > don't know what your performance requirements are, how often you're making > these calls, etc. so it's hard to provide any specific recommendations. > > > > > One of my colleagues found that occasionally closing and re-creating the > ClientRequestor avoids this problem, but I’d like to know why. > > > > I can't rightly say why you're seeing messages build up at this point. The > screenshot shows that there's a consumer on both queues. Perhaps these > consumers are no longer acknowledging the messages they receive. You say > you're only using a single ClientRequestor instance but there's two queues > which indicates two ClientRequestor instances working concurrently so > perhaps you're leaking ClientRequestors at some point. > > > > Do you have a way to reproduce this? If so, could you create a > reproducible test-case which I could use to see the same behavior? > > > > > > Justin > > > > On Mon, Jan 16, 2023 at 10:55 AM John Lilley < > john.lil...@redpointglobal.com.invalid> wrote: > > We do keep a singleton ClientRequestor around forever. Perhaps this is > just wrong? Is ClientRequestor cheap to make each time? > > > > One of my colleagues found that occasionally closing and re-creating the > ClientRequestor avoids this problem, but I’d like to know why. > > > > I think you can see that image here: > > > https://drive.google.com/file/d/1gKDrg4-tRs7WSPbsqQ7Bu-UfhBPu4Fdj/view?usp=sharing > <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fdrive.google.com%2ffile%2fd%2f1gKDrg4-tRs7WSPbsqQ7Bu-UfhBPu4Fdj%2fview%3fusp%3dsharing&c=E,1,u7_nPpg3Eo9y9jPflc6RRIEJocNVFDWY6NsGPar5xM7mkglP-pr0VVtdEjUwdkYbOCXZ0Y-5yjk8Kk98uZV7SF4ZpC1xWstZa5vvnWqODa3s&typo=1> > > > > john > > > > > > > > [image: rg] > <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,vfQKtAcb52qTSJksDEbpSPPMH35oLMHbFHh03U5Yjymdi6_74sLsjOSkxSqYP1iE7FxcrLU9BjLO5yYbNQBwawQCxa6TY2UFcPEWp1xShdsR&typo=1> > > *John Lilley * > > *Data Management Chief Architect, Redpoint Global Inc. * > > 888 Worcester Street, Suite 200 Wellesley, MA 02482 > > *M: *+1 7209385761 <+1%207209385761> | john.lil...@redpointglobal.com > > From: Justin Bertram <jbert...@apache.org> > *Sent:* Monday, January 16, 2023 9:48 AM > *To:* users@activemq.apache.org > *Subject:* Re: Messages accumulating in management queue > > > > **** [Caution] This email is from an external source. Please use caution > responding, opening attachments or clicking embedded links. **** > > > > > I don’t know if that image comes through the mailing list... > > > > I can't see your image. For what it's worth, attachments don't usually > make it through the list. > > > > Could you upload the image to another location and provide a link? > > > > Also, could you provide the full stack-trace of the NPE? > > > > I don't see in your code where you're actually closing the ClientRequestor > so that might be a problem. Without more details it's impossible to say. > > > > > > Justin > > > > On Mon, Jan 16, 2023 at 10:31 AM John Lilley < > john.lil...@redpointglobal.com.invalid> wrote: > > Greetings! > > > > We are using the Artemis queue-based management APIs, and noticed that one > of the two management queues that appears in the console starts to grow > without bound: > > > > > > I don’t know if that image comes through the mailing list, but it shows > two of the xn--activemq-2p3d.management > <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fxn--activemq-2p3d.management&c=E,1,yM5VuqjniGEfl7K4rM70Xa-2jC5pkpu2WMCuv2o_9ssgUAyqDzgAfPDtdn-xMCHxrbyaUxPnF2oUZkGoCF_o-vv_o42p3PRnI9YguDv7nT8RO5syVzebSQ,,&typo=1&ancr_add=1><UUID>” > queues, and one of the counts is in the 100s. It just keeps growing and > growing by about 300 per minute. > > > > I’ve tried to browse this queue on the management console, but the > browse() operation shows an NPE: > > java.lang.NullPointerException : Cannot invoke > "org.apache.activemq.artemis.core.paging.cursor.PageSubscription.iterator(boolean)" > because "this.this$0.pageSubscription" is null > > > > The calls we are making are like: > > String resourceName = ResourceNames.QUEUE + queueName; > > ClientRequestor client = getClient(); > > ClientSession clientSession = … > > ClientMessage request = clientSession.createMessage(false); > ManagementHelper.putAttribute(request, resourceName, attributeName); > ClientMessage reply = client.request(request); > > > > Or > > ClientRequestor client = getClient(); > > ClientMessage request = clientSession.createMessage(false); > ManagementHelper.putAttribute(request, resourceName, attributeName); > ClientMessage reply = client.request(request); > if (ManagementHelper.hasOperationSucceeded(reply)) { > for (var value : (Object[])ManagementHelper.getResult(reply, > Object.class)) { > result.add(value.toString()); > } > > > > Any idea what we are doing wrong? > > > > This happens on windows or linux > > > > Artemis version 2.27.0. > > > > java –version > > openjdk 17.0.3 2022-04-19 > > OpenJDK Runtime Environment Temurin-17.0.3+7 (build 17.0.3+7) > > OpenJDK 64-Bit Server VM Temurin-17.0.3+7 (build 17.0.3+7, mixed mode, > sharing) > > > > Thanks > > john > > > > > > > <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,y0ymtXAc9XiBAplLMyDSTkbZPAR64gUyHhSGTjs8RK2_en31lUYrNYUf9Da_DM9WCm4_pRY8h4_s7PAUvD5SJfK1UCvi3Clxrot9XAr38A,,&typo=1> > > *John Lilley > <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,y0ymtXAc9XiBAplLMyDSTkbZPAR64gUyHhSGTjs8RK2_en31lUYrNYUf9Da_DM9WCm4_pRY8h4_s7PAUvD5SJfK1UCvi3Clxrot9XAr38A,,&typo=1>* > > *Data Management Chief Architect, Redpoint Global Inc. > <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,y0ymtXAc9XiBAplLMyDSTkbZPAR64gUyHhSGTjs8RK2_en31lUYrNYUf9Da_DM9WCm4_pRY8h4_s7PAUvD5SJfK1UCvi3Clxrot9XAr38A,,&typo=1>* > > 888 Worcester Street, Suite 200 Wellesley, MA 02482 > <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,y0ymtXAc9XiBAplLMyDSTkbZPAR64gUyHhSGTjs8RK2_en31lUYrNYUf9Da_DM9WCm4_pRY8h4_s7PAUvD5SJfK1UCvi3Clxrot9XAr38A,,&typo=1> > > *M: +1 7209385761 | john.lil...@redpointglobal.com > <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,y0ymtXAc9XiBAplLMyDSTkbZPAR64gUyHhSGTjs8RK2_en31lUYrNYUf9Da_DM9WCm4_pRY8h4_s7PAUvD5SJfK1UCvi3Clxrot9XAr38A,,&typo=1>* > > > PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is > confidential and is intended solely for the use of the individual(s) to > whom it is addressed. If you believe you received this e-mail in error, > please notify the sender immediately, delete the e-mail from your computer > and do not copy, print or disclose it to anyone else. If you properly > received this e-mail as a customer, partner or vendor of Redpoint, you > should maintain its contents in confidence subject to the terms and > conditions of your agreement(s) with Redpoint. > <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,y0ymtXAc9XiBAplLMyDSTkbZPAR64gUyHhSGTjs8RK2_en31lUYrNYUf9Da_DM9WCm4_pRY8h4_s7PAUvD5SJfK1UCvi3Clxrot9XAr38A,,&typo=1> > > > PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is > confidential and is intended solely for the use of the individual(s) to > whom it is addressed. If you believe you received this e-mail in error, > please notify the sender immediately, delete the e-mail from your computer > and do not copy, print or disclose it to anyone else. If you properly > received this e-mail as a customer, partner or vendor of Redpoint, you > should maintain its contents in confidence subject to the terms and > conditions of your agreement(s) with Redpoint. > <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,y0ymtXAc9XiBAplLMyDSTkbZPAR64gUyHhSGTjs8RK2_en31lUYrNYUf9Da_DM9WCm4_pRY8h4_s7PAUvD5SJfK1UCvi3Clxrot9XAr38A,,&typo=1> > > > PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is > confidential and is intended solely for the use of the individual(s) to > whom it is addressed. If you believe you received this e-mail in error, > please notify the sender immediately, delete the e-mail from your computer > and do not copy, print or disclose it to anyone else. If you properly > received this e-mail as a customer, partner or vendor of Redpoint, you > should maintain its contents in confidence subject to the terms and > conditions of your agreement(s) with Redpoint. >
