Thanks!
[rg] <https://www.redpointglobal.com/> John Lilley Data Management Chief Architect, Redpoint Global Inc. 888 Worcester Street, Suite 200 Wellesley, MA 02482 M: +1 7209385761<tel:+1%207209385761> | john.lil...@redpointglobal.com<mailto:john.lil...@redpointglobal.com> From: Clebert Suconic <clebert.suco...@gmail.com> Sent: Friday, July 28, 2023 3:32 PM To: users@activemq.apache.org Subject: Re: Error enumerating large messages *** [Caution] This email is from an external source. Please use caution responding, opening attachments or clicking embedded links. *** I would upgrade to the latest version as I have done some substantial work with large messages and tests in 2.29.0: (e.g ARTEMIS-4233). Besides this is a bit difficult to identify what you're client doing. The only thing I can think of is these fixes I had in 2.29. so I would get 2.30. On Thu, Jul 27, 2023 at 8:38 PM John Lilley <john.lil...@redpointglobal.com.invalid<mailto:john.lil...@redpointglobal.com.invalid>> wrote: Greetings! I have recently started enumerating messages in a queue from time to time, because we’re trying to figure out if a unit of work is still pending. But I’m getting this error occasionally: 2023-07-27T21:38:30.007 [CanvasState-Abandoned] RpdmQueueUtils.browseStartMessages:159 [] ERROR - Error enumerating messages for pod {"message_type":"pod_specifier","pod_type":"project_execution","memory_mb":2048,"jvm_memory_mb":250} java.lang.RuntimeException: AMQ219023: The large message lost connection with its session, either because of a rollback or a closed session at org.apache.activemq.artemis.core.client.impl.ClientLargeMessageImpl.getBodyBuffer(ClientLargeMessageImpl.java:91<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientLargeMessageImpl.java%3a91&c=E,1,EMAB08TWdYYO4LFWRLqdddz9Q_iAB7ChMRHrrEIJNdpVgnHLMM2sPAlVQqpMEtpzpR_YJMplv0cpDe_Ah4298DOeO--OpEZY9zjP9BOF4fWg3LGve2qthw,,&typo=1&ancr_add=1>) at org.apache.activemq.artemis.jms.client.ActiveMQBytesMessage.readBytes(ActiveMQBytesMessage.java:217<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActiveMQBytesMessage.java%3a217&c=E,1,U8iiuF_0vvzFKT0OOuKIbN7FNmzqkVv5e8GTV98Lwki9ap5Ib0TJcYLCPgmyvpvwt1SI1hqGgxJ_dCfTCXyQSpi4uECf2uPKseho-csKsa0B&typo=1&ancr_add=1>) at net.redpoint.rpdm.services.RpdmQueueUtils.browseStartMessages(RpdmQueueUtils.java:155<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fRpdmQueueUtils.java%3a155&c=E,1,hFvYvhx-Bm7jX9I4bSkyUExXtlTOjbxHQ4ItkXy1yYQ5B7f9-WBh6CmX8Pv4pFUokof44Nji5MQQa6kvLxIGeJMGJHD-IrlgDvqxM6k7ukO3ShFOP34gW9caaw,,&typo=1&ancr_add=1>) at net.redpoint.rpdm.services.RpdmQueueUtils.getAllQueuedCanvasIds(RpdmQueueUtils.java:76<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fRpdmQueueUtils.java%3a76&c=E,1,PRn1WhZx6azSck3GU473g-9ktCSOR97NsDjrkiP3OJ5JQyMWj5C6UBEnwThoG_xk3DAN7ADM1uPtxdLVcvUr99q1B0UDxs4JYrId5pWmkKk3RDKT8y3rNuM,&typo=1&ancr_add=1>) at net.redpoint.rpdm.canvas_state_query.CanvasStateQueryServerImpl.doCanvasAbandoned(CanvasStateQueryServerImpl.java:282<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fCanvasStateQueryServerImpl.java%3a282&c=E,1,0MQh_Hg6m_VB9wkxppYn2HByHmbQBz_0Rx2NC4eYdhXUhKEm6SWA6_8sTphQnaVqeYVGCacM3y7iQrP8nXwxHY-fQlzhvAs1uw1QuD81ixb_ryTjCDtK&typo=1&ancr_add=1>) at net.redpoint.rpdm.canvas_state_query.CanvasStateQueryServerImpl$Abandoned.action(CanvasStateQueryServerImpl.java:265<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fCanvasStateQueryServerImpl.java%3a265&c=E,1,bvfVsjRdib5lJ43J88SQ29Sxp2Bpn3l1qL4J4SWoSaZ_O6Ctfc8E20zhvl_sVqGAsR1xPJyaebjiJcFq_ptrM_AFjm7-MPZtCnwwrns5_2bVkuYWVe37&typo=1&ancr_add=1>) at net.redpoint.rpdm.services.LazyStartPeriodicThread.run<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fnet.redpoint.rpdm.services.LazyStartPeriodicThread.run&c=E,1,bnecENh1w9bBi6pzB-qxVHg9uPXowUJ6I9704_7OYZedkdrLHQ7EkvUS2umxkDmrw6PRsrEWgD7GpjjxAUnFaZ6_tiO9CdHO6OzVZ_6VfYDrcVmOETkv&typo=1&ancr_add=1>(LazyStartPeriodicThread.java:91<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fLazyStartPeriodicThread.java%3a91&c=E,1,S_LcUAzeod145HEXSnkelQWJrrZecCtoxGDOL3xfr8mCp4-pAiPROYaP0iZMvvHC3RpC9cl3PQwadzu2NvYGyUKWSsXPy7XjofrmhXI-TS6rGQE,&typo=1&ancr_add=1>) at java.base/java.lang.Thread.run<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fjava.lang.Thread.run&c=E,1,d3f8dPziDguMd4SJKV6eafc--V1uw-wct5EuGNVEoYVRHrz72swuyAKlJn5O_VoMIZ9Mo7sa4fnd-CzYay_khdKTWcC0WZRFpbcJXkLfug,,&typo=1&ancr_add=1>(Thread.java:833<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThread.java%3a833&c=E,1,Hxd07ytuTRhxKVOBsq2S5bA38NzJ62DEPJQF07nZYOnP-eg9kNSlLY4B1rOds_M5IQddK2jmK5fcmKjiK9sGna3cl3fz13CmqweNhY3-TWKoOoMJ0K0,&typo=1&ancr_add=1>) Caused by: ActiveMQIllegalStateException[errorType=ILLEGAL_STATE message=AMQ219023: The large message lost connection with its session, either because of a rollback or a closed session] at org.apache.activemq.artemis.core.client.impl.LargeMessageControllerImpl.saveBuffer(LargeMessageControllerImpl.java:265<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fLargeMessageControllerImpl.java%3a265&c=E,1,1quz-1SHZC1IoSRTiHB03sEal2tuBmA7SJl_0sa7wzLcGksj_4dxP2uEAvHk2pXuxJ9nnURmiODisln7UJbTbUwmj_DIsYwAObKyM94iKS1L3c00SPPHsDP45t4,&typo=1&ancr_add=1>) at org.apache.activemq.artemis.core.client.impl.ClientLargeMessageImpl.checkBuffer(ClientLargeMessageImpl.java:157<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientLargeMessageImpl.java%3a157&c=E,1,vq27LzS9s5yRES-dOKOFpVtbLWXc1JtCHQr2YAwe1IJNzTEvwtCl3_8daFmvYG_2inn0BvXRzWxortesj76eFbMhrqlBD7bHWgI8d32ZYCDzu9u_5Q0,&typo=1&ancr_add=1>) at org.apache.activemq.artemis.core.client.impl.ClientLargeMessageImpl.getBodyBuffer(ClientLargeMessageImpl.java:89<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fClientLargeMessageImpl.java%3a89&c=E,1,osSEY4cd7T7FrSSE-EhUE3aPXoV7wA6Tu7J36SGo19afHPoONkFPd5vQa21XDrjEjQVgt2Wv1ECI_OOoNsIF9d0dfwkgmUB3sPO4WvyG_zUs3Q,,&typo=1&ancr_add=1>) This happens in two parts using JMS. The first enumerates the queue using browseMessages() to collect a List<Message>, and the second processes the Messages, reading their bodies and examining them. The problem occurs on this line in browseStartMessages when we read the body bytes bm.readBytes(requestBytes); Despite the error text, the session isn’t closed, or at least this error doesn’t always happen despite my not doing anything to re-create the session. I suspect that, given the ClientLargeMessageImpl.getBodyBuffer on the stack, something is going wrong that is specific to “large” messages. IIRC the large messages use a coat-check pattern to store the message on disk and bypass the in-memory queue. Can you suggest a solution? I’ve though of two potential changes: - Raise minLargeMessageSize to something big enough that our messages are never “large” - Read the body bytes immediately, while enumerating the queue, instead of later when looping over the List<Message> I was also trying to see if there is a way to attach “metadata” to a message and only read that? These messages are large, and presumably it is expensive to scan them, and I’m really only looking for a UUID embedded in all of that. JDK: Temurin 17 AMQ broker version: 2.28.0 AMQ JMS client dependency: <dependency> <groupId>org.apache.activemq</groupId> <artifactId>artemis-jms-client-all</artifactId> <version>2.28.0</version> </dependency> Thanks John public List<Message> browseMessages(String queueName) { // This helps avoid a known issue with queue message enumeration (in ActiveMQ "classic"), where after getting the final message // it hangs for a significant time, up to 20 seconds, before the Enumeration returns false. int count = getQueueEntryCount(queueName); List<Message> result = new ArrayList<>(); synchronized (lock) { try { Queue queue = getSession().createQueue(queueName); try (QueueBrowser browser = getSession().createBrowser(queue)) { Enumeration e = browser.getEnumeration(); while (count-- > 0 && e.hasMoreElements()) { try { result.add((Message) e.nextElement()); } catch (Exception ex) { LOG.warn("Error browsing queue '" + queueName + "'", ex); } } } } catch (Exception ex) { LOG.warn("Error browsing queue '" + queueName + "'", ex); } } return result; } The second processes the Messages: public List<RpcRequestPacket> browseStartMessages(PodSpecifier podSpecifier) { List<RpcRequestPacket> result = new ArrayList<>(); for (var message : browseMessages(getStartQueueName(podSpecifier))) { var bm = (BytesMessage)message; try { var requestBytes = new byte[(int) bm.getBodyLength()]; bm.readBytes(requestBytes); result.add(new RpcRequestPacket(requestBytes, new JmsRpcMessageMetadata(message.getJMSMessageID(), ""))); } catch (Exception ex) { LOG.error("Error enumerating messages for pod " + podSpecifier, ex); } } return result; } [rg]<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,dVmfqHI7KhMWjq-qYlMqDecRlRztXUKIY7-ob3A13WZmS4OA2MHx0VHKZyvbQBZC9TJd7QNsKJSxPBFM2w9iMo6FUIQwsQ_PM6uJg1w4pixLGB_5a6AVcL6-qmE,&typo=1> John Lilley Data Management Chief Architect, Redpoint Global Inc. 888 Worcester Street, Suite 200 Wellesley, MA 02482 M: +1 7209385761<tel:+1%207209385761> | john.lil...@redpointglobal.com<mailto:john.lil...@redpointglobal.com> PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is confidential and is intended solely for the use of the individual(s) to whom it is addressed. If you believe you received this e-mail in error, please notify the sender immediately, delete the e-mail from your computer and do not copy, print or disclose it to anyone else. If you properly received this e-mail as a customer, partner or vendor of Redpoint, you should maintain its contents in confidence subject to the terms and conditions of your agreement(s) with Redpoint. -- Clebert Suconic PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is confidential and is intended solely for the use of the individual(s) to whom it is addressed. If you believe you received this e-mail in error, please notify the sender immediately, delete the e-mail from your computer and do not copy, print or disclose it to anyone else. If you properly received this e-mail as a customer, partner or vendor of Redpoint, you should maintain its contents in confidence subject to the terms and conditions of your agreement(s) with Redpoint.
