Clebert, We're updating to 2.30.0 as you suggested , but even without that, I'm seeing huge improvement by reading the message body bytes while still in the enumeration loop instead waiting until later. I suspect that there is some promise that the body bytes for large messages are available in the enumeration loop, but might be purged later?
Regardless, this issue appears to be fixed now. Thanks John [rg] <https://www.redpointglobal.com/> John Lilley Data Management Chief Architect, Redpoint Global Inc. 888 Worcester Street, Suite 200 Wellesley, MA 02482 M: +1 7209385761<tel:+1%207209385761> | john.lil...@redpointglobal.com<mailto:john.lil...@redpointglobal.com> -----Original Message----- From: Clebert Suconic <clebert.suco...@gmail.com<mailto:clebert.suco...@gmail.com>> Sent: Sunday, July 30, 2023 4:07 PM To: users@activemq.apache.org<mailto:users@activemq.apache.org> Subject: Re: Unexpected Error Stopping Artemis 2.26.0 broker *** [Caution] This email is from an external source. Please use caution responding, opening attachments or clicking embedded links. *** there was a thread accessing paging as the server was stopping, what caused this exception. I would try with latest release on paging as page-cache was removed. On Mon, Jul 24, 2023 at 7:26 AM Shiv Kumar Dixit <shivkumardi...@eurofins.com.invalid<mailto:shivkumardi...@eurofins.com.invalid>> wrote: > > We see below error in artemis.log and it is stopping the broker. Broker > version is 2.26.0. Before broker stops, we see many entries where paging on > different addresses are starting and stopping. When paging related logs on > different addresses are done, we see message 'Cannot find iter to remove: > java.lang.RuntimeException:Cannot<ftp://java.lang.RuntimeException/Cannot> > find iter to remove' is logged and post that broker stops. Any input/lead > will be helpful. > > --------------Start of log------------------------- > 2023-07-17T12:22:12.999Z INFO --- [Thread-2 (Activ] > org.apache.activemq.artemis.core.server : AMQ224108: Stopped paging on > address 'XXXX'; size=0 bytes (0 messages); maxSize=-1 bytes (-1 messages); > globalSize=1035418769 bytes (76908 messages); globalMaxSize=1048576000 bytes > (-1 messages); .............. > ............. > .............. > ............... > > 2023-07-17T12:22:39.721Z WARN --- [Thread-1 (Activ] > org.apache.activemq.artemis.utils.actors.OrderedExecutor : Cannot find iter > to remove: > java.lang.RuntimeException:Cannot<ftp://java.lang.RuntimeException/Cannot> > find iter to remove > at > org.apache.activemq.artemis.core.paging.cursor.impl.PageSubscriptionImpl$https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fCursorIterator.next&c=E,1,UXJtBcA32yRhc_G7LwnwFjiEicJf9Q-z7CX0Aa-gPhOjOvLogJTGcr0VCTIBJEGIDaXkKC2bKm3R4zvfw7WZZKuhajH8sDzHIGaP8kpeilTwAqcsRgfVnucScbU,&typo=1(https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fPageSubscriptionImpl.java%3a1353&c=E,1,WGHdbv7lYDQk1CetmkaqPZ3mH4240gGXhJGopBcN4Pg0sa_7irgN34LdPumdLCI5zrkfhQL0ycFPfm4CTsUE9y-XWkKXUh9E8ju1n46fpCPA-3H8fALKkbl U&typo=1<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fPageSubscriptionImpl.java%3a1353&c=E,1,WGHdbv7lYDQk1CetmkaqPZ3mH4240gGXhJGopBcN4Pg0sa_7irgN34LdPumdLCI5zrkfhQL0ycFPfm4CTsUE9y-XWkKXUh9E8ju1n46fpCPA-3H8fALKkblU&typo=1>) [artemis-server-2.26.0.jar:2.26.0<ftp://artemis-server-2.26.0.jar/2.26.0>] > at > org.apache.activemq.artemis.core.paging.cursor.impl.PageSubscriptionImpl$CursorIterator.tryNext(https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fPageSubscriptionImpl.java%3a1492&c=E,1,3ufKq4Qo29ORxdtZS0M7yP2tEcYOXMA_GwjKt0YW-e_WVijfHK9WnmiM_ByohPCHYaUJUMuhz7bH1J_HrRlnKcEnQVD5eR76ufwichZL47zV&typo=1) > [artemis-server-2.26.0.jar:2.26.0<ftp://artemis-server-2.26.0.jar/2.26.0>] > at > org.apache.activemq.artemis.core.server.impl.QueueImpl$https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fExpiryScanner.run&c=E,1,KT2ZhSWrHEJzMxqZWthTe4xKQ7196PPO0K6QUA1ohSgbNESWEKHW-liI0XFkOK8vCwzygFy7FbVuD9oBQ95r168sjZK6knZ9EZJG0hirdLD0QurEwDKSvn97kEY,&typo=1(https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fQueueImpl.java%3a2522&c=E,1,foClFRFLzATGMuIO9UQuaZ7bCM5yW3PeUczUCPmeFsczz_C3fIy3G-xH5tMnCiRUoYH-9Kjyu82VL2U1EM78IJN-OROH6_VvvcHfiorloA,,&typo=1) > [ artemis-server-2.26.0.jar:2.26.0<ftp://artemis-server-2.26.0.jar/2.26.0>] > at > org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a56&c=E,1,Sc1xhb4LVLvknCsIT39Fhei8UVC8AiErtiX5koedLxHYXjB4x38lpSGJ-IcvDzBbpOvoHvzKflkBYhaBOomes8eJCcUhB0crkm7RkZYDsIA7YN4zWrpu7uE,&typo=1) > [artemis-commons-2.26.0.jar:] > at > org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fOrderedExecutor.java%3a31&c=E,1,Gbj0NBddIIH0zGVyUuxN1Qj2niqz6l6FfyjGkvTeubtbC0aI1esVnZ2gL43RgoqqJqxi_Y4pcStNcuLcEscMgf0F0OQm8TzOSFiRB7t6AH87BupBrADLI8E,&typo=1) > [artemis-commons-2.26.0.jar:] > at > org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fProcessorBase.java%3a67&c=E,1,ryEG2po7x9s5rHBasgQBjVqFh8mNrXMCN1IpAVB-GFUc77zCC7AB-k7OiaFiBk9gNhGu4HWHfoM4kYP4NnJz_tu0C7lWoIdhMQMHae7--nOdTLhQtTGgG08fJ2ru&typo=1) > [artemis-commons-2.26.0.jar:] > at > java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a1128&c=E,1,BdKWHsZ8bKTj_SPWCzOLuTM34hXx071gsh-LRKVLaFOg_okP665-WKuoveoIA3jF8xkrKT3TnO27Gd_nQVAa9wm9OTcSQqsjdHWRhSkM849tgQL9ka_sYl4bLg,,&typo=1) > [java.base:] > at > java.base/java.util.concurrent.ThreadPoolExecutor$https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fWorker.run&c=E,1,ebDF28XLJvH1xRzjqPwg0PFiS4VdfVsOUgsRoacdxobRTOCCNYSDTJJRWLYBqd-oiDFp7yxretlWHT-DY4FVA1sC1ouCdAeFpxeq220fa6KjSyXHDmOco3kV&typo=1(https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fThreadPoolExecutor.java%3a628&c=E,1,pg0K7aDtqrlp5v0SWMeP3NCZ_6KHHRmYD1lyWzYCdqrRgkRm3D_5-vy6lfIZHcXS1jgxAMaYPlLpky7ItajWVly5WvVFkg1w5ij8Cs81lu97lTAx1C7DPfMtXsE,&typo=1) > [java.base:] > at > org.apache.activemq.artemis.utils.ActiveMQThreadFactory$https://linkprotect.cudasvc.com/url?a=https%3a%2f%2f1.run&c=E,1,6vH2LkbXrdZ3JmhyoqiitxD4_tEJpbf0-tHiq4QJ2281LLLbZl_w692Jnxc1vMDQTBsdZSEkCNCp2BQ_dLjBruMGOPUlSo_V-OuRoW98edONP9bAENLdNJUO&typo=1(https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fActiveMQThreadFactory.java%3a118&c=E,1,XkvM8V_gVacV7pgTtr_o9ScOrw1IHf8lDyJ0p7hpj5fHhVvmudyRMArALaI8WrxQKdsXZoYDZXE1t9DH541odUJ8S1I7hutrEW-z6m3lGo8bKO4aR_9-gFb-&typo=1) > [artemis-commons-2.26.0.jar:] Caused by: > java.lang.IllegalStateException:Cannot<ftp: %20//java.lang.IllegalStateException/Cannot> find iter to remove > at > org.apache.activemq.artemis.core.paging.cursor.impl.PageSubscriptionImpl$CursorIterator.initPage(https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fPageSubscriptionImpl.java%3a1295&c=E,1,AXwLOqTZ1c43xvSxCl2yJk5A0eAzum-6JIwewNmzuAMGlS4cLmyCmERg6aJcRd-6-VsE7KfLAw6XnGUI79oJ7qxRjKykjzQ1GFN4adqQQFaNEA,,&typo=1) > [artemis-server-2.26.0.jar:2.26.0<ftp://artemis-server-2.26.0.jar/2.26.0>] > at > org.apache.activemq.artemis.core.paging.cursor.impl.PageSubscriptionImpl$CursorIterator.internalGetNext(https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fPageSubscriptionImpl.java%3a1473&c=E,1,76aBqGUgEsZPG_VKrfOvuWsVwnCqZmrpNHRqZdeEJ8DWlP8FssZFEw6yM7l2Tlb0h0TP74RpkT7xW5uTDgfX9Ilqd9OcxZP76mC4sAKJALQ2vDiIWyTi1uTg,,&typo=1<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fPageSubscriptionImpl.java%3a1473&c=E,1,76aBqGUgEsZPG_VKrfOvuWsVwnCq<u>Z</u>mrpNHRqZdeEJ8DWlP8FssZFEw6yM7l2Tlb0h0TP74RpkT7xW5uTDgfX9Ilqd9OcxZP76mC4sAKJALQ2vDiIWyTi1uTg,,&typo=1>) > [artemis-server-2.26.0.jar:2.26.0<ftp://artemis-server-2.26.0.jar/2.26.0>] > at > org.apache.activemq.artemis.core.paging.cursor.impl.PageSubscriptionImpl$CursorIterator.moveNext(https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fPageSubscriptionImpl.java%3a1378&c=E,1,JKSzZy1vOMSJ9hMxoUnE2Jq4-VqO50-rz_f6YZnKmEGRRWN3jGDfKyKPMB_VYMy-bRq2saeDCBPAxFnAJaIu8lbZyiffegUAf7UV-MlldKatvWG3&typo=1) > [artemis-server-2.26.0.jar:2.26.0<ftp://artemis-server-2.26.0.jar/2.26.0>] > at > org.apache.activemq.artemis.core.paging.cursor.impl.PageSubscriptionImpl$https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fCursorIterator.next&c=E,1,5mDw_GnLLPUac9TCpnv_TT0YF2g8bB37SmE8RkfE50qmy4VfEoNZOh3RZQwzLLobGXgedBAUj13xcaTZg-jKykcc8BnHZaFcbnVMMRvEne3OhsA,&typo=1(https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fPageSubscriptionImpl.java%3a1348&c=E,1,EVj8VBd8aXDgNkalSwF3J0qu1tjU7xPb4M374FF-NEojAiJBCB1DwaJ0z6zaqKwDwg7JtA4_4nchvLZvoUK_bqR1SpqhU4Q--2VX4-F1DYGNw86VUQ,,&typo=1) > [artemis-server-2.26.0.jar:2.26.0<ftp: %20//artemis-server-2.26.0.jar/2.26.0>] > ... 8 more > Caused by: > java.lang.IllegalStateException:Cannot<ftp://java.lang.IllegalStateException/Cannot> > find iter to remove > at > org.apache.activemq.artemis.utils.collections.LinkedListImpl.removeIter(https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fLinkedListImpl.java%3a370&c=E,1,O0beSCQ7FvpZXUpNPN0XPjrAUY6Jhb0zUHqqXmRy-Q4GZ-qqTzGsgrxMZvY6j39Qw2u7nMwheaRq9KKnA3FClnq8w5q3pTAuExMpzNPgoA,,&typo=1) > [artemis-commons-2.26.0.jar:] > at > org.apache.activemq.artemis.utils.collections.LinkedListImpl$Iterator.close(https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fLinkedListImpl.java%3a527&c=E,1,z2tGf1op3FGLdssjjfuaECpLwTM1jxCbjig1vSo6Bjt-hkXDe0F_4BNvg7qezgf4WDzUPM9l89rhVu-tqM8HYanQO8YVUHJB-NkuFP8LGYbmTQm6TEtw,,&typo=1<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fLinkedListImpl.java%3a527&c=E,1,z2tGf1op3FGLdssjjfuaECpLwTM1jxCbjig1vSo6Bjt-hkXDe0F_4BNvg7qezgf4WDzUPM9l89rhVu-tqM8H<u>Y</u>anQO8YVUHJB-NkuFP8LGYbmTQm6TEtw,,&typo=1>) > [artemis-commons-2.26.0.jar:] > at > org.apache.activemq.artemis.core.paging.cursor.impl.PageSubscriptionImpl$CursorIterator.initPage(https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fPageSubscriptionImpl.java%3a1286&c=E,1,FZ2Y0A-BjgwRM5CdPL-6K0JDGyOdE4PLF0UPcwsNVtI39cLzP1JkMX8Fc1v2FyJiTrL4uXBZzEFFyfJ4DwXM_ezCaJEOwY2hhYn59o3lNjycBw,,&typo=1) > [artemis-server-2.26.0.jar:2.26.0<ftp://artemis-server-2.26.0.jar/2.26.0>] > ... 11 more > > 2023-07-17T12:22:53.266Z INFO --- [Thread-27] > org.apache.activemq.artemis : AMQ241005: Stopping embedded web server > 2023-07-17T12:22:53.317Z INFO --- [Thread-27] > io.hawt.web.auth.AuthenticationFilter : Destroying hawtio > authentication filter 2023-07-17T12:22:53.323Z INFO --- [Thread-27] > io.hawt.HawtioContextListener : Destroying hawtio services > 2023-07-17T12:22:53.356Z INFO --- [Thread-27] > org.apache.activemq.hawtio.plugin.PluginContextListener : Destroyed > artemis-plugin plugin 2023-07-17T12:22:53.373Z INFO --- [Thread-27] > org.apache.activemq.hawtio.branding.PluginContextListener : Destroyed > activemq-branding plugin 2023-07-17T12:22:53.405Z INFO --- > [Thread-27] org.apache.activemq.artemis : AMQ241006: Stopped embedded > web server 2023-07-17T12:22:53.405Z INFO --- [Thread-27] > org.apache.activemq.artemis.core.server : AMQ221002: Apache ActiveMQ > Artemis Message Broker version 2.26.0 > [66ff429e-f80d-11eb-b357-0050568f9608] stopped, uptime 6 minutes > --------------End of log------------------------- > -- Clebert Suconic PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is confidential and is intended solely for the use of the individual(s) to whom it is addressed. If you believe you received this e-mail in error, please notify the sender immediately, delete the e-mail from your computer and do not copy, print or disclose it to anyone else. If you properly received this e-mail as a customer, partner or vendor of Redpoint, you should maintain its contents in confidence subject to the terms and conditions of your agreement(s) with Redpoint.
