Re: [artemis 2.1.0] taking 30+ minutes to boot & failover
thank you very much for your work on this. i really appreciate it. On Thu, Aug 23, 2018 at 12:26 PM Clebert Suconic wrote: > > what is the timeline on a 2.6.3 or 2.7.0 release? > > > > > I was supposed to release it next week based on my HEADS up. I'm > working on some extensive QA and dev at the moment.. but I should > make it by next week. >
Re: [artemis 2.1.0] taking 30+ minutes to boot & failover
> what is the timeline on a 2.6.3 or 2.7.0 release? > I was supposed to release it next week based on my HEADS up. I'm working on some extensive QA and dev at the moment.. but I should make it by next week.
Re: [artemis 2.1.0] taking 30+ minutes to boot & failover
i validated that the fix did in fact fix our upgrade path. i made a new 2.1.0 broker. i added some address settings via JMX. i attempted an upgrade to 2.6.2 and watched it fail. i pulled the code that you linked above. i packaged a release and then i upgraded to the released code and the server started just fine. thank you so much!! what is the timeline on a 2.6.3 or 2.7.0 release? On Fri, Aug 3, 2018 at 4:22 PM Clebert Suconic wrote: > I'm considering doing a 2.6.3 very soon.. it will include the fix.. > you should be able to move from 2.1.0 . let me know if you have any > other issues. > > On Fri, Aug 3, 2018 at 6:22 PM, Clebert Suconic > wrote: > > https://github.com/apache/activemq-artemis/pull/2214 > > > > > > I will make it into 2.6.x as soon as the PR tests are finished. > > > > > > If you could try a snapshot of master or 2.6.x (after I merge it of > > course). Let me know if you need help buildling it. > > > > On Wed, Jul 25, 2018 at 5:17 PM, Dan Langford > wrote: > >> Michael for the win! > >> @Clebert im not opposed to coding up a test case it is just that i am > >> swamped at the moment so im glad to hear that Michael already has a > bead on > >> this > >> > >> i did a *./artemis data exp* out of my 2.1.0 and then an *./artemis data > >> imp* on a clean 2.6.2. the new system didnt have Address Settings and > >> Security Settings that had previously been added via the jolokia api. i > >> should look into the export options more to see if i just failed to > provide > >> a command line argument. fortunately in my scenario it was pretty > trivial > >> to recreate those settings quickly. so i did that. the import created > all > >> the addresses, routes, queues, and messages that i needed. it went > pretty > >> well. so my cluster is now on 2.6.2 > >> > >> i plan to rework our usage of AddressSetting to not create a new entry > for > >> every Address we create but instead only create them for the few > exceptions > >> to our default configuration > >> > >> then i can start working on running the broker in CF with the cluster in > >> more of a Master-Master mode. > >> thanks again for all your help. feel free to respond if there is any > more > >> data you need from me. > >> > >> *side note: my broker is now spamming " AMQ224088: Timeout (10 seconds) > >> while handshaking has occurred. " but thats another issue i will figure > >> out. * > >> > >> my 30 minute startup times are now down to 30 seconds max. more like > 10-15 > >> seconds regularly. THANK YOU ALL SO MUCH FOR YOUR HELP. > >> > >> On Wed, Jul 25, 2018 at 2:44 PM michael.andre.pearce > >> wrote: > >> > >>> Clebert im able to recreate this and also worked out in code why this > >>> happens. Ill try catch you on irc tomorrow or friday. > >>> > >>> Sent from my Samsung Galaxy smartphone. > >>> Original message From: Clebert Suconic < > >>> clebert.suco...@gmail.com> Date: 25/07/2018 20:20 (GMT+00:00) To: > >>> users@activemq.apache.org Subject: Re: [artemis 2.1.0] taking 30+ > minutes > >>> to boot & failover > >>> Yes.. it should include it. > >>> > >>> > >>> Do you have an example of the API you used with 2.1? > >>> > >>> > >>> I can't write a compatibility test this week.. but if you provide me > >>> the example I will do it early next week. > >>> > >>> > >>> In a perfect world, if you did it .. it would be great :) > >>> > >>> > >>> On Wed, Jul 25, 2018 at 1:40 PM, Dan Langford > >>> wrote: > >>> > i tried 2.6.2 this morning to see if that was an improvement from > 2.5.0 i > >>> > tried a few months ago. unfortunately there was not. it may have > failed > >>> > much faster than 2.5.0 i dont recall the timing specifics but the > error > >>> is > >>> > the same > >>> > > >>> > 11:15:50,853 ERROR [org.apache.activemq.artemis.core.server] > AMQ224000: > >>> > Failure in initialisation: java.lang.NegativeArraySizeException > >>> > at > >>> > > >>> > org.apache.activemq.artemis.api.core.SimpleString.readSimpleString(SimpleString.java:182) > >>> > [art
Re: [artemis 2.1.0] taking 30+ minutes to boot & failover
I'm considering doing a 2.6.3 very soon.. it will include the fix.. you should be able to move from 2.1.0 . let me know if you have any other issues. On Fri, Aug 3, 2018 at 6:22 PM, Clebert Suconic wrote: > https://github.com/apache/activemq-artemis/pull/2214 > > > I will make it into 2.6.x as soon as the PR tests are finished. > > > If you could try a snapshot of master or 2.6.x (after I merge it of > course). Let me know if you need help buildling it. > > On Wed, Jul 25, 2018 at 5:17 PM, Dan Langford wrote: >> Michael for the win! >> @Clebert im not opposed to coding up a test case it is just that i am >> swamped at the moment so im glad to hear that Michael already has a bead on >> this >> >> i did a *./artemis data exp* out of my 2.1.0 and then an *./artemis data >> imp* on a clean 2.6.2. the new system didnt have Address Settings and >> Security Settings that had previously been added via the jolokia api. i >> should look into the export options more to see if i just failed to provide >> a command line argument. fortunately in my scenario it was pretty trivial >> to recreate those settings quickly. so i did that. the import created all >> the addresses, routes, queues, and messages that i needed. it went pretty >> well. so my cluster is now on 2.6.2 >> >> i plan to rework our usage of AddressSetting to not create a new entry for >> every Address we create but instead only create them for the few exceptions >> to our default configuration >> >> then i can start working on running the broker in CF with the cluster in >> more of a Master-Master mode. >> thanks again for all your help. feel free to respond if there is any more >> data you need from me. >> >> *side note: my broker is now spamming " AMQ224088: Timeout (10 seconds) >> while handshaking has occurred. " but thats another issue i will figure >> out. * >> >> my 30 minute startup times are now down to 30 seconds max. more like 10-15 >> seconds regularly. THANK YOU ALL SO MUCH FOR YOUR HELP. >> >> On Wed, Jul 25, 2018 at 2:44 PM michael.andre.pearce >> wrote: >> >>> Clebert im able to recreate this and also worked out in code why this >>> happens. Ill try catch you on irc tomorrow or friday. >>> >>> Sent from my Samsung Galaxy smartphone. >>> Original message From: Clebert Suconic < >>> clebert.suco...@gmail.com> Date: 25/07/2018 20:20 (GMT+00:00) To: >>> users@activemq.apache.org Subject: Re: [artemis 2.1.0] taking 30+ minutes >>> to boot & failover >>> Yes.. it should include it. >>> >>> >>> Do you have an example of the API you used with 2.1? >>> >>> >>> I can't write a compatibility test this week.. but if you provide me >>> the example I will do it early next week. >>> >>> >>> In a perfect world, if you did it .. it would be great :) >>> >>> >>> On Wed, Jul 25, 2018 at 1:40 PM, Dan Langford >>> wrote: >>> > i tried 2.6.2 this morning to see if that was an improvement from 2.5.0 i >>> > tried a few months ago. unfortunately there was not. it may have failed >>> > much faster than 2.5.0 i dont recall the timing specifics but the error >>> is >>> > the same >>> > >>> > 11:15:50,853 ERROR [org.apache.activemq.artemis.core.server] AMQ224000: >>> > Failure in initialisation: java.lang.NegativeArraySizeException >>> > at >>> > >>> org.apache.activemq.artemis.api.core.SimpleString.readSimpleString(SimpleString.java:182) >>> > [artemis-commons-2.6.2.jar:2.6.2] >>> > at >>> > >>> org.apache.activemq.artemis.api.core.SimpleString.readSimpleString(SimpleString.java:171) >>> > [artemis-commons-2.6.2.jar:2.6.2] >>> > at >>> > >>> org.apache.activemq.artemis.api.core.SimpleString.readNullableSimpleString(SimpleString.java:158) >>> > [artemis-commons-2.6.2.jar:2.6.2] >>> > at >>> > >>> org.apache.activemq.artemis.core.buffers.impl.ChannelBufferWrapper.readNullableSimpleString(ChannelBufferWrapper.java:69) >>> > [artemis-commons-2.6.2.jar:2.6.2] >>> > at >>> > >>> org.apache.activemq.artemis.core.settings.impl.AddressSettings.decode(AddressSettings.java:736) >>> > [artemis-server-2.6.2.jar:2.6.2] >>> > at >>> > >>> org.apache.activemq.artemis.core.persistence.config.PersistedAddressSetting.decode(Persisted
Re: [artemis 2.1.0] taking 30+ minutes to boot & failover
https://github.com/apache/activemq-artemis/pull/2214 I will make it into 2.6.x as soon as the PR tests are finished. If you could try a snapshot of master or 2.6.x (after I merge it of course). Let me know if you need help buildling it. On Wed, Jul 25, 2018 at 5:17 PM, Dan Langford wrote: > Michael for the win! > @Clebert im not opposed to coding up a test case it is just that i am > swamped at the moment so im glad to hear that Michael already has a bead on > this > > i did a *./artemis data exp* out of my 2.1.0 and then an *./artemis data > imp* on a clean 2.6.2. the new system didnt have Address Settings and > Security Settings that had previously been added via the jolokia api. i > should look into the export options more to see if i just failed to provide > a command line argument. fortunately in my scenario it was pretty trivial > to recreate those settings quickly. so i did that. the import created all > the addresses, routes, queues, and messages that i needed. it went pretty > well. so my cluster is now on 2.6.2 > > i plan to rework our usage of AddressSetting to not create a new entry for > every Address we create but instead only create them for the few exceptions > to our default configuration > > then i can start working on running the broker in CF with the cluster in > more of a Master-Master mode. > thanks again for all your help. feel free to respond if there is any more > data you need from me. > > *side note: my broker is now spamming " AMQ224088: Timeout (10 seconds) > while handshaking has occurred. " but thats another issue i will figure > out. * > > my 30 minute startup times are now down to 30 seconds max. more like 10-15 > seconds regularly. THANK YOU ALL SO MUCH FOR YOUR HELP. > > On Wed, Jul 25, 2018 at 2:44 PM michael.andre.pearce > wrote: > >> Clebert im able to recreate this and also worked out in code why this >> happens. Ill try catch you on irc tomorrow or friday. >> >> Sent from my Samsung Galaxy smartphone. >> Original message ----From: Clebert Suconic < >> clebert.suco...@gmail.com> Date: 25/07/2018 20:20 (GMT+00:00) To: >> users@activemq.apache.org Subject: Re: [artemis 2.1.0] taking 30+ minutes >> to boot & failover >> Yes.. it should include it. >> >> >> Do you have an example of the API you used with 2.1? >> >> >> I can't write a compatibility test this week.. but if you provide me >> the example I will do it early next week. >> >> >> In a perfect world, if you did it .. it would be great :) >> >> >> On Wed, Jul 25, 2018 at 1:40 PM, Dan Langford >> wrote: >> > i tried 2.6.2 this morning to see if that was an improvement from 2.5.0 i >> > tried a few months ago. unfortunately there was not. it may have failed >> > much faster than 2.5.0 i dont recall the timing specifics but the error >> is >> > the same >> > >> > 11:15:50,853 ERROR [org.apache.activemq.artemis.core.server] AMQ224000: >> > Failure in initialisation: java.lang.NegativeArraySizeException >> > at >> > >> org.apache.activemq.artemis.api.core.SimpleString.readSimpleString(SimpleString.java:182) >> > [artemis-commons-2.6.2.jar:2.6.2] >> > at >> > >> org.apache.activemq.artemis.api.core.SimpleString.readSimpleString(SimpleString.java:171) >> > [artemis-commons-2.6.2.jar:2.6.2] >> > at >> > >> org.apache.activemq.artemis.api.core.SimpleString.readNullableSimpleString(SimpleString.java:158) >> > [artemis-commons-2.6.2.jar:2.6.2] >> > at >> > >> org.apache.activemq.artemis.core.buffers.impl.ChannelBufferWrapper.readNullableSimpleString(ChannelBufferWrapper.java:69) >> > [artemis-commons-2.6.2.jar:2.6.2] >> > at >> > >> org.apache.activemq.artemis.core.settings.impl.AddressSettings.decode(AddressSettings.java:736) >> > [artemis-server-2.6.2.jar:2.6.2] >> > at >> > >> org.apache.activemq.artemis.core.persistence.config.PersistedAddressSetting.decode(PersistedAddressSetting.java:95) >> > [artemis-server-2.6.2.jar:2.6.2] >> > at >> > >> org.apache.activemq.artemis.core.persistence.impl.journal.AbstractJournalStorageManager.newAddressEncoding(AbstractJournalStorageManager.java:1925) >> > [artemis-server-2.6.2.jar:2.6.2] >> > at >> > >> org.apache.activemq.artemis.core.persistence.impl.journal.AbstractJournalStorageManager.loadBindingJournal(AbstractJournalStorageManager.java:1466) >> > [artemis-server-2.6.2.jar:2.6.2] >> > at >> > >> org.apache.activemq.art
Re: [artemis 2.1.0] taking 30+ minutes to boot & failover
Michael for the win! @Clebert im not opposed to coding up a test case it is just that i am swamped at the moment so im glad to hear that Michael already has a bead on this i did a *./artemis data exp* out of my 2.1.0 and then an *./artemis data imp* on a clean 2.6.2. the new system didnt have Address Settings and Security Settings that had previously been added via the jolokia api. i should look into the export options more to see if i just failed to provide a command line argument. fortunately in my scenario it was pretty trivial to recreate those settings quickly. so i did that. the import created all the addresses, routes, queues, and messages that i needed. it went pretty well. so my cluster is now on 2.6.2 i plan to rework our usage of AddressSetting to not create a new entry for every Address we create but instead only create them for the few exceptions to our default configuration then i can start working on running the broker in CF with the cluster in more of a Master-Master mode. thanks again for all your help. feel free to respond if there is any more data you need from me. *side note: my broker is now spamming " AMQ224088: Timeout (10 seconds) while handshaking has occurred. " but thats another issue i will figure out. * my 30 minute startup times are now down to 30 seconds max. more like 10-15 seconds regularly. THANK YOU ALL SO MUCH FOR YOUR HELP. On Wed, Jul 25, 2018 at 2:44 PM michael.andre.pearce wrote: > Clebert im able to recreate this and also worked out in code why this > happens. Ill try catch you on irc tomorrow or friday. > > Sent from my Samsung Galaxy smartphone. > Original message From: Clebert Suconic < > clebert.suco...@gmail.com> Date: 25/07/2018 20:20 (GMT+00:00) To: > users@activemq.apache.org Subject: Re: [artemis 2.1.0] taking 30+ minutes > to boot & failover > Yes.. it should include it. > > > Do you have an example of the API you used with 2.1? > > > I can't write a compatibility test this week.. but if you provide me > the example I will do it early next week. > > > In a perfect world, if you did it .. it would be great :) > > > On Wed, Jul 25, 2018 at 1:40 PM, Dan Langford > wrote: > > i tried 2.6.2 this morning to see if that was an improvement from 2.5.0 i > > tried a few months ago. unfortunately there was not. it may have failed > > much faster than 2.5.0 i dont recall the timing specifics but the error > is > > the same > > > > 11:15:50,853 ERROR [org.apache.activemq.artemis.core.server] AMQ224000: > > Failure in initialisation: java.lang.NegativeArraySizeException > > at > > > org.apache.activemq.artemis.api.core.SimpleString.readSimpleString(SimpleString.java:182) > > [artemis-commons-2.6.2.jar:2.6.2] > > at > > > org.apache.activemq.artemis.api.core.SimpleString.readSimpleString(SimpleString.java:171) > > [artemis-commons-2.6.2.jar:2.6.2] > > at > > > org.apache.activemq.artemis.api.core.SimpleString.readNullableSimpleString(SimpleString.java:158) > > [artemis-commons-2.6.2.jar:2.6.2] > > at > > > org.apache.activemq.artemis.core.buffers.impl.ChannelBufferWrapper.readNullableSimpleString(ChannelBufferWrapper.java:69) > > [artemis-commons-2.6.2.jar:2.6.2] > > at > > > org.apache.activemq.artemis.core.settings.impl.AddressSettings.decode(AddressSettings.java:736) > > [artemis-server-2.6.2.jar:2.6.2] > > at > > > org.apache.activemq.artemis.core.persistence.config.PersistedAddressSetting.decode(PersistedAddressSetting.java:95) > > [artemis-server-2.6.2.jar:2.6.2] > > at > > > org.apache.activemq.artemis.core.persistence.impl.journal.AbstractJournalStorageManager.newAddressEncoding(AbstractJournalStorageManager.java:1925) > > [artemis-server-2.6.2.jar:2.6.2] > > at > > > org.apache.activemq.artemis.core.persistence.impl.journal.AbstractJournalStorageManager.loadBindingJournal(AbstractJournalStorageManager.java:1466) > > [artemis-server-2.6.2.jar:2.6.2] > > at > > > org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.loadJournals(ActiveMQServerImpl.java:2601) > > [artemis-server-2.6.2.jar:2.6.2] > > at > > > org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.initialisePart2(ActiveMQServerImpl.java:2376) > > [artemis-server-2.6.2.jar:2.6.2] > > at > > > org.apache.activemq.artemis.core.server.impl.SharedNothingLiveActivation.run(SharedNothingLiveActivation.java:116) > > [artemis-server-2.6.2.jar:2.6.2] > > at > > > org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.internalStart(ActiveMQServerImpl.java:535) > > [artemis-server-2.6.2.jar:2.6.2] > > at > > > org.apache.activemq.artemis.core.server.impl.Activ
Re: [artemis 2.1.0] taking 30+ minutes to boot & failover
Clebert im able to recreate this and also worked out in code why this happens. Ill try catch you on irc tomorrow or friday. Sent from my Samsung Galaxy smartphone. Original message From: Clebert Suconic Date: 25/07/2018 20:20 (GMT+00:00) To: users@activemq.apache.org Subject: Re: [artemis 2.1.0] taking 30+ minutes to boot & failover Yes.. it should include it. Do you have an example of the API you used with 2.1? I can't write a compatibility test this week.. but if you provide me the example I will do it early next week. In a perfect world, if you did it .. it would be great :) On Wed, Jul 25, 2018 at 1:40 PM, Dan Langford wrote: > i tried 2.6.2 this morning to see if that was an improvement from 2.5.0 i > tried a few months ago. unfortunately there was not. it may have failed > much faster than 2.5.0 i dont recall the timing specifics but the error is > the same > > 11:15:50,853 ERROR [org.apache.activemq.artemis.core.server] AMQ224000: > Failure in initialisation: java.lang.NegativeArraySizeException > at > org.apache.activemq.artemis.api.core.SimpleString.readSimpleString(SimpleString.java:182) > [artemis-commons-2.6.2.jar:2.6.2] > at > org.apache.activemq.artemis.api.core.SimpleString.readSimpleString(SimpleString.java:171) > [artemis-commons-2.6.2.jar:2.6.2] > at > org.apache.activemq.artemis.api.core.SimpleString.readNullableSimpleString(SimpleString.java:158) > [artemis-commons-2.6.2.jar:2.6.2] > at > org.apache.activemq.artemis.core.buffers.impl.ChannelBufferWrapper.readNullableSimpleString(ChannelBufferWrapper.java:69) > [artemis-commons-2.6.2.jar:2.6.2] > at > org.apache.activemq.artemis.core.settings.impl.AddressSettings.decode(AddressSettings.java:736) > [artemis-server-2.6.2.jar:2.6.2] > at > org.apache.activemq.artemis.core.persistence.config.PersistedAddressSetting.decode(PersistedAddressSetting.java:95) > [artemis-server-2.6.2.jar:2.6.2] > at > org.apache.activemq.artemis.core.persistence.impl.journal.AbstractJournalStorageManager.newAddressEncoding(AbstractJournalStorageManager.java:1925) > [artemis-server-2.6.2.jar:2.6.2] > at > org.apache.activemq.artemis.core.persistence.impl.journal.AbstractJournalStorageManager.loadBindingJournal(AbstractJournalStorageManager.java:1466) > [artemis-server-2.6.2.jar:2.6.2] > at > org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.loadJournals(ActiveMQServerImpl.java:2601) > [artemis-server-2.6.2.jar:2.6.2] > at > org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.initialisePart2(ActiveMQServerImpl.java:2376) > [artemis-server-2.6.2.jar:2.6.2] > at > org.apache.activemq.artemis.core.server.impl.SharedNothingLiveActivation.run(SharedNothingLiveActivation.java:116) > [artemis-server-2.6.2.jar:2.6.2] > at > org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.internalStart(ActiveMQServerImpl.java:535) > [artemis-server-2.6.2.jar:2.6.2] > at > org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.start(ActiveMQServerImpl.java:474) > [artemis-server-2.6.2.jar:2.6.2] > at > org.apache.activemq.artemis.integration.FileBroker.start(FileBroker.java:111) > [artemis-cli-2.6.2.jar:2.6.2] > at org.apache.activemq.artemis.cli.commands.Run.execute(Run.java:82) > [artemis-cli-2.6.2.jar:2.6.2] > at > org.apache.activemq.artemis.cli.Artemis.internalExecute(Artemis.java:149) > [artemis-cli-2.6.2.jar:2.6.2] > at org.apache.activemq.artemis.cli.Artemis.execute(Artemis.java:97) > [artemis-cli-2.6.2.jar:2.6.2] > at org.apache.activemq.artemis.cli.Artemis.execute(Artemis.java:124) > [artemis-cli-2.6.2.jar:2.6.2] > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > [rt.jar:1.8.0_171] > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > [rt.jar:1.8.0_171] > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > [rt.jar:1.8.0_171] > at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_171] > at org.apache.activemq.artemis.boot.Artemis.execute(Artemis.java:129) > [artemis-boot.jar:2.6.2] > at org.apache.activemq.artemis.boot.Artemis.main(Artemis.java:49) > [artemis-boot.jar:2.6.2] > > > right there in the AddressSettings.decode > > > I am going to start looking into a journal export and then a journal import > and see what happens there. > are programmatically created addresses / queues stored in the data export > along with the address and security settings? > > > > > On Wed, Jul 25, 2018 at 10:49 AM Clebert Suconic > wrote: > >> @dan take a look at the compatibility tests in master.. We're help to >> help if you don't understand anything.. it's using ClassLoaders and >> Groovy so you can produce stuff with a combination o
Re: [artemis 2.1.0] taking 30+ minutes to boot & failover
Yes.. it should include it. Do you have an example of the API you used with 2.1? I can't write a compatibility test this week.. but if you provide me the example I will do it early next week. In a perfect world, if you did it .. it would be great :) On Wed, Jul 25, 2018 at 1:40 PM, Dan Langford wrote: > i tried 2.6.2 this morning to see if that was an improvement from 2.5.0 i > tried a few months ago. unfortunately there was not. it may have failed > much faster than 2.5.0 i dont recall the timing specifics but the error is > the same > > 11:15:50,853 ERROR [org.apache.activemq.artemis.core.server] AMQ224000: > Failure in initialisation: java.lang.NegativeArraySizeException > at > org.apache.activemq.artemis.api.core.SimpleString.readSimpleString(SimpleString.java:182) > [artemis-commons-2.6.2.jar:2.6.2] > at > org.apache.activemq.artemis.api.core.SimpleString.readSimpleString(SimpleString.java:171) > [artemis-commons-2.6.2.jar:2.6.2] > at > org.apache.activemq.artemis.api.core.SimpleString.readNullableSimpleString(SimpleString.java:158) > [artemis-commons-2.6.2.jar:2.6.2] > at > org.apache.activemq.artemis.core.buffers.impl.ChannelBufferWrapper.readNullableSimpleString(ChannelBufferWrapper.java:69) > [artemis-commons-2.6.2.jar:2.6.2] > at > org.apache.activemq.artemis.core.settings.impl.AddressSettings.decode(AddressSettings.java:736) > [artemis-server-2.6.2.jar:2.6.2] > at > org.apache.activemq.artemis.core.persistence.config.PersistedAddressSetting.decode(PersistedAddressSetting.java:95) > [artemis-server-2.6.2.jar:2.6.2] > at > org.apache.activemq.artemis.core.persistence.impl.journal.AbstractJournalStorageManager.newAddressEncoding(AbstractJournalStorageManager.java:1925) > [artemis-server-2.6.2.jar:2.6.2] > at > org.apache.activemq.artemis.core.persistence.impl.journal.AbstractJournalStorageManager.loadBindingJournal(AbstractJournalStorageManager.java:1466) > [artemis-server-2.6.2.jar:2.6.2] > at > org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.loadJournals(ActiveMQServerImpl.java:2601) > [artemis-server-2.6.2.jar:2.6.2] > at > org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.initialisePart2(ActiveMQServerImpl.java:2376) > [artemis-server-2.6.2.jar:2.6.2] > at > org.apache.activemq.artemis.core.server.impl.SharedNothingLiveActivation.run(SharedNothingLiveActivation.java:116) > [artemis-server-2.6.2.jar:2.6.2] > at > org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.internalStart(ActiveMQServerImpl.java:535) > [artemis-server-2.6.2.jar:2.6.2] > at > org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.start(ActiveMQServerImpl.java:474) > [artemis-server-2.6.2.jar:2.6.2] > at > org.apache.activemq.artemis.integration.FileBroker.start(FileBroker.java:111) > [artemis-cli-2.6.2.jar:2.6.2] > at org.apache.activemq.artemis.cli.commands.Run.execute(Run.java:82) > [artemis-cli-2.6.2.jar:2.6.2] > at > org.apache.activemq.artemis.cli.Artemis.internalExecute(Artemis.java:149) > [artemis-cli-2.6.2.jar:2.6.2] > at org.apache.activemq.artemis.cli.Artemis.execute(Artemis.java:97) > [artemis-cli-2.6.2.jar:2.6.2] > at org.apache.activemq.artemis.cli.Artemis.execute(Artemis.java:124) > [artemis-cli-2.6.2.jar:2.6.2] > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > [rt.jar:1.8.0_171] > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > [rt.jar:1.8.0_171] > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > [rt.jar:1.8.0_171] > at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_171] > at org.apache.activemq.artemis.boot.Artemis.execute(Artemis.java:129) > [artemis-boot.jar:2.6.2] > at org.apache.activemq.artemis.boot.Artemis.main(Artemis.java:49) > [artemis-boot.jar:2.6.2] > > > right there in the AddressSettings.decode > > > I am going to start looking into a journal export and then a journal import > and see what happens there. > are programmatically created addresses / queues stored in the data export > along with the address and security settings? > > > > > On Wed, Jul 25, 2018 at 10:49 AM Clebert Suconic > wrote: > >> @dan take a look at the compatibility tests in master.. We're help to >> help if you don't understand anything.. it's using ClassLoaders and >> Groovy so you can produce stuff with a combination of clients and >> servers on 2.1.0 and consume on the current version. >> >> On Wed, Jul 25, 2018 at 12:04 PM, Clebert Suconic >> wrote: >> > @Dan: what about this: >> > >> > you provide us a code path that could generate the incompatibility in >> 2.1.0... >> > >> > an ultimate deal would be if you produced a compatibility test with >> > 2.1.0 you add some code that will generate the journal with 2.1.0, >> > and then consume the messages on master (or 2.6.x)... >> > >> > >> > if you generate the compatibility test showing the issue, we can try >> fixing it. >> > >> > >> > What about this idea? that would help you migrate into 2.6.x
Re: [artemis 2.1.0] taking 30+ minutes to boot & failover
i tried 2.6.2 this morning to see if that was an improvement from 2.5.0 i tried a few months ago. unfortunately there was not. it may have failed much faster than 2.5.0 i dont recall the timing specifics but the error is the same 11:15:50,853 ERROR [org.apache.activemq.artemis.core.server] AMQ224000: Failure in initialisation: java.lang.NegativeArraySizeException at org.apache.activemq.artemis.api.core.SimpleString.readSimpleString(SimpleString.java:182) [artemis-commons-2.6.2.jar:2.6.2] at org.apache.activemq.artemis.api.core.SimpleString.readSimpleString(SimpleString.java:171) [artemis-commons-2.6.2.jar:2.6.2] at org.apache.activemq.artemis.api.core.SimpleString.readNullableSimpleString(SimpleString.java:158) [artemis-commons-2.6.2.jar:2.6.2] at org.apache.activemq.artemis.core.buffers.impl.ChannelBufferWrapper.readNullableSimpleString(ChannelBufferWrapper.java:69) [artemis-commons-2.6.2.jar:2.6.2] at org.apache.activemq.artemis.core.settings.impl.AddressSettings.decode(AddressSettings.java:736) [artemis-server-2.6.2.jar:2.6.2] at org.apache.activemq.artemis.core.persistence.config.PersistedAddressSetting.decode(PersistedAddressSetting.java:95) [artemis-server-2.6.2.jar:2.6.2] at org.apache.activemq.artemis.core.persistence.impl.journal.AbstractJournalStorageManager.newAddressEncoding(AbstractJournalStorageManager.java:1925) [artemis-server-2.6.2.jar:2.6.2] at org.apache.activemq.artemis.core.persistence.impl.journal.AbstractJournalStorageManager.loadBindingJournal(AbstractJournalStorageManager.java:1466) [artemis-server-2.6.2.jar:2.6.2] at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.loadJournals(ActiveMQServerImpl.java:2601) [artemis-server-2.6.2.jar:2.6.2] at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.initialisePart2(ActiveMQServerImpl.java:2376) [artemis-server-2.6.2.jar:2.6.2] at org.apache.activemq.artemis.core.server.impl.SharedNothingLiveActivation.run(SharedNothingLiveActivation.java:116) [artemis-server-2.6.2.jar:2.6.2] at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.internalStart(ActiveMQServerImpl.java:535) [artemis-server-2.6.2.jar:2.6.2] at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.start(ActiveMQServerImpl.java:474) [artemis-server-2.6.2.jar:2.6.2] at org.apache.activemq.artemis.integration.FileBroker.start(FileBroker.java:111) [artemis-cli-2.6.2.jar:2.6.2] at org.apache.activemq.artemis.cli.commands.Run.execute(Run.java:82) [artemis-cli-2.6.2.jar:2.6.2] at org.apache.activemq.artemis.cli.Artemis.internalExecute(Artemis.java:149) [artemis-cli-2.6.2.jar:2.6.2] at org.apache.activemq.artemis.cli.Artemis.execute(Artemis.java:97) [artemis-cli-2.6.2.jar:2.6.2] at org.apache.activemq.artemis.cli.Artemis.execute(Artemis.java:124) [artemis-cli-2.6.2.jar:2.6.2] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.8.0_171] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) [rt.jar:1.8.0_171] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_171] at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_171] at org.apache.activemq.artemis.boot.Artemis.execute(Artemis.java:129) [artemis-boot.jar:2.6.2] at org.apache.activemq.artemis.boot.Artemis.main(Artemis.java:49) [artemis-boot.jar:2.6.2] right there in the AddressSettings.decode I am going to start looking into a journal export and then a journal import and see what happens there. are programmatically created addresses / queues stored in the data export along with the address and security settings? On Wed, Jul 25, 2018 at 10:49 AM Clebert Suconic wrote: > @dan take a look at the compatibility tests in master.. We're help to > help if you don't understand anything.. it's using ClassLoaders and > Groovy so you can produce stuff with a combination of clients and > servers on 2.1.0 and consume on the current version. > > On Wed, Jul 25, 2018 at 12:04 PM, Clebert Suconic > wrote: > > @Dan: what about this: > > > > you provide us a code path that could generate the incompatibility in > 2.1.0... > > > > an ultimate deal would be if you produced a compatibility test with > > 2.1.0 you add some code that will generate the journal with 2.1.0, > > and then consume the messages on master (or 2.6.x)... > > > > > > if you generate the compatibility test showing the issue, we can try > fixing it. > > > > > > What about this idea? that would help you migrate into 2.6.x or master. > > > > On Tue, Jul 24, 2018 at 7:25 PM, Clebert Suconic > > wrote: > >> On Fri, Jul 20, 2018 at 4:47 PM, Dan Langford > wrote: > >>> Thank you that was very helpful. we actually do have an address > settings > >>> entry for each queue. there could be a better pattern for us. but > currently > >>> our automated system for creating queues creates an address setting at > the > >>> same time. i will look into improved patterns. > >>> > >>> as far as upgrading goes. i agree we really
Re: [artemis 2.1.0] taking 30+ minutes to boot & failover
@Dan: what about this: you provide us a code path that could generate the incompatibility in 2.1.0... an ultimate deal would be if you produced a compatibility test with 2.1.0 you add some code that will generate the journal with 2.1.0, and then consume the messages on master (or 2.6.x)... if you generate the compatibility test showing the issue, we can try fixing it. What about this idea? that would help you migrate into 2.6.x or master. On Tue, Jul 24, 2018 at 7:25 PM, Clebert Suconic wrote: > On Fri, Jul 20, 2018 at 4:47 PM, Dan Langford wrote: >> Thank you that was very helpful. we actually do have an address settings >> entry for each queue. there could be a better pattern for us. but currently >> our automated system for creating queues creates an address setting at the >> same time. i will look into improved patterns. >> >> as far as upgrading goes. i agree we really want to upgrade. until i can >> find a work around for the config-delete-queues deserialization bug >> introduced in 2.2.0 i brought up back in April we will not be able to >> easily move. > > You should be able to move to 2.6.2. if you're not able to I would > like to know where it failed. > > >> >> thanks again for all the help >> >> On Fri, Jul 20, 2018 at 8:56 AM Clebert Suconic >> wrote: >> >>> If you do not want to upgrade for any reason export the journal. Cleani >>> uo. Edit the text and remove the garbage (you will see) manually. Delete >>> all data and te import. >>> >>> (Make a backup to be safe of course) >>> >>> >>> But I still recommend the upgrade. >>> >>> On Fri, Jul 20, 2018 at 10:54 AM Clebert Suconic < >>> clebert.suco...@gmail.com> >>> wrote: >>> >>> > The address setting is the garbage I was talking about. Upgrade to the >>> > latest broker and there will be a cleanup done at the load before it >>> > starts. >>> > >>> > >>> > I highly recommend upgrade. >>> > >>> > On Fri, Jul 20, 2018 at 10:05 AM Justin Bertram >>> > wrote: >>> > >>> >> Analyzing thread dumps like this is pretty simple. I generally just >>> >> scroll >>> >> through and look for long stack-traces with lots of calls from >>> >> org.apache.activemq.artemis. In your case every single thread dump has >>> a >>> >> thread doing something like this: >>> >> >>> >> "main" #1 prio=5 os_prio=0 tid=0x7f902800eb20 nid=0x74fe runnable >>> >> [0x7f9031675000] >>> >>java.lang.Thread.State: RUNNABLE >>> >> at >>> >> >>> >> >>> org.apache.activemq.artemis.core.settings.impl.HierarchicalObjectRepository.getPossibleMatches(HierarchicalObjectRepository.java:373) >>> >> at >>> >> >>> >> >>> org.apache.activemq.artemis.core.settings.impl.HierarchicalObjectRepository.getMatch(HierarchicalObjectRepository.java:192) >>> >> at >>> >> >>> >> >>> org.apache.activemq.artemis.core.paging.impl.PagingManagerImpl.reapplySettings(PagingManagerImpl.java:113) >>> >> at >>> >> >>> >> >>> org.apache.activemq.artemis.core.paging.impl.PagingManagerImpl.onChange(PagingManagerImpl.java:108) >>> >> at >>> >> >>> >> >>> org.apache.activemq.artemis.core.settings.impl.HierarchicalObjectRepository.onChange(HierarchicalObjectRepository.java:348) >>> >> at >>> >> >>> >> >>> org.apache.activemq.artemis.core.settings.impl.HierarchicalObjectRepository.addMatch(HierarchicalObjectRepository.java:168) >>> >> at >>> >> >>> >> >>> org.apache.activemq.artemis.core.settings.impl.HierarchicalObjectRepository.addMatch(HierarchicalObjectRepository.java:147) >>> >> at >>> >> >>> >> >>> org.apache.activemq.artemis.core.settings.impl.HierarchicalObjectRepository.addMatch(HierarchicalObjectRepository.java:120) >>> >> at >>> >> >>> >> >>> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.recoverStoredConfigs(ActiveMQServerImpl.java:2424) >>> >> at >>> >> >>> >> >>> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.loadJournals(ActiveMQServerImpl.java:2374) >>> >> at >>> >> >>> >> >>> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.initialisePart2(ActiveMQServerImpl.java:2219) >>> >> - locked <0x80a8fce8> (a >>> >> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl) >>> >> at >>> >> >>> >> >>> org.apache.activemq.artemis.core.server.impl.SharedNothingLiveActivation.run(SharedNothingLiveActivation.java:109) >>> >> at >>> >> >>> >> >>> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.internalStart(ActiveMQServerImpl.java:518) >>> >> at >>> >> >>> >> >>> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.start(ActiveMQServerImpl.java:466) >>> >> - locked <0x80a8fce8> (a >>> >> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl) >>> >> at >>> >> >>> >> >>> org.apache.activemq.artemis.integration.FileBroker.start(FileBroker.java:111) >>> >> - locked <0x8098be80> (a >>> >> org.apache.activemq.artemis.integration.FileBroker) >>> >> at
Re: [artemis 2.1.0] taking 30+ minutes to boot & failover
@dan take a look at the compatibility tests in master.. We're help to help if you don't understand anything.. it's using ClassLoaders and Groovy so you can produce stuff with a combination of clients and servers on 2.1.0 and consume on the current version. On Wed, Jul 25, 2018 at 12:04 PM, Clebert Suconic wrote: > @Dan: what about this: > > you provide us a code path that could generate the incompatibility in 2.1.0... > > an ultimate deal would be if you produced a compatibility test with > 2.1.0 you add some code that will generate the journal with 2.1.0, > and then consume the messages on master (or 2.6.x)... > > > if you generate the compatibility test showing the issue, we can try fixing > it. > > > What about this idea? that would help you migrate into 2.6.x or master. > > On Tue, Jul 24, 2018 at 7:25 PM, Clebert Suconic > wrote: >> On Fri, Jul 20, 2018 at 4:47 PM, Dan Langford wrote: >>> Thank you that was very helpful. we actually do have an address settings >>> entry for each queue. there could be a better pattern for us. but currently >>> our automated system for creating queues creates an address setting at the >>> same time. i will look into improved patterns. >>> >>> as far as upgrading goes. i agree we really want to upgrade. until i can >>> find a work around for the config-delete-queues deserialization bug >>> introduced in 2.2.0 i brought up back in April we will not be able to >>> easily move. >> >> You should be able to move to 2.6.2. if you're not able to I would >> like to know where it failed. >> >> >>> >>> thanks again for all the help >>> >>> On Fri, Jul 20, 2018 at 8:56 AM Clebert Suconic >>> wrote: >>> If you do not want to upgrade for any reason export the journal. Cleani uo. Edit the text and remove the garbage (you will see) manually. Delete all data and te import. (Make a backup to be safe of course) But I still recommend the upgrade. On Fri, Jul 20, 2018 at 10:54 AM Clebert Suconic < clebert.suco...@gmail.com> wrote: > The address setting is the garbage I was talking about. Upgrade to the > latest broker and there will be a cleanup done at the load before it > starts. > > > I highly recommend upgrade. > > On Fri, Jul 20, 2018 at 10:05 AM Justin Bertram > wrote: > >> Analyzing thread dumps like this is pretty simple. I generally just >> scroll >> through and look for long stack-traces with lots of calls from >> org.apache.activemq.artemis. In your case every single thread dump has a >> thread doing something like this: >> >> "main" #1 prio=5 os_prio=0 tid=0x7f902800eb20 nid=0x74fe runnable >> [0x7f9031675000] >>java.lang.Thread.State: RUNNABLE >> at >> >> org.apache.activemq.artemis.core.settings.impl.HierarchicalObjectRepository.getPossibleMatches(HierarchicalObjectRepository.java:373) >> at >> >> org.apache.activemq.artemis.core.settings.impl.HierarchicalObjectRepository.getMatch(HierarchicalObjectRepository.java:192) >> at >> >> org.apache.activemq.artemis.core.paging.impl.PagingManagerImpl.reapplySettings(PagingManagerImpl.java:113) >> at >> >> org.apache.activemq.artemis.core.paging.impl.PagingManagerImpl.onChange(PagingManagerImpl.java:108) >> at >> >> org.apache.activemq.artemis.core.settings.impl.HierarchicalObjectRepository.onChange(HierarchicalObjectRepository.java:348) >> at >> >> org.apache.activemq.artemis.core.settings.impl.HierarchicalObjectRepository.addMatch(HierarchicalObjectRepository.java:168) >> at >> >> org.apache.activemq.artemis.core.settings.impl.HierarchicalObjectRepository.addMatch(HierarchicalObjectRepository.java:147) >> at >> >> org.apache.activemq.artemis.core.settings.impl.HierarchicalObjectRepository.addMatch(HierarchicalObjectRepository.java:120) >> at >> >> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.recoverStoredConfigs(ActiveMQServerImpl.java:2424) >> at >> >> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.loadJournals(ActiveMQServerImpl.java:2374) >> at >> >> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.initialisePart2(ActiveMQServerImpl.java:2219) >> - locked <0x80a8fce8> (a >> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl) >> at >> >> org.apache.activemq.artemis.core.server.impl.SharedNothingLiveActivation.run(SharedNothingLiveActivation.java:109) >> at >> >> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.internalStart(ActiveMQServerImpl.java:518) >> at >> >>
Re: [artemis 2.1.0] taking 30+ minutes to boot & failover
On Fri, Jul 20, 2018 at 4:47 PM, Dan Langford wrote: > Thank you that was very helpful. we actually do have an address settings > entry for each queue. there could be a better pattern for us. but currently > our automated system for creating queues creates an address setting at the > same time. i will look into improved patterns. > > as far as upgrading goes. i agree we really want to upgrade. until i can > find a work around for the config-delete-queues deserialization bug > introduced in 2.2.0 i brought up back in April we will not be able to > easily move. You should be able to move to 2.6.2. if you're not able to I would like to know where it failed. > > thanks again for all the help > > On Fri, Jul 20, 2018 at 8:56 AM Clebert Suconic > wrote: > >> If you do not want to upgrade for any reason export the journal. Cleani >> uo. Edit the text and remove the garbage (you will see) manually. Delete >> all data and te import. >> >> (Make a backup to be safe of course) >> >> >> But I still recommend the upgrade. >> >> On Fri, Jul 20, 2018 at 10:54 AM Clebert Suconic < >> clebert.suco...@gmail.com> >> wrote: >> >> > The address setting is the garbage I was talking about. Upgrade to the >> > latest broker and there will be a cleanup done at the load before it >> > starts. >> > >> > >> > I highly recommend upgrade. >> > >> > On Fri, Jul 20, 2018 at 10:05 AM Justin Bertram >> > wrote: >> > >> >> Analyzing thread dumps like this is pretty simple. I generally just >> >> scroll >> >> through and look for long stack-traces with lots of calls from >> >> org.apache.activemq.artemis. In your case every single thread dump has >> a >> >> thread doing something like this: >> >> >> >> "main" #1 prio=5 os_prio=0 tid=0x7f902800eb20 nid=0x74fe runnable >> >> [0x7f9031675000] >> >>java.lang.Thread.State: RUNNABLE >> >> at >> >> >> >> >> org.apache.activemq.artemis.core.settings.impl.HierarchicalObjectRepository.getPossibleMatches(HierarchicalObjectRepository.java:373) >> >> at >> >> >> >> >> org.apache.activemq.artemis.core.settings.impl.HierarchicalObjectRepository.getMatch(HierarchicalObjectRepository.java:192) >> >> at >> >> >> >> >> org.apache.activemq.artemis.core.paging.impl.PagingManagerImpl.reapplySettings(PagingManagerImpl.java:113) >> >> at >> >> >> >> >> org.apache.activemq.artemis.core.paging.impl.PagingManagerImpl.onChange(PagingManagerImpl.java:108) >> >> at >> >> >> >> >> org.apache.activemq.artemis.core.settings.impl.HierarchicalObjectRepository.onChange(HierarchicalObjectRepository.java:348) >> >> at >> >> >> >> >> org.apache.activemq.artemis.core.settings.impl.HierarchicalObjectRepository.addMatch(HierarchicalObjectRepository.java:168) >> >> at >> >> >> >> >> org.apache.activemq.artemis.core.settings.impl.HierarchicalObjectRepository.addMatch(HierarchicalObjectRepository.java:147) >> >> at >> >> >> >> >> org.apache.activemq.artemis.core.settings.impl.HierarchicalObjectRepository.addMatch(HierarchicalObjectRepository.java:120) >> >> at >> >> >> >> >> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.recoverStoredConfigs(ActiveMQServerImpl.java:2424) >> >> at >> >> >> >> >> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.loadJournals(ActiveMQServerImpl.java:2374) >> >> at >> >> >> >> >> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.initialisePart2(ActiveMQServerImpl.java:2219) >> >> - locked <0x80a8fce8> (a >> >> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl) >> >> at >> >> >> >> >> org.apache.activemq.artemis.core.server.impl.SharedNothingLiveActivation.run(SharedNothingLiveActivation.java:109) >> >> at >> >> >> >> >> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.internalStart(ActiveMQServerImpl.java:518) >> >> at >> >> >> >> >> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.start(ActiveMQServerImpl.java:466) >> >> - locked <0x80a8fce8> (a >> >> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl) >> >> at >> >> >> >> >> org.apache.activemq.artemis.integration.FileBroker.start(FileBroker.java:111) >> >> - locked <0x8098be80> (a >> >> org.apache.activemq.artemis.integration.FileBroker) >> >> at org.apache.activemq.artemis.cli.commands.Run.execute(Run.java:73) >> >> at >> >> >> org.apache.activemq.artemis.cli.Artemis.internalExecute(Artemis.java:148) >> >> at org.apache.activemq.artemis.cli.Artemis.execute(Artemis.java:95) >> >> at org.apache.activemq.artemis.cli.Artemis.execute(Artemis.java:122) >> >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >> >> at >> >> >> >> >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) >> >> at >> >> >> >> >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >> >> at java.lang.reflect.Method.invoke(Method.java:498) >> >> at >>
Re: [artemis 2.1.0] taking 30+ minutes to boot & failover
Thank you that was very helpful. we actually do have an address settings entry for each queue. there could be a better pattern for us. but currently our automated system for creating queues creates an address setting at the same time. i will look into improved patterns. as far as upgrading goes. i agree we really want to upgrade. until i can find a work around for the config-delete-queues deserialization bug introduced in 2.2.0 i brought up back in April we will not be able to easily move. thanks again for all the help On Fri, Jul 20, 2018 at 8:56 AM Clebert Suconic wrote: > If you do not want to upgrade for any reason export the journal. Cleani > uo. Edit the text and remove the garbage (you will see) manually. Delete > all data and te import. > > (Make a backup to be safe of course) > > > But I still recommend the upgrade. > > On Fri, Jul 20, 2018 at 10:54 AM Clebert Suconic < > clebert.suco...@gmail.com> > wrote: > > > The address setting is the garbage I was talking about. Upgrade to the > > latest broker and there will be a cleanup done at the load before it > > starts. > > > > > > I highly recommend upgrade. > > > > On Fri, Jul 20, 2018 at 10:05 AM Justin Bertram > > wrote: > > > >> Analyzing thread dumps like this is pretty simple. I generally just > >> scroll > >> through and look for long stack-traces with lots of calls from > >> org.apache.activemq.artemis. In your case every single thread dump has > a > >> thread doing something like this: > >> > >> "main" #1 prio=5 os_prio=0 tid=0x7f902800eb20 nid=0x74fe runnable > >> [0x7f9031675000] > >>java.lang.Thread.State: RUNNABLE > >> at > >> > >> > org.apache.activemq.artemis.core.settings.impl.HierarchicalObjectRepository.getPossibleMatches(HierarchicalObjectRepository.java:373) > >> at > >> > >> > org.apache.activemq.artemis.core.settings.impl.HierarchicalObjectRepository.getMatch(HierarchicalObjectRepository.java:192) > >> at > >> > >> > org.apache.activemq.artemis.core.paging.impl.PagingManagerImpl.reapplySettings(PagingManagerImpl.java:113) > >> at > >> > >> > org.apache.activemq.artemis.core.paging.impl.PagingManagerImpl.onChange(PagingManagerImpl.java:108) > >> at > >> > >> > org.apache.activemq.artemis.core.settings.impl.HierarchicalObjectRepository.onChange(HierarchicalObjectRepository.java:348) > >> at > >> > >> > org.apache.activemq.artemis.core.settings.impl.HierarchicalObjectRepository.addMatch(HierarchicalObjectRepository.java:168) > >> at > >> > >> > org.apache.activemq.artemis.core.settings.impl.HierarchicalObjectRepository.addMatch(HierarchicalObjectRepository.java:147) > >> at > >> > >> > org.apache.activemq.artemis.core.settings.impl.HierarchicalObjectRepository.addMatch(HierarchicalObjectRepository.java:120) > >> at > >> > >> > org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.recoverStoredConfigs(ActiveMQServerImpl.java:2424) > >> at > >> > >> > org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.loadJournals(ActiveMQServerImpl.java:2374) > >> at > >> > >> > org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.initialisePart2(ActiveMQServerImpl.java:2219) > >> - locked <0x80a8fce8> (a > >> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl) > >> at > >> > >> > org.apache.activemq.artemis.core.server.impl.SharedNothingLiveActivation.run(SharedNothingLiveActivation.java:109) > >> at > >> > >> > org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.internalStart(ActiveMQServerImpl.java:518) > >> at > >> > >> > org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.start(ActiveMQServerImpl.java:466) > >> - locked <0x80a8fce8> (a > >> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl) > >> at > >> > >> > org.apache.activemq.artemis.integration.FileBroker.start(FileBroker.java:111) > >> - locked <0x8098be80> (a > >> org.apache.activemq.artemis.integration.FileBroker) > >> at org.apache.activemq.artemis.cli.commands.Run.execute(Run.java:73) > >> at > >> > org.apache.activemq.artemis.cli.Artemis.internalExecute(Artemis.java:148) > >> at org.apache.activemq.artemis.cli.Artemis.execute(Artemis.java:95) > >> at org.apache.activemq.artemis.cli.Artemis.execute(Artemis.java:122) > >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > >> at > >> > >> > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > >> at > >> > >> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > >> at java.lang.reflect.Method.invoke(Method.java:498) > >> at > org.apache.activemq.artemis.boot.Artemis.execute(Artemis.java:129) > >> at org.apache.activemq.artemis.boot.Artemis.main(Artemis.java:49) > >> > >> In every case it's the "main" thread which isn't surprising as that is > the > >> thread responsible for starting the broker. Also, you can pretty >
Re: [artemis 2.1.0] taking 30+ minutes to boot & failover
If you do not want to upgrade for any reason export the journal. Cleani uo. Edit the text and remove the garbage (you will see) manually. Delete all data and te import. (Make a backup to be safe of course) But I still recommend the upgrade. On Fri, Jul 20, 2018 at 10:54 AM Clebert Suconic wrote: > The address setting is the garbage I was talking about. Upgrade to the > latest broker and there will be a cleanup done at the load before it > starts. > > > I highly recommend upgrade. > > On Fri, Jul 20, 2018 at 10:05 AM Justin Bertram > wrote: > >> Analyzing thread dumps like this is pretty simple. I generally just >> scroll >> through and look for long stack-traces with lots of calls from >> org.apache.activemq.artemis. In your case every single thread dump has a >> thread doing something like this: >> >> "main" #1 prio=5 os_prio=0 tid=0x7f902800eb20 nid=0x74fe runnable >> [0x7f9031675000] >>java.lang.Thread.State: RUNNABLE >> at >> >> org.apache.activemq.artemis.core.settings.impl.HierarchicalObjectRepository.getPossibleMatches(HierarchicalObjectRepository.java:373) >> at >> >> org.apache.activemq.artemis.core.settings.impl.HierarchicalObjectRepository.getMatch(HierarchicalObjectRepository.java:192) >> at >> >> org.apache.activemq.artemis.core.paging.impl.PagingManagerImpl.reapplySettings(PagingManagerImpl.java:113) >> at >> >> org.apache.activemq.artemis.core.paging.impl.PagingManagerImpl.onChange(PagingManagerImpl.java:108) >> at >> >> org.apache.activemq.artemis.core.settings.impl.HierarchicalObjectRepository.onChange(HierarchicalObjectRepository.java:348) >> at >> >> org.apache.activemq.artemis.core.settings.impl.HierarchicalObjectRepository.addMatch(HierarchicalObjectRepository.java:168) >> at >> >> org.apache.activemq.artemis.core.settings.impl.HierarchicalObjectRepository.addMatch(HierarchicalObjectRepository.java:147) >> at >> >> org.apache.activemq.artemis.core.settings.impl.HierarchicalObjectRepository.addMatch(HierarchicalObjectRepository.java:120) >> at >> >> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.recoverStoredConfigs(ActiveMQServerImpl.java:2424) >> at >> >> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.loadJournals(ActiveMQServerImpl.java:2374) >> at >> >> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.initialisePart2(ActiveMQServerImpl.java:2219) >> - locked <0x80a8fce8> (a >> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl) >> at >> >> org.apache.activemq.artemis.core.server.impl.SharedNothingLiveActivation.run(SharedNothingLiveActivation.java:109) >> at >> >> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.internalStart(ActiveMQServerImpl.java:518) >> at >> >> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.start(ActiveMQServerImpl.java:466) >> - locked <0x80a8fce8> (a >> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl) >> at >> >> org.apache.activemq.artemis.integration.FileBroker.start(FileBroker.java:111) >> - locked <0x8098be80> (a >> org.apache.activemq.artemis.integration.FileBroker) >> at org.apache.activemq.artemis.cli.commands.Run.execute(Run.java:73) >> at >> org.apache.activemq.artemis.cli.Artemis.internalExecute(Artemis.java:148) >> at org.apache.activemq.artemis.cli.Artemis.execute(Artemis.java:95) >> at org.apache.activemq.artemis.cli.Artemis.execute(Artemis.java:122) >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >> at >> >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) >> at >> >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >> at java.lang.reflect.Method.invoke(Method.java:498) >> at org.apache.activemq.artemis.boot.Artemis.execute(Artemis.java:129) >> at org.apache.activemq.artemis.boot.Artemis.main(Artemis.java:49) >> >> In every case it's the "main" thread which isn't surprising as that is the >> thread responsible for starting the broker. Also, you can pretty clearly >> see in the trace that this is the thread starting the broker, it's loading >> the journals, & restoring stored configuration (either address settings or >> security settings). I've seen high broker start times when there are lots >> and lots of addresses and lots of and lots of settings. Do either (or >> both) of these situations apply to you? >> >> >> Justin >> >> On Fri, Jul 20, 2018 at 1:16 AM, Dan Langford >> wrote: >> >> > Dang I can’t easily upgrade past 2.1.0 because of the >> config-delete-queues >> > deserialization bug introduced in 2.2.0. Unless that bug was squashed in >> > 2.6+. I don’t think I made a jira for it (vacation and work load) but we >> > discussed it back in April. I should go confirm that bug on 2.6 and >> make a >> > jira for that >> > >> > Thanks >> > On Thu, Jul 19, 2018 at 5:46 PM Clebert Suconic < >>
Re: [artemis 2.1.0] taking 30+ minutes to boot & failover
The address setting is the garbage I was talking about. Upgrade to the latest broker and there will be a cleanup done at the load before it starts. I highly recommend upgrade. On Fri, Jul 20, 2018 at 10:05 AM Justin Bertram wrote: > Analyzing thread dumps like this is pretty simple. I generally just scroll > through and look for long stack-traces with lots of calls from > org.apache.activemq.artemis. In your case every single thread dump has a > thread doing something like this: > > "main" #1 prio=5 os_prio=0 tid=0x7f902800eb20 nid=0x74fe runnable > [0x7f9031675000] >java.lang.Thread.State: RUNNABLE > at > > org.apache.activemq.artemis.core.settings.impl.HierarchicalObjectRepository.getPossibleMatches(HierarchicalObjectRepository.java:373) > at > > org.apache.activemq.artemis.core.settings.impl.HierarchicalObjectRepository.getMatch(HierarchicalObjectRepository.java:192) > at > > org.apache.activemq.artemis.core.paging.impl.PagingManagerImpl.reapplySettings(PagingManagerImpl.java:113) > at > > org.apache.activemq.artemis.core.paging.impl.PagingManagerImpl.onChange(PagingManagerImpl.java:108) > at > > org.apache.activemq.artemis.core.settings.impl.HierarchicalObjectRepository.onChange(HierarchicalObjectRepository.java:348) > at > > org.apache.activemq.artemis.core.settings.impl.HierarchicalObjectRepository.addMatch(HierarchicalObjectRepository.java:168) > at > > org.apache.activemq.artemis.core.settings.impl.HierarchicalObjectRepository.addMatch(HierarchicalObjectRepository.java:147) > at > > org.apache.activemq.artemis.core.settings.impl.HierarchicalObjectRepository.addMatch(HierarchicalObjectRepository.java:120) > at > > org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.recoverStoredConfigs(ActiveMQServerImpl.java:2424) > at > > org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.loadJournals(ActiveMQServerImpl.java:2374) > at > > org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.initialisePart2(ActiveMQServerImpl.java:2219) > - locked <0x80a8fce8> (a > org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl) > at > > org.apache.activemq.artemis.core.server.impl.SharedNothingLiveActivation.run(SharedNothingLiveActivation.java:109) > at > > org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.internalStart(ActiveMQServerImpl.java:518) > at > > org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.start(ActiveMQServerImpl.java:466) > - locked <0x80a8fce8> (a > org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl) > at > > org.apache.activemq.artemis.integration.FileBroker.start(FileBroker.java:111) > - locked <0x8098be80> (a > org.apache.activemq.artemis.integration.FileBroker) > at org.apache.activemq.artemis.cli.commands.Run.execute(Run.java:73) > at > org.apache.activemq.artemis.cli.Artemis.internalExecute(Artemis.java:148) > at org.apache.activemq.artemis.cli.Artemis.execute(Artemis.java:95) > at org.apache.activemq.artemis.cli.Artemis.execute(Artemis.java:122) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at org.apache.activemq.artemis.boot.Artemis.execute(Artemis.java:129) > at org.apache.activemq.artemis.boot.Artemis.main(Artemis.java:49) > > In every case it's the "main" thread which isn't surprising as that is the > thread responsible for starting the broker. Also, you can pretty clearly > see in the trace that this is the thread starting the broker, it's loading > the journals, & restoring stored configuration (either address settings or > security settings). I've seen high broker start times when there are lots > and lots of addresses and lots of and lots of settings. Do either (or > both) of these situations apply to you? > > > Justin > > On Fri, Jul 20, 2018 at 1:16 AM, Dan Langford > wrote: > > > Dang I can’t easily upgrade past 2.1.0 because of the > config-delete-queues > > deserialization bug introduced in 2.2.0. Unless that bug was squashed in > > 2.6+. I don’t think I made a jira for it (vacation and work load) but we > > discussed it back in April. I should go confirm that bug on 2.6 and make > a > > jira for that > > > > Thanks > > On Thu, Jul 19, 2018 at 5:46 PM Clebert Suconic < > clebert.suco...@gmail.com > > > > > wrote: > > > > > There is an issue I remember where the journal would have some dirt > that > > > was fixed on 2.3/0. > > > > > > I would ipgrade to 2.6.2. > > > > > > On Thu, Jul 19, 2018 at 6:34 PM Dan Langford > > > wrote: > > > > > > > would you be willing to help me translate these thread dumps? > > > > > > > > i attached a Zip file with some thread dumps in them. i will also >
Re: [artemis 2.1.0] taking 30+ minutes to boot & failover
Analyzing thread dumps like this is pretty simple. I generally just scroll through and look for long stack-traces with lots of calls from org.apache.activemq.artemis. In your case every single thread dump has a thread doing something like this: "main" #1 prio=5 os_prio=0 tid=0x7f902800eb20 nid=0x74fe runnable [0x7f9031675000] java.lang.Thread.State: RUNNABLE at org.apache.activemq.artemis.core.settings.impl.HierarchicalObjectRepository.getPossibleMatches(HierarchicalObjectRepository.java:373) at org.apache.activemq.artemis.core.settings.impl.HierarchicalObjectRepository.getMatch(HierarchicalObjectRepository.java:192) at org.apache.activemq.artemis.core.paging.impl.PagingManagerImpl.reapplySettings(PagingManagerImpl.java:113) at org.apache.activemq.artemis.core.paging.impl.PagingManagerImpl.onChange(PagingManagerImpl.java:108) at org.apache.activemq.artemis.core.settings.impl.HierarchicalObjectRepository.onChange(HierarchicalObjectRepository.java:348) at org.apache.activemq.artemis.core.settings.impl.HierarchicalObjectRepository.addMatch(HierarchicalObjectRepository.java:168) at org.apache.activemq.artemis.core.settings.impl.HierarchicalObjectRepository.addMatch(HierarchicalObjectRepository.java:147) at org.apache.activemq.artemis.core.settings.impl.HierarchicalObjectRepository.addMatch(HierarchicalObjectRepository.java:120) at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.recoverStoredConfigs(ActiveMQServerImpl.java:2424) at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.loadJournals(ActiveMQServerImpl.java:2374) at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.initialisePart2(ActiveMQServerImpl.java:2219) - locked <0x80a8fce8> (a org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl) at org.apache.activemq.artemis.core.server.impl.SharedNothingLiveActivation.run(SharedNothingLiveActivation.java:109) at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.internalStart(ActiveMQServerImpl.java:518) at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.start(ActiveMQServerImpl.java:466) - locked <0x80a8fce8> (a org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl) at org.apache.activemq.artemis.integration.FileBroker.start(FileBroker.java:111) - locked <0x8098be80> (a org.apache.activemq.artemis.integration.FileBroker) at org.apache.activemq.artemis.cli.commands.Run.execute(Run.java:73) at org.apache.activemq.artemis.cli.Artemis.internalExecute(Artemis.java:148) at org.apache.activemq.artemis.cli.Artemis.execute(Artemis.java:95) at org.apache.activemq.artemis.cli.Artemis.execute(Artemis.java:122) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.activemq.artemis.boot.Artemis.execute(Artemis.java:129) at org.apache.activemq.artemis.boot.Artemis.main(Artemis.java:49) In every case it's the "main" thread which isn't surprising as that is the thread responsible for starting the broker. Also, you can pretty clearly see in the trace that this is the thread starting the broker, it's loading the journals, & restoring stored configuration (either address settings or security settings). I've seen high broker start times when there are lots and lots of addresses and lots of and lots of settings. Do either (or both) of these situations apply to you? Justin On Fri, Jul 20, 2018 at 1:16 AM, Dan Langford wrote: > Dang I can’t easily upgrade past 2.1.0 because of the config-delete-queues > deserialization bug introduced in 2.2.0. Unless that bug was squashed in > 2.6+. I don’t think I made a jira for it (vacation and work load) but we > discussed it back in April. I should go confirm that bug on 2.6 and make a > jira for that > > Thanks > On Thu, Jul 19, 2018 at 5:46 PM Clebert Suconic > > wrote: > > > There is an issue I remember where the journal would have some dirt that > > was fixed on 2.3/0. > > > > I would ipgrade to 2.6.2. > > > > On Thu, Jul 19, 2018 at 6:34 PM Dan Langford > > wrote: > > > > > would you be willing to help me translate these thread dumps? > > > > > > i attached a Zip file with some thread dumps in them. i will also share > > > the fasthread.io links for each file. (i was struggling getting > > > fastthread to do a combo report with the threads in the correct order) > > > > > > artemis04-20180719-1525 https://goo.gl/d88azU > > > artemis04-20180719-1530 https://goo.gl/G78qn3 > > > artemis04-20180719-1535 https://goo.gl/aMBSBw > > > artemis04-20180719-1540 https://goo.gl/brKxxk > > > artemis04-20180719-1545 https://goo.gl/RaXXCs > > > artemis04-20180719-1550 https://goo.gl/r5dndK > > >
Re: [artemis 2.1.0] taking 30+ minutes to boot & failover
Dang I can’t easily upgrade past 2.1.0 because of the config-delete-queues deserialization bug introduced in 2.2.0. Unless that bug was squashed in 2.6+. I don’t think I made a jira for it (vacation and work load) but we discussed it back in April. I should go confirm that bug on 2.6 and make a jira for that Thanks On Thu, Jul 19, 2018 at 5:46 PM Clebert Suconic wrote: > There is an issue I remember where the journal would have some dirt that > was fixed on 2.3/0. > > I would ipgrade to 2.6.2. > > On Thu, Jul 19, 2018 at 6:34 PM Dan Langford > wrote: > > > would you be willing to help me translate these thread dumps? > > > > i attached a Zip file with some thread dumps in them. i will also share > > the fasthread.io links for each file. (i was struggling getting > > fastthread to do a combo report with the threads in the correct order) > > > > artemis04-20180719-1525 https://goo.gl/d88azU > > artemis04-20180719-1530 https://goo.gl/G78qn3 > > artemis04-20180719-1535 https://goo.gl/aMBSBw > > artemis04-20180719-1540 https://goo.gl/brKxxk > > artemis04-20180719-1545 https://goo.gl/RaXXCs > > artemis04-20180719-1550 https://goo.gl/r5dndK > > artemis04-20180719-1555 https://goo.gl/YJRLxe > > > > at :35, :45, and :55 the young+old gen space gets bigger than at the > other > > sample times. but i dont know what to look for in here to determine what > > the broker is actually during during this time. > > > > thanks > > > > On Fri, Jul 6, 2018 at 10:42 AM Justin Bertram > > wrote: > > > >> The first place I would start is grabbing thread dumps every so often to > >> see what the broker is actually doing during the 30+ minutes. > >> > >> > >> Justin > >> > >> On Fri, Jul 6, 2018 at 11:34 AM, Dan Langford > >> wrote: > >> > >> > so my server startup times and failover times are growing pretty big. > >> but i > >> > dont really know where to start looking. > >> > > >> > here is a snippet of some logs to show you the time stamps: > >> > > >> > 08:11:31,801 INFO [org.apache.activemq.artemis.integration.bootstrap] > >> > AMQ101000: Starting ActiveMQ Artemis Server > >> > > >> > 08:11:31,818 INFO [org.apache.activemq.artemis.core.server] > AMQ221000: > >> > live Message Broker is starting with configuration Broker > Configuration > >> > (clustered=true,journalDirectory=./data/journal,bindingsDirectory=./ > >> > data/bindings,largeMessagesDirectory=./data/large-messages, > >> > pagingDirectory=./data/paging) > >> > > >> > 08:11:34,462 INFO [org.apache.activemq.artemis.core.server] > AMQ221012: > >> > Using AIO Journal > >> > > >> > 08:11:34,493 INFO [org.apache.activemq.artemis.core.server] > AMQ221057: > >> > Global Max Size is being adjusted to 1/2 of the JVM max size (-Xmx). > >> being > >> > defined as 1,073,741,824 > >> > > >> > 08:11:34,555 INFO [org.apache.activemq.artemis.core.server] > AMQ221043: > >> > Protocol module found: [artemis-server]. Adding protocol support for: > >> CORE > >> > > >> > 08:11:34,555 INFO [org.apache.activemq.artemis.core.server] > AMQ221043: > >> > Protocol module found: [artemis-amqp-protocol]. Adding protocol > support > >> > for: AMQP > >> > > >> > 08:11:34,556 INFO [org.apache.activemq.artemis.core.server] > AMQ221043: > >> > Protocol module found: [artemis-hornetq-protocol]. Adding protocol > >> support > >> > for: HORNETQ > >> > > >> > 08:11:34,556 INFO [org.apache.activemq.artemis.core.server] > AMQ221043: > >> > Protocol module found: [artemis-mqtt-protocol]. Adding protocol > support > >> > for: MQTT > >> > > >> > 08:11:34,556 INFO [org.apache.activemq.artemis.core.server] > AMQ221043: > >> > Protocol module found: [artemis-openwire-protocol]. Adding protocol > >> support > >> > for: OPENWIRE > >> > > >> > 08:11:34,556 INFO [org.apache.activemq.artemis.core.server] > AMQ221043: > >> > Protocol module found: [artemis-stomp-protocol]. Adding protocol > support > >> > for: STOMP > >> > > >> > 08:41:38,963 WARN [org.apache.activemq.artemis.core.server] > AMQ222165: > >> No > >> > Dead Letter Address configured for queue DLQ in AddressSettings > >> > > >> > 08:41:38,963 WARN [org.apache.activemq.artemis.core.server] > AMQ222166: > >> No > >> > Expiry Address configured for queue DLQ in AddressSettings > >> > > >> > 08:41:38,983 WARN [org.apache.activemq.artemis.core.server] > AMQ222165: > >> No > >> > Dead Letter Address configured for queue ExpiryQueue in > AddressSettings > >> > > >> > 08:41:38,983 WARN [org.apache.activemq.artemis.core.server] > AMQ222166: > >> No > >> > Expiry Address configured for queue ExpiryQueue in AddressSettings > >> > > >> > 08:41:38,984 WARN [org.apache.activemq.artemis.core.server] > AMQ222165: > >> No > >> > Dead Letter Address configured for queue example in AddressSettings > >> > > >> > 08:41:38,985 WARN [org.apache.activemq.artemis.core.server] > AMQ222166: > >> No > >> > Expiry Address configured for queue example in AddressSettings > >> > > >> > 08:41:38,985 WARN [org.apache.activemq.artemis.core.server] > AMQ222165: > >> No
Re: [artemis 2.1.0] taking 30+ minutes to boot & failover
There is an issue I remember where the journal would have some dirt that was fixed on 2.3/0. I would ipgrade to 2.6.2. On Thu, Jul 19, 2018 at 6:34 PM Dan Langford wrote: > would you be willing to help me translate these thread dumps? > > i attached a Zip file with some thread dumps in them. i will also share > the fasthread.io links for each file. (i was struggling getting > fastthread to do a combo report with the threads in the correct order) > > artemis04-20180719-1525 https://goo.gl/d88azU > artemis04-20180719-1530 https://goo.gl/G78qn3 > artemis04-20180719-1535 https://goo.gl/aMBSBw > artemis04-20180719-1540 https://goo.gl/brKxxk > artemis04-20180719-1545 https://goo.gl/RaXXCs > artemis04-20180719-1550 https://goo.gl/r5dndK > artemis04-20180719-1555 https://goo.gl/YJRLxe > > at :35, :45, and :55 the young+old gen space gets bigger than at the other > sample times. but i dont know what to look for in here to determine what > the broker is actually during during this time. > > thanks > > On Fri, Jul 6, 2018 at 10:42 AM Justin Bertram > wrote: > >> The first place I would start is grabbing thread dumps every so often to >> see what the broker is actually doing during the 30+ minutes. >> >> >> Justin >> >> On Fri, Jul 6, 2018 at 11:34 AM, Dan Langford >> wrote: >> >> > so my server startup times and failover times are growing pretty big. >> but i >> > dont really know where to start looking. >> > >> > here is a snippet of some logs to show you the time stamps: >> > >> > 08:11:31,801 INFO [org.apache.activemq.artemis.integration.bootstrap] >> > AMQ101000: Starting ActiveMQ Artemis Server >> > >> > 08:11:31,818 INFO [org.apache.activemq.artemis.core.server] AMQ221000: >> > live Message Broker is starting with configuration Broker Configuration >> > (clustered=true,journalDirectory=./data/journal,bindingsDirectory=./ >> > data/bindings,largeMessagesDirectory=./data/large-messages, >> > pagingDirectory=./data/paging) >> > >> > 08:11:34,462 INFO [org.apache.activemq.artemis.core.server] AMQ221012: >> > Using AIO Journal >> > >> > 08:11:34,493 INFO [org.apache.activemq.artemis.core.server] AMQ221057: >> > Global Max Size is being adjusted to 1/2 of the JVM max size (-Xmx). >> being >> > defined as 1,073,741,824 >> > >> > 08:11:34,555 INFO [org.apache.activemq.artemis.core.server] AMQ221043: >> > Protocol module found: [artemis-server]. Adding protocol support for: >> CORE >> > >> > 08:11:34,555 INFO [org.apache.activemq.artemis.core.server] AMQ221043: >> > Protocol module found: [artemis-amqp-protocol]. Adding protocol support >> > for: AMQP >> > >> > 08:11:34,556 INFO [org.apache.activemq.artemis.core.server] AMQ221043: >> > Protocol module found: [artemis-hornetq-protocol]. Adding protocol >> support >> > for: HORNETQ >> > >> > 08:11:34,556 INFO [org.apache.activemq.artemis.core.server] AMQ221043: >> > Protocol module found: [artemis-mqtt-protocol]. Adding protocol support >> > for: MQTT >> > >> > 08:11:34,556 INFO [org.apache.activemq.artemis.core.server] AMQ221043: >> > Protocol module found: [artemis-openwire-protocol]. Adding protocol >> support >> > for: OPENWIRE >> > >> > 08:11:34,556 INFO [org.apache.activemq.artemis.core.server] AMQ221043: >> > Protocol module found: [artemis-stomp-protocol]. Adding protocol support >> > for: STOMP >> > >> > 08:41:38,963 WARN [org.apache.activemq.artemis.core.server] AMQ222165: >> No >> > Dead Letter Address configured for queue DLQ in AddressSettings >> > >> > 08:41:38,963 WARN [org.apache.activemq.artemis.core.server] AMQ222166: >> No >> > Expiry Address configured for queue DLQ in AddressSettings >> > >> > 08:41:38,983 WARN [org.apache.activemq.artemis.core.server] AMQ222165: >> No >> > Dead Letter Address configured for queue ExpiryQueue in AddressSettings >> > >> > 08:41:38,983 WARN [org.apache.activemq.artemis.core.server] AMQ222166: >> No >> > Expiry Address configured for queue ExpiryQueue in AddressSettings >> > >> > 08:41:38,984 WARN [org.apache.activemq.artemis.core.server] AMQ222165: >> No >> > Dead Letter Address configured for queue example in AddressSettings >> > >> > 08:41:38,985 WARN [org.apache.activemq.artemis.core.server] AMQ222166: >> No >> > Expiry Address configured for queue example in AddressSettings >> > >> > 08:41:38,985 WARN [org.apache.activemq.artemis.core.server] AMQ222165: >> No >> > Dead Letter Address configured for queue exampleQueue in AddressSettings >> > >> > 08:41:38,986 WARN [org.apache.activemq.artemis.core.server] AMQ222166: >> No >> > Expiry Address configured for queue exampleQueue in AddressSettings >> > >> > >> > and it continues. i have 138 queues. i wonder if i need to be looking at >> > PAGE configuration or some cache sizes. do i need to be looking at the >> > number of messages persisted on these queues? where would you look to >> > determine why the startup times are so long? >> > >> > also, i know i need to upgrade but i cannot upgrade off of 2.1.0 due to >> > some deseralization changes
Re: [artemis 2.1.0] taking 30+ minutes to boot & failover
The first place I would start is grabbing thread dumps every so often to see what the broker is actually doing during the 30+ minutes. Justin On Fri, Jul 6, 2018 at 11:34 AM, Dan Langford wrote: > so my server startup times and failover times are growing pretty big. but i > dont really know where to start looking. > > here is a snippet of some logs to show you the time stamps: > > 08:11:31,801 INFO [org.apache.activemq.artemis.integration.bootstrap] > AMQ101000: Starting ActiveMQ Artemis Server > > 08:11:31,818 INFO [org.apache.activemq.artemis.core.server] AMQ221000: > live Message Broker is starting with configuration Broker Configuration > (clustered=true,journalDirectory=./data/journal,bindingsDirectory=./ > data/bindings,largeMessagesDirectory=./data/large-messages, > pagingDirectory=./data/paging) > > 08:11:34,462 INFO [org.apache.activemq.artemis.core.server] AMQ221012: > Using AIO Journal > > 08:11:34,493 INFO [org.apache.activemq.artemis.core.server] AMQ221057: > Global Max Size is being adjusted to 1/2 of the JVM max size (-Xmx). being > defined as 1,073,741,824 > > 08:11:34,555 INFO [org.apache.activemq.artemis.core.server] AMQ221043: > Protocol module found: [artemis-server]. Adding protocol support for: CORE > > 08:11:34,555 INFO [org.apache.activemq.artemis.core.server] AMQ221043: > Protocol module found: [artemis-amqp-protocol]. Adding protocol support > for: AMQP > > 08:11:34,556 INFO [org.apache.activemq.artemis.core.server] AMQ221043: > Protocol module found: [artemis-hornetq-protocol]. Adding protocol support > for: HORNETQ > > 08:11:34,556 INFO [org.apache.activemq.artemis.core.server] AMQ221043: > Protocol module found: [artemis-mqtt-protocol]. Adding protocol support > for: MQTT > > 08:11:34,556 INFO [org.apache.activemq.artemis.core.server] AMQ221043: > Protocol module found: [artemis-openwire-protocol]. Adding protocol support > for: OPENWIRE > > 08:11:34,556 INFO [org.apache.activemq.artemis.core.server] AMQ221043: > Protocol module found: [artemis-stomp-protocol]. Adding protocol support > for: STOMP > > 08:41:38,963 WARN [org.apache.activemq.artemis.core.server] AMQ222165: No > Dead Letter Address configured for queue DLQ in AddressSettings > > 08:41:38,963 WARN [org.apache.activemq.artemis.core.server] AMQ222166: No > Expiry Address configured for queue DLQ in AddressSettings > > 08:41:38,983 WARN [org.apache.activemq.artemis.core.server] AMQ222165: No > Dead Letter Address configured for queue ExpiryQueue in AddressSettings > > 08:41:38,983 WARN [org.apache.activemq.artemis.core.server] AMQ222166: No > Expiry Address configured for queue ExpiryQueue in AddressSettings > > 08:41:38,984 WARN [org.apache.activemq.artemis.core.server] AMQ222165: No > Dead Letter Address configured for queue example in AddressSettings > > 08:41:38,985 WARN [org.apache.activemq.artemis.core.server] AMQ222166: No > Expiry Address configured for queue example in AddressSettings > > 08:41:38,985 WARN [org.apache.activemq.artemis.core.server] AMQ222165: No > Dead Letter Address configured for queue exampleQueue in AddressSettings > > 08:41:38,986 WARN [org.apache.activemq.artemis.core.server] AMQ222166: No > Expiry Address configured for queue exampleQueue in AddressSettings > > > and it continues. i have 138 queues. i wonder if i need to be looking at > PAGE configuration or some cache sizes. do i need to be looking at the > number of messages persisted on these queues? where would you look to > determine why the startup times are so long? > > also, i know i need to upgrade but i cannot upgrade off of 2.1.0 due to > some deseralization changes introduced in 2.2.0. i think i have another > thread on here (that i need to update) regarding that upgrade issue. > > thanks for any ideas or insight you have for me >
[artemis 2.1.0] taking 30+ minutes to boot & failover
so my server startup times and failover times are growing pretty big. but i dont really know where to start looking. here is a snippet of some logs to show you the time stamps: 08:11:31,801 INFO [org.apache.activemq.artemis.integration.bootstrap] AMQ101000: Starting ActiveMQ Artemis Server 08:11:31,818 INFO [org.apache.activemq.artemis.core.server] AMQ221000: live Message Broker is starting with configuration Broker Configuration (clustered=true,journalDirectory=./data/journal,bindingsDirectory=./data/bindings,largeMessagesDirectory=./data/large-messages,pagingDirectory=./data/paging) 08:11:34,462 INFO [org.apache.activemq.artemis.core.server] AMQ221012: Using AIO Journal 08:11:34,493 INFO [org.apache.activemq.artemis.core.server] AMQ221057: Global Max Size is being adjusted to 1/2 of the JVM max size (-Xmx). being defined as 1,073,741,824 08:11:34,555 INFO [org.apache.activemq.artemis.core.server] AMQ221043: Protocol module found: [artemis-server]. Adding protocol support for: CORE 08:11:34,555 INFO [org.apache.activemq.artemis.core.server] AMQ221043: Protocol module found: [artemis-amqp-protocol]. Adding protocol support for: AMQP 08:11:34,556 INFO [org.apache.activemq.artemis.core.server] AMQ221043: Protocol module found: [artemis-hornetq-protocol]. Adding protocol support for: HORNETQ 08:11:34,556 INFO [org.apache.activemq.artemis.core.server] AMQ221043: Protocol module found: [artemis-mqtt-protocol]. Adding protocol support for: MQTT 08:11:34,556 INFO [org.apache.activemq.artemis.core.server] AMQ221043: Protocol module found: [artemis-openwire-protocol]. Adding protocol support for: OPENWIRE 08:11:34,556 INFO [org.apache.activemq.artemis.core.server] AMQ221043: Protocol module found: [artemis-stomp-protocol]. Adding protocol support for: STOMP 08:41:38,963 WARN [org.apache.activemq.artemis.core.server] AMQ222165: No Dead Letter Address configured for queue DLQ in AddressSettings 08:41:38,963 WARN [org.apache.activemq.artemis.core.server] AMQ222166: No Expiry Address configured for queue DLQ in AddressSettings 08:41:38,983 WARN [org.apache.activemq.artemis.core.server] AMQ222165: No Dead Letter Address configured for queue ExpiryQueue in AddressSettings 08:41:38,983 WARN [org.apache.activemq.artemis.core.server] AMQ222166: No Expiry Address configured for queue ExpiryQueue in AddressSettings 08:41:38,984 WARN [org.apache.activemq.artemis.core.server] AMQ222165: No Dead Letter Address configured for queue example in AddressSettings 08:41:38,985 WARN [org.apache.activemq.artemis.core.server] AMQ222166: No Expiry Address configured for queue example in AddressSettings 08:41:38,985 WARN [org.apache.activemq.artemis.core.server] AMQ222165: No Dead Letter Address configured for queue exampleQueue in AddressSettings 08:41:38,986 WARN [org.apache.activemq.artemis.core.server] AMQ222166: No Expiry Address configured for queue exampleQueue in AddressSettings and it continues. i have 138 queues. i wonder if i need to be looking at PAGE configuration or some cache sizes. do i need to be looking at the number of messages persisted on these queues? where would you look to determine why the startup times are so long? also, i know i need to upgrade but i cannot upgrade off of 2.1.0 due to some deseralization changes introduced in 2.2.0. i think i have another thread on here (that i need to update) regarding that upgrade issue. thanks for any ideas or insight you have for me