Michael for the win! @Clebert im not opposed to coding up a test case it is just that i am swamped at the moment so im glad to hear that Michael already has a bead on this
i did a *./artemis data exp* out of my 2.1.0 and then an *./artemis data imp* on a clean 2.6.2. the new system didnt have Address Settings and Security Settings that had previously been added via the jolokia api. i should look into the export options more to see if i just failed to provide a command line argument. fortunately in my scenario it was pretty trivial to recreate those settings quickly. so i did that. the import created all the addresses, routes, queues, and messages that i needed. it went pretty well. so my cluster is now on 2.6.2 i plan to rework our usage of AddressSetting to not create a new entry for every Address we create but instead only create them for the few exceptions to our default configuration then i can start working on running the broker in CF with the cluster in more of a Master-Master mode. thanks again for all your help. feel free to respond if there is any more data you need from me. *side note: my broker is now spamming " AMQ224088: Timeout (10 seconds) while handshaking has occurred. " but thats another issue i will figure out. * my 30 minute startup times are now down to 30 seconds max. more like 10-15 seconds regularly. THANK YOU ALL SO MUCH FOR YOUR HELP. On Wed, Jul 25, 2018 at 2:44 PM michael.andre.pearce <michael.andre.pea...@me.com.invalid> wrote: > Clebert im able to recreate this and also worked out in code why this > happens. Ill try catch you on irc tomorrow or friday. > > Sent from my Samsung Galaxy smartphone. > -------- Original message --------From: Clebert Suconic < > clebert.suco...@gmail.com> Date: 25/07/2018 20:20 (GMT+00:00) To: > users@activemq.apache.org Subject: Re: [artemis 2.1.0] taking 30+ minutes > to boot & failover > Yes.. it should include it. > > > Do you have an example of the API you used with 2.1? > > > I can't write a compatibility test this week.. but if you provide me > the example I will do it early next week. > > > In a perfect world, if you did it .. it would be great :) > > > On Wed, Jul 25, 2018 at 1:40 PM, Dan Langford <danlangf...@gmail.com> > wrote: > > i tried 2.6.2 this morning to see if that was an improvement from 2.5.0 i > > tried a few months ago. unfortunately there was not. it may have failed > > much faster than 2.5.0 i dont recall the timing specifics but the error > is > > the same > > > > 11:15:50,853 ERROR [org.apache.activemq.artemis.core.server] AMQ224000: > > Failure in initialisation: java.lang.NegativeArraySizeException > > at > > > org.apache.activemq.artemis.api.core.SimpleString.readSimpleString(SimpleString.java:182) > > [artemis-commons-2.6.2.jar:2.6.2] > > at > > > org.apache.activemq.artemis.api.core.SimpleString.readSimpleString(SimpleString.java:171) > > [artemis-commons-2.6.2.jar:2.6.2] > > at > > > org.apache.activemq.artemis.api.core.SimpleString.readNullableSimpleString(SimpleString.java:158) > > [artemis-commons-2.6.2.jar:2.6.2] > > at > > > org.apache.activemq.artemis.core.buffers.impl.ChannelBufferWrapper.readNullableSimpleString(ChannelBufferWrapper.java:69) > > [artemis-commons-2.6.2.jar:2.6.2] > > at > > > org.apache.activemq.artemis.core.settings.impl.AddressSettings.decode(AddressSettings.java:736) > > [artemis-server-2.6.2.jar:2.6.2] > > at > > > org.apache.activemq.artemis.core.persistence.config.PersistedAddressSetting.decode(PersistedAddressSetting.java:95) > > [artemis-server-2.6.2.jar:2.6.2] > > at > > > org.apache.activemq.artemis.core.persistence.impl.journal.AbstractJournalStorageManager.newAddressEncoding(AbstractJournalStorageManager.java:1925) > > [artemis-server-2.6.2.jar:2.6.2] > > at > > > org.apache.activemq.artemis.core.persistence.impl.journal.AbstractJournalStorageManager.loadBindingJournal(AbstractJournalStorageManager.java:1466) > > [artemis-server-2.6.2.jar:2.6.2] > > at > > > org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.loadJournals(ActiveMQServerImpl.java:2601) > > [artemis-server-2.6.2.jar:2.6.2] > > at > > > org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.initialisePart2(ActiveMQServerImpl.java:2376) > > [artemis-server-2.6.2.jar:2.6.2] > > at > > > org.apache.activemq.artemis.core.server.impl.SharedNothingLiveActivation.run(SharedNothingLiveActivation.java:116) > > [artemis-server-2.6.2.jar:2.6.2] > > at > > > org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.internalStart(ActiveMQServerImpl.java:535) > > [artemis-server-2.6.2.jar:2.6.2] > > at > > > org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.start(ActiveMQServerImpl.java:474) > > [artemis-server-2.6.2.jar:2.6.2] > > at > > > org.apache.activemq.artemis.integration.FileBroker.start(FileBroker.java:111) > > [artemis-cli-2.6.2.jar:2.6.2] > > at org.apache.activemq.artemis.cli.commands.Run.execute(Run.java:82) > > [artemis-cli-2.6.2.jar:2.6.2] > > at > > org.apache.activemq.artemis.cli.Artemis.internalExecute(Artemis.java:149) > > [artemis-cli-2.6.2.jar:2.6.2] > > at org.apache.activemq.artemis.cli.Artemis.execute(Artemis.java:97) > > [artemis-cli-2.6.2.jar:2.6.2] > > at org.apache.activemq.artemis.cli.Artemis.execute(Artemis.java:124) > > [artemis-cli-2.6.2.jar:2.6.2] > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > [rt.jar:1.8.0_171] > > at > > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > > [rt.jar:1.8.0_171] > > at > > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > [rt.jar:1.8.0_171] > > at java.lang.reflect.Method.invoke(Method.java:498) [rt.jar:1.8.0_171] > > at org.apache.activemq.artemis.boot.Artemis.execute(Artemis.java:129) > > [artemis-boot.jar:2.6.2] > > at org.apache.activemq.artemis.boot.Artemis.main(Artemis.java:49) > > [artemis-boot.jar:2.6.2] > > > > > > right there in the AddressSettings.decode > > > > > > I am going to start looking into a journal export and then a journal > import > > and see what happens there. > > are programmatically created addresses / queues stored in the data export > > along with the address and security settings? > > > > > > > > > > On Wed, Jul 25, 2018 at 10:49 AM Clebert Suconic < > clebert.suco...@gmail.com> > > wrote: > > > >> @dan take a look at the compatibility tests in master.. We're help to > >> help if you don't understand anything.. it's using ClassLoaders and > >> Groovy so you can produce stuff with a combination of clients and > >> servers on 2.1.0 and consume on the current version. > >> > >> On Wed, Jul 25, 2018 at 12:04 PM, Clebert Suconic > >> <clebert.suco...@gmail.com> wrote: > >> > @Dan: what about this: > >> > > >> > you provide us a code path that could generate the incompatibility in > >> 2.1.0... > >> > > >> > an ultimate deal would be if you produced a compatibility test with > >> > 2.1.0.... you add some code that will generate the journal with 2.1.0, > >> > and then consume the messages on master (or 2.6.x)... > >> > > >> > > >> > if you generate the compatibility test showing the issue, we can try > >> fixing it. > >> > > >> > > >> > What about this idea? that would help you migrate into 2.6.x or > master. > >> > > >> > On Tue, Jul 24, 2018 at 7:25 PM, Clebert Suconic > >> > <clebert.suco...@gmail.com> wrote: > >> >> On Fri, Jul 20, 2018 at 4:47 PM, Dan Langford <danlangf...@gmail.com > > > >> wrote: > >> >>> Thank you that was very helpful. we actually do have an address > >> settings > >> >>> entry for each queue. there could be a better pattern for us. but > >> currently > >> >>> our automated system for creating queues creates an address setting > at > >> the > >> >>> same time. i will look into improved patterns. > >> >>> > >> >>> as far as upgrading goes. i agree we really want to upgrade. until i > >> can > >> >>> find a work around for the config-delete-queues deserialization bug > >> >>> introduced in 2.2.0 i brought up back in April we will not be able > to > >> >>> easily move. > >> >> > >> >> You should be able to move to 2.6.2. if you're not able to I would > >> >> like to know where it failed. > >> >> > >> >> > >> >>> > >> >>> thanks again for all the help > >> >>> > >> >>> On Fri, Jul 20, 2018 at 8:56 AM Clebert Suconic < > >> clebert.suco...@gmail.com> > >> >>> wrote: > >> >>> > >> >>>> If you do not want to upgrade for any reason export the journal. > >> Cleani > >> >>>> uo. Edit the text and remove the garbage (you will see) manually. > >> Delete > >> >>>> all data and te import. > >> >>>> > >> >>>> (Make a backup to be safe of course) > >> >>>> > >> >>>> > >> >>>> But I still recommend the upgrade. > >> >>>> > >> >>>> On Fri, Jul 20, 2018 at 10:54 AM Clebert Suconic < > >> >>>> clebert.suco...@gmail.com> > >> >>>> wrote: > >> >>>> > >> >>>> > The address setting is the garbage I was talking about. Upgrade > to > >> the > >> >>>> > latest broker and there will be a cleanup done at the load > before it > >> >>>> > starts. > >> >>>> > > >> >>>> > > >> >>>> > I highly recommend upgrade. > >> >>>> > > >> >>>> > On Fri, Jul 20, 2018 at 10:05 AM Justin Bertram < > >> jbert...@apache.org> > >> >>>> > wrote: > >> >>>> > > >> >>>> >> Analyzing thread dumps like this is pretty simple. I generally > >> just > >> >>>> >> scroll > >> >>>> >> through and look for long stack-traces with lots of calls from > >> >>>> >> org.apache.activemq.artemis. In your case every single thread > >> dump has > >> >>>> a > >> >>>> >> thread doing something like this: > >> >>>> >> > >> >>>> >> "main" #1 prio=5 os_prio=0 tid=0x00007f902800eb20 nid=0x74fe > >> runnable > >> >>>> >> [0x00007f9031675000] > >> >>>> >> java.lang.Thread.State: RUNNABLE > >> >>>> >> at > >> >>>> >> > >> >>>> >> > >> >>>> > >> > org.apache.activemq.artemis.core.settings.impl.HierarchicalObjectRepository.getPossibleMatches(HierarchicalObjectRepository.java:373) > >> >>>> >> at > >> >>>> >> > >> >>>> >> > >> >>>> > >> > org.apache.activemq.artemis.core.settings.impl.HierarchicalObjectRepository.getMatch(HierarchicalObjectRepository.java:192) > >> >>>> >> at > >> >>>> >> > >> >>>> >> > >> >>>> > >> > org.apache.activemq.artemis.core.paging.impl.PagingManagerImpl.reapplySettings(PagingManagerImpl.java:113) > >> >>>> >> at > >> >>>> >> > >> >>>> >> > >> >>>> > >> > org.apache.activemq.artemis.core.paging.impl.PagingManagerImpl.onChange(PagingManagerImpl.java:108) > >> >>>> >> at > >> >>>> >> > >> >>>> >> > >> >>>> > >> > org.apache.activemq.artemis.core.settings.impl.HierarchicalObjectRepository.onChange(HierarchicalObjectRepository.java:348) > >> >>>> >> at > >> >>>> >> > >> >>>> >> > >> >>>> > >> > org.apache.activemq.artemis.core.settings.impl.HierarchicalObjectRepository.addMatch(HierarchicalObjectRepository.java:168) > >> >>>> >> at > >> >>>> >> > >> >>>> >> > >> >>>> > >> > org.apache.activemq.artemis.core.settings.impl.HierarchicalObjectRepository.addMatch(HierarchicalObjectRepository.java:147) > >> >>>> >> at > >> >>>> >> > >> >>>> >> > >> >>>> > >> > org.apache.activemq.artemis.core.settings.impl.HierarchicalObjectRepository.addMatch(HierarchicalObjectRepository.java:120) > >> >>>> >> at > >> >>>> >> > >> >>>> >> > >> >>>> > >> > org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.recoverStoredConfigs(ActiveMQServerImpl.java:2424) > >> >>>> >> at > >> >>>> >> > >> >>>> >> > >> >>>> > >> > org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.loadJournals(ActiveMQServerImpl.java:2374) > >> >>>> >> at > >> >>>> >> > >> >>>> >> > >> >>>> > >> > org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.initialisePart2(ActiveMQServerImpl.java:2219) > >> >>>> >> - locked <0x0000000080a8fce8> (a > >> >>>> >> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl) > >> >>>> >> at > >> >>>> >> > >> >>>> >> > >> >>>> > >> > org.apache.activemq.artemis.core.server.impl.SharedNothingLiveActivation.run(SharedNothingLiveActivation.java:109) > >> >>>> >> at > >> >>>> >> > >> >>>> >> > >> >>>> > >> > org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.internalStart(ActiveMQServerImpl.java:518) > >> >>>> >> at > >> >>>> >> > >> >>>> >> > >> >>>> > >> > org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.start(ActiveMQServerImpl.java:466) > >> >>>> >> - locked <0x0000000080a8fce8> (a > >> >>>> >> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl) > >> >>>> >> at > >> >>>> >> > >> >>>> >> > >> >>>> > >> > org.apache.activemq.artemis.integration.FileBroker.start(FileBroker.java:111) > >> >>>> >> - locked <0x000000008098be80> (a > >> >>>> >> org.apache.activemq.artemis.integration.FileBroker) > >> >>>> >> at > >> org.apache.activemq.artemis.cli.commands.Run.execute(Run.java:73) > >> >>>> >> at > >> >>>> >> > >> >>>> > >> > org.apache.activemq.artemis.cli.Artemis.internalExecute(Artemis.java:148) > >> >>>> >> at > >> org.apache.activemq.artemis.cli.Artemis.execute(Artemis.java:95) > >> >>>> >> at > >> org.apache.activemq.artemis.cli.Artemis.execute(Artemis.java:122) > >> >>>> >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native > Method) > >> >>>> >> at > >> >>>> >> > >> >>>> >> > >> >>>> > >> > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > >> >>>> >> at > >> >>>> >> > >> >>>> >> > >> >>>> > >> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > >> >>>> >> at java.lang.reflect.Method.invoke(Method.java:498) > >> >>>> >> at > >> >>>> org.apache.activemq.artemis.boot.Artemis.execute(Artemis.java:129) > >> >>>> >> at > >> org.apache.activemq.artemis.boot.Artemis.main(Artemis.java:49) > >> >>>> >> > >> >>>> >> In every case it's the "main" thread which isn't surprising as > >> that is > >> >>>> the > >> >>>> >> thread responsible for starting the broker. Also, you can > pretty > >> >>>> clearly > >> >>>> >> see in the trace that this is the thread starting the broker, > it's > >> >>>> loading > >> >>>> >> the journals, & restoring stored configuration (either address > >> settings > >> >>>> or > >> >>>> >> security settings). I've seen high broker start times when > there > >> are > >> >>>> lots > >> >>>> >> and lots of addresses and lots of and lots of settings. Do > either > >> (or > >> >>>> >> both) of these situations apply to you? > >> >>>> >> > >> >>>> >> > >> >>>> >> Justin > >> >>>> >> > >> >>>> >> On Fri, Jul 20, 2018 at 1:16 AM, Dan Langford < > >> danlangf...@gmail.com> > >> >>>> >> wrote: > >> >>>> >> > >> >>>> >> > Dang I can’t easily upgrade past 2.1.0 because of the > >> >>>> >> config-delete-queues > >> >>>> >> > deserialization bug introduced in 2.2.0. Unless that bug was > >> squashed > >> >>>> in > >> >>>> >> > 2.6+. I don’t think I made a jira for it (vacation and work > >> load) but > >> >>>> we > >> >>>> >> > discussed it back in April. I should go confirm that bug on > 2.6 > >> and > >> >>>> >> make a > >> >>>> >> > jira for that > >> >>>> >> > > >> >>>> >> > Thanks > >> >>>> >> > On Thu, Jul 19, 2018 at 5:46 PM Clebert Suconic < > >> >>>> >> clebert.suco...@gmail.com > >> >>>> >> > > > >> >>>> >> > wrote: > >> >>>> >> > > >> >>>> >> > > There is an issue I remember where the journal would have > some > >> dirt > >> >>>> >> that > >> >>>> >> > > was fixed on 2.3/0. > >> >>>> >> > > > >> >>>> >> > > I would ipgrade to 2.6.2. > >> >>>> >> > > > >> >>>> >> > > On Thu, Jul 19, 2018 at 6:34 PM Dan Langford < > >> danlangf...@gmail.com > >> >>>> > > >> >>>> >> > > wrote: > >> >>>> >> > > > >> >>>> >> > > > would you be willing to help me translate these thread > dumps? > >> >>>> >> > > > > >> >>>> >> > > > i attached a Zip file with some thread dumps in them. i > will > >> also > >> >>>> >> share > >> >>>> >> > > > the fasthread.io links for each file. (i was struggling > >> getting > >> >>>> >> > > > fastthread to do a combo report with the threads in the > >> correct > >> >>>> >> order) > >> >>>> >> > > > > >> >>>> >> > > > artemis04-20180719-1525 https://goo.gl/d88azU > >> >>>> >> > > > artemis04-20180719-1530 https://goo.gl/G78qn3 > >> >>>> >> > > > artemis04-20180719-1535 https://goo.gl/aMBSBw > >> >>>> >> > > > artemis04-20180719-1540 https://goo.gl/brKxxk > >> >>>> >> > > > artemis04-20180719-1545 https://goo.gl/RaXXCs > >> >>>> >> > > > artemis04-20180719-1550 https://goo.gl/r5dndK > >> >>>> >> > > > artemis04-20180719-1555 https://goo.gl/YJRLxe > >> >>>> >> > > > > >> >>>> >> > > > at :35, :45, and :55 the young+old gen space gets bigger > >> than at > >> >>>> the > >> >>>> >> > > other > >> >>>> >> > > > sample times. but i dont know what to look for in here to > >> >>>> determine > >> >>>> >> > what > >> >>>> >> > > > the broker is actually during during this time. > >> >>>> >> > > > > >> >>>> >> > > > thanks > >> >>>> >> > > > > >> >>>> >> > > > On Fri, Jul 6, 2018 at 10:42 AM Justin Bertram < > >> >>>> jbert...@apache.org > >> >>>> >> > > >> >>>> >> > > > wrote: > >> >>>> >> > > > > >> >>>> >> > > >> The first place I would start is grabbing thread dumps > >> every so > >> >>>> >> often > >> >>>> >> > to > >> >>>> >> > > >> see what the broker is actually doing during the 30+ > >> minutes. > >> >>>> >> > > >> > >> >>>> >> > > >> > >> >>>> >> > > >> Justin > >> >>>> >> > > >> > >> >>>> >> > > >> On Fri, Jul 6, 2018 at 11:34 AM, Dan Langford < > >> >>>> >> danlangf...@gmail.com> > >> >>>> >> > > >> wrote: > >> >>>> >> > > >> > >> >>>> >> > > >> > so my server startup times and failover times are > growing > >> >>>> pretty > >> >>>> >> > big. > >> >>>> >> > > >> but i > >> >>>> >> > > >> > dont really know where to start looking. > >> >>>> >> > > >> > > >> >>>> >> > > >> > here is a snippet of some logs to show you the time > >> stamps: > >> >>>> >> > > >> > > >> >>>> >> > > >> > 08:11:31,801 INFO [org.apache.activemq.artemis. > >> >>>> >> > integration.bootstrap] > >> >>>> >> > > >> > AMQ101000: Starting ActiveMQ Artemis Server > >> >>>> >> > > >> > > >> >>>> >> > > >> > 08:11:31,818 INFO > >> [org.apache.activemq.artemis.core.server] > >> >>>> >> > > AMQ221000: > >> >>>> >> > > >> > live Message Broker is starting with configuration > Broker > >> >>>> >> > > Configuration > >> >>>> >> > > >> > (clustered=true,journalDirectory=./data/ > >> >>>> >> > journal,bindingsDirectory=./ > >> >>>> >> > > >> > > >> data/bindings,largeMessagesDirectory=./data/large-messages, > >> >>>> >> > > >> > pagingDirectory=./data/paging) > >> >>>> >> > > >> > > >> >>>> >> > > >> > 08:11:34,462 INFO > >> [org.apache.activemq.artemis.core.server] > >> >>>> >> > > AMQ221012: > >> >>>> >> > > >> > Using AIO Journal > >> >>>> >> > > >> > > >> >>>> >> > > >> > 08:11:34,493 INFO > >> [org.apache.activemq.artemis.core.server] > >> >>>> >> > > AMQ221057: > >> >>>> >> > > >> > Global Max Size is being adjusted to 1/2 of the JVM max > >> size > >> >>>> >> (-Xmx). > >> >>>> >> > > >> being > >> >>>> >> > > >> > defined as 1,073,741,824 > >> >>>> >> > > >> > > >> >>>> >> > > >> > 08:11:34,555 INFO > >> [org.apache.activemq.artemis.core.server] > >> >>>> >> > > AMQ221043: > >> >>>> >> > > >> > Protocol module found: [artemis-server]. Adding > protocol > >> >>>> support > >> >>>> >> > for: > >> >>>> >> > > >> CORE > >> >>>> >> > > >> > > >> >>>> >> > > >> > 08:11:34,555 INFO > >> [org.apache.activemq.artemis.core.server] > >> >>>> >> > > AMQ221043: > >> >>>> >> > > >> > Protocol module found: [artemis-amqp-protocol]. Adding > >> protocol > >> >>>> >> > > support > >> >>>> >> > > >> > for: AMQP > >> >>>> >> > > >> > > >> >>>> >> > > >> > 08:11:34,556 INFO > >> [org.apache.activemq.artemis.core.server] > >> >>>> >> > > AMQ221043: > >> >>>> >> > > >> > Protocol module found: [artemis-hornetq-protocol]. > Adding > >> >>>> >> protocol > >> >>>> >> > > >> support > >> >>>> >> > > >> > for: HORNETQ > >> >>>> >> > > >> > > >> >>>> >> > > >> > 08:11:34,556 INFO > >> [org.apache.activemq.artemis.core.server] > >> >>>> >> > > AMQ221043: > >> >>>> >> > > >> > Protocol module found: [artemis-mqtt-protocol]. Adding > >> protocol > >> >>>> >> > > support > >> >>>> >> > > >> > for: MQTT > >> >>>> >> > > >> > > >> >>>> >> > > >> > 08:11:34,556 INFO > >> [org.apache.activemq.artemis.core.server] > >> >>>> >> > > AMQ221043: > >> >>>> >> > > >> > Protocol module found: [artemis-openwire-protocol]. > Adding > >> >>>> >> protocol > >> >>>> >> > > >> support > >> >>>> >> > > >> > for: OPENWIRE > >> >>>> >> > > >> > > >> >>>> >> > > >> > 08:11:34,556 INFO > >> [org.apache.activemq.artemis.core.server] > >> >>>> >> > > AMQ221043: > >> >>>> >> > > >> > Protocol module found: [artemis-stomp-protocol]. Adding > >> >>>> protocol > >> >>>> >> > > support > >> >>>> >> > > >> > for: STOMP > >> >>>> >> > > >> > > >> >>>> >> > > >> > 08:41:38,963 WARN > >> [org.apache.activemq.artemis.core.server] > >> >>>> >> > > AMQ222165: > >> >>>> >> > > >> No > >> >>>> >> > > >> > Dead Letter Address configured for queue DLQ in > >> AddressSettings > >> >>>> >> > > >> > > >> >>>> >> > > >> > 08:41:38,963 WARN > >> [org.apache.activemq.artemis.core.server] > >> >>>> >> > > AMQ222166: > >> >>>> >> > > >> No > >> >>>> >> > > >> > Expiry Address configured for queue DLQ in > AddressSettings > >> >>>> >> > > >> > > >> >>>> >> > > >> > 08:41:38,983 WARN > >> [org.apache.activemq.artemis.core.server] > >> >>>> >> > > AMQ222165: > >> >>>> >> > > >> No > >> >>>> >> > > >> > Dead Letter Address configured for queue ExpiryQueue in > >> >>>> >> > > AddressSettings > >> >>>> >> > > >> > > >> >>>> >> > > >> > 08:41:38,983 WARN > >> [org.apache.activemq.artemis.core.server] > >> >>>> >> > > AMQ222166: > >> >>>> >> > > >> No > >> >>>> >> > > >> > Expiry Address configured for queue ExpiryQueue in > >> >>>> >> AddressSettings > >> >>>> >> > > >> > > >> >>>> >> > > >> > 08:41:38,984 WARN > >> [org.apache.activemq.artemis.core.server] > >> >>>> >> > > AMQ222165: > >> >>>> >> > > >> No > >> >>>> >> > > >> > Dead Letter Address configured for queue example in > >> >>>> >> AddressSettings > >> >>>> >> > > >> > > >> >>>> >> > > >> > 08:41:38,985 WARN > >> [org.apache.activemq.artemis.core.server] > >> >>>> >> > > AMQ222166: > >> >>>> >> > > >> No > >> >>>> >> > > >> > Expiry Address configured for queue example in > >> AddressSettings > >> >>>> >> > > >> > > >> >>>> >> > > >> > 08:41:38,985 WARN > >> [org.apache.activemq.artemis.core.server] > >> >>>> >> > > AMQ222165: > >> >>>> >> > > >> No > >> >>>> >> > > >> > Dead Letter Address configured for queue exampleQueue > in > >> >>>> >> > > AddressSettings > >> >>>> >> > > >> > > >> >>>> >> > > >> > 08:41:38,986 WARN > >> [org.apache.activemq.artemis.core.server] > >> >>>> >> > > AMQ222166: > >> >>>> >> > > >> No > >> >>>> >> > > >> > Expiry Address configured for queue exampleQueue in > >> >>>> >> AddressSettings > >> >>>> >> > > >> > > >> >>>> >> > > >> > > >> >>>> >> > > >> > and it continues. i have 138 queues. i wonder if i need > >> to be > >> >>>> >> > looking > >> >>>> >> > > at > >> >>>> >> > > >> > PAGE configuration or some cache sizes. do i need to be > >> looking > >> >>>> >> at > >> >>>> >> > the > >> >>>> >> > > >> > number of messages persisted on these queues? where > would > >> you > >> >>>> >> look > >> >>>> >> > to > >> >>>> >> > > >> > determine why the startup times are so long? > >> >>>> >> > > >> > > >> >>>> >> > > >> > also, i know i need to upgrade but i cannot upgrade > off of > >> >>>> 2.1.0 > >> >>>> >> due > >> >>>> >> > > to > >> >>>> >> > > >> > some deseralization changes introduced in 2.2.0. i > think > >> i have > >> >>>> >> > > another > >> >>>> >> > > >> > thread on here (that i need to update) regarding that > >> upgrade > >> >>>> >> issue. > >> >>>> >> > > >> > > >> >>>> >> > > >> > thanks for any ideas or insight you have for me > >> >>>> >> > > >> > > >> >>>> >> > > >> > >> >>>> >> > > > -- > >> >>>> >> > > Clebert Suconic > >> >>>> >> > > > >> >>>> >> > > >> >>>> >> > >> >>>> > -- > >> >>>> > Clebert Suconic > >> >>>> > > >> >>>> -- > >> >>>> Clebert Suconic > >> >>>> > >> >> > >> >> > >> >> > >> >> -- > >> >> Clebert Suconic > >> > > >> > > >> > > >> > -- > >> > Clebert Suconic > >> > >> > >> > >> -- > >> Clebert Suconic > >> > > > > -- > Clebert Suconic >