@dan take a look at the compatibility tests in master.. We're help to help if you don't understand anything.. it's using ClassLoaders and Groovy so you can produce stuff with a combination of clients and servers on 2.1.0 and consume on the current version.
On Wed, Jul 25, 2018 at 12:04 PM, Clebert Suconic <clebert.suco...@gmail.com> wrote: > @Dan: what about this: > > you provide us a code path that could generate the incompatibility in 2.1.0... > > an ultimate deal would be if you produced a compatibility test with > 2.1.0.... you add some code that will generate the journal with 2.1.0, > and then consume the messages on master (or 2.6.x)... > > > if you generate the compatibility test showing the issue, we can try fixing > it. > > > What about this idea? that would help you migrate into 2.6.x or master. > > On Tue, Jul 24, 2018 at 7:25 PM, Clebert Suconic > <clebert.suco...@gmail.com> wrote: >> On Fri, Jul 20, 2018 at 4:47 PM, Dan Langford <danlangf...@gmail.com> wrote: >>> Thank you that was very helpful. we actually do have an address settings >>> entry for each queue. there could be a better pattern for us. but currently >>> our automated system for creating queues creates an address setting at the >>> same time. i will look into improved patterns. >>> >>> as far as upgrading goes. i agree we really want to upgrade. until i can >>> find a work around for the config-delete-queues deserialization bug >>> introduced in 2.2.0 i brought up back in April we will not be able to >>> easily move. >> >> You should be able to move to 2.6.2. if you're not able to I would >> like to know where it failed. >> >> >>> >>> thanks again for all the help >>> >>> On Fri, Jul 20, 2018 at 8:56 AM Clebert Suconic <clebert.suco...@gmail.com> >>> wrote: >>> >>>> If you do not want to upgrade for any reason export the journal. Cleani >>>> uo. Edit the text and remove the garbage (you will see) manually. Delete >>>> all data and te import. >>>> >>>> (Make a backup to be safe of course) >>>> >>>> >>>> But I still recommend the upgrade. >>>> >>>> On Fri, Jul 20, 2018 at 10:54 AM Clebert Suconic < >>>> clebert.suco...@gmail.com> >>>> wrote: >>>> >>>> > The address setting is the garbage I was talking about. Upgrade to the >>>> > latest broker and there will be a cleanup done at the load before it >>>> > starts. >>>> > >>>> > >>>> > I highly recommend upgrade. >>>> > >>>> > On Fri, Jul 20, 2018 at 10:05 AM Justin Bertram <jbert...@apache.org> >>>> > wrote: >>>> > >>>> >> Analyzing thread dumps like this is pretty simple. I generally just >>>> >> scroll >>>> >> through and look for long stack-traces with lots of calls from >>>> >> org.apache.activemq.artemis. In your case every single thread dump has >>>> a >>>> >> thread doing something like this: >>>> >> >>>> >> "main" #1 prio=5 os_prio=0 tid=0x00007f902800eb20 nid=0x74fe runnable >>>> >> [0x00007f9031675000] >>>> >> java.lang.Thread.State: RUNNABLE >>>> >> at >>>> >> >>>> >> >>>> org.apache.activemq.artemis.core.settings.impl.HierarchicalObjectRepository.getPossibleMatches(HierarchicalObjectRepository.java:373) >>>> >> at >>>> >> >>>> >> >>>> org.apache.activemq.artemis.core.settings.impl.HierarchicalObjectRepository.getMatch(HierarchicalObjectRepository.java:192) >>>> >> at >>>> >> >>>> >> >>>> org.apache.activemq.artemis.core.paging.impl.PagingManagerImpl.reapplySettings(PagingManagerImpl.java:113) >>>> >> at >>>> >> >>>> >> >>>> org.apache.activemq.artemis.core.paging.impl.PagingManagerImpl.onChange(PagingManagerImpl.java:108) >>>> >> at >>>> >> >>>> >> >>>> org.apache.activemq.artemis.core.settings.impl.HierarchicalObjectRepository.onChange(HierarchicalObjectRepository.java:348) >>>> >> at >>>> >> >>>> >> >>>> org.apache.activemq.artemis.core.settings.impl.HierarchicalObjectRepository.addMatch(HierarchicalObjectRepository.java:168) >>>> >> at >>>> >> >>>> >> >>>> org.apache.activemq.artemis.core.settings.impl.HierarchicalObjectRepository.addMatch(HierarchicalObjectRepository.java:147) >>>> >> at >>>> >> >>>> >> >>>> org.apache.activemq.artemis.core.settings.impl.HierarchicalObjectRepository.addMatch(HierarchicalObjectRepository.java:120) >>>> >> at >>>> >> >>>> >> >>>> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.recoverStoredConfigs(ActiveMQServerImpl.java:2424) >>>> >> at >>>> >> >>>> >> >>>> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.loadJournals(ActiveMQServerImpl.java:2374) >>>> >> at >>>> >> >>>> >> >>>> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.initialisePart2(ActiveMQServerImpl.java:2219) >>>> >> - locked <0x0000000080a8fce8> (a >>>> >> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl) >>>> >> at >>>> >> >>>> >> >>>> org.apache.activemq.artemis.core.server.impl.SharedNothingLiveActivation.run(SharedNothingLiveActivation.java:109) >>>> >> at >>>> >> >>>> >> >>>> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.internalStart(ActiveMQServerImpl.java:518) >>>> >> at >>>> >> >>>> >> >>>> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.start(ActiveMQServerImpl.java:466) >>>> >> - locked <0x0000000080a8fce8> (a >>>> >> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl) >>>> >> at >>>> >> >>>> >> >>>> org.apache.activemq.artemis.integration.FileBroker.start(FileBroker.java:111) >>>> >> - locked <0x000000008098be80> (a >>>> >> org.apache.activemq.artemis.integration.FileBroker) >>>> >> at org.apache.activemq.artemis.cli.commands.Run.execute(Run.java:73) >>>> >> at >>>> >> >>>> org.apache.activemq.artemis.cli.Artemis.internalExecute(Artemis.java:148) >>>> >> at org.apache.activemq.artemis.cli.Artemis.execute(Artemis.java:95) >>>> >> at org.apache.activemq.artemis.cli.Artemis.execute(Artemis.java:122) >>>> >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >>>> >> at >>>> >> >>>> >> >>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) >>>> >> at >>>> >> >>>> >> >>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>>> >> at java.lang.reflect.Method.invoke(Method.java:498) >>>> >> at >>>> org.apache.activemq.artemis.boot.Artemis.execute(Artemis.java:129) >>>> >> at org.apache.activemq.artemis.boot.Artemis.main(Artemis.java:49) >>>> >> >>>> >> In every case it's the "main" thread which isn't surprising as that is >>>> the >>>> >> thread responsible for starting the broker. Also, you can pretty >>>> clearly >>>> >> see in the trace that this is the thread starting the broker, it's >>>> loading >>>> >> the journals, & restoring stored configuration (either address settings >>>> or >>>> >> security settings). I've seen high broker start times when there are >>>> lots >>>> >> and lots of addresses and lots of and lots of settings. Do either (or >>>> >> both) of these situations apply to you? >>>> >> >>>> >> >>>> >> Justin >>>> >> >>>> >> On Fri, Jul 20, 2018 at 1:16 AM, Dan Langford <danlangf...@gmail.com> >>>> >> wrote: >>>> >> >>>> >> > Dang I can’t easily upgrade past 2.1.0 because of the >>>> >> config-delete-queues >>>> >> > deserialization bug introduced in 2.2.0. Unless that bug was squashed >>>> in >>>> >> > 2.6+. I don’t think I made a jira for it (vacation and work load) but >>>> we >>>> >> > discussed it back in April. I should go confirm that bug on 2.6 and >>>> >> make a >>>> >> > jira for that >>>> >> > >>>> >> > Thanks >>>> >> > On Thu, Jul 19, 2018 at 5:46 PM Clebert Suconic < >>>> >> clebert.suco...@gmail.com >>>> >> > > >>>> >> > wrote: >>>> >> > >>>> >> > > There is an issue I remember where the journal would have some dirt >>>> >> that >>>> >> > > was fixed on 2.3/0. >>>> >> > > >>>> >> > > I would ipgrade to 2.6.2. >>>> >> > > >>>> >> > > On Thu, Jul 19, 2018 at 6:34 PM Dan Langford <danlangf...@gmail.com >>>> > >>>> >> > > wrote: >>>> >> > > >>>> >> > > > would you be willing to help me translate these thread dumps? >>>> >> > > > >>>> >> > > > i attached a Zip file with some thread dumps in them. i will also >>>> >> share >>>> >> > > > the fasthread.io links for each file. (i was struggling getting >>>> >> > > > fastthread to do a combo report with the threads in the correct >>>> >> order) >>>> >> > > > >>>> >> > > > artemis04-20180719-1525 https://goo.gl/d88azU >>>> >> > > > artemis04-20180719-1530 https://goo.gl/G78qn3 >>>> >> > > > artemis04-20180719-1535 https://goo.gl/aMBSBw >>>> >> > > > artemis04-20180719-1540 https://goo.gl/brKxxk >>>> >> > > > artemis04-20180719-1545 https://goo.gl/RaXXCs >>>> >> > > > artemis04-20180719-1550 https://goo.gl/r5dndK >>>> >> > > > artemis04-20180719-1555 https://goo.gl/YJRLxe >>>> >> > > > >>>> >> > > > at :35, :45, and :55 the young+old gen space gets bigger than at >>>> the >>>> >> > > other >>>> >> > > > sample times. but i dont know what to look for in here to >>>> determine >>>> >> > what >>>> >> > > > the broker is actually during during this time. >>>> >> > > > >>>> >> > > > thanks >>>> >> > > > >>>> >> > > > On Fri, Jul 6, 2018 at 10:42 AM Justin Bertram < >>>> jbert...@apache.org >>>> >> > >>>> >> > > > wrote: >>>> >> > > > >>>> >> > > >> The first place I would start is grabbing thread dumps every so >>>> >> often >>>> >> > to >>>> >> > > >> see what the broker is actually doing during the 30+ minutes. >>>> >> > > >> >>>> >> > > >> >>>> >> > > >> Justin >>>> >> > > >> >>>> >> > > >> On Fri, Jul 6, 2018 at 11:34 AM, Dan Langford < >>>> >> danlangf...@gmail.com> >>>> >> > > >> wrote: >>>> >> > > >> >>>> >> > > >> > so my server startup times and failover times are growing >>>> pretty >>>> >> > big. >>>> >> > > >> but i >>>> >> > > >> > dont really know where to start looking. >>>> >> > > >> > >>>> >> > > >> > here is a snippet of some logs to show you the time stamps: >>>> >> > > >> > >>>> >> > > >> > 08:11:31,801 INFO [org.apache.activemq.artemis. >>>> >> > integration.bootstrap] >>>> >> > > >> > AMQ101000: Starting ActiveMQ Artemis Server >>>> >> > > >> > >>>> >> > > >> > 08:11:31,818 INFO [org.apache.activemq.artemis.core.server] >>>> >> > > AMQ221000: >>>> >> > > >> > live Message Broker is starting with configuration Broker >>>> >> > > Configuration >>>> >> > > >> > (clustered=true,journalDirectory=./data/ >>>> >> > journal,bindingsDirectory=./ >>>> >> > > >> > data/bindings,largeMessagesDirectory=./data/large-messages, >>>> >> > > >> > pagingDirectory=./data/paging) >>>> >> > > >> > >>>> >> > > >> > 08:11:34,462 INFO [org.apache.activemq.artemis.core.server] >>>> >> > > AMQ221012: >>>> >> > > >> > Using AIO Journal >>>> >> > > >> > >>>> >> > > >> > 08:11:34,493 INFO [org.apache.activemq.artemis.core.server] >>>> >> > > AMQ221057: >>>> >> > > >> > Global Max Size is being adjusted to 1/2 of the JVM max size >>>> >> (-Xmx). >>>> >> > > >> being >>>> >> > > >> > defined as 1,073,741,824 >>>> >> > > >> > >>>> >> > > >> > 08:11:34,555 INFO [org.apache.activemq.artemis.core.server] >>>> >> > > AMQ221043: >>>> >> > > >> > Protocol module found: [artemis-server]. Adding protocol >>>> support >>>> >> > for: >>>> >> > > >> CORE >>>> >> > > >> > >>>> >> > > >> > 08:11:34,555 INFO [org.apache.activemq.artemis.core.server] >>>> >> > > AMQ221043: >>>> >> > > >> > Protocol module found: [artemis-amqp-protocol]. Adding protocol >>>> >> > > support >>>> >> > > >> > for: AMQP >>>> >> > > >> > >>>> >> > > >> > 08:11:34,556 INFO [org.apache.activemq.artemis.core.server] >>>> >> > > AMQ221043: >>>> >> > > >> > Protocol module found: [artemis-hornetq-protocol]. Adding >>>> >> protocol >>>> >> > > >> support >>>> >> > > >> > for: HORNETQ >>>> >> > > >> > >>>> >> > > >> > 08:11:34,556 INFO [org.apache.activemq.artemis.core.server] >>>> >> > > AMQ221043: >>>> >> > > >> > Protocol module found: [artemis-mqtt-protocol]. Adding protocol >>>> >> > > support >>>> >> > > >> > for: MQTT >>>> >> > > >> > >>>> >> > > >> > 08:11:34,556 INFO [org.apache.activemq.artemis.core.server] >>>> >> > > AMQ221043: >>>> >> > > >> > Protocol module found: [artemis-openwire-protocol]. Adding >>>> >> protocol >>>> >> > > >> support >>>> >> > > >> > for: OPENWIRE >>>> >> > > >> > >>>> >> > > >> > 08:11:34,556 INFO [org.apache.activemq.artemis.core.server] >>>> >> > > AMQ221043: >>>> >> > > >> > Protocol module found: [artemis-stomp-protocol]. Adding >>>> protocol >>>> >> > > support >>>> >> > > >> > for: STOMP >>>> >> > > >> > >>>> >> > > >> > 08:41:38,963 WARN [org.apache.activemq.artemis.core.server] >>>> >> > > AMQ222165: >>>> >> > > >> No >>>> >> > > >> > Dead Letter Address configured for queue DLQ in AddressSettings >>>> >> > > >> > >>>> >> > > >> > 08:41:38,963 WARN [org.apache.activemq.artemis.core.server] >>>> >> > > AMQ222166: >>>> >> > > >> No >>>> >> > > >> > Expiry Address configured for queue DLQ in AddressSettings >>>> >> > > >> > >>>> >> > > >> > 08:41:38,983 WARN [org.apache.activemq.artemis.core.server] >>>> >> > > AMQ222165: >>>> >> > > >> No >>>> >> > > >> > Dead Letter Address configured for queue ExpiryQueue in >>>> >> > > AddressSettings >>>> >> > > >> > >>>> >> > > >> > 08:41:38,983 WARN [org.apache.activemq.artemis.core.server] >>>> >> > > AMQ222166: >>>> >> > > >> No >>>> >> > > >> > Expiry Address configured for queue ExpiryQueue in >>>> >> AddressSettings >>>> >> > > >> > >>>> >> > > >> > 08:41:38,984 WARN [org.apache.activemq.artemis.core.server] >>>> >> > > AMQ222165: >>>> >> > > >> No >>>> >> > > >> > Dead Letter Address configured for queue example in >>>> >> AddressSettings >>>> >> > > >> > >>>> >> > > >> > 08:41:38,985 WARN [org.apache.activemq.artemis.core.server] >>>> >> > > AMQ222166: >>>> >> > > >> No >>>> >> > > >> > Expiry Address configured for queue example in AddressSettings >>>> >> > > >> > >>>> >> > > >> > 08:41:38,985 WARN [org.apache.activemq.artemis.core.server] >>>> >> > > AMQ222165: >>>> >> > > >> No >>>> >> > > >> > Dead Letter Address configured for queue exampleQueue in >>>> >> > > AddressSettings >>>> >> > > >> > >>>> >> > > >> > 08:41:38,986 WARN [org.apache.activemq.artemis.core.server] >>>> >> > > AMQ222166: >>>> >> > > >> No >>>> >> > > >> > Expiry Address configured for queue exampleQueue in >>>> >> AddressSettings >>>> >> > > >> > >>>> >> > > >> > >>>> >> > > >> > and it continues. i have 138 queues. i wonder if i need to be >>>> >> > looking >>>> >> > > at >>>> >> > > >> > PAGE configuration or some cache sizes. do i need to be looking >>>> >> at >>>> >> > the >>>> >> > > >> > number of messages persisted on these queues? where would you >>>> >> look >>>> >> > to >>>> >> > > >> > determine why the startup times are so long? >>>> >> > > >> > >>>> >> > > >> > also, i know i need to upgrade but i cannot upgrade off of >>>> 2.1.0 >>>> >> due >>>> >> > > to >>>> >> > > >> > some deseralization changes introduced in 2.2.0. i think i have >>>> >> > > another >>>> >> > > >> > thread on here (that i need to update) regarding that upgrade >>>> >> issue. >>>> >> > > >> > >>>> >> > > >> > thanks for any ideas or insight you have for me >>>> >> > > >> > >>>> >> > > >> >>>> >> > > > -- >>>> >> > > Clebert Suconic >>>> >> > > >>>> >> > >>>> >> >>>> > -- >>>> > Clebert Suconic >>>> > >>>> -- >>>> Clebert Suconic >>>> >> >> >> >> -- >> Clebert Suconic > > > > -- > Clebert Suconic -- Clebert Suconic
