This is new, so please submit a bug in JIRA. Please include as much information as you can about your configuration and how your broker is being used; because you're seeing problems while no one else is and this appears to be code everyone should be hitting, it's possible that there's something unique about your setup or your usage that triggers the problem, and whoever investigates the issue will benefit from having as much detail as possible.
Tim On Jul 13, 2016 12:26 AM, "Johan Carlquist" <jo...@su.se> wrote: Hi! We have two brokers esb-test-mq01 and esb-test-mq02 configured in a network of brokers. Each broker has configuration like this: <networkConnector networkTTL="-1" conduitSubscriptions="false" decreaseNetworkConsumerPriority="true" uri="static:(ssl://${activemq.networkPartner}:61616)" userName="${activemq.username}" password="${activemq.password}" /> Both brokers running ActiveMQ 5.13.0, Java 1.8.0_45 and Ubuntu 14.04 (3.13.0-87-generic x86_64). Sometimes (we havn't found any patterns for when it occurs) one of the connection between the hosts closes with the following message: "Network connection between vm://esb-test-mq01.it.su.se#18 and ssl:// esb-test-mq02.it.su.se/77.238.37.208:61616 shutdown due to a local error: java.lang.ArithmeticException: / by zero" After this the broker never calls back. Yesterday we finally caught this division with TRACE turned on, but we can't make anything out of the logs. We now wondering, is this a known bug that we haven't found any information on, just missconfiguration or should we report the bug on issues.apache.org? Greatful for any input. -- Johan And the log lines (sorry for the json encryption): Jul 12 05:41:37 esb-test-mq01.it.su.se {"timeMillis": 1468294897046,"thread":"ActiveMQ BrokerService[esb-test-mq01.it.su.se] Task-3082","level":"DEBUG","loggerName":"org.apache.activemq.network.DemandForwardingBridgeSupport","message":"bridging (esb-test-mq01.it.su.se -> esb-test-mq02.it.su.se), consumer: ActiveMQBytesMessage {commandId = 1218913, responseRequired = true, messageId = ID:esb-test-mq02-43870-1468234158631-7:21062:0:0:1, originalDestination = null, originalTransactionId = null, producerId = esb-test-mq01.it.su.se->esb-test-mq02.it.su.se-45448-1467985313001-17:2:1:1, destination = queue://aliveness-test, transactionId = null, expiration = 0, timestamp = 1468286832000, arrival = 0, brokerInTime = 1468294896175, brokerOutTime = 1468294897046, correlationId = null, replyTo = null, persistent = true, type = null, priority = 4, groupID = null, groupSequence = 0, targetConsumerId = null, compressed = false, userID = null, content = org.apache.activemq.util.ByteSequence@4bd15e41, marshalledProperties = org.apache.activemq.util.ByteSequence@44e78a09, dataStructure = null, redeliveryCounter = 865, size = 1193, properties = {JMS_AMQP_NATIVE=true, JMS_AMQP_MESSAGE_FORMAT=0, JMS_AMQP_FirstAcquirer=false}, readOnlyProperties = false, readOnlyBody = false, droppable = false, jmsXGroupFirstForConsumer = false} ActiveMQBytesMessage{ bytesOut = null, dataOut = null, dataIn = null }, destination: esb-test-mq01.it.su.se->esb-test-mq02.it.su.se-45448-1467985313001-17:1:1:36663, brokerPath: queue://aliveness-test, message: [esb-test-mq02.it.su.se, esb-test-mq01.it.su.se, esb-test-mq02.it.su.se, esb-test-mq01.it.su.se, esb-test-mq02.it.su.se, esb-test-mq01.it.su.se, esb-test-mq02.it.su.se, esb-test-mq01.it.su.se, esb-test-mq02.it.su.se, esb-test-mq01.it.su.se, esb-test-mq02.it.su.se, esb-test-mq01.it.su.se, esb-test-mq02.it.su.se, esb-test-mq01.it.su.se, esb-test-mq02.it.su.se, esb-test-mq01.it.su.se, esb-test-mq02.it.su.se, esb-test-mq01.it.su.se, esb-test-mq02.it.su.se, esb-test-mq01.it.su.se, esb-test-mq02.it.su.se, esb-test-m Jul 12 05:41:37 esb-test-mq01.it.su.se {"timeMillis": 1468294897057,"thread":"ActiveMQ Transport: ssl:// esb-test-mq02.it.su.se/77.238.37.208:61616","level":"TRACE","loggerName":"org.apache.activemq.network.DemandForwardingBridgeSupport","message":"serviceLocalException: disposed false ex","thrown":{"commonElementCount":0,"localizedMessage":"/ by zero","message":"/ by zero","name":"java.lang.ArithmeticException","extendedStackTrace":[{"class":"org.apache.activemq.broker.region.AbstractSubscription","method":"getInFlightUsage","file":"AbstractSubscription.java","line":240,"exact":false,"location":"activemq-broker-5.13.0.jar","version":"5.13.0"},{"class":"org.apache.activemq.broker.region.Queue","method":"isOptimizeStorage","file":"Queue.java","line":2322,"exact":false,"location":"activemq-broker-5.13.0.jar","version":"5.13.0"},{"class":"org.apache.activemq.broker.region.Queue","method":"doMessageSend","file":"Queue.java","line":839,"exact":false,"location":"activemq-broker-5.13.0.jar","version":"5.13.0"},{"class":"org.apache.activemq.broker.region.Queue","method":"send","file":"Queue.java","line":731,"exact":false,"location":"activemq-broker-5.13.0.jar","version":"5.13.0"},{"class":"org.apache.activemq.broker.region.DestinationFilter","method":"send","file":"DestinationFilter.java","line":132,"exact":false,"location":"activemq-broker-5.13.0.jar","version":"5.13.0"},{"class":"org.apache.activemq.broker.region.AbstractRegion","method":"send","file":"AbstractRegion.java","line":503,"exact":false,"location":"activemq-broker-5.13.0.jar","version":"5.13.0"},{"class":"org.apache.activemq.broker.region.RegionBroker","method":"send","file":"RegionBroker.java","line":468,"exact":false,"location":"activemq-broker-5.13.0.jar","version":"5.13.0"},{"class":"org.apache.activemq.broker.jmx.ManagedRegionBroker","method":"send","file":"ManagedRegionBroker.java","line":296,"exact":false,"location":"activemq-broker-5.13.0.jar","version":"5.13.0"},{"class":"org.apache.activemq.broker.BrokerFilter","method":"send","file":"BrokerFilter.java","li Jul 12 05:41:37 esb-test-mq01.it.su.se {"timeMillis": 1468294897060,"thread":"ActiveMQ Transport: ssl:// esb-test-mq02.it.su.se/77.238.37.208:61616","level":"INFO","loggerName":"org.apache.activemq.network.DemandForwardingBridgeSupport","message":"Network connection between vm://esb-test-mq01.it.su.se#18 and ssl:// esb-test-mq02.it.su.se/77.238.37.208:61616 shutdown due to a local error: java.lang.ArithmeticException: / by zero","endOfBatch":false,"loggerFqcn":"org.slf4j.impl.Log4jLoggerAdapter"} Jul 12 05:41:37 esb-test-mq01.it.su.se {"timeMillis": 1468294897060,"thread":"ActiveMQ Transport: ssl:// esb-test-mq02.it.su.se/77.238.37.208:61616","level":"DEBUG","loggerName":"org.apache.activemq.network.DemandForwardingBridgeSupport","message":"The local Exception was: java.lang.ArithmeticException: / by zero","thrown":{"commonElementCount":0,"localizedMessage":"/ by zero","message":"/ by zero","name":"java.lang.ArithmeticException","extendedStackTrace":[{"class":"org.apache.activemq.broker.region.AbstractSubscription","method":"getInFlightUsage","file":"AbstractSubscription.java","line":240,"exact":false,"location":"activemq-broker-5.13.0.jar","version":"5.13.0"},{"class":"org.apache.activemq.broker.region.Queue","method":"isOptimizeStorage","file":"Queue.java","line":2322,"exact":false,"location":"activemq-broker-5.13.0.jar","version":"5.13.0"},{"class":"org.apache.activemq.broker.region.Queue","method":"doMessageSend","file":"Queue.java","line":839,"exact":false,"location":"activemq-broker-5.13.0.jar","version":"5.13.0"},{"class":"org.apache.activemq.broker.region.Queue","method":"send","file":"Queue.java","line":731,"exact":false,"location":"activemq-broker-5.13.0.jar","version":"5.13.0"},{"class":"org.apache.activemq.broker.region.DestinationFilter","method":"send","file":"DestinationFilter.java","line":132,"exact":false,"location":"activemq-broker-5.13.0.jar","version":"5.13.0"},{"class":"org.apache.activemq.broker.region.AbstractRegion","method":"send","file":"AbstractRegion.java","line":503,"exact":false,"location":"activemq-broker-5.13.0.jar","version":"5.13.0"},{"class":"org.apache.activemq.broker.region.RegionBroker","method":"send","file":"RegionBroker.java","line":468,"exact":false,"location":"activemq-broker-5.13.0.jar","version":"5.13.0"},{"class":"org.apache.activemq.broker.jmx.ManagedRegionBroker","method":"send","file":"ManagedRegionBroker.java","line":296,"exact":false,"location":"activemq-broker-5.13.0.jar","version":"5.13.0"},{"class":"org.apache.activemq.broker.BrokerFilter","method":"send","file Jul 12 05:41:37 esb-test-mq01.it.su.se {"timeMillis": 1468294897062,"thread":"ActiveMQ BrokerService[esb-test-mq01.it.su.se] Task-3079","level":"DEBUG","loggerName":"org.apache.activemq.network.DemandForwardingBridgeSupport","message":" stopping esb-test-mq01.it.su.se bridge to esb-test-mq02.it.su.se ","endOfBatch":false,"loggerFqcn":"org.slf4j.impl.Log4jLoggerAdapter"} Jul 12 05:41:37 esb-test-mq01.it.su.se {"timeMillis": 1468294897062,"thread":"ActiveMQ BrokerService[esb-test-mq01.it.su.se] Task-3079","level":"TRACE","loggerName":"org.apache.activemq.network.DemandForwardingBridgeSupport","message":" esb-test-mq01.it.su.se unregister bridge (org.apache.activemq.network.DemandForwardingBridge@199245a1) to esb-test-mq02.it.su.se ","endOfBatch":false,"loggerFqcn":"org.slf4j.impl.Log4jLoggerAdapter"} Jul 12 05:41:37 esb-test-mq01.it.su.se {"timeMillis": 1468294897062,"thread":"ActiveMQ BrokerService[esb-test-mq01.it.su.se] Task-3084","level":"TRACE","loggerName":"org.apache.activemq.network.DemandForwardingBridgeSupport","message":"serviceLocalException: disposed true ex","thrown":{"commonElementCount":0,"localizedMessage":"peer (vm://esb-test-mq01.it.su.se#19) stopped.","message":"peer (vm:// esb-test-mq01.it.su.se#19) stopped.","name":"org.apache.activemq.transport.TransportDisposedIOException","extendedStackTrace":[{"class":"org.apache.activemq.transport.vm.VMTransport","method":"stop","file":"VMTransport.java","line":230,"exact":true,"location":"activemq-broker-5.13.0.jar","version":"5.13.0"},{"class":"org.apache.activemq.transport.TransportFilter","method":"stop","file":"TransportFilter.java","line":65,"exact":true,"location":"activemq-client-5.13.0.jar","version":"5.13.0"},{"class":"org.apache.activemq.transport.TransportFilter","method":"stop","file":"TransportFilter.java","line":65,"exact":true,"location":"activemq-client-5.13.0.jar","version":"5.13.0"},{"class":"org.apache.activemq.transport.ResponseCorrelator","method":"stop","file":"ResponseCorrelator.java","line":132,"exact":true,"location":"activemq-client-5.13.0.jar","version":"5.13.0"},{"class":"org.apache.activemq.broker.TransportConnection","method":"doStop","file":"TransportConnection.java","line":1165,"exact":true,"location":"activemq-broker-5.13.0.jar","version":"5.13.0"},{"class":"org.apache.activemq.broker.TransportConnection$4","method":"run","file":"TransportConnection.java","line":1131,"exact":true,"location":"activemq-broker-5.13.0.jar","version":"5.13.0"},{"class":"java.util.concurrent.ThreadPoolExecutor","method":"runWorker","file":"ThreadPoolExecutor.java","line":1142,"exact":true,"location":"?","version":"1.8.0_45"},{"class":"java.util.concurrent.ThreadPoolExecutor$Worker","method":"run","file":"ThreadPoolExecutor.java","line":617,"exact":true,"location":"?","version":"1.8.0_45"},{"class":"java.lang.Thread","method":"run","file":"Thread.j Jul 12 05:41:37 esb-test-mq01.it.su.se {"timeMillis": 1468294897062,"thread":"ActiveMQ BrokerService[esb-test-mq01.it.su.se] Task-3079","level":"TRACE","loggerName":"org.apache.activemq.network.DemandForwardingBridgeSupport","message":"serviceLocalException: disposed true ex","thrown":{"commonElementCount":0,"localizedMessage":"Stopped.","message":"Stopped.","name":"java.io.IOException","extendedStackTrace":[{"class":"org.apache.activemq.transport.ResponseCorrelator","method":"stop","file":"ResponseCorrelator.java","line":131,"exact":true,"location":"activemq-client-5.13.0.jar","version":"5.13.0"},{"class":"org.apache.activemq.util.ServiceStopper","method":"stop","file":"ServiceStopper.java","line":41,"exact":true,"location":"activemq-client-5.13.0.jar","version":"5.13.0"},{"class":"org.apache.activemq.network.DemandForwardingBridgeSupport","method":"stop","file":"DemandForwardingBridgeSupport.java","line":305,"exact":true,"location":"activemq-broker-5.13.0.jar","version":"5.13.0"},{"class":"org.apache.activemq.util.ServiceSupport","method":"dispose","file":"ServiceSupport.java","line":43,"exact":true,"location":"activemq-client-5.13.0.jar","version":"5.13.0"},{"class":"org.apache.activemq.network.DemandForwardingBridgeSupport$9","method":"run","file":"DemandForwardingBridgeSupport.java","line":878,"exact":true,"location":"activemq-broker-5.13.0.jar","version":"5.13.0"},{"class":"java.util.concurrent.ThreadPoolExecutor","method":"runWorker","file":"ThreadPoolExecutor.java","line":1142,"exact":true,"location":"?","version":"1.8.0_45"},{"class":"java.util.concurrent.ThreadPoolExecutor$Worker","method":"run","file":"ThreadPoolExecutor.java","line":617,"exact":true,"location":"?","version":"1.8.0_45"},{"class":"java.lang.Thread","method":"run","file":"Thread.java","line":745,"exact":true,"location":"?","version":"1.8.0_45"}]},"endOfBatch":false,"loggerFqcn":"org.slf4j.impl.Log4jLoggerAdapter"} Jul 12 05:41:37 esb-test-mq01.it.su.se {"timeMillis": 1468294897061,"thread":"ActiveMQ Transport: ssl:// esb-test-mq02.it.su.se/77.238.37.208:61616","level":"TRACE","loggerName":"org.apache.activemq.network.DemandForwardingBridgeSupport","message":"fire bridge failed, listener: org.apache.activemq.network.DiscoveryNetworkConnector$1DiscoverNetworkBridgeListener@67a83e67","thrown":{"commonElementCount":0,"localizedMessage":"/ by zero","message":"/ by zero","name":"java.lang.ArithmeticException","extendedStackTrace":[{"class":"org.apache.activemq.broker.region.AbstractSubscription","method":"getInFlightUsage","file":"AbstractSubscription.java","line":240,"exact":false,"location":"activemq-broker-5.13.0.jar","version":"5.13.0"},{"class":"org.apache.activemq.broker.region.Queue","method":"isOptimizeStorage","file":"Queue.java","line":2322,"exact":false,"location":"activemq-broker-5.13.0.jar","version":"5.13.0"},{"class":"org.apache.activemq.broker.region.Queue","method":"doMessageSend","file":"Queue.java","line":839,"exact":false,"location":"activemq-broker-5.13.0.jar","version":"5.13.0"},{"class":"org.apache.activemq.broker.region.Queue","method":"send","file":"Queue.java","line":731,"exact":false,"location":"activemq-broker-5.13.0.jar","version":"5.13.0"},{"class":"org.apache.activemq.broker.region.DestinationFilter","method":"send","file":"DestinationFilter.java","line":132,"exact":false,"location":"activemq-broker-5.13.0.jar","version":"5.13.0"},{"class":"org.apache.activemq.broker.region.AbstractRegion","method":"send","file":"AbstractRegion.java","line":503,"exact":false,"location":"activemq-broker-5.13.0.jar","version":"5.13.0"},{"class":"org.apache.activemq.broker.region.RegionBroker","method":"send","file":"RegionBroker.java","line":468,"exact":false,"location":"activemq-broker-5.13.0.jar","version":"5.13.0"},{"class":"org.apache.activemq.broker.jmx.ManagedRegionBroker","method":"send","file":"ManagedRegionBroker.java","line":296,"exact":false,"location":"activemq-broker-5.13.0.jar","version":"5.13.0"},{"class":"org. Jul 12 05:41:37 esb-test-mq01.it.su.se {"timeMillis": 1468294897063,"thread":"ActiveMQ Task-3","level":"INFO","loggerName":"org.apache.activemq.network.DiscoveryNetworkConnector","message":"Establishing network connection from vm://esb-test-mq01.it.su.se?async=false to ssl:// esb-test-mq02.it.su.se:61616 ","endOfBatch":false,"loggerFqcn":"org.slf4j.impl.Log4jLoggerAdapter"} Jul 12 05:41:37 esb-test-mq01.it.su.se {"timeMillis": 1468294897062,"thread":"ActiveMQ BrokerService[esb-test-mq01.it.su.se] Task-3079","level":"TRACE","loggerName":"org.apache.activemq.network.DemandForwardingBridgeSupport","message":"serviceLocalException: disposed true ex","thrown":{"commonElementCount":0,"localizedMessage":"Stopped.","message":"Stopped.","name":"java.io.IOException","extendedStackTrace":[{"class":"org.apache.activemq.transport.ResponseCorrelator","method":"stop","file":"ResponseCorrelator.java","line":131,"exact":true,"location":"activemq-client-5.13.0.jar","version":"5.13.0"},{"class":"org.apache.activemq.util.ServiceStopper","method":"stop","file":"ServiceStopper.java","line":41,"exact":true,"location":"activemq-client-5.13.0.jar","version":"5.13.0"},{"class":"org.apache.activemq.network.DemandForwardingBridgeSupport","method":"stop","file":"DemandForwardingBridgeSupport.java","line":305,"exact":true,"location":"activemq-broker-5.13.0.jar","version":"5.13.0"},{"class":"org.apache.activemq.util.ServiceSupport","method":"dispose","file":"ServiceSupport.java","line":43,"exact":true,"location":"activemq-client-5.13.0.jar","version":"5.13.0"},{"class":"org.apache.activemq.network.DemandForwardingBridgeSupport$9","method":"run","file":"DemandForwardingBridgeSupport.java","line":878,"exact":true,"location":"activemq-broker-5.13.0.jar","version":"5.13.0"},{"class":"java.util.concurrent.ThreadPoolExecutor","method":"runWorker","file":"ThreadPoolExecutor.java","line":1142,"exact":true,"location":"?","version":"1.8.0_45"},{"class":"java.util.concurrent.ThreadPoolExecutor$Worker","method":"run","file":"ThreadPoolExecutor.java","line":617,"exact":true,"location":"?","version":"1.8.0_45"},{"class":"java.lang.Thread","method":"run","file":"Thread.java","line":745,"exact":true,"location":"?","version":"1.8.0_45"}]},"endOfBatch":false,"loggerFqcn":"org.slf4j.impl.Log4jLoggerAdapter"}