Hi!
We have two brokers esb-test-mq01 and esb-test-mq02 configured in a
network of brokers. Each broker has configuration like this:
<networkConnector
networkTTL="-1"
conduitSubscriptions="false"
decreaseNetworkConsumerPriority="true"
uri="static:(ssl://${activemq.networkPartner}:61616)"
userName="${activemq.username}"
password="${activemq.password}"
/>
Both brokers running ActiveMQ 5.13.0, Java 1.8.0_45 and Ubuntu 14.04
(3.13.0-87-generic x86_64).
Sometimes (we havn't found any patterns for when it occurs) one of the
connection between the hosts closes with the following message:
"Network connection between vm://esb-test-mq01.it.su.se#18 and
ssl://esb-test-mq02.it.su.se/77.238.37.208:61616 shutdown due to a local error:
java.lang.ArithmeticException: / by zero"
After this the broker never calls back. Yesterday we finally caught this
division with TRACE turned on, but we can't make anything out of the
logs. We now wondering, is this a known bug that we haven't found any
information on, just missconfiguration or should we report the bug on
issues.apache.org?
Greatful for any input.
--
Johan
And the log lines (sorry for the json encryption):
Jul 12 05:41:37 esb-test-mq01.it.su.se {"timeMillis":
1468294897046,"thread":"ActiveMQ BrokerService[esb-test-mq01.it.su.se]
Task-3082","level":"DEBUG","loggerName":"org.apache.activemq.network.DemandForwardingBridgeSupport","message":"bridging
(esb-test-mq01.it.su.se -> esb-test-mq02.it.su.se), consumer:
ActiveMQBytesMessage {commandId = 1218913, responseRequired = true, messageId =
ID:esb-test-mq02-43870-1468234158631-7:21062:0:0:1, originalDestination = null,
originalTransactionId = null, producerId =
esb-test-mq01.it.su.se->esb-test-mq02.it.su.se-45448-1467985313001-17:2:1:1,
destination = queue://aliveness-test, transactionId = null, expiration = 0,
timestamp = 1468286832000, arrival = 0, brokerInTime = 1468294896175,
brokerOutTime = 1468294897046, correlationId = null, replyTo = null, persistent
= true, type = null, priority = 4, groupID = null, groupSequence = 0,
targetConsumerId = null, compressed = false, userID = null, content =
org.apache.activemq.util.ByteSequence@4bd15e41, marshalledProperties =
org.apache.activemq.util.ByteSequence@44e78a09, dataStructure = null,
redeliveryCounter = 865, size = 1193, properties = {JMS_AMQP_NATIVE=true,
JMS_AMQP_MESSAGE_FORMAT=0, JMS_AMQP_FirstAcquirer=false}, readOnlyProperties =
false, readOnlyBody = false, droppable = false, jmsXGroupFirstForConsumer =
false} ActiveMQBytesMessage{ bytesOut = null, dataOut = null, dataIn = null },
destination:
esb-test-mq01.it.su.se->esb-test-mq02.it.su.se-45448-1467985313001-17:1:1:36663,
brokerPath: queue://aliveness-test, message: [esb-test-mq02.it.su.se,
esb-test-mq01.it.su.se, esb-test-mq02.it.su.se, esb-test-mq01.it.su.se,
esb-test-mq02.it.su.se, esb-test-mq01.it.su.se, esb-test-mq02.it.su.se,
esb-test-mq01.it.su.se, esb-test-mq02.it.su.se, esb-test-mq01.it.su.se,
esb-test-mq02.it.su.se, esb-test-mq01.it.su.se, esb-test-mq02.it.su.se,
esb-test-mq01.it.su.se, esb-test-mq02.it.su.se, esb-test-mq01.it.su.se,
esb-test-mq02.it.su.se, esb-test-mq01.it.su.se, esb-test-mq02.it.su.se,
esb-test-mq01.it.su.se, esb-test-mq02.it.su.se, esb-test-m
Jul 12 05:41:37 esb-test-mq01.it.su.se {"timeMillis":
1468294897057,"thread":"ActiveMQ Transport:
ssl://esb-test-mq02.it.su.se/77.238.37.208:61616","level":"TRACE","loggerName":"org.apache.activemq.network.DemandForwardingBridgeSupport","message":"serviceLocalException:
disposed false ex","thrown":{"commonElementCount":0,"localizedMessage":"/ by
zero","message":"/ by
zero","name":"java.lang.ArithmeticException","extendedStackTrace":[{"class":"org.apache.activemq.broker.region.AbstractSubscription","method":"getInFlightUsage","file":"AbstractSubscription.java","line":240,"exact":false,"location":"activemq-broker-5.13.0.jar","version":"5.13.0"},{"class":"org.apache.activemq.broker.region.Queue","method":"isOptimizeStorage","file":"Queue.java","line":2322,"exact":false,"location":"activemq-broker-5.13.0.jar","version":"5.13.0"},{"class":"org.apache.activemq.broker.region.Queue","method":"doMessageSend","file":"Queue.java","line":839,"exact":false,"location":"activemq-broker-5.13.0.jar","version":"5.13.0"},{"class":"org.apache.activemq.broker.region.Queue","method":"send","file":"Queue.java","line":731,"exact":false,"location":"activemq-broker-5.13.0.jar","version":"5.13.0"},{"class":"org.apache.activemq.broker.region.DestinationFilter","method":"send","file":"DestinationFilter.java","line":132,"exact":false,"location":"activemq-broker-5.13.0.jar","version":"5.13.0"},{"class":"org.apache.activemq.broker.region.AbstractRegion","method":"send","file":"AbstractRegion.java","line":503,"exact":false,"location":"activemq-broker-5.13.0.jar","version":"5.13.0"},{"class":"org.apache.activemq.broker.region.RegionBroker","method":"send","file":"RegionBroker.java","line":468,"exact":false,"location":"activemq-broker-5.13.0.jar","version":"5.13.0"},{"class":"org.apache.activemq.broker.jmx.ManagedRegionBroker","method":"send","file":"ManagedRegionBroker.java","line":296,"exact":false,"location":"activemq-broker-5.13.0.jar","version":"5.13.0"},{"class":"org.apache.activemq.broker.BrokerFilter","method":"send","file":"BrokerFilter.java","li
Jul 12 05:41:37 esb-test-mq01.it.su.se {"timeMillis":
1468294897060,"thread":"ActiveMQ Transport:
ssl://esb-test-mq02.it.su.se/77.238.37.208:61616","level":"INFO","loggerName":"org.apache.activemq.network.DemandForwardingBridgeSupport","message":"Network
connection between vm://esb-test-mq01.it.su.se#18 and
ssl://esb-test-mq02.it.su.se/77.238.37.208:61616 shutdown due to a local error:
java.lang.ArithmeticException: / by
zero","endOfBatch":false,"loggerFqcn":"org.slf4j.impl.Log4jLoggerAdapter"}
Jul 12 05:41:37 esb-test-mq01.it.su.se {"timeMillis":
1468294897060,"thread":"ActiveMQ Transport:
ssl://esb-test-mq02.it.su.se/77.238.37.208:61616","level":"DEBUG","loggerName":"org.apache.activemq.network.DemandForwardingBridgeSupport","message":"The
local Exception was: java.lang.ArithmeticException: / by
zero","thrown":{"commonElementCount":0,"localizedMessage":"/ by
zero","message":"/ by
zero","name":"java.lang.ArithmeticException","extendedStackTrace":[{"class":"org.apache.activemq.broker.region.AbstractSubscription","method":"getInFlightUsage","file":"AbstractSubscription.java","line":240,"exact":false,"location":"activemq-broker-5.13.0.jar","version":"5.13.0"},{"class":"org.apache.activemq.broker.region.Queue","method":"isOptimizeStorage","file":"Queue.java","line":2322,"exact":false,"location":"activemq-broker-5.13.0.jar","version":"5.13.0"},{"class":"org.apache.activemq.broker.region.Queue","method":"doMessageSend","file":"Queue.java","line":839,"exact":false,"location":"activemq-broker-5.13.0.jar","version":"5.13.0"},{"class":"org.apache.activemq.broker.region.Queue","method":"send","file":"Queue.java","line":731,"exact":false,"location":"activemq-broker-5.13.0.jar","version":"5.13.0"},{"class":"org.apache.activemq.broker.region.DestinationFilter","method":"send","file":"DestinationFilter.java","line":132,"exact":false,"location":"activemq-broker-5.13.0.jar","version":"5.13.0"},{"class":"org.apache.activemq.broker.region.AbstractRegion","method":"send","file":"AbstractRegion.java","line":503,"exact":false,"location":"activemq-broker-5.13.0.jar","version":"5.13.0"},{"class":"org.apache.activemq.broker.region.RegionBroker","method":"send","file":"RegionBroker.java","line":468,"exact":false,"location":"activemq-broker-5.13.0.jar","version":"5.13.0"},{"class":"org.apache.activemq.broker.jmx.ManagedRegionBroker","method":"send","file":"ManagedRegionBroker.java","line":296,"exact":false,"location":"activemq-broker-5.13.0.jar","version":"5.13.0"},{"class":"org.apache.activemq.broker.BrokerFilter","method":"send","file
Jul 12 05:41:37 esb-test-mq01.it.su.se {"timeMillis":
1468294897062,"thread":"ActiveMQ BrokerService[esb-test-mq01.it.su.se]
Task-3079","level":"DEBUG","loggerName":"org.apache.activemq.network.DemandForwardingBridgeSupport","message":"
stopping esb-test-mq01.it.su.se bridge to
esb-test-mq02.it.su.se","endOfBatch":false,"loggerFqcn":"org.slf4j.impl.Log4jLoggerAdapter"}
Jul 12 05:41:37 esb-test-mq01.it.su.se {"timeMillis":
1468294897062,"thread":"ActiveMQ BrokerService[esb-test-mq01.it.su.se]
Task-3079","level":"TRACE","loggerName":"org.apache.activemq.network.DemandForwardingBridgeSupport","message":"esb-test-mq01.it.su.se
unregister bridge
(org.apache.activemq.network.DemandForwardingBridge@199245a1) to
esb-test-mq02.it.su.se","endOfBatch":false,"loggerFqcn":"org.slf4j.impl.Log4jLoggerAdapter"}
Jul 12 05:41:37 esb-test-mq01.it.su.se {"timeMillis":
1468294897062,"thread":"ActiveMQ BrokerService[esb-test-mq01.it.su.se]
Task-3084","level":"TRACE","loggerName":"org.apache.activemq.network.DemandForwardingBridgeSupport","message":"serviceLocalException:
disposed true ex","thrown":{"commonElementCount":0,"localizedMessage":"peer
(vm://esb-test-mq01.it.su.se#19) stopped.","message":"peer
(vm://esb-test-mq01.it.su.se#19)
stopped.","name":"org.apache.activemq.transport.TransportDisposedIOException","extendedStackTrace":[{"class":"org.apache.activemq.transport.vm.VMTransport","method":"stop","file":"VMTransport.java","line":230,"exact":true,"location":"activemq-broker-5.13.0.jar","version":"5.13.0"},{"class":"org.apache.activemq.transport.TransportFilter","method":"stop","file":"TransportFilter.java","line":65,"exact":true,"location":"activemq-client-5.13.0.jar","version":"5.13.0"},{"class":"org.apache.activemq.transport.TransportFilter","method":"stop","file":"TransportFilter.java","line":65,"exact":true,"location":"activemq-client-5.13.0.jar","version":"5.13.0"},{"class":"org.apache.activemq.transport.ResponseCorrelator","method":"stop","file":"ResponseCorrelator.java","line":132,"exact":true,"location":"activemq-client-5.13.0.jar","version":"5.13.0"},{"class":"org.apache.activemq.broker.TransportConnection","method":"doStop","file":"TransportConnection.java","line":1165,"exact":true,"location":"activemq-broker-5.13.0.jar","version":"5.13.0"},{"class":"org.apache.activemq.broker.TransportConnection$4","method":"run","file":"TransportConnection.java","line":1131,"exact":true,"location":"activemq-broker-5.13.0.jar","version":"5.13.0"},{"class":"java.util.concurrent.ThreadPoolExecutor","method":"runWorker","file":"ThreadPoolExecutor.java","line":1142,"exact":true,"location":"?","version":"1.8.0_45"},{"class":"java.util.concurrent.ThreadPoolExecutor$Worker","method":"run","file":"ThreadPoolExecutor.java","line":617,"exact":true,"location":"?","version":"1.8.0_45"},{"class":"java.lang.Thread","method":"run","file":"Thread.j
Jul 12 05:41:37 esb-test-mq01.it.su.se {"timeMillis":
1468294897062,"thread":"ActiveMQ BrokerService[esb-test-mq01.it.su.se]
Task-3079","level":"TRACE","loggerName":"org.apache.activemq.network.DemandForwardingBridgeSupport","message":"serviceLocalException:
disposed true
ex","thrown":{"commonElementCount":0,"localizedMessage":"Stopped.","message":"Stopped.","name":"java.io.IOException","extendedStackTrace":[{"class":"org.apache.activemq.transport.ResponseCorrelator","method":"stop","file":"ResponseCorrelator.java","line":131,"exact":true,"location":"activemq-client-5.13.0.jar","version":"5.13.0"},{"class":"org.apache.activemq.util.ServiceStopper","method":"stop","file":"ServiceStopper.java","line":41,"exact":true,"location":"activemq-client-5.13.0.jar","version":"5.13.0"},{"class":"org.apache.activemq.network.DemandForwardingBridgeSupport","method":"stop","file":"DemandForwardingBridgeSupport.java","line":305,"exact":true,"location":"activemq-broker-5.13.0.jar","version":"5.13.0"},{"class":"org.apache.activemq.util.ServiceSupport","method":"dispose","file":"ServiceSupport.java","line":43,"exact":true,"location":"activemq-client-5.13.0.jar","version":"5.13.0"},{"class":"org.apache.activemq.network.DemandForwardingBridgeSupport$9","method":"run","file":"DemandForwardingBridgeSupport.java","line":878,"exact":true,"location":"activemq-broker-5.13.0.jar","version":"5.13.0"},{"class":"java.util.concurrent.ThreadPoolExecutor","method":"runWorker","file":"ThreadPoolExecutor.java","line":1142,"exact":true,"location":"?","version":"1.8.0_45"},{"class":"java.util.concurrent.ThreadPoolExecutor$Worker","method":"run","file":"ThreadPoolExecutor.java","line":617,"exact":true,"location":"?","version":"1.8.0_45"},{"class":"java.lang.Thread","method":"run","file":"Thread.java","line":745,"exact":true,"location":"?","version":"1.8.0_45"}]},"endOfBatch":false,"loggerFqcn":"org.slf4j.impl.Log4jLoggerAdapter"}
Jul 12 05:41:37 esb-test-mq01.it.su.se {"timeMillis":
1468294897061,"thread":"ActiveMQ Transport:
ssl://esb-test-mq02.it.su.se/77.238.37.208:61616","level":"TRACE","loggerName":"org.apache.activemq.network.DemandForwardingBridgeSupport","message":"fire
bridge failed, listener:
org.apache.activemq.network.DiscoveryNetworkConnector$1DiscoverNetworkBridgeListener@67a83e67","thrown":{"commonElementCount":0,"localizedMessage":"/
by zero","message":"/ by
zero","name":"java.lang.ArithmeticException","extendedStackTrace":[{"class":"org.apache.activemq.broker.region.AbstractSubscription","method":"getInFlightUsage","file":"AbstractSubscription.java","line":240,"exact":false,"location":"activemq-broker-5.13.0.jar","version":"5.13.0"},{"class":"org.apache.activemq.broker.region.Queue","method":"isOptimizeStorage","file":"Queue.java","line":2322,"exact":false,"location":"activemq-broker-5.13.0.jar","version":"5.13.0"},{"class":"org.apache.activemq.broker.region.Queue","method":"doMessageSend","file":"Queue.java","line":839,"exact":false,"location":"activemq-broker-5.13.0.jar","version":"5.13.0"},{"class":"org.apache.activemq.broker.region.Queue","method":"send","file":"Queue.java","line":731,"exact":false,"location":"activemq-broker-5.13.0.jar","version":"5.13.0"},{"class":"org.apache.activemq.broker.region.DestinationFilter","method":"send","file":"DestinationFilter.java","line":132,"exact":false,"location":"activemq-broker-5.13.0.jar","version":"5.13.0"},{"class":"org.apache.activemq.broker.region.AbstractRegion","method":"send","file":"AbstractRegion.java","line":503,"exact":false,"location":"activemq-broker-5.13.0.jar","version":"5.13.0"},{"class":"org.apache.activemq.broker.region.RegionBroker","method":"send","file":"RegionBroker.java","line":468,"exact":false,"location":"activemq-broker-5.13.0.jar","version":"5.13.0"},{"class":"org.apache.activemq.broker.jmx.ManagedRegionBroker","method":"send","file":"ManagedRegionBroker.java","line":296,"exact":false,"location":"activemq-broker-5.13.0.jar","version":"5.13.0"},{"class":"org.
Jul 12 05:41:37 esb-test-mq01.it.su.se {"timeMillis":
1468294897063,"thread":"ActiveMQ
Task-3","level":"INFO","loggerName":"org.apache.activemq.network.DiscoveryNetworkConnector","message":"Establishing
network connection from vm://esb-test-mq01.it.su.se?async=false to
ssl://esb-test-mq02.it.su.se:61616","endOfBatch":false,"loggerFqcn":"org.slf4j.impl.Log4jLoggerAdapter"}
Jul 12 05:41:37 esb-test-mq01.it.su.se {"timeMillis":
1468294897062,"thread":"ActiveMQ BrokerService[esb-test-mq01.it.su.se]
Task-3079","level":"TRACE","loggerName":"org.apache.activemq.network.DemandForwardingBridgeSupport","message":"serviceLocalException:
disposed true
ex","thrown":{"commonElementCount":0,"localizedMessage":"Stopped.","message":"Stopped.","name":"java.io.IOException","extendedStackTrace":[{"class":"org.apache.activemq.transport.ResponseCorrelator","method":"stop","file":"ResponseCorrelator.java","line":131,"exact":true,"location":"activemq-client-5.13.0.jar","version":"5.13.0"},{"class":"org.apache.activemq.util.ServiceStopper","method":"stop","file":"ServiceStopper.java","line":41,"exact":true,"location":"activemq-client-5.13.0.jar","version":"5.13.0"},{"class":"org.apache.activemq.network.DemandForwardingBridgeSupport","method":"stop","file":"DemandForwardingBridgeSupport.java","line":305,"exact":true,"location":"activemq-broker-5.13.0.jar","version":"5.13.0"},{"class":"org.apache.activemq.util.ServiceSupport","method":"dispose","file":"ServiceSupport.java","line":43,"exact":true,"location":"activemq-client-5.13.0.jar","version":"5.13.0"},{"class":"org.apache.activemq.network.DemandForwardingBridgeSupport$9","method":"run","file":"DemandForwardingBridgeSupport.java","line":878,"exact":true,"location":"activemq-broker-5.13.0.jar","version":"5.13.0"},{"class":"java.util.concurrent.ThreadPoolExecutor","method":"runWorker","file":"ThreadPoolExecutor.java","line":1142,"exact":true,"location":"?","version":"1.8.0_45"},{"class":"java.util.concurrent.ThreadPoolExecutor$Worker","method":"run","file":"ThreadPoolExecutor.java","line":617,"exact":true,"location":"?","version":"1.8.0_45"},{"class":"java.lang.Thread","method":"run","file":"Thread.java","line":745,"exact":true,"location":"?","version":"1.8.0_45"}]},"endOfBatch":false,"loggerFqcn":"org.slf4j.impl.Log4jLoggerAdapter"}
