This is new, so please submit a bug in JIRA. Please include as much
information as you can about your configuration and how your broker is
being used; because you're seeing problems while no one else is and this
appears to be code everyone should be hitting, it's possible that there's
something unique about your setup or your usage that triggers the problem,
and whoever investigates the issue will benefit from having as much detail
as possible.
Tim
On Jul 13, 2016 12:26 AM, "Johan Carlquist" <jo...@su.se> wrote:
Hi!
We have two brokers esb-test-mq01 and esb-test-mq02 configured in a
network of brokers. Each broker has configuration like this:
<networkConnector
networkTTL="-1"
conduitSubscriptions="false"
decreaseNetworkConsumerPriority="true"
uri="static:(ssl://${activemq.networkPartner}:61616)"
userName="${activemq.username}"
password="${activemq.password}"
/>
Both brokers running ActiveMQ 5.13.0, Java 1.8.0_45 and Ubuntu 14.04
(3.13.0-87-generic x86_64).
Sometimes (we havn't found any patterns for when it occurs) one of the
connection between the hosts closes with the following message:
"Network connection between vm://esb-test-mq01.it.su.se#18 and ssl://
esb-test-mq02.it.su.se/77.238.37.208:61616 shutdown due to a local error:
java.lang.ArithmeticException: / by zero"
After this the broker never calls back. Yesterday we finally caught this
division with TRACE turned on, but we can't make anything out of the
logs. We now wondering, is this a known bug that we haven't found any
information on, just missconfiguration or should we report the bug on
issues.apache.org?
Greatful for any input.
--
Johan
And the log lines (sorry for the json encryption):
Jul 12 05:41:37 esb-test-mq01.it.su.se {"timeMillis":
1468294897046,"thread":"ActiveMQ BrokerService[esb-test-mq01.it.su.se]
Task-3082","level":"DEBUG","loggerName":"org.apache.activemq.network.DemandForwardingBridgeSupport","message":"bridging
(esb-test-mq01.it.su.se -> esb-test-mq02.it.su.se), consumer:
ActiveMQBytesMessage {commandId = 1218913, responseRequired = true,
messageId = ID:esb-test-mq02-43870-1468234158631-7:21062:0:0:1,
originalDestination = null, originalTransactionId = null, producerId =
esb-test-mq01.it.su.se->esb-test-mq02.it.su.se-45448-1467985313001-17:2:1:1,
destination = queue://aliveness-test, transactionId = null, expiration = 0,
timestamp = 1468286832000, arrival = 0, brokerInTime = 1468294896175,
brokerOutTime = 1468294897046, correlationId = null, replyTo = null,
persistent = true, type = null, priority = 4, groupID = null, groupSequence
= 0, targetConsumerId = null, compressed = false, userID = null, content =
org.apache.activemq.util.ByteSequence@4bd15e41, marshalledProperties =
org.apache.activemq.util.ByteSequence@44e78a09, dataStructure = null,
redeliveryCounter = 865, size = 1193, properties = {JMS_AMQP_NATIVE=true,
JMS_AMQP_MESSAGE_FORMAT=0, JMS_AMQP_FirstAcquirer=false},
readOnlyProperties = false, readOnlyBody = false, droppable = false,
jmsXGroupFirstForConsumer = false} ActiveMQBytesMessage{ bytesOut = null,
dataOut = null, dataIn = null }, destination:
esb-test-mq01.it.su.se->esb-test-mq02.it.su.se-45448-1467985313001-17:1:1:36663,
brokerPath: queue://aliveness-test, message: [esb-test-mq02.it.su.se,
esb-test-mq01.it.su.se, esb-test-mq02.it.su.se, esb-test-mq01.it.su.se,
esb-test-mq02.it.su.se, esb-test-mq01.it.su.se, esb-test-mq02.it.su.se,
esb-test-mq01.it.su.se, esb-test-mq02.it.su.se, esb-test-mq01.it.su.se,
esb-test-mq02.it.su.se, esb-test-mq01.it.su.se, esb-test-mq02.it.su.se,
esb-test-mq01.it.su.se, esb-test-mq02.it.su.se, esb-test-mq01.it.su.se,
esb-test-mq02.it.su.se, esb-test-mq01.it.su.se, esb-test-mq02.it.su.se,
esb-test-mq01.it.su.se, esb-test-mq02.it.su.se, esb-test-m
Jul 12 05:41:37 esb-test-mq01.it.su.se {"timeMillis":
1468294897057,"thread":"ActiveMQ Transport: ssl://
esb-test-mq02.it.su.se/77.238.37.208:61616","level":"TRACE","loggerName":"org.apache.activemq.network.DemandForwardingBridgeSupport","message":"serviceLocalException:
disposed false ex","thrown":{"commonElementCount":0,"localizedMessage":"/
by zero","message":"/ by
zero","name":"java.lang.ArithmeticException","extendedStackTrace":[{"class":"org.apache.activemq.broker.region.AbstractSubscription","method":"getInFlightUsage","file":"AbstractSubscription.java","line":240,"exact":false,"location":"activemq-broker-5.13.0.jar","version":"5.13.0"},{"class":"org.apache.activemq.broker.region.Queue","method":"isOptimizeStorage","file":"Queue.java","line":2322,"exact":false,"location":"activemq-broker-5.13.0.jar","version":"5.13.0"},{"class":"org.apache.activemq.broker.region.Queue","method":"doMessageSend","file":"Queue.java","line":839,"exact":false,"location":"activemq-broker-5.13.0.jar","version":"5.13.0"},{"class":"org.apache.activemq.broker.region.Queue","method":"send","file":"Queue.java","line":731,"exact":false,"location":"activemq-broker-5.13.0.jar","version":"5.13.0"},{"class":"org.apache.activemq.broker.region.DestinationFilter","method":"send","file":"DestinationFilter.java","line":132,"exact":false,"location":"activemq-broker-5.13.0.jar","version":"5.13.0"},{"class":"org.apache.activemq.broker.region.AbstractRegion","method":"send","file":"AbstractRegion.java","line":503,"exact":false,"location":"activemq-broker-5.13.0.jar","version":"5.13.0"},{"class":"org.apache.activemq.broker.region.RegionBroker","method":"send","file":"RegionBroker.java","line":468,"exact":false,"location":"activemq-broker-5.13.0.jar","version":"5.13.0"},{"class":"org.apache.activemq.broker.jmx.ManagedRegionBroker","method":"send","file":"ManagedRegionBroker.java","line":296,"exact":false,"location":"activemq-broker-5.13.0.jar","version":"5.13.0"},{"class":"org.apache.activemq.broker.BrokerFilter","method":"send","file":"BrokerFilter.java","li
Jul 12 05:41:37 esb-test-mq01.it.su.se {"timeMillis":
1468294897060,"thread":"ActiveMQ Transport: ssl://
esb-test-mq02.it.su.se/77.238.37.208:61616","level":"INFO","loggerName":"org.apache.activemq.network.DemandForwardingBridgeSupport","message":"Network
connection between vm://esb-test-mq01.it.su.se#18 and ssl://
esb-test-mq02.it.su.se/77.238.37.208:61616 shutdown due to a local error:
java.lang.ArithmeticException: / by
zero","endOfBatch":false,"loggerFqcn":"org.slf4j.impl.Log4jLoggerAdapter"}
Jul 12 05:41:37 esb-test-mq01.it.su.se {"timeMillis":
1468294897060,"thread":"ActiveMQ Transport: ssl://
esb-test-mq02.it.su.se/77.238.37.208:61616","level":"DEBUG","loggerName":"org.apache.activemq.network.DemandForwardingBridgeSupport","message":"The
local Exception was: java.lang.ArithmeticException: / by
zero","thrown":{"commonElementCount":0,"localizedMessage":"/ by
zero","message":"/ by
zero","name":"java.lang.ArithmeticException","extendedStackTrace":[{"class":"org.apache.activemq.broker.region.AbstractSubscription","method":"getInFlightUsage","file":"AbstractSubscription.java","line":240,"exact":false,"location":"activemq-broker-5.13.0.jar","version":"5.13.0"},{"class":"org.apache.activemq.broker.region.Queue","method":"isOptimizeStorage","file":"Queue.java","line":2322,"exact":false,"location":"activemq-broker-5.13.0.jar","version":"5.13.0"},{"class":"org.apache.activemq.broker.region.Queue","method":"doMessageSend","file":"Queue.java","line":839,"exact":false,"location":"activemq-broker-5.13.0.jar","version":"5.13.0"},{"class":"org.apache.activemq.broker.region.Queue","method":"send","file":"Queue.java","line":731,"exact":false,"location":"activemq-broker-5.13.0.jar","version":"5.13.0"},{"class":"org.apache.activemq.broker.region.DestinationFilter","method":"send","file":"DestinationFilter.java","line":132,"exact":false,"location":"activemq-broker-5.13.0.jar","version":"5.13.0"},{"class":"org.apache.activemq.broker.region.AbstractRegion","method":"send","file":"AbstractRegion.java","line":503,"exact":false,"location":"activemq-broker-5.13.0.jar","version":"5.13.0"},{"class":"org.apache.activemq.broker.region.RegionBroker","method":"send","file":"RegionBroker.java","line":468,"exact":false,"location":"activemq-broker-5.13.0.jar","version":"5.13.0"},{"class":"org.apache.activemq.broker.jmx.ManagedRegionBroker","method":"send","file":"ManagedRegionBroker.java","line":296,"exact":false,"location":"activemq-broker-5.13.0.jar","version":"5.13.0"},{"class":"org.apache.activemq.broker.BrokerFilter","method":"send","file
Jul 12 05:41:37 esb-test-mq01.it.su.se {"timeMillis":
1468294897062,"thread":"ActiveMQ BrokerService[esb-test-mq01.it.su.se]
Task-3079","level":"DEBUG","loggerName":"org.apache.activemq.network.DemandForwardingBridgeSupport","message":"
stopping esb-test-mq01.it.su.se bridge to esb-test-mq02.it.su.se
","endOfBatch":false,"loggerFqcn":"org.slf4j.impl.Log4jLoggerAdapter"}
Jul 12 05:41:37 esb-test-mq01.it.su.se {"timeMillis":
1468294897062,"thread":"ActiveMQ BrokerService[esb-test-mq01.it.su.se]
Task-3079","level":"TRACE","loggerName":"org.apache.activemq.network.DemandForwardingBridgeSupport","message":"
esb-test-mq01.it.su.se unregister bridge
(org.apache.activemq.network.DemandForwardingBridge@199245a1) to
esb-test-mq02.it.su.se
","endOfBatch":false,"loggerFqcn":"org.slf4j.impl.Log4jLoggerAdapter"}
Jul 12 05:41:37 esb-test-mq01.it.su.se {"timeMillis":
1468294897062,"thread":"ActiveMQ BrokerService[esb-test-mq01.it.su.se]
Task-3084","level":"TRACE","loggerName":"org.apache.activemq.network.DemandForwardingBridgeSupport","message":"serviceLocalException:
disposed true ex","thrown":{"commonElementCount":0,"localizedMessage":"peer
(vm://esb-test-mq01.it.su.se#19) stopped.","message":"peer (vm://
esb-test-mq01.it.su.se#19)
stopped.","name":"org.apache.activemq.transport.TransportDisposedIOException","extendedStackTrace":[{"class":"org.apache.activemq.transport.vm.VMTransport","method":"stop","file":"VMTransport.java","line":230,"exact":true,"location":"activemq-broker-5.13.0.jar","version":"5.13.0"},{"class":"org.apache.activemq.transport.TransportFilter","method":"stop","file":"TransportFilter.java","line":65,"exact":true,"location":"activemq-client-5.13.0.jar","version":"5.13.0"},{"class":"org.apache.activemq.transport.TransportFilter","method":"stop","file":"TransportFilter.java","line":65,"exact":true,"location":"activemq-client-5.13.0.jar","version":"5.13.0"},{"class":"org.apache.activemq.transport.ResponseCorrelator","method":"stop","file":"ResponseCorrelator.java","line":132,"exact":true,"location":"activemq-client-5.13.0.jar","version":"5.13.0"},{"class":"org.apache.activemq.broker.TransportConnection","method":"doStop","file":"TransportConnection.java","line":1165,"exact":true,"location":"activemq-broker-5.13.0.jar","version":"5.13.0"},{"class":"org.apache.activemq.broker.TransportConnection$4","method":"run","file":"TransportConnection.java","line":1131,"exact":true,"location":"activemq-broker-5.13.0.jar","version":"5.13.0"},{"class":"java.util.concurrent.ThreadPoolExecutor","method":"runWorker","file":"ThreadPoolExecutor.java","line":1142,"exact":true,"location":"?","version":"1.8.0_45"},{"class":"java.util.concurrent.ThreadPoolExecutor$Worker","method":"run","file":"ThreadPoolExecutor.java","line":617,"exact":true,"location":"?","version":"1.8.0_45"},{"class":"java.lang.Thread","method":"run","file":"Thread.j
Jul 12 05:41:37 esb-test-mq01.it.su.se {"timeMillis":
1468294897062,"thread":"ActiveMQ BrokerService[esb-test-mq01.it.su.se]
Task-3079","level":"TRACE","loggerName":"org.apache.activemq.network.DemandForwardingBridgeSupport","message":"serviceLocalException:
disposed true
ex","thrown":{"commonElementCount":0,"localizedMessage":"Stopped.","message":"Stopped.","name":"java.io.IOException","extendedStackTrace":[{"class":"org.apache.activemq.transport.ResponseCorrelator","method":"stop","file":"ResponseCorrelator.java","line":131,"exact":true,"location":"activemq-client-5.13.0.jar","version":"5.13.0"},{"class":"org.apache.activemq.util.ServiceStopper","method":"stop","file":"ServiceStopper.java","line":41,"exact":true,"location":"activemq-client-5.13.0.jar","version":"5.13.0"},{"class":"org.apache.activemq.network.DemandForwardingBridgeSupport","method":"stop","file":"DemandForwardingBridgeSupport.java","line":305,"exact":true,"location":"activemq-broker-5.13.0.jar","version":"5.13.0"},{"class":"org.apache.activemq.util.ServiceSupport","method":"dispose","file":"ServiceSupport.java","line":43,"exact":true,"location":"activemq-client-5.13.0.jar","version":"5.13.0"},{"class":"org.apache.activemq.network.DemandForwardingBridgeSupport$9","method":"run","file":"DemandForwardingBridgeSupport.java","line":878,"exact":true,"location":"activemq-broker-5.13.0.jar","version":"5.13.0"},{"class":"java.util.concurrent.ThreadPoolExecutor","method":"runWorker","file":"ThreadPoolExecutor.java","line":1142,"exact":true,"location":"?","version":"1.8.0_45"},{"class":"java.util.concurrent.ThreadPoolExecutor$Worker","method":"run","file":"ThreadPoolExecutor.java","line":617,"exact":true,"location":"?","version":"1.8.0_45"},{"class":"java.lang.Thread","method":"run","file":"Thread.java","line":745,"exact":true,"location":"?","version":"1.8.0_45"}]},"endOfBatch":false,"loggerFqcn":"org.slf4j.impl.Log4jLoggerAdapter"}
Jul 12 05:41:37 esb-test-mq01.it.su.se {"timeMillis":
1468294897061,"thread":"ActiveMQ Transport: ssl://
esb-test-mq02.it.su.se/77.238.37.208:61616","level":"TRACE","loggerName":"org.apache.activemq.network.DemandForwardingBridgeSupport","message":"fire
bridge failed, listener:
org.apache.activemq.network.DiscoveryNetworkConnector$1DiscoverNetworkBridgeListener@67a83e67","thrown":{"commonElementCount":0,"localizedMessage":"/
by zero","message":"/ by
zero","name":"java.lang.ArithmeticException","extendedStackTrace":[{"class":"org.apache.activemq.broker.region.AbstractSubscription","method":"getInFlightUsage","file":"AbstractSubscription.java","line":240,"exact":false,"location":"activemq-broker-5.13.0.jar","version":"5.13.0"},{"class":"org.apache.activemq.broker.region.Queue","method":"isOptimizeStorage","file":"Queue.java","line":2322,"exact":false,"location":"activemq-broker-5.13.0.jar","version":"5.13.0"},{"class":"org.apache.activemq.broker.region.Queue","method":"doMessageSend","file":"Queue.java","line":839,"exact":false,"location":"activemq-broker-5.13.0.jar","version":"5.13.0"},{"class":"org.apache.activemq.broker.region.Queue","method":"send","file":"Queue.java","line":731,"exact":false,"location":"activemq-broker-5.13.0.jar","version":"5.13.0"},{"class":"org.apache.activemq.broker.region.DestinationFilter","method":"send","file":"DestinationFilter.java","line":132,"exact":false,"location":"activemq-broker-5.13.0.jar","version":"5.13.0"},{"class":"org.apache.activemq.broker.region.AbstractRegion","method":"send","file":"AbstractRegion.java","line":503,"exact":false,"location":"activemq-broker-5.13.0.jar","version":"5.13.0"},{"class":"org.apache.activemq.broker.region.RegionBroker","method":"send","file":"RegionBroker.java","line":468,"exact":false,"location":"activemq-broker-5.13.0.jar","version":"5.13.0"},{"class":"org.apache.activemq.broker.jmx.ManagedRegionBroker","method":"send","file":"ManagedRegionBroker.java","line":296,"exact":false,"location":"activemq-broker-5.13.0.jar","version":"5.13.0"},{"class":"org.
Jul 12 05:41:37 esb-test-mq01.it.su.se {"timeMillis":
1468294897063,"thread":"ActiveMQ
Task-3","level":"INFO","loggerName":"org.apache.activemq.network.DiscoveryNetworkConnector","message":"Establishing
network connection from vm://esb-test-mq01.it.su.se?async=false to ssl://
esb-test-mq02.it.su.se:61616
","endOfBatch":false,"loggerFqcn":"org.slf4j.impl.Log4jLoggerAdapter"}
Jul 12 05:41:37 esb-test-mq01.it.su.se {"timeMillis":
1468294897062,"thread":"ActiveMQ BrokerService[esb-test-mq01.it.su.se]
Task-3079","level":"TRACE","loggerName":"org.apache.activemq.network.DemandForwardingBridgeSupport","message":"serviceLocalException:
disposed true
ex","thrown":{"commonElementCount":0,"localizedMessage":"Stopped.","message":"Stopped.","name":"java.io.IOException","extendedStackTrace":[{"class":"org.apache.activemq.transport.ResponseCorrelator","method":"stop","file":"ResponseCorrelator.java","line":131,"exact":true,"location":"activemq-client-5.13.0.jar","version":"5.13.0"},{"class":"org.apache.activemq.util.ServiceStopper","method":"stop","file":"ServiceStopper.java","line":41,"exact":true,"location":"activemq-client-5.13.0.jar","version":"5.13.0"},{"class":"org.apache.activemq.network.DemandForwardingBridgeSupport","method":"stop","file":"DemandForwardingBridgeSupport.java","line":305,"exact":true,"location":"activemq-broker-5.13.0.jar","version":"5.13.0"},{"class":"org.apache.activemq.util.ServiceSupport","method":"dispose","file":"ServiceSupport.java","line":43,"exact":true,"location":"activemq-client-5.13.0.jar","version":"5.13.0"},{"class":"org.apache.activemq.network.DemandForwardingBridgeSupport$9","method":"run","file":"DemandForwardingBridgeSupport.java","line":878,"exact":true,"location":"activemq-broker-5.13.0.jar","version":"5.13.0"},{"class":"java.util.concurrent.ThreadPoolExecutor","method":"runWorker","file":"ThreadPoolExecutor.java","line":1142,"exact":true,"location":"?","version":"1.8.0_45"},{"class":"java.util.concurrent.ThreadPoolExecutor$Worker","method":"run","file":"ThreadPoolExecutor.java","line":617,"exact":true,"location":"?","version":"1.8.0_45"},{"class":"java.lang.Thread","method":"run","file":"Thread.java","line":745,"exact":true,"location":"?","version":"1.8.0_45"}]},"endOfBatch":false,"loggerFqcn":"org.slf4j.impl.Log4jLoggerAdapter"}
