Hello. activemq-5.7.0-1, java version "1.6.0_24". We have 16 activemq nodes in a mesh network. The only current application using the activemq network is MCollective.
I'm very new to Activemq and am hoping to get some pointers as to wether I am going down the right path in troubleshooting the symptoms we are seeing. >From the client application point of view we are seeing the following symptoms. Without getting into too much details about what mcollective is let me generically describe the symptoms like this. - We do not see responses from the nodes connected to 1 of the brokers - Requests to some nodes never make it to the destination node - Responses from some nodes never make it back to the client On all of the brokers we see these log messages from activemq 2013-06-27 15:51:03,608 [160.70.27:57274] WARN TopicSubscription - TopicSubscription: consumer=ID:mco-broker8-57609-1372346502469-88:1:1:1, destinations=144, dispatched=1000, delivered=0, matched=1001, discarded=0: has twice its prefetch limit pending, without an ack; it appears to be slow We have adjusted the prefetch from the default of 1000 to 5000 and 15000 without any change. I set optimizeAcknowledge = true without any obvious benefits. I can't actually see via the attribute optimizeAcknowledge when I browse via jmx so while it's in the xml file it's possible I put it in the wrong spot? Basically i'm hoping someone can tell me that yes the warnings above are related to the symptoms i've described and I should focus my efforts on trying to understand/fix whatever is causing this. What's strange is that these log messages occur even when the only activity going on with mcollective is essentially just a heartbeat from each node (10K nodes) every 5 minutes. Any ideas on how I can move forward with figuring out what the problem is. I'm figuring I just don't understand activemq enough and have some configurations which are causing this situation. I've included an activemq.xml here. IT's the same on all 16 nodes. I've stripped out the auth section. TIA! Derek. <beans xmlns="http://www.springframework.org/schema/beans"; xmlns:amq="http://activemq.apache.org/schema/core"; xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-2.0.xsd http://activemq.apache.org/schema/core http://activemq.apache.org/schema/core/activemq-core.xsd http://activemq.apache.org/camel/schema/spring http://activemq.apache.org/camel/schema/spring/camel-spring.xsd";> <bean class="org.apache.activemq.spring.ActiveMQConnectionFactory"> <property name="optimizeAcknowledge"> <value>true</value> </property> </bean> <bean class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer"> <property name="locations"> <value>file:${activemq.base}/conf/credentials.properties</value> </property> </bean> <broker xmlns="http://activemq.apache.org/schema/core"; brokerName="mco-broker2.dc1.domain.name" useJmx="true" schedulePeriodForDestinationPurge="60000" networkConnectorStartAsync="true" deleteAllMessagesOnStartup="true"> <destinationPolicy> <policyMap> <policyEntries> <policyEntry topic=">" producerFlowControl="false"/> <policyEntry queue="*.reply.>" gcInactiveDestinations="true" inactiveTimoutBeforeGC="300000" /> </policyEntries> </policyMap> </destinationPolicy> <managementContext> <managementContext createConnector="false"/> </managementContext> <networkConnectors> <networkConnector name="mco-broker2.dc1.domain.name-mco-broker1.dc1.domain.name-topics" uri="static:(ssl://mco-broker1.dc1.domain.name:6167)" userName="XXXX" password="XXX" duplex="false" prefetchSize="15000" decreaseNetworkConsumerPriority="true" networkTTL="1" dynamicOnly="true"> <excludedDestinations> <queue physicalName=">" /> </excludedDestinations> </networkConnector> <networkConnector name="mco-broker2.dc1.domain.name-mco-broker1.dc1.domain.name-queues" uri="static:(ssl://mco-broker1.dc1.domain.name:6167)" userName="XXXX" password="XXX" duplex="false" decreaseNetworkConsumerPriority="true" networkTTL="1" dynamicOnly="true" conduitSubscriptions="false"> <excludedDestinations> <topic physicalName=">" /> </excludedDestinations> </networkConnector> <networkConnector name="mco-broker2.dc1.domain.name-mco-broker1.bc1.domain.name-topics" uri="static:(ssl://mco-broker1.bc1.domain.name:6167)" userName="XXXX" password="XXX" duplex="false" prefetchSize="15000" decreaseNetworkConsumerPriority="true" networkTTL="1" dynamicOnly="true"> <excludedDestinations> <queue physicalName=">" /> </excludedDestinations> </networkConnector> <networkConnector name="mco-broker2.dc1.domain.name-mco-broker1.bc1.domain.name-queues" uri="static:(ssl://mco-broker1.bc1.domain.name:6167)" userName="XXXX" password="XXX" duplex="false" decreaseNetworkConsumerPriority="true" networkTTL="1" dynamicOnly="true" conduitSubscriptions="false"> <excludedDestinations> <topic physicalName=">" /> </excludedDestinations> </networkConnector> <networkConnector name="mco-broker2.dc1.domain.name-mco-broker10.bc1.domain.name-topics" uri="static:(ssl://mco-broker10.bc1.domain.name:6167)" userName="XXXX" password="XXX" duplex="false" prefetchSize="15000" decreaseNetworkConsumerPriority="true" networkTTL="1" dynamicOnly="true"> <excludedDestinations> <queue physicalName=">" /> </excludedDestinations> </networkConnector> <networkConnector name="mco-broker2.dc1.domain.name-mco-broker10.bc1.domain.name-queues" uri="static:(ssl://mco-broker10.bc1.domain.name:6167)" userName="XXXX" password="XXX" duplex="false" decreaseNetworkConsumerPriority="true" networkTTL="1" dynamicOnly="true" conduitSubscriptions="false"> <excludedDestinations> <topic physicalName=">" /> </excludedDestinations> </networkConnector> <networkConnector name="mco-broker2.dc1.domain.name-mco-broker11.bc1.domain.name-topics" uri="static:(ssl://mco-broker11.bc1.domain.name:6167)" userName="XXXX" password="XXX" duplex="false" prefetchSize="15000" decreaseNetworkConsumerPriority="true" networkTTL="1" dynamicOnly="true"> <excludedDestinations> <queue physicalName=">" /> </excludedDestinations> </networkConnector> <networkConnector name="mco-broker2.dc1.domain.name-mco-broker11.bc1.domain.name-queues" uri="static:(ssl://mco-broker11.bc1.domain.name:6167)" userName="XXXX" password="XXX" duplex="false" decreaseNetworkConsumerPriority="true" networkTTL="1" dynamicOnly="true" conduitSubscriptions="false"> <excludedDestinations> <topic physicalName=">" /> </excludedDestinations> </networkConnector> <networkConnector name="mco-broker2.dc1.domain.name-mco-broker12.bc1.domain.name-topics" uri="static:(ssl://mco-broker12.bc1.domain.name:6167)" userName="XXXX" password="XXX" duplex="false" prefetchSize="15000" decreaseNetworkConsumerPriority="true" networkTTL="1" dynamicOnly="true"> <excludedDestinations> <queue physicalName=">" /> </excludedDestinations> </networkConnector> <networkConnector name="mco-broker2.dc1.domain.name-mco-broker12.bc1.domain.name-queues" uri="static:(ssl://mco-broker12.bc1.domain.name:6167)" userName="XXXX" password="XXX" duplex="false" decreaseNetworkConsumerPriority="true" networkTTL="1" dynamicOnly="true" conduitSubscriptions="false"> <excludedDestinations> <topic physicalName=">" /> </excludedDestinations> </networkConnector> <networkConnector name="mco-broker2.dc1.domain.name-mco-broker2.bc1.domain.name-topics" uri="static:(ssl://mco-broker2.bc1.domain.name:6167)" userName="XXXX" password="XXX" duplex="false" prefetchSize="15000" decreaseNetworkConsumerPriority="true" networkTTL="1" dynamicOnly="true"> <excludedDestinations> <queue physicalName=">" /> </excludedDestinations> </networkConnector> <networkConnector name="mco-broker2.dc1.domain.name-mco-broker2.bc1.domain.name-queues" uri="static:(ssl://mco-broker2.bc1.domain.name:6167)" userName="XXXX" password="XXX" duplex="false" decreaseNetworkConsumerPriority="true" networkTTL="1" dynamicOnly="true" conduitSubscriptions="false"> <excludedDestinations> <topic physicalName=">" /> </excludedDestinations> </networkConnector> <networkConnector name="mco-broker2.dc1.domain.name-mco-broker3.dc1.domain.name-topics" uri="static:(ssl://mco-broker3.dc1.domain.name:6167)" userName="XXXX" password="XXX" duplex="false" prefetchSize="15000" decreaseNetworkConsumerPriority="true" networkTTL="1" dynamicOnly="true"> <excludedDestinations> <queue physicalName=">" /> </excludedDestinations> </networkConnector> <networkConnector name="mco-broker2.dc1.domain.name-mco-broker3.dc1.domain.name-queues" uri="static:(ssl://mco-broker3.dc1.domain.name:6167)" userName="XXXX" password="XXX" duplex="false" decreaseNetworkConsumerPriority="true" networkTTL="1" dynamicOnly="true" conduitSubscriptions="false"> <excludedDestinations> <topic physicalName=">" /> </excludedDestinations> </networkConnector> <networkConnector name="mco-broker2.dc1.domain.name-mco-broker3.bc1.domain.name-topics" uri="static:(ssl://mco-broker3.bc1.domain.name:6167)" userName="XXXX" password="XXX" duplex="false" prefetchSize="15000" decreaseNetworkConsumerPriority="true" networkTTL="1" dynamicOnly="true"> <excludedDestinations> <queue physicalName=">" /> </excludedDestinations> </networkConnector> <networkConnector name="mco-broker2.dc1.domain.name-mco-broker3.bc1.domain.name-queues" uri="static:(ssl://mco-broker3.bc1.domain.name:6167)" userName="XXXX" password="XXX" duplex="false" decreaseNetworkConsumerPriority="true" networkTTL="1" dynamicOnly="true" conduitSubscriptions="false"> <excludedDestinations> <topic physicalName=">" /> </excludedDestinations> </networkConnector> <networkConnector name="mco-broker2.dc1.domain.name-mco-broker4.dc1.domain.name-topics" uri="static:(ssl://mco-broker4.dc1.domain.name:6167)" userName="XXXX" password="XXX" duplex="false" prefetchSize="15000" decreaseNetworkConsumerPriority="true" networkTTL="1" dynamicOnly="true"> <excludedDestinations> <queue physicalName=">" /> </excludedDestinations> </networkConnector> <networkConnector name="mco-broker2.dc1.domain.name-mco-broker4.dc1.domain.name-queues" uri="static:(ssl://mco-broker4.dc1.domain.name:6167)" userName="XXXX" password="XXX" duplex="false" decreaseNetworkConsumerPriority="true" networkTTL="1" dynamicOnly="true" conduitSubscriptions="false"> <excludedDestinations> <topic physicalName=">" /> </excludedDestinations> </networkConnector> <networkConnector name="mco-broker2.dc1.domain.name-mco-broker4.bc1.domain.name-topics" uri="static:(ssl://mco-broker4.bc1.domain.name:6167)" userName="XXXX" password="XXX" duplex="false" prefetchSize="15000" decreaseNetworkConsumerPriority="true" networkTTL="1" dynamicOnly="true"> <excludedDestinations> <queue physicalName=">" /> </excludedDestinations> </networkConnector> <networkConnector name="mco-broker2.dc1.domain.name-mco-broker4.bc1.domain.name-queues" uri="static:(ssl://mco-broker4.bc1.domain.name:6167)" userName="XXXX" password="XXX" duplex="false" decreaseNetworkConsumerPriority="true" networkTTL="1" dynamicOnly="true" conduitSubscriptions="false"> <excludedDestinations> <topic physicalName=">" /> </excludedDestinations> </networkConnector> <networkConnector name="mco-broker2.dc1.domain.name-mco-broker5.bc1.domain.name-topics" uri="static:(ssl://mco-broker5.bc1.domain.name:6167)" userName="XXXX" password="XXX" duplex="false" prefetchSize="15000" decreaseNetworkConsumerPriority="true" networkTTL="1" dynamicOnly="true"> <excludedDestinations> <queue physicalName=">" /> </excludedDestinations> </networkConnector> <networkConnector name="mco-broker2.dc1.domain.name-mco-broker5.bc1.domain.name-queues" uri="static:(ssl://mco-broker5.bc1.domain.name:6167)" userName="XXXX" password="XXX" duplex="false" decreaseNetworkConsumerPriority="true" networkTTL="1" dynamicOnly="true" conduitSubscriptions="false"> <excludedDestinations> <topic physicalName=">" /> </excludedDestinations> </networkConnector> <networkConnector name="mco-broker2.dc1.domain.name-mco-broker6.bc1.domain.name-topics" uri="static:(ssl://mco-broker6.bc1.domain.name:6167)" userName="XXXX" password="XXX" duplex="false" prefetchSize="15000" decreaseNetworkConsumerPriority="true" networkTTL="1" dynamicOnly="true"> <excludedDestinations> <queue physicalName=">" /> </excludedDestinations> </networkConnector> <networkConnector name="mco-broker2.dc1.domain.name-mco-broker6.bc1.domain.name-queues" uri="static:(ssl://mco-broker6.bc1.domain.name:6167)" userName="XXXX" password="XXX" duplex="false" decreaseNetworkConsumerPriority="true" networkTTL="1" dynamicOnly="true" conduitSubscriptions="false"> <excludedDestinations> <topic physicalName=">" /> </excludedDestinations> </networkConnector> <networkConnector name="mco-broker2.dc1.domain.name-mco-broker7.bc1.domain.name-topics" uri="static:(ssl://mco-broker7.bc1.domain.name:6167)" userName="XXXX" password="XXX" duplex="false" prefetchSize="15000" decreaseNetworkConsumerPriority="true" networkTTL="1" dynamicOnly="true"> <excludedDestinations> <queue physicalName=">" /> </excludedDestinations> </networkConnector> <networkConnector name="mco-broker2.dc1.domain.name-mco-broker7.bc1.domain.name-queues" uri="static:(ssl://mco-broker7.bc1.domain.name:6167)" userName="XXXX" password="XXX" duplex="false" decreaseNetworkConsumerPriority="true" networkTTL="1" dynamicOnly="true" conduitSubscriptions="false"> <excludedDestinations> <topic physicalName=">" /> </excludedDestinations> </networkConnector> <networkConnector name="mco-broker2.dc1.domain.name-mco-broker8.bc1.domain.name-topics" uri="static:(ssl://mco-broker8.bc1.domain.name:6167)" userName="XXXX" password="XXX" duplex="false" prefetchSize="15000" decreaseNetworkConsumerPriority="true" networkTTL="1" dynamicOnly="true"> <excludedDestinations> <queue physicalName=">" /> </excludedDestinations> </networkConnector> <networkConnector name="mco-broker2.dc1.domain.name-mco-broker8.bc1.domain.name-queues" uri="static:(ssl://mco-broker8.bc1.domain.name:6167)" userName="XXXX" password="XXX" duplex="false" decreaseNetworkConsumerPriority="true" networkTTL="1" dynamicOnly="true" conduitSubscriptions="false"> <excludedDestinations> <topic physicalName=">" /> </excludedDestinations> </networkConnector> <networkConnector name="mco-broker2.dc1.domain.name-mco-broker9.bc1.domain.name-topics" uri="static:(ssl://mco-broker9.bc1.domain.name:6167)" userName="XXXX" password="XXX" duplex="false" prefetchSize="15000" decreaseNetworkConsumerPriority="true" networkTTL="1" dynamicOnly="true"> <excludedDestinations> <queue physicalName=">" /> </excludedDestinations> </networkConnector> <networkConnector name="mco-broker2.dc1.domain.name-mco-broker9.bc1.domain.name-queues" uri="static:(ssl://mco-broker9.bc1.domain.name:6167)" userName="XXXX" password="XXX" duplex="false" decreaseNetworkConsumerPriority="true" networkTTL="1" dynamicOnly="true" conduitSubscriptions="false"> <excludedDestinations> <topic physicalName=">" /> </excludedDestinations> </networkConnector> </networkConnectors> <persistenceAdapter> <kahaDB directory="/var/cache/activemq/data" journalMaxFileLength="64mb"/> </persistenceAdapter> <plugins> <statisticsBrokerPlugin/> <simpleAuthenticationPlugin> </simpleAuthenticationPlugin> <authorizationPlugin> <map> <authorizationMap> <authorizationEntries> <authorizationEntry queue=">" write="admins" read="admins" admin="admins" /> <authorizationEntry queue="global.reply.>" write="allcollectives" admin="allcollectives" /> <authorizationEntry queue="eu.reply.>" write="eu" admin="eu" /> <authorizationEntry queue="eu_infra.reply.>" write="eu_infra" admin="eu_infra" /> <authorizationEntry queue="eu_services.reply.>" write="eu_services" admin="eu_services" /> <authorizationEntry queue="eu_customer.reply.>" write="eu_customer" admin="eu_customer" /> <authorizationEntry queue="global.nodes" read="allcollectives" admin="allcollectives" /> <authorizationEntry queue="eu.nodes" read="eu" admin="eu" /> <authorizationEntry queue="eu_infra.nodes" read="eu_infra" admin="eu_infra" /> <authorizationEntry queue="eu_services.nodes" read="eu_services" admin="eu_services" /> <authorizationEntry queue="eu_customer.nodes" read="eu_customer" admin="eu_customer" /> <authorizationEntry topic=">" write="admins" read="admins" admin="admins" /> <authorizationEntry topic="global.registration.agent" write="allcollectives" read="allcollectives" admin="allcollectives" /> <authorizationEntry topic="global.*.agent" read="allcollectives" admin="allcollectives" /> <authorizationEntry topic="eu.registration.agent" write="eu" read="eu" admin="eu" /> <authorizationEntry topic="eu.*.agent" read="eu" admin="eu" /> <authorizationEntry topic="eu_infra.*.agent" read="eu_infra" admin="eu_infra" /> <authorizationEntry topic="eu_services.*.agent" read="eu_services" admin="eu_services" /> <authorizationEntry topic="eu_customer.*.agent" read="eu_customer" admin="eu_customer" /> <authorizationEntry topic="ActiveMQ.Advisory.>" write="everyone" read="everyone" admin="everyone" /> </authorizationEntries> </authorizationMap> </map> </authorizationPlugin> </plugins> <sslContext> <sslContext /> </sslContext> <systemUsage> <systemUsage> <memoryUsage> <memoryUsage limit="2 gb"/> </memoryUsage> <storeUsage> <storeUsage limit="10 gb" name="foo"/> </storeUsage> <tempUsage> <tempUsage limit="1 gb"/> </tempUsage> </systemUsage> </systemUsage> <transportConnectors> <transportConnector name="openwiressl" uri="ssl://0.0.0.0:6167"/> <transportConnector name="stompssl" uri="stomp+ssl://0.0.0.0:6164"/> </transportConnectors> </broker> <import resource="jetty.xml"/> </beans> -- View this message in context: http://activemq.2283324.n4.nabble.com/Broker-to-Broker-prefetch-problems-tp4668660.html Sent from the ActiveMQ - User mailing list archive at Nabble.com.
