Hi everyone, I am quiet new to ActiveMQ and try to monitor the overall queue size. I found a nice Nagios/Icinga plugin to do so, which uses the StatisticsPlugin. Currently I am a bit puzzles, because it seems the same configuration yields different result on different host. On one host (my VM), everything works as expected, whereas on my EC2-machine I only get a timeout (no answer at all).
My software-stack reads as follows: - CentOS 5.6 - Java version "1.6.0_24", Java(TM) SE Runtime Environment (build 1.6.0_24-b07) - Ruby 1.8.6 - Rubygem-Stomp 1.1.8 - ActiveMQ 5.5 I am pretty sure that I run the same software on both hosts installed from the same RPMs. I know this is not a "hardcore" ActiveMQ issue, but very appreciate any help. On both hosts, the configuration is -------- <beans xmlns="http://www.springframework.org/schema/beans"; xmlns:amq="http://activemq.apache.org/schema/core"; xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-2.0.xsd http://activemq.apache.org/schema/core http://activemq.apache.org/schema/core/activemq-core.xsd http://activemq.apache.org/camel/schema/spring http://activemq.apache.org/camel/schema/spring/camel-spring.xsd";> <broker xmlns="http://activemq.apache.org/schema/core"; brokerName="localhost" useJmx="true"> <managementContext> <managementContext createConnector="false"/> </managementContext> <plugins> <statisticsBrokerPlugin/> <simpleAuthenticationPlugin> <users> <authenticationUser username="nf-user" password="XXXX" groups="newsfeed-read,newsfeed-write,newsfeed-admin,everyone"/> <authenticationUser username="mc-admin" password="XXXX" groups="mcollective-read,mcollective-write,mcollective-admin,everyone"/> <authenticationUser username="mc-user" password="XXXX" groups="mcollective-read,mcollective-write,mcollective-admin,everyone"/> <authenticationUser username="icinga" password="247fbvbvvn390j4mvebnhb7g4bquwnvjdshkudhdsufssannz" groups="monitoring-read,monitoring-write,monitoring-admin,everyone"/> </users> </simpleAuthenticationPlugin> <authorizationPlugin> <map> <authorizationMap> <authorizationEntries> <authorizationEntry queue=">" write="admins" read="admins" admin="admins" /> <authorizationEntry topic=">" write="admins" read="admins" admin="admins" /> <authorizationEntry topic="ActiveMQ.Advisory.>" read="everyone" write="everyone" admin="everyone"/> <authorizationEntry queue="ActiveMQ.Statistics.Destination.>" read="monitoring-read" write="monitoring-write" admin="monitoring-admin" /> <authorizationEntry topic="nagios.>" read="monitoring-read" write="monitoring-write" admin="monitoring-admin" /> <authorizationEntry topic="mcollective.>" write="mcollective-write" read="mcollective-read" admin="mcollective-admin" /> <authorizationEntry queue="mcollective.>" write="mcollective-write" read="mcollective-read" admin="mcollective-admin" /> <authorizationEntry topic="newsfeed.>" write="newsfeed-write" read="newsfeed-read" admin="newsfeed-admin" /> <authorizationEntry queue="newsfeed.>" write="newsfeed-write" read="newsfeed-read" admin="newsfeed-admin" /> </authorizationEntries> </authorizationMap> </map> </authorizationPlugin> </plugins> <systemUsage> <systemUsage> <memoryUsage> <memoryUsage limit="20 mb"/> </memoryUsage> <storeUsage> <storeUsage limit="1 gb" name="foo"/> </storeUsage> <tempUsage> <tempUsage limit="100 mb"/> </tempUsage> </systemUsage> </systemUsage> <transportConnectors> <transportConnector name="openwire" uri="tcp://0.0.0.0:6166"/> <transportConnector name="stomp" uri="stomp://0.0.0.0:6163"/> </transportConnectors> </broker> </beans> ---------- In the working scenario, the tcpdump trace reads like: ------- CONNECT login:icinga content-length:0 passcode:247fbvbvvn390j4mvebnhb7g4bquwnvjdshkudhdsufssannz content-type: text/plain; charset=UTF-8 .CONNECTED session:ID:centos-vm-51950-1309785600907-3:12 . SUBSCRIBE destination:/topic/nagios.statresults.centos-vm content-length:0 transformation:jms-map-xml content-type: text/plain; charset=UTF-8 .SEND reply-to:/topic/nagios.statresults.centos-vm destination:/queue/ActiveMQ.Statistics.Destination.> content-length:0 content-type: text/plain; charset=UTF-8 .MESSAGE message-id:ID:centos-vm-51950-1309785600907-2:1:0:0:23 type:Advisory transformation:jms-map-xml destination:/topic/nagios.statresults.centos-vm timestamp:0 expires:0 priority:0 <map> <entry> <string>memoryUsage</string> <long>0</long> </entry> <entry> <string>dequeueCount</string> <long>0</long> </entry> <entry> <string>inflightCount</string> <long>0</long> </entry> <entry> <string>messagesCached</string> <long>0</long> </entry> <entry> <string>averageEnqueueTime</string> <double>0.0</double> </entry> <entry> <string>destinationName</string> <string>queue://ActiveMQ.Statistics.Destination.stompcat</string> </entry> <entry> <string>size</string> <long>0</long> </entry> <entry> <string>memoryPercentUsage</string> <int>0</int> </entry> <entry> -------- In the non-working scenario the trace is: -------- CONNECT passcode:247fbvbvvn390j4mvebnhb7g4bquwnvjdshkudhdsufssannz content-length:0 login:icinga content-type: text/plain; charset=UTF-8 .CONNECTED session:ID:ip-10-49-29-146-34975-1309785697533-3:43 . SUBSCRIBE destination:/topic/nagios.statresults.ip-10-49-29-146 content-length:0 transformation:jms-map-xml content-type: text/plain; charset=UTF-8 .SEND reply-to:/topic/nagios.statresults.ip-10-49-29-146 destination:/queue/ActiveMQ.Statistics.Destination.> content-length:0 content-type: text/plain; charset=UTF-8 . ------- Notice the difference in the connect-message (field order of passcode/login). Logfile in the working scenario is: --------- ==> /var/log/activemq/wrapper.log <== INFO | jvm 1 | 2011/07/04 16:15:10 | DEBUG MessageDatabase - Checkpoint started. INFO | jvm 1 | 2011/07/04 16:15:10 | DEBUG MessageDatabase - Checkpoint done. ==> /var/log/activemq/activemq.log <== 2011-07-04 16:15:15,073 [eckpoint Worker] DEBUG MessageDatabase - Checkpoint started. 2011-07-04 16:15:15,110 [eckpoint Worker] DEBUG MessageDatabase - Checkpoint done. 2011-07-04 16:15:15,133 [127.0.0.1:53160] DEBUG TransportConnection - Setting up new connection id: ID:centos-vm-51950-1309785600907-3:13, address: /127.0.0.1:53160 2011-07-04 16:15:15,175 [127.0.0.1:53160] DEBUG AbstractRegion - localhost adding consumer: ID:centos-vm-51950-1309785600907-3:13:-1:1 for destination: topic://nagios.statresults.centos-vm 2011-07-04 16:15:15,216 [127.0.0.1:53160] DEBUG TransportConnection - remove connection id: ID:centos-vm-51950-1309785600907-3:13 2011-07-04 16:15:15,216 [127.0.0.1:53160] DEBUG AbstractRegion - localhost removing consumer: ID:centos-vm-51950-1309785600907-3:13:-1:1 for destination: topic://nagios.statresults.centos-vm 2011-07-04 16:15:15,217 [.0.0.1:53160-38] DEBUG TransportConnection - Stopping connection: /127.0.0.1:53160 2011-07-04 16:15:15,218 [.0.0.1:53160-38] DEBUG TcpTransport - Stopping transport tcp:///127.0.0.1:53160 2011-07-04 16:15:15,218 [.0.0.1:53160-38] DEBUG TransportConnection - Stopped transport: /127.0.0.1:53160 2011-07-04 16:15:15,219 [.0.0.1:53160-38] DEBUG TransportConnection - Connection Stopped: /127.0.0.1:53160 ==> /var/log/activemq/wrapper.log <== INFO | jvm 1 | 2011/07/04 16:15:15 | DEBUG MessageDatabase - Checkpoint started. INFO | jvm 1 | 2011/07/04 16:15:15 | DEBUG MessageDatabase - Checkpoint done. INFO | jvm 1 | 2011/07/04 16:15:15 | DEBUG TransportConnection - Setting up new connection id: ID:centos-vm-51950-1309785600907-3:13, address: /127.0.0.1:53160 INFO | jvm 1 | 2011/07/04 16:15:15 | DEBUG AbstractRegion - localhost adding consumer: ID:centos-vm-51950-1309785600907-3:13:-1:1 for destination: topic://nagios.statresults.centos-vm INFO | jvm 1 | 2011/07/04 16:15:15 | DEBUG TransportConnection - remove connection id: ID:centos-vm-51950-1309785600907-3:13 INFO | jvm 1 | 2011/07/04 16:15:15 | DEBUG AbstractRegion - localhost removing consumer: ID:centos-vm-51950-1309785600907-3:13:-1:1 for destination: topic://nagios.statresults.centos-vm INFO | jvm 1 | 2011/07/04 16:15:15 | DEBUG TransportConnection - Stopping connection: /127.0.0.1:53160 INFO | jvm 1 | 2011/07/04 16:15:15 | DEBUG TcpTransport - Stopping transport tcp:///127.0.0.1:53160 INFO | jvm 1 | 2011/07/04 16:15:15 | DEBUG TransportConnection - Stopped transport: /127.0.0.1:53160 INFO | jvm 1 | 2011/07/04 16:15:15 | DEBUG TransportConnection - Connection Stopped: /127.0.0.1:53160 ----------- Whereas in the non-working it is: ----------- ==> /var/log/activemq/wrapper.log <== INFO | jvm 1 | 2011/07/04 16:16:58 | DEBUG TransportConnection - Setting up new connection id: ID:ip-10-49-29-146-34975-1309785697533-3:70, address: /127.0.0.1:60654 INFO | jvm 1 | 2011/07/04 16:16:58 | DEBUG AbstractRegion - localhost adding consumer: ID:ip-10-49-29-146-34975-1309785697533-3:70:-1:1 for destination: topic://nagios.statresults.ip-10-49-29-146 ==> /var/log/activemq/activemq.log <== 2011-07-04 16:17:00,912 [eckpoint Worker] DEBUG MessageDatabase - Checkpoint started. 2011-07-04 16:17:00,917 [eckpoint Worker] DEBUG MessageDatabase - Checkpoint done. ==> /var/log/activemq/wrapper.log <== INFO | jvm 1 | 2011/07/04 16:17:01 | DEBUG MessageDatabase - Checkpoint started. INFO | jvm 1 | 2011/07/04 16:17:01 | DEBUG MessageDatabase - Checkpoint done. ==> /var/log/activemq/activemq.log <== 2011-07-04 16:17:03,372 [127.0.0.1:60654] DEBUG Transport - Transport failed: java.io.EOFException java.io.EOFException at java.io.DataInputStream.readByte(DataInputStream.java:250) at org.apache.activemq.transport.stomp.StompWireFormat.readLine(StompWireFormat.java:146) at org.apache.activemq.transport.stomp.StompWireFormat.parseAction(StompWireFormat.java:162) at org.apache.activemq.transport.stomp.StompWireFormat.unmarshal(StompWireFormat.java:92) at org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:227) at org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:219) at org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:202) at java.lang.Thread.run(Thread.java:662) 2011-07-04 16:17:03,373 [0.0.1:60654-366] DEBUG TransportConnection - Stopping connection: /127.0.0.1:60654 2011-07-04 16:17:03,373 [0.0.1:60654-366] DEBUG TcpTransport - Stopping transport tcp:///127.0.0.1:60654 2011-07-04 16:17:03,374 [0.0.1:60654-366] DEBUG TransportConnection - Stopped transport: /127.0.0.1:60654 2011-07-04 16:17:03,375 [0.0.1:60654-366] DEBUG TransportConnection - Cleaning up connection resources: /127.0.0.1:60654 2011-07-04 16:17:03,375 [0.0.1:60654-366] DEBUG TransportConnection - remove connection id: ID:ip-10-49-29-146-34975-1309785697533-3:70 2011-07-04 16:17:03,375 [0.0.1:60654-366] DEBUG AbstractRegion - localhost removing consumer: ID:ip-10-49-29-146-34975-1309785697533-3:70:-1:1 for destination: topic://nagios.statresults.ip-10-49-29-146 2011-07-04 16:17:03,376 [0.0.1:60654-366] DEBUG TransportConnection - Connection Stopped: /127.0.0.1:60654 ==> /var/log/activemq/wrapper.log <== INFO | jvm 1 | 2011/07/04 16:17:03 | DEBUG Transport - Transport failed: java.io.EOFException INFO | jvm 1 | 2011/07/04 16:17:03 | java.io.EOFException INFO | jvm 1 | 2011/07/04 16:17:03 | at java.io.DataInputStream.readByte(DataInputStream.java:250) INFO | jvm 1 | 2011/07/04 16:17:03 | at org.apache.activemq.transport.stomp.StompWireFormat.readLine(StompWireFormat.java:146) INFO | jvm 1 | 2011/07/04 16:17:03 | at org.apache.activemq.transport.stomp.StompWireFormat.parseAction(StompWireFormat.java:162) INFO | jvm 1 | 2011/07/04 16:17:03 | at org.apache.activemq.transport.stomp.StompWireFormat.unmarshal(StompWireFormat.java:92) INFO | jvm 1 | 2011/07/04 16:17:03 | at org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:227) INFO | jvm 1 | 2011/07/04 16:17:03 | at org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:219) INFO | jvm 1 | 2011/07/04 16:17:03 | at org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:202) INFO | jvm 1 | 2011/07/04 16:17:03 | at java.lang.Thread.run(Thread.java:662) INFO | jvm 1 | 2011/07/04 16:17:03 | DEBUG TransportConnection - Stopping connection: /127.0.0.1:60654 INFO | jvm 1 | 2011/07/04 16:17:03 | DEBUG TcpTransport - Stopping transport tcp:///127.0.0.1:60654 INFO | jvm 1 | 2011/07/04 16:17:03 | DEBUG TransportConnection - Stopped transport: /127.0.0.1:60654 INFO | jvm 1 | 2011/07/04 16:17:03 | DEBUG TransportConnection - Cleaning up connection resources: /127.0.0.1:60654 INFO | jvm 1 | 2011/07/04 16:17:03 | DEBUG TransportConnection - remove connection id: ID:ip-10-49-29-146-34975-1309785697533-3:70 INFO | jvm 1 | 2011/07/04 16:17:03 | DEBUG AbstractRegion - localhost removing consumer: ID:ip-10-49-29-146-34975-1309785697533-3:70:-1:1 for destination: topic://nagios.statresults.ip-10-49-29-146 INFO | jvm 1 | 2011/07/04 16:17:03 | DEBUG TransportConnection - Connection Stopped: /127.0.0.1:60654 -------------- Thanks a lot for your help, Jens
