Having lots of stability issues (5.8.0 on Win64 2008 Server 64GB RAM). Messages are IBytesMessages, typically around 400k. Typically 3 to 4 queues running at once. 200 consumers. Checked the log and seeing a TON of exceptions:
1) INFO | jvm 1 | 2013/08/26 15:09:32 | WARN | Transport Connection to: tcp://x.x.x.x:59339 failed: java.net.SocketException: Software caused connection abort: socket write error 2) INFO | jvm 1 | 2013/08/26 15:09:33 | WARN | Transport Connection to: tcp://x.x.x.x:59700 failed: java.io.EOFException 3) INFO | jvm 1 | 2013/08/26 15:09:33 | WARN | Transport Connection to: tcp://x.x.x.x:62316 failed: java.net.SocketException: Software caused connection abort: recv failed 4) INFO | jvm 1 | 2013/08/26 15:09:33 | WARN | Transport Connection to: tcp://x.x.x.x:62316 failed: org.apache.activemq.transport.InactivityIOException: Cannot send, channel has already failed: tcp://x.x.x.x:62316 Hundreds / Thousands of these for various IPs and ports. 5) INFO | jvm 1 | 2013/08/26 16:05:48 | WARN | Async error occurred: javax.jms.JMSException: Destination still has an active subscription: queue://x-caf311ac-edb9-4ff1-8d7b-890d320cece8 INFO | jvm 1 | 2013/08/26 16:05:48 | javax.jms.JMSException: Destination still has an active subscription: queue://x-caf311ac-edb9-4ff1-8d7b-890d320cece8 INFO | jvm 1 | 2013/08/26 16:05:48 | at org.apache.activemq.broker.region.AbstractRegion.removeDestination(AbstractRegion.java:186) INFO | jvm 1 | 2013/08/26 16:05:48 | at org.apache.activemq.broker.jmx.ManagedQueueRegion.removeDestination(ManagedQueueRegion.java:62) INFO | jvm 1 | 2013/08/26 16:05:48 | at org.apache.activemq.broker.region.RegionBroker.removeDestination(RegionBroker.java:288) INFO | jvm 1 | 2013/08/26 16:05:48 | at org.apache.activemq.broker.region.RegionBroker.removeDestinationInfo(RegionBroker.java:301) INFO | jvm 1 | 2013/08/26 16:05:48 | at org.apache.activemq.broker.BrokerFilter.removeDestinationInfo(BrokerFilter.java:221) INFO | jvm 1 | 2013/08/26 16:05:48 | at org.apache.activemq.advisory.AdvisoryBroker.removeDestinationInfo(AdvisoryBroker.java:223) INFO | jvm 1 | 2013/08/26 16:05:48 | at org.apache.activemq.broker.BrokerFilter.removeDestinationInfo(BrokerFilter.java:221) INFO | jvm 1 | 2013/08/26 16:05:48 | at org.apache.activemq.broker.BrokerFilter.removeDestinationInfo(BrokerFilter.java:221) INFO | jvm 1 | 2013/08/26 16:05:48 | at org.apache.activemq.broker.MutableBrokerFilter.removeDestinationInfo(MutableBrokerFilter.java:228) INFO | jvm 1 | 2013/08/26 16:05:48 | at org.apache.activemq.broker.TransportConnection.processRemoveDestination(TransportConnection.java:537) INFO | jvm 1 | 2013/08/26 16:05:48 | at org.apache.activemq.command.DestinationInfo.visit(DestinationInfo.java:124) INFO | jvm 1 | 2013/08/26 16:05:48 | at org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:329) INFO | jvm 1 | 2013/08/26 16:05:48 | at org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:184) INFO | jvm 1 | 2013/08/26 16:05:48 | at org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50) INFO | jvm 1 | 2013/08/26 16:05:48 | at org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:113) INFO | jvm 1 | 2013/08/26 16:05:48 | at org.apache.activemq.transport.AbstractInactivityMonitor.onCommand(AbstractInactivityMonitor.java:288) INFO | jvm 1 | 2013/08/26 16:05:48 | at org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:83) INFO | jvm 1 | 2013/08/26 16:05:48 | at org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:214) INFO | jvm 1 | 2013/08/26 16:05:48 | at org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:196) INFO | jvm 1 | 2013/08/26 16:05:48 | at java.lang.Thread.run(Unknown Source) 6) INFO | jvm 2 | 2013/08/27 09:17:17 | WARN | Async error occurred: java.lang.IllegalStateException: Timer already cancelled. INFO | jvm 2 | 2013/08/27 09:17:17 | java.lang.IllegalStateException: Timer already cancelled. INFO | jvm 2 | 2013/08/27 09:17:17 | at java.util.Timer.sched(Unknown Source) INFO | jvm 2 | 2013/08/27 09:17:17 | at java.util.Timer.schedule(Unknown Source) INFO | jvm 2 | 2013/08/27 09:17:17 | at org.apache.activemq.thread.Scheduler.schedualPeriodically(Scheduler.java:50) INFO | jvm 2 | 2013/08/27 09:17:17 | at org.apache.activemq.broker.region.Queue.start(Queue.java:889) INFO | jvm 2 | 2013/08/27 09:17:17 | at org.apache.activemq.broker.region.AbstractRegion.addDestination(AbstractRegion.java:142) INFO | jvm 2 | 2013/08/27 09:17:17 | at org.apache.activemq.broker.region.RegionBroker.addDestination(RegionBroker.java:277) INFO | jvm 2 | 2013/08/27 09:17:17 | at org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145) INFO | jvm 2 | 2013/08/27 09:17:17 | at org.apache.activemq.advisory.AdvisoryBroker.addDestination(AdvisoryBroker.java:174) INFO | jvm 2 | 2013/08/27 09:17:17 | at org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145) INFO | jvm 2 | 2013/08/27 09:17:17 | at org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145) INFO | jvm 2 | 2013/08/27 09:17:17 | at org.apache.activemq.broker.MutableBrokerFilter.addDestination(MutableBrokerFilter.java:151) INFO | jvm 2 | 2013/08/27 09:17:17 | at org.apache.activemq.broker.region.RegionBroker.addProducer(RegionBroker.java:324) INFO | jvm 2 | 2013/08/27 09:17:17 | at org.apache.activemq.broker.jmx.ManagedRegionBroker.addProducer(ManagedRegionBroker.java:252) INFO | jvm 2 | 2013/08/27 09:17:17 | at org.apache.activemq.broker.BrokerFilter.addProducer(BrokerFilter.java:93) INFO | jvm 2 | 2013/08/27 09:17:17 | at org.apache.activemq.advisory.AdvisoryBroker.addProducer(AdvisoryBroker.java:162) INFO | jvm 2 | 2013/08/27 09:17:17 | at org.apache.activemq.broker.CompositeDestinationBroker.addProducer(CompositeDestinationBroker.java:56) INFO | jvm 2 | 2013/08/27 09:17:17 | at org.apache.activemq.broker.BrokerFilter.addProducer(BrokerFilter.java:93) INFO | jvm 2 | 2013/08/27 09:17:17 | at org.apache.activemq.broker.MutableBrokerFilter.addProducer(MutableBrokerFilter.java:99) INFO | jvm 2 | 2013/08/27 09:17:17 | at org.apache.activemq.broker.TransportConnection.processAddProducer(TransportConnection.java:566) INFO | jvm 2 | 2013/08/27 09:17:17 | at org.apache.activemq.command.ProducerInfo.visit(ProducerInfo.java:105) INFO | jvm 2 | 2013/08/27 09:17:17 | at org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:329) INFO | jvm 2 | 2013/08/27 09:17:17 | at org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:184) INFO | jvm 2 | 2013/08/27 09:17:17 | at org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50) INFO | jvm 2 | 2013/08/27 09:17:17 | at org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:113) INFO | jvm 2 | 2013/08/27 09:17:17 | at org.apache.activemq.transport.AbstractInactivityMonitor.onCommand(AbstractInactivityMonitor.java:288) INFO | jvm 2 | 2013/08/27 09:17:17 | at org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:83) INFO | jvm 2 | 2013/08/27 09:17:17 | at org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:214) INFO | jvm 2 | 2013/08/27 09:17:17 | at org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:196) INFO | jvm 2 | 2013/08/27 09:17:17 | at java.lang.Thread.run(Unknown Source) 7) INFO | jvm 2 | 2013/08/27 09:17:17 | WARN | Async error occurred: java.lang.IllegalStateException: Cannot remove a consumer that had not been registered: ID:x-w01-49130-635131918368230097-1:0:1:1 INFO | jvm 2 | 2013/08/27 09:17:17 | java.lang.IllegalStateException: Cannot remove a consumer that had not been registered: ID:x-w01-49130-635131918368230097-1:0:1:1 INFO | jvm 2 | 2013/08/27 09:17:17 | at org.apache.activemq.broker.TransportConnection.processRemoveConsumer(TransportConnection.java:647) INFO | jvm 2 | 2013/08/27 09:17:17 | at org.apache.activemq.command.RemoveInfo.visit(RemoveInfo.java:76) INFO | jvm 2 | 2013/08/27 09:17:17 | at org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:329) INFO | jvm 2 | 2013/08/27 09:17:17 | at org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:184) INFO | jvm 2 | 2013/08/27 09:17:17 | at org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50) INFO | jvm 2 | 2013/08/27 09:17:17 | at org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:113) INFO | jvm 2 | 2013/08/27 09:17:17 | at org.apache.activemq.transport.AbstractInactivityMonitor.onCommand(AbstractInactivityMonitor.java:288) INFO | jvm 2 | 2013/08/27 09:17:17 | at org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:83) INFO | jvm 2 | 2013/08/27 09:17:17 | at org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:214) INFO | jvm 2 | 2013/08/27 09:17:17 | at org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:196) INFO | jvm 2 | 2013/08/27 09:17:17 | at java.lang.Thread.run(Unknown Source) 8) INFO | jvm 2 | 2013/08/27 09:17:17 | WARN | Transport Connection to: tcp://x.x.x.x:49129 failed: java.io.EOFException INFO | jvm 2 | 2013/08/27 09:18:03 | WARN | Transport Connection to: tcp://x.x.x.x:57520 failed: java.net.SocketException: Software caused connection abort: socket write error INFO | jvm 2 | 2013/08/27 09:35:03 | WARN | /admin/topics.jsp INFO | jvm 2 | 2013/08/27 09:35:03 | javax.el.ELException: java.lang.reflect.UndeclaredThrowableException INFO | jvm 2 | 2013/08/27 09:35:03 | at javax.el.BeanELResolver.getValue(BeanELResolver.java:298) INFO | jvm 2 | 2013/08/27 09:35:03 | at javax.el.CompositeELResolver.getValue(CompositeELResolver.java:175) INFO | jvm 2 | 2013/08/27 09:35:03 | at com.sun.el.parser.AstValue.getValue(AstValue.java:138) INFO | jvm 2 | 2013/08/27 09:35:03 | at com.sun.el.ValueExpressionImpl.getValue(ValueExpressionImpl.java:206) INFO | jvm 2 | 2013/08/27 09:35:03 | at org.apache.jasper.runtime.PageContextImpl.evaluateExpression(PageContextImpl.java:1001) INFO | jvm 2 | 2013/08/27 09:35:03 | at org.apache.jsp.topics_jsp._jspx_meth_c_param_0(org.apache.jsp.topics_jsp:265) INFO | jvm 2 | 2013/08/27 09:35:03 | at org.apache.jsp.topics_jsp._jspx_meth_c_url_0(org.apache.jsp.topics_jsp:234) INFO | jvm 2 | 2013/08/27 09:35:03 | at org.apache.jsp.topics_jsp._jspx_meth_c_forEach_0(org.apache.jsp.topics_jsp:168) INFO | jvm 2 | 2013/08/27 09:35:03 | at org.apache.jsp.topics_jsp._jspService(org.apache.jsp.topics_jsp:110) INFO | jvm 2 | 2013/08/27 09:35:03 | at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:109) INFO | jvm 2 | 2013/08/27 09:35:03 | at javax.servlet.http.HttpServlet.service(HttpServlet.java:806) INFO | jvm 2 | 2013/08/27 09:35:03 | at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:389) INFO | jvm 2 | 2013/08/27 09:35:03 | at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:486) INFO | jvm 2 | 2013/08/27 09:35:03 | at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:380) INFO | jvm 2 | 2013/08/27 09:35:03 | at javax.servlet.http.HttpServlet.service(HttpServlet.java:806) INFO | jvm 2 | 2013/08/27 09:35:03 | at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:652) INFO | jvm 2 | 2013/08/27 09:35:03 | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1329) INFO | jvm 2 | 2013/08/27 09:35:03 | at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:83) INFO | jvm 2 | 2013/08/27 09:35:03 | at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:76) INFO | jvm 2 | 2013/08/27 09:35:03 | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1300) INFO | jvm 2 | 2013/08/27 09:35:03 | at org.apache.activemq.web.SessionFilter.doFilter(SessionFilter.java:45) INFO | jvm 2 | 2013/08/27 09:35:03 | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1300) INFO | jvm 2 | 2013/08/27 09:35:03 | at org.apache.activemq.web.filter.ApplicationContextFilter.doFilter(ApplicationContextFilter.java:102) INFO | jvm 2 | 2013/08/27 09:35:03 | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1300) INFO | jvm 2 | 2013/08/27 09:35:03 | at com.opensymphony.sitemesh.webapp.SiteMeshFilter.obtainContent(SiteMeshFilter.java:129) INFO | jvm 2 | 2013/08/27 09:35:03 | at com.opensymphony.sitemesh.webapp.SiteMeshFilter.doFilter(SiteMeshFilter.java:77) INFO | jvm 2 | 2013/08/27 09:35:03 | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1300) INFO | jvm 2 | 2013/08/27 09:35:03 | at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:445) INFO | jvm 2 | 2013/08/27 09:35:03 | at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137) INFO | jvm 2 | 2013/08/27 09:35:03 | at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:521) INFO | jvm 2 | 2013/08/27 09:35:03 | at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:227) INFO | jvm 2 | 2013/08/27 09:35:03 | at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1038) INFO | jvm 2 | 2013/08/27 09:35:03 | at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:374) INFO | jvm 2 | 2013/08/27 09:35:03 | at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:189) INFO | jvm 2 | 2013/08/27 09:35:03 | at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:972) INFO | jvm 2 | 2013/08/27 09:35:03 | at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135) INFO | jvm 2 | 2013/08/27 09:35:03 | at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154) INFO | jvm 2 | 2013/08/27 09:35:03 | at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:521) INFO | jvm 2 | 2013/08/27 09:35:03 | at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154) INFO | jvm 2 | 2013/08/27 09:35:03 | at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116) INFO | jvm 2 | 2013/08/27 09:35:03 | at org.eclipse.jetty.server.Server.handle(Server.java:363) INFO | jvm 2 | 2013/08/27 09:35:03 | at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:483) INFO | jvm 2 | 2013/08/27 09:35:03 | at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:920) INFO | jvm 2 | 2013/08/27 09:35:03 | at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:982) INFO | jvm 2 | 2013/08/27 09:35:03 | at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:635) INFO | jvm 2 | 2013/08/27 09:35:03 | at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235) INFO | jvm 2 | 2013/08/27 09:35:03 | at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82) INFO | jvm 2 | 2013/08/27 09:35:03 | at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:627) INFO | jvm 2 | 2013/08/27 09:35:03 | at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:51) INFO | jvm 2 | 2013/08/27 09:35:03 | at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608) INFO | jvm 2 | 2013/08/27 09:35:03 | at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543) INFO | jvm 2 | 2013/08/27 09:35:03 | at java.lang.Thread.run(Unknown Source) INFO | jvm 2 | 2013/08/27 09:35:03 | Caused by: java.lang.reflect.UndeclaredThrowableException INFO | jvm 2 | 2013/08/27 09:35:03 | at com.sun.proxy.$Proxy6.getName(Unknown Source) INFO | jvm 2 | 2013/08/27 09:35:03 | at sun.reflect.GeneratedMethodAccessor40.invoke(Unknown Source) INFO | jvm 2 | 2013/08/27 09:35:03 | at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) INFO | jvm 2 | 2013/08/27 09:35:03 | at java.lang.reflect.Method.invoke(Unknown Source) INFO | jvm 2 | 2013/08/27 09:35:03 | at javax.el.BeanELResolver.getValue(BeanELResolver.java:293) INFO | jvm 2 | 2013/08/27 09:35:03 | ... 51 more INFO | jvm 2 | 2013/08/27 09:35:03 | Caused by: javax.management.InstanceNotFoundException: org.apache.activemq:type=Broker,brokerName=dtresna9shyp029.datatree.local,destinationType=Topic,destinationName=ActiveMQ.Advisory.Consumer.Queue.x-3b0e195e-b400-420c-8d30-1c641b6dffa8 INFO | jvm 2 | 2013/08/27 09:35:03 | at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getMBean(Unknown Source) INFO | jvm 2 | 2013/08/27 09:35:03 | at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(Unknown Source) INFO | jvm 2 | 2013/08/27 09:35:03 | at com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(Unknown Source) INFO | jvm 2 | 2013/08/27 09:35:03 | at javax.management.MBeanServerInvocationHandler.invoke(Unknown Source) INFO | jvm 2 | 2013/08/27 09:35:03 | ... 56 more 9) also gettign some about: INFO | jvm 2 | 2013/08/27 09:39:29 | java.lang.UnsupportedOperationException: Possible CSRF attack What in the world is going on here? -- View this message in context: http://activemq.2283324.n4.nabble.com/Can-you-help-with-these-exceptions-tp4670752.html Sent from the ActiveMQ - User mailing list archive at Nabble.com.
