Are you having network issues? Or are there firewalls in between client and broker?
On Tue, Aug 27, 2013 at 9:55 AM, SledgeHammer <gro...@firstam.com> wrote: > Having lots of stability issues (5.8.0 on Win64 2008 Server 64GB RAM). > Messages are IBytesMessages, typically around 400k. Typically 3 to 4 queues > running at once. 200 consumers. Checked the log and seeing a TON of > exceptions: > > 1) INFO | jvm 1 | 2013/08/26 15:09:32 | WARN | Transport Connection > to: tcp://x.x.x.x:59339 failed: java.net.SocketException: Software caused > connection abort: socket write error > 2) INFO | jvm 1 | 2013/08/26 15:09:33 | WARN | Transport Connection > to: tcp://x.x.x.x:59700 failed: java.io.EOFException > 3) INFO | jvm 1 | 2013/08/26 15:09:33 | WARN | Transport Connection > to: tcp://x.x.x.x:62316 failed: java.net.SocketException: Software caused > connection abort: recv failed > 4) INFO | jvm 1 | 2013/08/26 15:09:33 | WARN | Transport Connection > to: tcp://x.x.x.x:62316 failed: > org.apache.activemq.transport.InactivityIOException: Cannot send, channel > has already failed: tcp://x.x.x.x:62316 > > Hundreds / Thousands of these for various IPs and ports. > > 5) INFO | jvm 1 | 2013/08/26 16:05:48 | WARN | Async error occurred: > javax.jms.JMSException: Destination still has an active subscription: > queue://x-caf311ac-edb9-4ff1-8d7b-890d320cece8 > INFO | jvm 1 | 2013/08/26 16:05:48 | javax.jms.JMSException: > Destination still has an active subscription: > queue://x-caf311ac-edb9-4ff1-8d7b-890d320cece8 > INFO | jvm 1 | 2013/08/26 16:05:48 | at > > org.apache.activemq.broker.region.AbstractRegion.removeDestination(AbstractRegion.java:186) > INFO | jvm 1 | 2013/08/26 16:05:48 | at > > org.apache.activemq.broker.jmx.ManagedQueueRegion.removeDestination(ManagedQueueRegion.java:62) > INFO | jvm 1 | 2013/08/26 16:05:48 | at > > org.apache.activemq.broker.region.RegionBroker.removeDestination(RegionBroker.java:288) > INFO | jvm 1 | 2013/08/26 16:05:48 | at > > org.apache.activemq.broker.region.RegionBroker.removeDestinationInfo(RegionBroker.java:301) > INFO | jvm 1 | 2013/08/26 16:05:48 | at > > org.apache.activemq.broker.BrokerFilter.removeDestinationInfo(BrokerFilter.java:221) > INFO | jvm 1 | 2013/08/26 16:05:48 | at > > org.apache.activemq.advisory.AdvisoryBroker.removeDestinationInfo(AdvisoryBroker.java:223) > INFO | jvm 1 | 2013/08/26 16:05:48 | at > > org.apache.activemq.broker.BrokerFilter.removeDestinationInfo(BrokerFilter.java:221) > INFO | jvm 1 | 2013/08/26 16:05:48 | at > > org.apache.activemq.broker.BrokerFilter.removeDestinationInfo(BrokerFilter.java:221) > INFO | jvm 1 | 2013/08/26 16:05:48 | at > > org.apache.activemq.broker.MutableBrokerFilter.removeDestinationInfo(MutableBrokerFilter.java:228) > INFO | jvm 1 | 2013/08/26 16:05:48 | at > > org.apache.activemq.broker.TransportConnection.processRemoveDestination(TransportConnection.java:537) > INFO | jvm 1 | 2013/08/26 16:05:48 | at > org.apache.activemq.command.DestinationInfo.visit(DestinationInfo.java:124) > INFO | jvm 1 | 2013/08/26 16:05:48 | at > > org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:329) > INFO | jvm 1 | 2013/08/26 16:05:48 | at > > org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:184) > INFO | jvm 1 | 2013/08/26 16:05:48 | at > > org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50) > INFO | jvm 1 | 2013/08/26 16:05:48 | at > > org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:113) > INFO | jvm 1 | 2013/08/26 16:05:48 | at > > org.apache.activemq.transport.AbstractInactivityMonitor.onCommand(AbstractInactivityMonitor.java:288) > INFO | jvm 1 | 2013/08/26 16:05:48 | at > > org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:83) > INFO | jvm 1 | 2013/08/26 16:05:48 | at > org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:214) > INFO | jvm 1 | 2013/08/26 16:05:48 | at > org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:196) > INFO | jvm 1 | 2013/08/26 16:05:48 | at > java.lang.Thread.run(Unknown > Source) > > 6) > INFO | jvm 2 | 2013/08/27 09:17:17 | WARN | Async error occurred: > java.lang.IllegalStateException: Timer already cancelled. > INFO | jvm 2 | 2013/08/27 09:17:17 | java.lang.IllegalStateException: > Timer already cancelled. > INFO | jvm 2 | 2013/08/27 09:17:17 | at > java.util.Timer.sched(Unknown > Source) > INFO | jvm 2 | 2013/08/27 09:17:17 | at > java.util.Timer.schedule(Unknown Source) > INFO | jvm 2 | 2013/08/27 09:17:17 | at > > org.apache.activemq.thread.Scheduler.schedualPeriodically(Scheduler.java:50) > INFO | jvm 2 | 2013/08/27 09:17:17 | at > org.apache.activemq.broker.region.Queue.start(Queue.java:889) > INFO | jvm 2 | 2013/08/27 09:17:17 | at > > org.apache.activemq.broker.region.AbstractRegion.addDestination(AbstractRegion.java:142) > INFO | jvm 2 | 2013/08/27 09:17:17 | at > > org.apache.activemq.broker.region.RegionBroker.addDestination(RegionBroker.java:277) > INFO | jvm 2 | 2013/08/27 09:17:17 | at > > org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145) > INFO | jvm 2 | 2013/08/27 09:17:17 | at > > org.apache.activemq.advisory.AdvisoryBroker.addDestination(AdvisoryBroker.java:174) > INFO | jvm 2 | 2013/08/27 09:17:17 | at > > org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145) > INFO | jvm 2 | 2013/08/27 09:17:17 | at > > org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145) > INFO | jvm 2 | 2013/08/27 09:17:17 | at > > org.apache.activemq.broker.MutableBrokerFilter.addDestination(MutableBrokerFilter.java:151) > INFO | jvm 2 | 2013/08/27 09:17:17 | at > > org.apache.activemq.broker.region.RegionBroker.addProducer(RegionBroker.java:324) > INFO | jvm 2 | 2013/08/27 09:17:17 | at > > org.apache.activemq.broker.jmx.ManagedRegionBroker.addProducer(ManagedRegionBroker.java:252) > INFO | jvm 2 | 2013/08/27 09:17:17 | at > org.apache.activemq.broker.BrokerFilter.addProducer(BrokerFilter.java:93) > INFO | jvm 2 | 2013/08/27 09:17:17 | at > > org.apache.activemq.advisory.AdvisoryBroker.addProducer(AdvisoryBroker.java:162) > INFO | jvm 2 | 2013/08/27 09:17:17 | at > > org.apache.activemq.broker.CompositeDestinationBroker.addProducer(CompositeDestinationBroker.java:56) > INFO | jvm 2 | 2013/08/27 09:17:17 | at > org.apache.activemq.broker.BrokerFilter.addProducer(BrokerFilter.java:93) > INFO | jvm 2 | 2013/08/27 09:17:17 | at > > org.apache.activemq.broker.MutableBrokerFilter.addProducer(MutableBrokerFilter.java:99) > INFO | jvm 2 | 2013/08/27 09:17:17 | at > > org.apache.activemq.broker.TransportConnection.processAddProducer(TransportConnection.java:566) > INFO | jvm 2 | 2013/08/27 09:17:17 | at > org.apache.activemq.command.ProducerInfo.visit(ProducerInfo.java:105) > INFO | jvm 2 | 2013/08/27 09:17:17 | at > > org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:329) > INFO | jvm 2 | 2013/08/27 09:17:17 | at > > org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:184) > INFO | jvm 2 | 2013/08/27 09:17:17 | at > > org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50) > INFO | jvm 2 | 2013/08/27 09:17:17 | at > > org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:113) > INFO | jvm 2 | 2013/08/27 09:17:17 | at > > org.apache.activemq.transport.AbstractInactivityMonitor.onCommand(AbstractInactivityMonitor.java:288) > INFO | jvm 2 | 2013/08/27 09:17:17 | at > > org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:83) > INFO | jvm 2 | 2013/08/27 09:17:17 | at > org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:214) > INFO | jvm 2 | 2013/08/27 09:17:17 | at > org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:196) > INFO | jvm 2 | 2013/08/27 09:17:17 | at > java.lang.Thread.run(Unknown > Source) > > 7) INFO | jvm 2 | 2013/08/27 09:17:17 | WARN | Async error occurred: > java.lang.IllegalStateException: Cannot remove a consumer that had not been > registered: ID:x-w01-49130-635131918368230097-1:0:1:1 > INFO | jvm 2 | 2013/08/27 09:17:17 | java.lang.IllegalStateException: > Cannot remove a consumer that had not been registered: > ID:x-w01-49130-635131918368230097-1:0:1:1 > INFO | jvm 2 | 2013/08/27 09:17:17 | at > > org.apache.activemq.broker.TransportConnection.processRemoveConsumer(TransportConnection.java:647) > INFO | jvm 2 | 2013/08/27 09:17:17 | at > org.apache.activemq.command.RemoveInfo.visit(RemoveInfo.java:76) > INFO | jvm 2 | 2013/08/27 09:17:17 | at > > org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:329) > INFO | jvm 2 | 2013/08/27 09:17:17 | at > > org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:184) > INFO | jvm 2 | 2013/08/27 09:17:17 | at > > org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50) > INFO | jvm 2 | 2013/08/27 09:17:17 | at > > org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:113) > INFO | jvm 2 | 2013/08/27 09:17:17 | at > > org.apache.activemq.transport.AbstractInactivityMonitor.onCommand(AbstractInactivityMonitor.java:288) > INFO | jvm 2 | 2013/08/27 09:17:17 | at > > org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:83) > INFO | jvm 2 | 2013/08/27 09:17:17 | at > org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:214) > INFO | jvm 2 | 2013/08/27 09:17:17 | at > org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:196) > INFO | jvm 2 | 2013/08/27 09:17:17 | at > java.lang.Thread.run(Unknown > Source) > > 8) INFO | jvm 2 | 2013/08/27 09:17:17 | WARN | Transport Connection > to: tcp://x.x.x.x:49129 failed: java.io.EOFException > INFO | jvm 2 | 2013/08/27 09:18:03 | WARN | Transport Connection to: > tcp://x.x.x.x:57520 failed: java.net.SocketException: Software caused > connection abort: socket write error > INFO | jvm 2 | 2013/08/27 09:35:03 | WARN | /admin/topics.jsp > INFO | jvm 2 | 2013/08/27 09:35:03 | javax.el.ELException: > java.lang.reflect.UndeclaredThrowableException > INFO | jvm 2 | 2013/08/27 09:35:03 | at > javax.el.BeanELResolver.getValue(BeanELResolver.java:298) > INFO | jvm 2 | 2013/08/27 09:35:03 | at > javax.el.CompositeELResolver.getValue(CompositeELResolver.java:175) > INFO | jvm 2 | 2013/08/27 09:35:03 | at > com.sun.el.parser.AstValue.getValue(AstValue.java:138) > INFO | jvm 2 | 2013/08/27 09:35:03 | at > com.sun.el.ValueExpressionImpl.getValue(ValueExpressionImpl.java:206) > INFO | jvm 2 | 2013/08/27 09:35:03 | at > > org.apache.jasper.runtime.PageContextImpl.evaluateExpression(PageContextImpl.java:1001) > INFO | jvm 2 | 2013/08/27 09:35:03 | at > > org.apache.jsp.topics_jsp._jspx_meth_c_param_0(org.apache.jsp.topics_jsp:265) > INFO | jvm 2 | 2013/08/27 09:35:03 | at > org.apache.jsp.topics_jsp._jspx_meth_c_url_0(org.apache.jsp.topics_jsp:234) > INFO | jvm 2 | 2013/08/27 09:35:03 | at > > org.apache.jsp.topics_jsp._jspx_meth_c_forEach_0(org.apache.jsp.topics_jsp:168) > INFO | jvm 2 | 2013/08/27 09:35:03 | at > org.apache.jsp.topics_jsp._jspService(org.apache.jsp.topics_jsp:110) > INFO | jvm 2 | 2013/08/27 09:35:03 | at > org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:109) > INFO | jvm 2 | 2013/08/27 09:35:03 | at > javax.servlet.http.HttpServlet.service(HttpServlet.java:806) > INFO | jvm 2 | 2013/08/27 09:35:03 | at > > org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:389) > INFO | jvm 2 | 2013/08/27 09:35:03 | at > org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:486) > INFO | jvm 2 | 2013/08/27 09:35:03 | at > org.apache.jasper.servlet.JspServlet.service(JspServlet.java:380) > INFO | jvm 2 | 2013/08/27 09:35:03 | at > javax.servlet.http.HttpServlet.service(HttpServlet.java:806) > INFO | jvm 2 | 2013/08/27 09:35:03 | at > org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:652) > INFO | jvm 2 | 2013/08/27 09:35:03 | at > > org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1329) > INFO | jvm 2 | 2013/08/27 09:35:03 | at > > org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:83) > INFO | jvm 2 | 2013/08/27 09:35:03 | at > > org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:76) > INFO | jvm 2 | 2013/08/27 09:35:03 | at > > org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1300) > INFO | jvm 2 | 2013/08/27 09:35:03 | at > org.apache.activemq.web.SessionFilter.doFilter(SessionFilter.java:45) > INFO | jvm 2 | 2013/08/27 09:35:03 | at > > org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1300) > INFO | jvm 2 | 2013/08/27 09:35:03 | at > > org.apache.activemq.web.filter.ApplicationContextFilter.doFilter(ApplicationContextFilter.java:102) > INFO | jvm 2 | 2013/08/27 09:35:03 | at > > org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1300) > INFO | jvm 2 | 2013/08/27 09:35:03 | at > > com.opensymphony.sitemesh.webapp.SiteMeshFilter.obtainContent(SiteMeshFilter.java:129) > INFO | jvm 2 | 2013/08/27 09:35:03 | at > > com.opensymphony.sitemesh.webapp.SiteMeshFilter.doFilter(SiteMeshFilter.java:77) > INFO | jvm 2 | 2013/08/27 09:35:03 | at > > org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1300) > INFO | jvm 2 | 2013/08/27 09:35:03 | at > org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:445) > INFO | jvm 2 | 2013/08/27 09:35:03 | at > > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137) > INFO | jvm 2 | 2013/08/27 09:35:03 | at > org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:521) > INFO | jvm 2 | 2013/08/27 09:35:03 | at > > org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:227) > INFO | jvm 2 | 2013/08/27 09:35:03 | at > > org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1038) > INFO | jvm 2 | 2013/08/27 09:35:03 | at > org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:374) > INFO | jvm 2 | 2013/08/27 09:35:03 | at > > org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:189) > INFO | jvm 2 | 2013/08/27 09:35:03 | at > > org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:972) > INFO | jvm 2 | 2013/08/27 09:35:03 | at > > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135) > INFO | jvm 2 | 2013/08/27 09:35:03 | at > > org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154) > INFO | jvm 2 | 2013/08/27 09:35:03 | at > org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:521) > INFO | jvm 2 | 2013/08/27 09:35:03 | at > > org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154) > INFO | jvm 2 | 2013/08/27 09:35:03 | at > > org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116) > INFO | jvm 2 | 2013/08/27 09:35:03 | at > org.eclipse.jetty.server.Server.handle(Server.java:363) > INFO | jvm 2 | 2013/08/27 09:35:03 | at > > org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:483) > INFO | jvm 2 | 2013/08/27 09:35:03 | at > > org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:920) > INFO | jvm 2 | 2013/08/27 09:35:03 | at > > org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:982) > INFO | jvm 2 | 2013/08/27 09:35:03 | at > org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:635) > INFO | jvm 2 | 2013/08/27 09:35:03 | at > org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235) > INFO | jvm 2 | 2013/08/27 09:35:03 | at > > org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82) > INFO | jvm 2 | 2013/08/27 09:35:03 | at > > org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:627) > INFO | jvm 2 | 2013/08/27 09:35:03 | at > > org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:51) > INFO | jvm 2 | 2013/08/27 09:35:03 | at > > org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608) > INFO | jvm 2 | 2013/08/27 09:35:03 | at > > org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543) > INFO | jvm 2 | 2013/08/27 09:35:03 | at > java.lang.Thread.run(Unknown > Source) > INFO | jvm 2 | 2013/08/27 09:35:03 | Caused by: > java.lang.reflect.UndeclaredThrowableException > INFO | jvm 2 | 2013/08/27 09:35:03 | at > com.sun.proxy.$Proxy6.getName(Unknown Source) > INFO | jvm 2 | 2013/08/27 09:35:03 | at > sun.reflect.GeneratedMethodAccessor40.invoke(Unknown Source) > INFO | jvm 2 | 2013/08/27 09:35:03 | at > sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) > INFO | jvm 2 | 2013/08/27 09:35:03 | at > java.lang.reflect.Method.invoke(Unknown Source) > INFO | jvm 2 | 2013/08/27 09:35:03 | at > javax.el.BeanELResolver.getValue(BeanELResolver.java:293) > INFO | jvm 2 | 2013/08/27 09:35:03 | ... 51 more > INFO | jvm 2 | 2013/08/27 09:35:03 | Caused by: > javax.management.InstanceNotFoundException: > > org.apache.activemq:type=Broker,brokerName=dtresna9shyp029.datatree.local,destinationType=Topic,destinationName=ActiveMQ.Advisory.Consumer.Queue.x-3b0e195e-b400-420c-8d30-1c641b6dffa8 > INFO | jvm 2 | 2013/08/27 09:35:03 | at > com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getMBean(Unknown > Source) > INFO | jvm 2 | 2013/08/27 09:35:03 | at > com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(Unknown > Source) > INFO | jvm 2 | 2013/08/27 09:35:03 | at > com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(Unknown Source) > INFO | jvm 2 | 2013/08/27 09:35:03 | at > javax.management.MBeanServerInvocationHandler.invoke(Unknown Source) > INFO | jvm 2 | 2013/08/27 09:35:03 | ... 56 more > > 9) also gettign some about: > > INFO | jvm 2 | 2013/08/27 09:39:29 | > java.lang.UnsupportedOperationException: Possible CSRF attack > > What in the world is going on here? > > > > > -- > View this message in context: > http://activemq.2283324.n4.nabble.com/Can-you-help-with-these-exceptions-tp4670752.html > Sent from the ActiveMQ - User mailing list archive at Nabble.com. > -- *Christian Posta* http://www.christianposta.com/blog twitter: @christianposta
