I use the API heavily for automation purposes. While load starts to increase on the application server, we reach a breaking point where Spacewalk becomes unresponsive, and throws 500 internal server errors. Listed at the bottom are some of the errors I see when this happens. I'm looking for advice on how to better diagnose and\or tune my settings to optimize Tomcat performance. I feel like the system is beefy enough to handle this load, but Tomcat appears to be my bottleneck.. How should I go about resolving this? I have played around with the AJP connector settings quite a bit, but can't seem to find suitable parameters.
Spacewalk 2.2 OS: RHEL6.6 RAM: 64GB (Please note, under heavy load, we are not utilizing even 50% of RAM.. it seems to be all CPU, but I'm not sure how to get Tomcat to utilize more RAM other than boosting the Xmx settings which I have already done) CPU: 16 (vCPU) Tomcat: apache-tomcat-apis-0.1-1.el6.noarch tomcat6-el-2.1-api-6.0.24-83.el6_6.x86_64 tomcat6-lib-6.0.24-83.el6_6.x86_64 tomcat6-servlet-2.5-api-6.0.24-83.el6_6.x86_64 tomcat6-6.0.24-83.el6_6.x86_64 tomcat5-jsp-2.0-api-5.5.27-7.jpp5.noarch tomcat6-jsp-2.1-api-6.0.24-83.el6_6.x86_64 tomcat5-servlet-2.4-api-5.5.27-7.jpp5.noarch Java: java-1.6.0-openjdk-1.6.0.36-1.13.8.1.el6_7.x86_64 Oracle client: oracle-instantclient11.2-basic-11.2.0.3.0-1.x86_64 JAVA_OPTS="-ea -Xms512m -Xmx4096m -Djava.awt.headless=true -Dorg.xml.sax.driver=org.apache.xerces.parsers.SAXParser -Dorg.apache.tomcat.util.http.Parameters.MAX_COUNT=1024 -XX:MaxNewSize=256 -XX:-UseConcMarkSweepGC -Dnet.sf.ehcache.skipUpdateCheck=true -Djavax.sql.DataSource.Factory=org.apache.commons.dbcp.BasicDataSourceFactory -Djava.library.path=\${system_property:java.library.path}:/usr/lib:/usr/lib64/oracle/11.2/client/lib" Connector settings: <Connector port="8080" protocol="HTTP/1.1" connectionTimeout="20000" redirectPort="8443" URIEncoding="UTF-8" address="127.0.0.1" maxThreads="1024" maxKeepAliveRequests="1000"/> <!-- Define an AJP 1.3 Connector on port 8009 --> <Connector port="8009" protocol="AJP/1.3" redirectPort="8443" URIEncoding="UTF-8" address="127.0.0.1" maxThreads="1024"/> <Connector port="8009" protocol="AJP/1.3" redirectPort="8443" URIEncoding="UTF-8" address="::1" maxThreads="1024"/> >From /var/log/tomcat6/catalina.out Aug 25, 2015 1:33:23 AM org.apache.jk.core.MsgContext action WARNING: Unable to send headers java.net.SocketException: Broken pipe at java.net.SocketOutputStream.socketWrite0(Native Method) at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109) at java.net.SocketOutputStream.write(SocketOutputStream.java:153) at org.apache.jk.common.ChannelSocket.send(ChannelSocket.java:532) at org.apache.jk.common.JkInputStream.appendHead(JkInputStream.java:326) at org.apache.jk.core.MsgContext.action(MsgContext.java:266) at org.apache.coyote.Response.action(Response.java:183) at org.apache.coyote.Response.sendHeaders(Response.java:379) at org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:305) at org.apache.catalina.connector.OutputBuffer.flush(OutputBuffer.java:288) at org.apache.catalina.connector.CoyoteWriter.flush(CoyoteWriter.java:95) at org.apache.jasper.runtime.JspWriterImpl.flush(JspWriterImpl.java:175) at org.apache.jasper.runtime.JspRuntimeLibrary.include(JspRuntimeLibrary.java:956) at org.apache.jasper.runtime.PageContextImpl.include(PageContextImpl.java:622) at com.opensymphony.module.sitemesh.taglib.page.ApplyDecoratorTag.doEndTag(ApplyDecoratorTag.java:258) at org.apache.jsp.WEB_002dINF.pages.common.errors._500_jsp._jspx_meth_page_005fapplyDecorator_005f0(Unknown Source) at org.apache.jsp.WEB_002dINF.pages.common.errors._500_jsp._jspService(Unknown Source) at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70) at javax.servlet.http.HttpServlet.service(HttpServlet.java:717) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:646) at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:438) at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:374) at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:302) at org.apache.catalina.core.StandardHostValve.custom(StandardHostValve.java:415) at org.apache.catalina.core.StandardHostValve.status(StandardHostValve.java:342) at org.apache.catalina.core.StandardHostValve.throwable(StandardHostValve.java:286) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:141) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298) at org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:190) at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:291) at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:769) at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:698) at org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:891) at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:690) at java.lang.Thread.run(Thread.java:701) SEVERE: Exception initializing page context java.lang.IllegalStateException: Cannot create a session after the response has been committed at org.apache.catalina.connector.Request.doGetSession(Request.java:2402) at org.apache.catalina.connector.Request.getSession(Request.java:2119) at org.apache.catalina.connector.RequestFacade.getSession(RequestFacade.java:833) at javax.servlet.http.HttpServletRequestWrapper.getSession(HttpServletRequestWrapper.java:216) at org.apache.catalina.core.ApplicationHttpRequest.getSession(ApplicationHttpRequest.java:544) at javax.servlet.http.HttpServletRequestWrapper.getSession(HttpServletRequestWrapper.java:216) at org.apache.catalina.core.ApplicationHttpRequest.getSession(ApplicationHttpRequest.java:544) at javax.servlet.http.HttpServletRequestWrapper.getSession(HttpServletRequestWrapper.java:216) at org.apache.catalina.core.ApplicationHttpRequest.getSession(ApplicationHttpRequest.java:544) at org.apache.catalina.core.ApplicationHttpRequest.getSession(ApplicationHttpRequest.java:493) at org.apache.jasper.runtime.PageContextImpl._initialize(PageContextImpl.java:146) at org.apache.jasper.runtime.PageContextImpl.initialize(PageContextImpl.java:124) at org.apache.jasper.runtime.JspFactoryImpl.internalGetPageContext(JspFactoryImpl.java:107) at org.apache.jasper.runtime.JspFactoryImpl.getPageContext(JspFactoryImpl.java:63) at org.apache.jsp.WEB_002dINF.includes.footer_jsp._jspService(Unknown Source) at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70) at javax.servlet.http.HttpServlet.service(HttpServlet.java:717) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:646) at org.apache.catalina.core.ApplicationDispatcher.doInclude(ApplicationDispatcher.java:551) at org.apache.catalina.core.ApplicationDispatcher.include(ApplicationDispatcher.java:488) at org.apache.jasper.runtime.JspRuntimeLibrary.include(JspRuntimeLibrary.java:968) at org.apache.jsp.WEB_002dINF.decorators.layout_005fc_jsp._jspService(Unknown Source) at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70) at javax.servlet.http.HttpServlet.service(HttpServlet.java:717) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:646) at org.apache.catalina.core.ApplicationDispatcher.doInclude(ApplicationDispatcher.java:551) at org.apache.catalina.core.ApplicationDispatcher.include(ApplicationDispatcher.java:488) at org.apache.jasper.runtime.JspRuntimeLibrary.include(JspRuntimeLibrary.java:968) at org.apache.jasper.runtime.PageContextImpl.include(PageContextImpl.java:622) at com.opensymphony.module.sitemesh.taglib.page.ApplyDecoratorTag.doEndTag(ApplyDecoratorTag.java:258) at org.apache.jsp.WEB_002dINF.pages.common.errors._500_jsp._jspx_meth_page_005fapplyDecorator_005f0(Unknown Source) at org.apache.jsp.WEB_002dINF.pages.common.errors._500_jsp._jspService(Unknown Source) at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70) at javax.servlet.http.HttpServlet.service(HttpServlet.java:717) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:646) at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:438) at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:374) at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:302) at org.apache.catalina.core.StandardHostValve.custom(StandardHostValve.java:415) at org.apache.catalina.core.StandardHostValve.status(StandardHostValve.java:342) at org.apache.catalina.core.StandardHostValve.throwable(StandardHostValve.java:286) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:141) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298) at org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:190) at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:291) at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:769) at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:698) at org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:891) at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:690) at java.lang.Thread.run(Thread.java:701) I also see lots of AJP timeouts in the apache error log [Tue Aug 25 04:21:18 2015] [error] (70007)The timeout specified has expired: proxy: read response failed from (null) (localhost) [Tue Aug 25 04:21:18 2015] [error] (70007)The timeout specified has expired: ajp_ilink_receive() can't receive header [Tue Aug 25 04:21:18 2015] [error] ajp_read_header: ajp_ilink_receive failed [Tue Aug 25 04:21:18 2015] [error] (70007)The timeout specified has expired: proxy: read response failed from (null) (localhost) [Tue Aug 25 04:21:18 2015] [error] (70007)The timeout specified has expired: ajp_ilink_receive() can't receive header [Tue Aug 25 04:21:18 2015] [error] ajp_read_header: ajp_ilink_receive failed [Tue Aug 25 04:21:18 2015] [error] (70007)The timeout specified has expired: proxy: read response failed from (null) (localhost) [Tue Aug 25 04:21:18 2015] [error] (70007)The timeout specified has expired: ajp_ilink_receive() can't receive header [Tue Aug 25 04:21:18 2015] [error] ajp_read_header: ajp_ilink_receive failed
_______________________________________________ Spacewalk-list mailing list Spacewalk-list@redhat.com https://www.redhat.com/mailman/listinfo/spacewalk-list