Hi I have a production web application running in Tomcat. The web application uses struts 2 as MVC layer.
We had an issue wherein one of the web servers spiked to 100% cpu usage. This issue lasted for over several hours. I took the thread dump and saw over several hundred threads in runnable state and the dumps show the following stack trace for most of the threads. "TP-Processor480" daemon prio=10 tid=0x00002aab8c1ff400 nid=0xf23 runnable [0x00000000656af000..0x00000000656b3d90] java.lang.Thread.State: RUNNABLE at java.util.HashMap.get(HashMap.java:303) at com.opensymphony.xwork2.util.LocalizedTextUtil.buildMessageFormat(LocalizedTextUtil.java:620) at com.opensymphony.xwork2.util.LocalizedTextUtil.getDefaultMessage(LocalizedTextUtil.java:588) at com.opensymphony.xwork2.util.LocalizedTextUtil.findText(LocalizedTextUtil.java:461) at com.opensymphony.xwork2.TextProviderSupport.getText(TextProviderSupport.java:224) at com.opensymphony.xwork2.ActionSupport.getText(ActionSupport.java:99) at org.apache.struts2.components.Text.end(Text.java:158) at org.apache.struts2.views.jsp.ComponentTagSupport.doEndTag(ComponentTagSupport.java:43) at org.apache.jsp.tag.web.tabs_tag._jspx_meth_s_005ftext_005f2(tabs_tag.java:565) at org.apache.jsp.tag.web.tabs_tag._jspx_meth_s_005fif_005f0(tabs_tag.java:280) at org.apache.jsp.tag.web.tabs_tag.doTag(tabs_tag.java:113) at org.apache.jsp.myjsp_jsp._jspService(myjsp_jsp.java:379) at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:98) at javax.servlet.http.HttpServlet.service(HttpServlet.java:729) at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:331) at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:329) at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:265) at javax.servlet.http.HttpServlet.service(HttpServlet.java:729) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188) at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:679) at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:461) at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:399) at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:301) at org.apache.struts2.dispatcher.ServletDispatcherResult.doExecute(ServletDispatcherResult.java:139) at org.apache.struts2.dispatcher.StrutsResultSupport.execute(StrutsResultSupport.java:178) at com.opensymphony.xwork2.DefaultActionInvocation.executeResult(DefaultActionInvocation.java:348) at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:253) at com.opensymphony.xwork2.interceptor.DefaultWorkflowInterceptor.doIntercept(DefaultWorkflowInterceptor.java:221) at com.opensymphony.xwork2.interceptor.MethodFilterInterceptor.intercept(MethodFilterInterceptor.java:86) at com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:224) at com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:223) at com.opensymphony.xwork2.util.profiling.UtilTimerStack.profile(UtilTimerStack.java:455) at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:221) at com.opensymphony.xwork2.validator.ValidationInterceptor.doIntercept(ValidationInterceptor.java:150) at org.apache.struts2.interceptor.validation.AnnotationValidationInterceptor.doIntercept(AnnotationValidationInterceptor.java:48) at com.opensymphony.xwork2.interceptor.MethodFilterInterceptor.intercept(MethodFilterInterceptor.java:86) at com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:224) at com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:223) at com.opensymphony.xwork2.util.profiling.UtilTimerStack.profile(UtilTimerStack.java:455) at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:221) Now the code in question uses struts s:text tag and just fetches the value from a property file. I am not sure why so many threads should get stuck over there (threads are in runnable state).java.util.HashMap is not threadsafe, and if accessed concurrently by multiple threads, one particular mode of failure is an infinite loop within a particular bucket<http://mailinator.blogspot.com/2009/06/beautiful-race-condition.html>, which could be causing this issue. This issue happens only once after our deployment, and if we restart the server it will never occur. https://issues.apache.org/jira/browse/ww-2808 mentions that using s:text could have performance issues. They mentioned configuring freemarker cache could improve the performance. We are using Struts 2.0.11 and don't use freemarker template. Will updating the Struts to latest version and switching to freemarker template could help us in resolving this issue? (as freemarker template can be configured to use a internal ConcurrentHashMap). I never used freemarker template and we don't want to shift to this unless we are sure that this could fix the issue. Did anyone experienced such issue before. Thanks Abhilash