Hi
I have a production web application running in Tomcat. The web application
uses struts 2 as MVC layer.
We had an issue wherein one of the web servers spiked to 100% cpu usage.
This issue lasted for over several hours. I took the thread dump and saw
over several hundred threads in runnable state and the dumps show the
following stack trace for most of the threads.
"TP-Processor480" daemon prio=10 tid=0x00002aab8c1ff400 nid=0xf23 runnable
[0x00000000656af000..0x00000000656b3d90]
java.lang.Thread.State: RUNNABLE
at java.util.HashMap.get(HashMap.java:303)
at
com.opensymphony.xwork2.util.LocalizedTextUtil.buildMessageFormat(LocalizedTextUtil.java:620)
at
com.opensymphony.xwork2.util.LocalizedTextUtil.getDefaultMessage(LocalizedTextUtil.java:588)
at
com.opensymphony.xwork2.util.LocalizedTextUtil.findText(LocalizedTextUtil.java:461)
at
com.opensymphony.xwork2.TextProviderSupport.getText(TextProviderSupport.java:224)
at
com.opensymphony.xwork2.ActionSupport.getText(ActionSupport.java:99)
at org.apache.struts2.components.Text.end(Text.java:158)
at
org.apache.struts2.views.jsp.ComponentTagSupport.doEndTag(ComponentTagSupport.java:43)
at
org.apache.jsp.tag.web.tabs_tag._jspx_meth_s_005ftext_005f2(tabs_tag.java:565)
at
org.apache.jsp.tag.web.tabs_tag._jspx_meth_s_005fif_005f0(tabs_tag.java:280)
at org.apache.jsp.tag.web.tabs_tag.doTag(tabs_tag.java:113)
at org.apache.jsp.myjsp_jsp._jspService(myjsp_jsp.java:379)
at
org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:98)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
at
org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:331)
at
org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:329)
at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:265)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
at
org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:679)
at
org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:461)
at
org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:399)
at
org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:301)
at
org.apache.struts2.dispatcher.ServletDispatcherResult.doExecute(ServletDispatcherResult.java:139)
at
org.apache.struts2.dispatcher.StrutsResultSupport.execute(StrutsResultSupport.java:178)
at
com.opensymphony.xwork2.DefaultActionInvocation.executeResult(DefaultActionInvocation.java:348)
at
com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:253)
at
com.opensymphony.xwork2.interceptor.DefaultWorkflowInterceptor.doIntercept(DefaultWorkflowInterceptor.java:221)
at
com.opensymphony.xwork2.interceptor.MethodFilterInterceptor.intercept(MethodFilterInterceptor.java:86)
at
com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:224)
at
com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:223)
at
com.opensymphony.xwork2.util.profiling.UtilTimerStack.profile(UtilTimerStack.java:455)
at
com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:221)
at
com.opensymphony.xwork2.validator.ValidationInterceptor.doIntercept(ValidationInterceptor.java:150)
at
org.apache.struts2.interceptor.validation.AnnotationValidationInterceptor.doIntercept(AnnotationValidationInterceptor.java:48)
at
com.opensymphony.xwork2.interceptor.MethodFilterInterceptor.intercept(MethodFilterInterceptor.java:86)
at
com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:224)
at
com.opensymphony.xwork2.DefaultActionInvocation$2.doProfiling(DefaultActionInvocation.java:223)
at
com.opensymphony.xwork2.util.profiling.UtilTimerStack.profile(UtilTimerStack.java:455)
at
com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:221)
Now the code in question uses struts s:text tag and just fetches the value
from a property file. I am not sure why so many threads should get stuck
over there (threads are in runnable state).java.util.HashMap is not
threadsafe, and if accessed concurrently by multiple threads, one particular
mode of failure is an infinite loop within a particular
bucket<http://mailinator.blogspot.com/2009/06/beautiful-race-condition.html>,
which could be causing this issue.
This issue happens only once after our deployment, and if we restart the
server it will never occur.
https://issues.apache.org/jira/browse/ww-2808 mentions that using s:text
could have performance issues. They mentioned configuring freemarker cache
could improve the performance.
We are using Struts 2.0.11 and don't use freemarker template. Will updating
the Struts to latest version and switching to freemarker template could help
us in resolving this issue? (as freemarker template can be configured to
use a internal ConcurrentHashMap).
I never used freemarker template and we don't want to shift to this unless
we are sure that this could fix the issue.
Did anyone experienced such issue before.
Thanks
Abhilash
