-----Christopher Schultz <ch...@christopherschultz.net> wrote: -----
To: Tomcat Users List <users@tomcat.apache.org>
From: Christopher Schultz <ch...@christopherschultz.net>
Date: 04/24/2015 07:14PM
Subject: Re: Tomcat Thread issue
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
Felix,
On 4/24/15 3:19 AM, Felix Schumacher wrote:
> Am 24. April 2015 09:08:08 MESZ, schrieb Subhro Paul
> <subhro.p...@tcs.com>:
>>
>>
>> -----Subhro Paul <subhro.p...@tcs.com> wrote: ----- To:
>> users@tomcat.apache.org From: Subhro Paul <subhro.p...@tcs.com>
>> Date: 04/23/2015 06:20PM Subject: Re: Tomcat Thread issue
>>
>> -----Daniel Mikusa <dmik...@pivotal.io> wrote: ----- To: Tomcat
>> Users List <users@tomcat.apache.org> From: Daniel Mikusa
>> <dmik...@pivotal.io> Date: 04/23/2015 05:01PM Subject: Re: Tomcat
>> Thread issue
>>
>> On Thu, Apr 23, 2015 at 7:15 AM, Subhro Paul
>> <subhro.p...@tcs.com> wrote:
>>
>>> Dear Team,
>>>
>>> One of our client's website stopped working yesterday. We
>>> observed
>> that
>>> Tomcat servers were not working properly during that time. We
>>> have
>> checked
>>> the memory usage of the server was fine but in the Catalina.out
>>> log
>> we
>>> found it was already reached to max thread which is 512 though
>>> the
>> number
>>> of connections to the server was normal. We took a thread dump
>>> from
>> the
>>> server using VisualVM and we got the below message from
>>> threaddump:
>>>
>>
>> Since a thread dump is a point in time snapshot, you should
>> always take multiple thread dumps, with a few seconds in between
>> each one. This gives you additional perspective as to what's
>> happening with the threads over a period of time.
>>
>>
>>>
>>> "http-8080-1" - Thread t@22
>>>
>>> java.lang.Thread.State: BLOCKED
>>>
>>> at java.util.Vector$1.nextElement(Vector.java:320)
>>>
>>> - waiting to lock <37749687> (a java.util.Vector) owned
>> by
>>> "http-8080-116" t@161
>>>
>>> at
>>> org.apache.jsp.includes.header_jsp.isExcludePath(header_jsp.java:116
)
>>>
>>>
>>>
at
>>> org.apache.jsp.includes.header_jsp._jspService(header_jsp.java:314)
>>>
>>
>>
>>>
Look at what header.jsp is doing. It seems to be doing something with
>> the Vector class which is causing the thread to block.
>>
>>
>>>
>>> at
>>> org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
>>>
>>>
>>>
at
>> javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
>>>
>>> at
>>>
>> org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper
.java:377)
>>>
>>>
>>
at
>>>
>> org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:3
13)
>>>
>>>
>>
at
>>> org.apache.jasper.servlet.JspServlet.service(JspServlet.java:260)
>>>
>>>
>>>
at
>> javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
>>>
>>> at
>>>
>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Appl
icationFilterChain.java:290)
>>>
>>>
>>
at
>>>
>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationF
ilterChain.java:206)
>>>
>>>
>>
at
>>>
>> org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDisp
atcher.java:646)
>>>
>>>
>>
at
>>>
>> org.apache.catalina.core.ApplicationDispatcher.doInclude(ApplicationD
ispatcher.java:551)
>>>
>>>
>>
at
>>>
>> org.apache.catalina.core.ApplicationDispatcher.include(ApplicationDis
patcher.java:488)
>>>
>>>
>>
at
>>>
>> org.apache.jasper.runtime.JspRuntimeLibrary.include(JspRuntimeLibrary
.java:968)
>>>
>>>
>>
at
>>>
>> org.apache.jsp.home.customer_005fservice.bill.my_005fbill_jsp._jspSer
vice(my_005fbill_jsp.java:126)
>>>
>>>
>>
at
>>> org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
>>>
>>>
>>>
at
>> javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
>>>
>>> at
>>>
>> org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper
.java:377)
>>>
>>>
>>
at
>>>
>> org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:3
13)
>>>
>>>
>>
at
>>> org.apache.jasper.servlet.JspServlet.service(JspServlet.java:260)
>>>
>>>
>>>
at
>> javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
>>>
>>> at
>>>
>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Appl
icationFilterChain.java:290)
>>>
>>>
>>
at
>>>
>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationF
ilterChain.java:206)
>>>
>>>
>>
at
>>>
>> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperV
alve.java:233)
>>>
>>>
>>
at
>>>
>> org.apache.catalina.core.StandardContextValve.invoke(StandardContextV
alve.java:191)
>>>
>>>
>>
at
>>>
>> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.j
ava:127)
>>>
>>>
>>
at
>>>
>> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.j
ava:102)
>>>
>>>
>>
at
>>>
>> org.apache.catalina.valves.RequestFilterValve.process(RequestFilterVa
lve.java:269)
>>>
>>>
>>
at
>>>
>> org.apache.catalina.valves.RemoteHostValve.invoke(RemoteHostValve.jav
a:81)
>>>
>>>
>>
at
>>>
>> org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:
555)
>>>
>>>
>>
at
>>>
>> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineVal
ve.java:109)
>>>
>>>
>>
at
>>>
>> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.jav
a:298)
>>>
>>>
>>
at
>>>
>> org.apache.coyote.http11.Http11Processor.process(Http11Processor.java
:857)
>>>
>>>
>>
at
>>>
>> org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.proce
ss(Http11Protocol.java:588)
>>>
>>>
>>
at
>>>
>> org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:48
9)
>>>
>>>
>>
at java.lang.Thread.run(Thread.java:701)
>>>
>>>
>>>
>>> Locked ownable synchronizers:
>>>
>>> - ÿ ÿ ÿ ÿ ÿNone
>>>
>>>
>>>
>>> This was coming for different threads. Once we restarted the
>>> servers,
>> the
>>> website back to normal again but we got the below exception in
>>> the
>> log :
>>>
>>>
>>>
>>> Apr 22, 2015 11:15:28 AM
>>> org.apache.catalina.loader.WebappClassLoader
>>> clearReferencesThreads
>>>
>>> SEVERE: A web application appears to have started a thread
>>> named [http-8080-1] but has failed to stop it. This is very
>>> likely to
>> create a
>>> memory leak.
>>>
>>>
>> This means your app started a thread and failed to properly stop
>> it. ˜If you stopped the process completely, this is not going to
>> cause any problems because the entire process would have exited
>> and so this thread would go away any way. ˜If you're doing
>> hot redeploy's (i.e. deplying apps without restarting the
>> process), this could cause problems. ˜Either way, when you
>> have a moment you should try to investigate what is creating
>> that thread and setup something to properly close it.
>>
>>
>>>
>>>
>>> So, we want to know while the thread is blocked is it like
>>> deadlock condition for which all threads were unavailable?
>>
>>
>> The thread dump simply lists the thread as "blocked". ˜This
>> generally means that at some point it will complete, so it's not
>> technically a deadlock. If you had multiple thread dumps, you
>> could watch the thread across all of them and see how it
>> progresses.
>>
>>
>>> Current thread count I about 190 but after few days this will
>>> reach
>> to
>>> 500+ again even if the concurrent users are not high. Memory
>>> usage of
>> the
>>> server was normal during this issue. This problem is happening
>>> from
>> last 2
>>> 3 months.
>>>
>>
>> Look at the header.jsp, especially anything that has changed in
>> the last 2 - 3 months.
>>
>> Dan
>>
>>
>>>
>>>
>>>
>>> Thanks & Regards,
>>>
>>> Subhro Paul
>>
>>
>> Dear Team,
>>
>> Thanks for the reply.
>>
>> Next thing I want to know, creating a vector and adding objects
>> to the same in header.jsp which is loaded
>> and˜referred˜by˜almost˜all the JSP's(more
>> than 1000) in an application, will that leads to memory leak.
>>
>> Thanks & Regards, Subhro Paul
>>
>>
>>
>> Hi all,
>>
>> Is there any way to understand why the vector object was locked
>> by thread "http-8080-116" for long? From the thread dump we can
>> see there is around 140 threads were blocked because of the
>> thread "http-8080-116" acquired locked on the vector object.˜
>
> Look at the full thread dump. There you will see where the lock is
> held. We can't tell you just from looking at the stacktrace.
+1
My question would be what that Vector is being used for. If all
threads have to consult that Vector, and its synchronized (which is
it: it's a java.util.Vector), then every thread will be waiting on
every other thread.
If this Vector is filled with read-only data, and the contents don't
need to be changed, then consider using another concrete List
implementation there. If you use ArrayList or LinkedList or whatever,
your page accesses will speed-up significantly.
If I were you, I'd investigate the possibility of switching
implementations. I would also wrap the unsynchronized, read-only list
in a formally-read-only list implementation like this:
public class Startup implements ServletContextListener {
ÿÿpublic void contextInitialized(...) {
ÿÿ ÿ List<Thing> listOfThings = new ArrayList<>(...);
ÿÿ ÿ // fill listOfThings with things
ÿÿ ÿ listOfThings = java.util.Collections.unmodifiableList(listOfThings)
;
ÿÿ ÿ getServletContext().setAttribute("listOfThings");
ÿÿ}
}
This will give you a structure that will throw an error if you try to
modify it.
////
If you ARE trying to modify it, you have to ask yourself why you are
using a single structure that every request coming into the
application is going to have to fight-over.
In either case, can you explain the use-case here?
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2
Comment: GPGTools - http://gpgtools.org
Hi Team,
ÿ
While staring tomcat we can observeÿÿthrough Jconsole thatÿsome threads are
automatically created by Tomcat. Once we put load on the server, depending on
the load Tomcat increases the thread count but if we stop putting load on the
server we can see theÿstillÿThread count on the server remain same even after
long time. My assumption was it would destroy some threads once there is a
little load or no load which is not the actual scenario. We can see that the
unused threads are in sleep mode. Is it a normal feature of Tomcat or some
configuration is required in Tomcat to destroy inactive threads. FYI we are
using “HTTP 1.1” connector.
Thanks & Regards,
Subhro Paul
=====-----=====-----=====
Notice: The information contained in this e-mail
message and/or attachments to it may contain
confidential or privileged information. If you are
not the intended recipient, any dissemination, use,
review, distribution, printing or copying of the
information contained in this e-mail message
and/or attachments to it are strictly prohibited. If
you have received this communication in error,
please notify us by reply e-mail or telephone and
immediately and permanently delete the message
and any attachments. Thank you
