Thanks Thomas,
I just wanted to know in tomcat is there any way to detect such
long running thread and kill them.
On 6 October 2015 at 01:07, Mark Thomas <[email protected]> wrote:
> On 05/10/2015 16:00, Yogesh Patel wrote:
> > Other 199 process are also for /solr430/update?wt=xml&version=2.2
> HTTP/1.1
> > only
>
> That is surprising. That could be a Solr issue or a Tomcat issue. Best
> to take three thread dumps each 10 to 30 seconds apart and then take a
> look to see what is holding up those threads. Best guess is a deadlock
> or a database issue.
>
> (You still have the potential for thread starvation that will come back
> to bite you later after you fix this issue unless you fix it)
>
> Mark
>
> >
> > On 5 October 2015 at 20:18, Mark Thomas <[email protected]> wrote:
> >
> >> On 05/10/2015 12:58, Yogesh Patel wrote:
> >>> Hi Mark Thomas,
> >>>
> >>> in image it shows Tomcat Manager screen:
> >>> Under "ajp-apr-10003" Section in tomcat manager:
> >>> Max threads: 200 Current thread count:200 Current thread busy :200
> Keeped
> >>> alive socket count :0
> >>>
> >>> *Stage Time BSent BRecv Client VHost
> >>> Request*
> >>> S 2884466 ms 0 KB 184063 KB machineip host
> >> POST
> >>> /solr430/update?wt=xml&version=2.2 HTTP/1.1
> >>
> >> And the other 199 processors?
> >>
> >> Mark
> >>
> >>
> >>>
> >>> ..........
> >>> ............
> >>>
> >>>
> >>> On 5 October 2015 at 16:12, Yogesh Patel <[email protected]>
> >> wrote:
> >>>
> >>>> Hi , let me try sending image using attachment , if still its not
> >> viewable
> >>>> then i will find another way.
> >>>>
> >>>> On 5 October 2015 at 16:06, Mark Thomas <[email protected]> wrote:
> >>>>
> >>>>> On 05/10/2015 11:28, Yogesh Patel wrote:
> >>>>>> Hi Thomas ,
> >>>>>> Please see this image ...have a look at Time
> column
> >>>>>
> >>>>> The list strips images. If you really want us to look at the image
> (not
> >>>>> that I think it will be remotely relevant) upload it somewhere and
> post
> >>>>> the URL (make sure it is publicly accessible and doe snot require any
> >>>>> form of registration to view it).
> >>>>>
> >>>>> Mark
> >>>>>
> >>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> On 5 October 2015 at 14:50, Mark Thomas <[email protected]
> >>>>>> <mailto:[email protected]>> wrote:
> >>>>>>
> >>>>>> On 05/10/2015 10:09, Yogesh Patel wrote:
> >>>>>> > Hi Thomas,
> >>>>>> >
> >>>>>> > Connector configuration is like :
> >>>>>> > <Connector port="10004" protocol="AJP/1.3" redirectPort="8443"
> >> />
> >>>>>>
> >>>>>> That means you are using the BIO AJP connector.
> >>>>>>
> >>>>>> You don't have a problem with long running requests.
> >>>>>>
> >>>>>> You have a problem with thread starvation.
> >>>>>>
> >>>>>> AJP uses persistent connections.
> >>>>>>
> >>>>>> BIO requires one thread per connection, regardless of whether or
> >> nor
> >>>>>> that connection is processing a request or waiting for the next
> >> one.
> >>>>>>
> >>>>>> httpd will (eventually, assuming mostly default config) create
> one
> >>>>> AJP
> >>>>>> connection for each httpd thread. If you have more httpd threads
> >>>>> than
> >>>>>> Tomcat threads you will eventually reach the point where httpd
> >> can't
> >>>>>> create a Tomcat thread it requires and at that point it will
> >> appear
> >>>>>> to hang.
> >>>>>>
> >>>>>> There are several possible fixes:
> >>>>>> a) disable connection re-use in httpd for AJP connections
> >>>>>> b) use the NIO AJP connector in Tomcat
> >>>>>> c) increase maxThreads in Tomcat so it is > max threads in httpd
> >>>>>>
> >>>>>> For more explanation read this:
> >>>>>>
> >>>>>
> >>
> http://people.apache.org/~markt/presentations/2015-04-15-Tomcat-clustering-part-1-reverse-proxies.pdf
> >>>>>> <
> >>>>>
> >>
> http://people.apache.org/%7Emarkt/presentations/2015-04-15-Tomcat-clustering-part-1-reverse-proxies.pdf
> >>>>>>
> >>>>>>
> >>>>>> from slide 29 to 33
> >>>>>>
> >>>>>> Mark
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> >
> >>>>>> > On 5 October 2015 at 14:17, Mark Thomas <[email protected]
> >>>>>> <mailto:[email protected]>> wrote:
> >>>>>> >
> >>>>>> >> On 05/10/2015 09:07, Yogesh Patel wrote:
> >>>>>> >>> Thanks Mark Thomas ,
> >>>>>> >>>
> >>>>>> >>> Our application is access by Apache TO Tomcat using AJP
> >>>>>> Connector.
> >>>>>> >>
> >>>>>> >> OK. That answers one of the questions I asked. However, you
> >>>>> haven't
> >>>>>> >> provided the connector configuration.
> >>>>>> >>
> >>>>>> >>> Problem :
> >>>>>> >>> Our application was mostly hanged,after looking at
> tomcat
> >>>>>> manager it
> >>>>>> >>> shown there are so many long running threads shown.
> >>>>>> >>
> >>>>>> >> The Tomcat Manager app does not show long running threads. It
> >>>>>> does show
> >>>>>> >> you how many requests are busy but again, a busy thread is
> not
> >>>>>> >> necessarily processing a request.
> >>>>>> >>
> >>>>>> >>> We want to recognize why so many threads are running
> since
> >>>>>> long time.
> >>>>>> >>
> >>>>>> >> Threads are always "running". The key question is are those
> >>>>> threads
> >>>>>> >> 'idle' or 'busy'. An idle thread is waiting to be allocated
> >> work.
> >>>>>> A busy
> >>>>>> >> thread has been allocated work but the manager application
> >> can't
> >>>>>> >> distinguish if that work is 'wait for a request to process'
> or
> >>>>>> >> 'processing a request'.
> >>>>>> >>
> >>>>>> >>> We want to detect such thread and want to kill these
> >> stucked
> >>>>>> thread.
> >>>>>> >>
> >>>>>> >> You continue to make the (increasingly unlikely) assumption
> >> that
> >>>>> 200
> >>>>>> >> busy threads mean you have 200 threads processing long
> running
> >>>>>> requests.
> >>>>>> >> Had you provided the connector configuration I asked for (by
> >> that
> >>>>>> I mean
> >>>>>> >> the <Connector ... /> elements in server.xml) then I'd be in
> a
> >>>>> better
> >>>>>> >> position to tell you what is wrong.
> >>>>>> >>
> >>>>>> >> If you do have 200 long running requests then the
> >>>>>> >> StuckThreadDetectionValve is how you detect them. That fact
> >> that
> >>>>> that
> >>>>>> >> valve is not reporting any long running requests should be a
> >> big
> >>>>> clue
> >>>>>> >> that your assumptions about what is going on are not correct.
> >>>>>> >>
> >>>>>> >> If, and it is a big if, you did have 200 concurrent long
> >> running
> >>>>>> >> requests then there is no guaranteed way to stop them. If
> your
> >>>>>> >> application checks for interrupts (most don't) then the
> >>>>>> >> StuckThreadDetectionValve can interrupt them which should
> >>>>>> terminate the
> >>>>>> >> processing of that request but the time it would take to code
> >> the
> >>>>>> >> application to handle that properly would normally be better
> >>>>> spent
> >>>>>> >> fixing the root cause of the long running request.
> >>>>>> >>
> >>>>>> >> Mark
> >>>>>> >>
> >>>>>> >>>
> >>>>>> >>>
> >>>>>> >>>
> >>>>>> >>>
> >>>>>> >>>
> >>>>>> >>> On 5 October 2015 at 13:11, Mark Thomas <[email protected]
> >>>>>> <mailto:[email protected]>> wrote:
> >>>>>> >>>
> >>>>>> >>>> On 05/10/2015 07:54, Yogesh Patel wrote:
> >>>>>> >>>>> We are facing issues with long running thread in
> >>>>>> tomcat . we
> >>>>>> >> are
> >>>>>> >>>>> using Tomcat-7.0.47.Tomcat manager shows Current busy
> >> threads
> >>>>>> : 200,
> >>>>>> >>>>> application gets stucked due to these long running
> threads.
> >>>>>> >>>>
> >>>>>> >>>> What makes you think you have issues with long running
> >> threads?
> >>>>>> A busy
> >>>>>> >>>> thread isn't necessarily processing a request.
> Configuration
> >>>>> errors
> >>>>>> >>>> leading to thread starvation are a more typical cause of
> the
> >>>>>> symptom you
> >>>>>> >>>> describe rather than long running threads in the
> application.
> >>>>>> >>>>
> >>>>>> >>>>> We implemented StuckThreadDetectionValve in
> >>>>>> server.xml( <Valve
> >>>>>> >>>>>
> >>>>>> >>>>>
> >>>>>>
> className="org.apache.catalina.valves.StuckThreadDetectionValve"
> >>>>>> >>>>>
> >>>>>> >>>>> threshold="60" />), but it could not help out.
> >>>>>> >>>>
> >>>>>> >>>> Why not? Again, this suggests that long running threads
> >> aren't
> >>>>> the
> >>>>>> >> problem.
> >>>>>> >>>>
> >>>>>> >>>>> So we implemented custom StuckThreadDetectionValve
> by
> >>>>>> extending
> >>>>>> >>>>> StuckThreadDetectionValve from
> >>>>>> >>>>> catalina, it only goes to "constructor","initInternal",and
> >> in
> >>>>>> >>>> "invoke"(when
> >>>>>> >>>>> action fires), it does not go to function
> >>>>>> "getStuckThreadNames()" even
> >>>>>> >>>>> after threshold time.How to achieve the same.Is there any
> >> way
> >>>>>> to detect
> >>>>>> >>>>> stucked thread and kill them?
> >>>>>> >>>>
> >>>>>> >>>> First of all your invoke() method fails to call
> >> super.invoke()
> >>>>>> so the
> >>>>>> >>>> Valve is never going to do anything.
> >>>>>> >>>>
> >>>>>> >>>> Secondly, if the original valve didn't work adding a bunch
> of
> >>>>>> >>>> System.out.println() lines isn't going to magically make it
> >>>>> work.
> >>>>>> >>>>
> >>>>>> >>>> Thirdly, getStuckThreadNames() is never called by any
> Tomcat
> >>>>>> code so it
> >>>>>> >>>> is no surprise that you never see a call to that method.
> That
> >>>>>> method is
> >>>>>> >>>> intended for use via JMX.
> >>>>>> >>>>
> >>>>>> >>>> I suggest you start again and tell us more about the
> problem
> >>>>>> you are
> >>>>>> >>>> trying to solve (lack of response) and your configuration.
> In
> >>>>>> particular
> >>>>>> >>>> we need to know your connector configuration and how users
> >>>>>> access the
> >>>>>> >>>> application. Do they connect directly to Tomcat or do they
> go
> >>>>> via a
> >>>>>> >>>> reverse proxy?
> >>>>>> >>>>
> >>>>>> >>>> Mark
> >>>>>> >>>>
> >>>>>> >>>>
> >>>>>> >>>>>
> >>>>>> >>>>> My custom Valve is like following :
> >>>>>> >>>>>
> >>>>>> >>>>> public class StuckThreadDetection extends
> >>>>>> StuckThreadDetectionValve
> >>>>>> >>>>> {
> >>>>>> >>>>> StuckThreadDetection stuckThreadDetection;
> >>>>>> >>>>>
> >>>>>> >>>>> public StuckThreadDetection()
> >>>>>> >>>>> {
> >>>>>> >>>>> System.out.println("in StuckThreadDetection constructor");
> >>>>>> >>>>>
> >>>>>> >>>>> }
> >>>>>> >>>>>
> >>>>>> >>>>> public void invoke(Request request, Response response)
> >> throws
> >>>>>> >>>> IOException,
> >>>>>> >>>>> ServletException
> >>>>>> >>>>> {
> >>>>>> >>>>> System.out.println("in invoke...");
> >>>>>> >>>>>
> >>>>>> >>>>> getNext().invoke(request, response);
> >>>>>> >>>>> }
> >>>>>> >>>>>
> >>>>>> >>>>> @Override
> >>>>>> >>>>> protected void initInternal() throws LifecycleException
> >>>>>> >>>>> {
> >>>>>> >>>>> System.out.println("In initInternal");
> >>>>>> >>>>> super.initInternal();
> >>>>>> >>>>> }
> >>>>>> >>>>>
> >>>>>> >>>>> @Override
> >>>>>> >>>>> public String[] getStuckThreadNames()
> >>>>>> >>>>> {
> >>>>>> >>>>> System.out.println("in getStuckThreadNames...");
> >>>>>> >>>>> String[] listStuckedThread = this.getStuckThreadNames();
> >>>>>> >>>>>
> >>>>>> >>>>> for (String threadName : listStuckedThread)
> >>>>>> >>>>> {
> >>>>>> >>>>> System.out.println(threadName);
> >>>>>> >>>>> }
> >>>>>> >>>>>
> >>>>>> >>>>> return listStuckedThread;
> >>>>>> >>>>>
> >>>>>> >>>>> }
> >>>>>> >>>>>
> >>>>>> >>>>> @Override
> >>>>>> >>>>> public String getInfo()
> >>>>>> >>>>> {
> >>>>>> >>>>> System.out.println("In getInfo");
> >>>>>> >>>>> return super.getInfo();
> >>>>>> >>>>> }
> >>>>>> >>>>> }
> >>>>>> >>>>>
> >>>>>> >>>>>
> >>>>>> >>>>>
> >>>>>> >>>>>
> >>>>>> >>>>
> >>>>>> >>>>
> >>>>>> >>>>
> >>>>>>
> >>>>>
> ---------------------------------------------------------------------
> >>>>>> >>>> To unsubscribe, e-mail:
> [email protected]
> >>>>>> <mailto:[email protected]>
> >>>>>> >>>> For additional commands, e-mail:
> >> [email protected]
> >>>>>> <mailto:[email protected]>
> >>>>>> >>>>
> >>>>>> >>>>
> >>>>>> >>>
> >>>>>> >>>
> >>>>>> >>
> >>>>>> >>
> >>>>>> >>
> >>>>> ---------------------------------------------------------------------
> >>>>>> >> To unsubscribe, e-mail: [email protected]
> >>>>>> <mailto:[email protected]>
> >>>>>> >> For additional commands, e-mail:
> [email protected]
> >>>>>> <mailto:[email protected]>
> >>>>>> >>
> >>>>>> >>
> >>>>>> >
> >>>>>> >
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>
> ---------------------------------------------------------------------
> >>>>>> To unsubscribe, e-mail: [email protected]
> >>>>>> <mailto:[email protected]>
> >>>>>> For additional commands, e-mail: [email protected]
> >>>>>> <mailto:[email protected]>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> --
> >>>>>> /Thanks & Regards,/
> >>>>>> / /
> >>>>>> / Yogesh Patel/
> >>>>>
> >>>>>
> >>>>> ---------------------------------------------------------------------
> >>>>> To unsubscribe, e-mail: [email protected]
> >>>>> For additional commands, e-mail: [email protected]
> >>>>>
> >>>>>
> >>>>
> >>>>
> >>>> --
> >>>> *Thanks & Regards,*
> >>>>
> >>>> * Yogesh Patel*
> >>>>
> >>>
> >>>
> >>>
> >>
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: [email protected]
> >> For additional commands, e-mail: [email protected]
> >>
> >>
> >
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [email protected]
> For additional commands, e-mail: [email protected]
>
>
--
*Thanks & Regards,*
* Yogesh Patel*
