Other 199 process are also for /solr430/update?wt=xml&version=2.2 HTTP/1.1 only
On 5 October 2015 at 20:18, Mark Thomas <ma...@apache.org> wrote: > On 05/10/2015 12:58, Yogesh Patel wrote: > > Hi Mark Thomas, > > > > in image it shows Tomcat Manager screen: > > Under "ajp-apr-10003" Section in tomcat manager: > > Max threads: 200 Current thread count:200 Current thread busy :200 Keeped > > alive socket count :0 > > > > *Stage Time BSent BRecv Client VHost > > Request* > > S 2884466 ms 0 KB 184063 KB machineip host > POST > > /solr430/update?wt=xml&version=2.2 HTTP/1.1 > > And the other 199 processors? > > Mark > > > > > > .......... > > ............ > > > > > > On 5 October 2015 at 16:12, Yogesh Patel <yogesh.r.pa...@highq.com> > wrote: > > > >> Hi , let me try sending image using attachment , if still its not > viewable > >> then i will find another way. > >> > >> On 5 October 2015 at 16:06, Mark Thomas <ma...@apache.org> wrote: > >> > >>> On 05/10/2015 11:28, Yogesh Patel wrote: > >>>> Hi Thomas , > >>>> Please see this image ...have a look at Time column > >>> > >>> The list strips images. If you really want us to look at the image (not > >>> that I think it will be remotely relevant) upload it somewhere and post > >>> the URL (make sure it is publicly accessible and doe snot require any > >>> form of registration to view it). > >>> > >>> Mark > >>> > >>> > >>>> > >>>> > >>>> > >>>> > >>>> On 5 October 2015 at 14:50, Mark Thomas <ma...@apache.org > >>>> <mailto:ma...@apache.org>> wrote: > >>>> > >>>> On 05/10/2015 10:09, Yogesh Patel wrote: > >>>> > Hi Thomas, > >>>> > > >>>> > Connector configuration is like : > >>>> > <Connector port="10004" protocol="AJP/1.3" redirectPort="8443" > /> > >>>> > >>>> That means you are using the BIO AJP connector. > >>>> > >>>> You don't have a problem with long running requests. > >>>> > >>>> You have a problem with thread starvation. > >>>> > >>>> AJP uses persistent connections. > >>>> > >>>> BIO requires one thread per connection, regardless of whether or > nor > >>>> that connection is processing a request or waiting for the next > one. > >>>> > >>>> httpd will (eventually, assuming mostly default config) create one > >>> AJP > >>>> connection for each httpd thread. If you have more httpd threads > >>> than > >>>> Tomcat threads you will eventually reach the point where httpd > can't > >>>> create a Tomcat thread it requires and at that point it will > appear > >>>> to hang. > >>>> > >>>> There are several possible fixes: > >>>> a) disable connection re-use in httpd for AJP connections > >>>> b) use the NIO AJP connector in Tomcat > >>>> c) increase maxThreads in Tomcat so it is > max threads in httpd > >>>> > >>>> For more explanation read this: > >>>> > >>> > http://people.apache.org/~markt/presentations/2015-04-15-Tomcat-clustering-part-1-reverse-proxies.pdf > >>>> < > >>> > http://people.apache.org/%7Emarkt/presentations/2015-04-15-Tomcat-clustering-part-1-reverse-proxies.pdf > >>>> > >>>> > >>>> from slide 29 to 33 > >>>> > >>>> Mark > >>>> > >>>> > >>>> > >>>> > > >>>> > On 5 October 2015 at 14:17, Mark Thomas <ma...@apache.org > >>>> <mailto:ma...@apache.org>> wrote: > >>>> > > >>>> >> On 05/10/2015 09:07, Yogesh Patel wrote: > >>>> >>> Thanks Mark Thomas , > >>>> >>> > >>>> >>> Our application is access by Apache TO Tomcat using AJP > >>>> Connector. > >>>> >> > >>>> >> OK. That answers one of the questions I asked. However, you > >>> haven't > >>>> >> provided the connector configuration. > >>>> >> > >>>> >>> Problem : > >>>> >>> Our application was mostly hanged,after looking at tomcat > >>>> manager it > >>>> >>> shown there are so many long running threads shown. > >>>> >> > >>>> >> The Tomcat Manager app does not show long running threads. It > >>>> does show > >>>> >> you how many requests are busy but again, a busy thread is not > >>>> >> necessarily processing a request. > >>>> >> > >>>> >>> We want to recognize why so many threads are running since > >>>> long time. > >>>> >> > >>>> >> Threads are always "running". The key question is are those > >>> threads > >>>> >> 'idle' or 'busy'. An idle thread is waiting to be allocated > work. > >>>> A busy > >>>> >> thread has been allocated work but the manager application > can't > >>>> >> distinguish if that work is 'wait for a request to process' or > >>>> >> 'processing a request'. > >>>> >> > >>>> >>> We want to detect such thread and want to kill these > stucked > >>>> thread. > >>>> >> > >>>> >> You continue to make the (increasingly unlikely) assumption > that > >>> 200 > >>>> >> busy threads mean you have 200 threads processing long running > >>>> requests. > >>>> >> Had you provided the connector configuration I asked for (by > that > >>>> I mean > >>>> >> the <Connector ... /> elements in server.xml) then I'd be in a > >>> better > >>>> >> position to tell you what is wrong. > >>>> >> > >>>> >> If you do have 200 long running requests then the > >>>> >> StuckThreadDetectionValve is how you detect them. That fact > that > >>> that > >>>> >> valve is not reporting any long running requests should be a > big > >>> clue > >>>> >> that your assumptions about what is going on are not correct. > >>>> >> > >>>> >> If, and it is a big if, you did have 200 concurrent long > running > >>>> >> requests then there is no guaranteed way to stop them. If your > >>>> >> application checks for interrupts (most don't) then the > >>>> >> StuckThreadDetectionValve can interrupt them which should > >>>> terminate the > >>>> >> processing of that request but the time it would take to code > the > >>>> >> application to handle that properly would normally be better > >>> spent > >>>> >> fixing the root cause of the long running request. > >>>> >> > >>>> >> Mark > >>>> >> > >>>> >>> > >>>> >>> > >>>> >>> > >>>> >>> > >>>> >>> > >>>> >>> On 5 October 2015 at 13:11, Mark Thomas <ma...@apache.org > >>>> <mailto:ma...@apache.org>> wrote: > >>>> >>> > >>>> >>>> On 05/10/2015 07:54, Yogesh Patel wrote: > >>>> >>>>> We are facing issues with long running thread in > >>>> tomcat . we > >>>> >> are > >>>> >>>>> using Tomcat-7.0.47.Tomcat manager shows Current busy > threads > >>>> : 200, > >>>> >>>>> application gets stucked due to these long running threads. > >>>> >>>> > >>>> >>>> What makes you think you have issues with long running > threads? > >>>> A busy > >>>> >>>> thread isn't necessarily processing a request. Configuration > >>> errors > >>>> >>>> leading to thread starvation are a more typical cause of the > >>>> symptom you > >>>> >>>> describe rather than long running threads in the application. > >>>> >>>> > >>>> >>>>> We implemented StuckThreadDetectionValve in > >>>> server.xml( <Valve > >>>> >>>>> > >>>> >>>>> > >>>> className="org.apache.catalina.valves.StuckThreadDetectionValve" > >>>> >>>>> > >>>> >>>>> threshold="60" />), but it could not help out. > >>>> >>>> > >>>> >>>> Why not? Again, this suggests that long running threads > aren't > >>> the > >>>> >> problem. > >>>> >>>> > >>>> >>>>> So we implemented custom StuckThreadDetectionValve by > >>>> extending > >>>> >>>>> StuckThreadDetectionValve from > >>>> >>>>> catalina, it only goes to "constructor","initInternal",and > in > >>>> >>>> "invoke"(when > >>>> >>>>> action fires), it does not go to function > >>>> "getStuckThreadNames()" even > >>>> >>>>> after threshold time.How to achieve the same.Is there any > way > >>>> to detect > >>>> >>>>> stucked thread and kill them? > >>>> >>>> > >>>> >>>> First of all your invoke() method fails to call > super.invoke() > >>>> so the > >>>> >>>> Valve is never going to do anything. > >>>> >>>> > >>>> >>>> Secondly, if the original valve didn't work adding a bunch of > >>>> >>>> System.out.println() lines isn't going to magically make it > >>> work. > >>>> >>>> > >>>> >>>> Thirdly, getStuckThreadNames() is never called by any Tomcat > >>>> code so it > >>>> >>>> is no surprise that you never see a call to that method. That > >>>> method is > >>>> >>>> intended for use via JMX. > >>>> >>>> > >>>> >>>> I suggest you start again and tell us more about the problem > >>>> you are > >>>> >>>> trying to solve (lack of response) and your configuration. In > >>>> particular > >>>> >>>> we need to know your connector configuration and how users > >>>> access the > >>>> >>>> application. Do they connect directly to Tomcat or do they go > >>> via a > >>>> >>>> reverse proxy? > >>>> >>>> > >>>> >>>> Mark > >>>> >>>> > >>>> >>>> > >>>> >>>>> > >>>> >>>>> My custom Valve is like following : > >>>> >>>>> > >>>> >>>>> public class StuckThreadDetection extends > >>>> StuckThreadDetectionValve > >>>> >>>>> { > >>>> >>>>> StuckThreadDetection stuckThreadDetection; > >>>> >>>>> > >>>> >>>>> public StuckThreadDetection() > >>>> >>>>> { > >>>> >>>>> System.out.println("in StuckThreadDetection constructor"); > >>>> >>>>> > >>>> >>>>> } > >>>> >>>>> > >>>> >>>>> public void invoke(Request request, Response response) > throws > >>>> >>>> IOException, > >>>> >>>>> ServletException > >>>> >>>>> { > >>>> >>>>> System.out.println("in invoke..."); > >>>> >>>>> > >>>> >>>>> getNext().invoke(request, response); > >>>> >>>>> } > >>>> >>>>> > >>>> >>>>> @Override > >>>> >>>>> protected void initInternal() throws LifecycleException > >>>> >>>>> { > >>>> >>>>> System.out.println("In initInternal"); > >>>> >>>>> super.initInternal(); > >>>> >>>>> } > >>>> >>>>> > >>>> >>>>> @Override > >>>> >>>>> public String[] getStuckThreadNames() > >>>> >>>>> { > >>>> >>>>> System.out.println("in getStuckThreadNames..."); > >>>> >>>>> String[] listStuckedThread = this.getStuckThreadNames(); > >>>> >>>>> > >>>> >>>>> for (String threadName : listStuckedThread) > >>>> >>>>> { > >>>> >>>>> System.out.println(threadName); > >>>> >>>>> } > >>>> >>>>> > >>>> >>>>> return listStuckedThread; > >>>> >>>>> > >>>> >>>>> } > >>>> >>>>> > >>>> >>>>> @Override > >>>> >>>>> public String getInfo() > >>>> >>>>> { > >>>> >>>>> System.out.println("In getInfo"); > >>>> >>>>> return super.getInfo(); > >>>> >>>>> } > >>>> >>>>> } > >>>> >>>>> > >>>> >>>>> > >>>> >>>>> > >>>> >>>>> > >>>> >>>> > >>>> >>>> > >>>> >>>> > >>>> > >>> --------------------------------------------------------------------- > >>>> >>>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > >>>> <mailto:users-unsubscr...@tomcat.apache.org> > >>>> >>>> For additional commands, e-mail: > users-h...@tomcat.apache.org > >>>> <mailto:users-h...@tomcat.apache.org> > >>>> >>>> > >>>> >>>> > >>>> >>> > >>>> >>> > >>>> >> > >>>> >> > >>>> >> > >>> --------------------------------------------------------------------- > >>>> >> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > >>>> <mailto:users-unsubscr...@tomcat.apache.org> > >>>> >> For additional commands, e-mail: users-h...@tomcat.apache.org > >>>> <mailto:users-h...@tomcat.apache.org> > >>>> >> > >>>> >> > >>>> > > >>>> > > >>>> > >>>> > >>>> > >>> --------------------------------------------------------------------- > >>>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > >>>> <mailto:users-unsubscr...@tomcat.apache.org> > >>>> For additional commands, e-mail: users-h...@tomcat.apache.org > >>>> <mailto:users-h...@tomcat.apache.org> > >>>> > >>>> > >>>> > >>>> > >>>> -- > >>>> /Thanks & Regards,/ > >>>> / / > >>>> / Yogesh Patel/ > >>> > >>> > >>> --------------------------------------------------------------------- > >>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > >>> For additional commands, e-mail: users-h...@tomcat.apache.org > >>> > >>> > >> > >> > >> -- > >> *Thanks & Regards,* > >> > >> * Yogesh Patel* > >> > > > > > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > > -- *Thanks & Regards,* * Yogesh Patel*
