Hi Maik,
On 2012-09-10, at 11:04 AM, Maik Musall wrote: > Hi Chuck, > > Am 10.09.2012 um 19:23 schrieb Chuck Hill <[email protected]>: > >> Hi Maik, >> >> "WorkerThread207" that many worker threads indicates two things to me: >> 1. Your app configuration is too high. I'd use a max of 6-10 and a listen >> queue size of around 4 (adjusted to suit your specific needs). A WO app is >> very, very unlikely to recover from a 200 worker thread backlog in any way >> that is useful to the users > > You may be right, they were at 16/512/8/128. I just set them to 4/8/8/6 and > am eager to watch the behaviour tomorrow. You should at least know when there is a problem sooner. Then as quickly as you can, get a thread dump with jstack. > > There are up to 100 users concurrently (it's a backoffice app), although > concurrently running requests are typically not more than 2-3, plus 1-2 > DirectActions, plus possibly 1-2 long response pages running statistics stuff. OK, the 4/8/8/6 numbers you have seem reasonable for that load. >> 2. You have a thread that is taking a long time to return a result. If you >> are dispatching requests concurrently, then this is most likely stuck in >> EOControl/EOAccess (e.g. waiting for a slow query result) or connecting to >> some external process. You could also have a deadlock. If you are not >> dispatching requests concurrently, then this delay could be in other code. > > When that situation occurs, the app is not using CPU any more, neither is the > database. It often doesn't respond to SIGTERM any more and needs SIGKILL to > terminate so we can restart. That sounds like what a blocked non-daemon thread would cause. >> The traces below do not show the problem. If you want to send a full dump, >> I am willing to look at it. It is possible that the problem had resolved by >> the time you took this dump. What you show below is normal for a lot of >> worker threads. WorkerThread206 is waiting for a new request, >> WorkerThread207 is idle waiting for something to do in the future. > > Thanks for the offer; here is the full jstack output: > http://akaihi.selbstdenker.com/~maik/jstack_powerd_20120910.txt Other than having a large number of idle worker threads, there is nothing in that trace that indicates the problem. In my experience, that means that they problem has resolved itself and the application recovered. You will need to run jstack closer to the start of the problem even to capture what is going wrong. Chuck >> On 2012-09-10, at 8:03 AM, Maik Musall wrote: >> >>> Hi, >>> >>> in an app with high concurrency, the app sometimes becomes unresponsive to >>> everything but DirectActions at the time of day with the most concurrency. >>> All users aren't seeing responses any more. In jstack I see hundreds of >>> these: >>> >>>> "WorkerThread207" prio=5 tid=131e0a800 nid=0x151aa2000 waiting for monitor >>>> entry [151aa1000] >>>> java.lang.Thread.State: BLOCKED (on object monitor) >>>> at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:406) >>>> - waiting to lock <20d3da450> (a java.net.SocksSocketImpl) >>>> at java.net.ServerSocket.implAccept(ServerSocket.java:462) >>>> at java.net.ServerSocket.accept(ServerSocket.java:430) >>>> at >>>> com.webobjects.appserver._private.WOWorkerThread.run(WOWorkerThread.java:210) >>>> at java.lang.Thread.run(Thread.java:680) >>> >>> all waiting on the same lock 20d3da450, and one thread holding that lock: >>> >>>> "WorkerThread206" prio=5 tid=131d79800 nid=0x15199f000 runnable [15199e000] >>>> java.lang.Thread.State: RUNNABLE >>>> at java.net.PlainSocketImpl.socketAccept(Native Method) >>>> at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:408) >>>> - locked <20d3da450> (a java.net.SocksSocketImpl) >>>> at java.net.ServerSocket.implAccept(ServerSocket.java:462) >>>> at java.net.ServerSocket.accept(ServerSocket.java:430) >>>> at >>>> com.webobjects.appserver._private.WOWorkerThread.run(WOWorkerThread.java:210) >>>> at java.lang.Thread.run(Thread.java:680) >>> >>> Anyone familiar with this problem? >>> >>> Maik >>> _______________________________________________ >>> Do not post admin requests to the list. They will be ignored. >>> Webobjects-dev mailing list ([email protected]) >>> Help/Unsubscribe/Update your Subscription: >>> https://lists.apple.com/mailman/options/webobjects-dev/chill%40global-village.net >>> >>> This email sent to [email protected] >> >> -- >> Chuck Hill Senior Consultant / VP Development >> >> Practical WebObjects - for developers who want to increase their overall >> knowledge of WebObjects or who are trying to solve specific problems. >> http://www.global-village.net/gvc/practical_webobjects >> >> >> >> >> >> >> >> > -- Chuck Hill Senior Consultant / VP Development Practical WebObjects - for developers who want to increase their overall knowledge of WebObjects or who are trying to solve specific problems. http://www.global-village.net/gvc/practical_webobjects _______________________________________________ Do not post admin requests to the list. They will be ignored. Webobjects-dev mailing list ([email protected]) Help/Unsubscribe/Update your Subscription: https://lists.apple.com/mailman/options/webobjects-dev/archive%40mail-archive.com This email sent to [email protected]
