Hi guys,
I know I'm a little late on this, but I'm also seeing the same
behavior. It's not a long running query I don't think because I'm logging long
queries in postgres and nothing is running over 10 seconds. Can you explain
why having a max of 256 worker threads is too high? Any other things I should
look at? The customers are not happy! My last problem did turn out to be a
bunch of deadlocks, which all now seem to be resolved. It had to with setting
er.extensions.ERXObjectStoreCoordinatorPool.maxCoordinators=4 which should be
seamless (you would think) but causes issues with fetch specs that have EOs
crossing OSCs. I had to pull all EOs local, seems like something that should
be handled inside wonder automatically (so I consider it a bug, whether it is
or not could be argued I guess). Anyway, after those all got fixed, I'm now
running into this. Much harder to figure out since I don't even know what the
lock is held on.
BTW Chuck and Quinton, I owe you guys a beer. Thanks for pointing me in the
right direction on the last problem.
Thanks for any help.
-Mike
-----Original Message-----
From: [email protected]
[mailto:[email protected]] On Behalf
Of Chuck Hill
Sent: Monday, September 10, 2012 1:24 PM
To: Maik Musall
Cc: [email protected] WebObjects
Subject: Re: WOWorkerThread deadlocks
Hi Maik,
"WorkerThread207" that many worker threads indicates two things to me:
1. Your app configuration is too high. I'd use a max of 6-10 and a listen
queue size of around 4 (adjusted to suit your specific needs). A WO app is
very, very unlikely to recover from a 200 worker thread backlog in any way that
is useful to the users
2. You have a thread that is taking a long time to return a result. If you are
dispatching requests concurrently, then this is most likely stuck in
EOControl/EOAccess (e.g. waiting for a slow query result) or connecting to some
external process. You could also have a deadlock. If you are not dispatching
requests concurrently, then this delay could be in other code.
The traces below do not show the problem. If you want to send a full dump, I
am willing to look at it. It is possible that the problem had resolved by the
time you took this dump. What you show below is normal for a lot of worker
threads. WorkerThread206 is waiting for a new request, WorkerThread207 is idle
waiting for something to do in the future.
Chuck
On 2012-09-10, at 8:03 AM, Maik Musall wrote:
> Hi,
>
> in an app with high concurrency, the app sometimes becomes unresponsive to
> everything but DirectActions at the time of day with the most concurrency.
> All users aren't seeing responses any more. In jstack I see hundreds of these:
>
>> "WorkerThread207" prio=5 tid=131e0a800 nid=0x151aa2000 waiting for monitor
>> entry [151aa1000]
>> java.lang.Thread.State: BLOCKED (on object monitor)
>> at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:406)
>> - waiting to lock <20d3da450> (a java.net.SocksSocketImpl)
>> at java.net.ServerSocket.implAccept(ServerSocket.java:462)
>> at java.net.ServerSocket.accept(ServerSocket.java:430)
>> at
>> com.webobjects.appserver._private.WOWorkerThread.run(WOWorkerThread.java:210)
>> at java.lang.Thread.run(Thread.java:680)
>
> all waiting on the same lock 20d3da450, and one thread holding that lock:
>
>> "WorkerThread206" prio=5 tid=131d79800 nid=0x15199f000 runnable [15199e000]
>> java.lang.Thread.State: RUNNABLE
>> at java.net.PlainSocketImpl.socketAccept(Native Method)
>> at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:408)
>> - locked <20d3da450> (a java.net.SocksSocketImpl)
>> at java.net.ServerSocket.implAccept(ServerSocket.java:462)
>> at java.net.ServerSocket.accept(ServerSocket.java:430)
>> at
>> com.webobjects.appserver._private.WOWorkerThread.run(WOWorkerThread.java:210)
>> at java.lang.Thread.run(Thread.java:680)
>
> Anyone familiar with this problem?
>
> Maik
> _______________________________________________
> Do not post admin requests to the list. They will be ignored.
> Webobjects-dev mailing list ([email protected])
> Help/Unsubscribe/Update your Subscription:
> https://lists.apple.com/mailman/options/webobjects-dev/chill%40global-village.net
>
> This email sent to [email protected]
--
Chuck Hill Senior Consultant / VP Development
Practical WebObjects - for developers who want to increase their overall
knowledge of WebObjects or who are trying to solve specific problems.
http://www.global-village.net/gvc/practical_webobjects
_______________________________________________
Do not post admin requests to the list. They will be ignored.
Webobjects-dev mailing list ([email protected])
Help/Unsubscribe/Update your Subscription:
https://lists.apple.com/mailman/options/webobjects-dev/mgargano%40escholar.com
This email sent to [email protected]
_______________________________________________
Do not post admin requests to the list. They will be ignored.
Webobjects-dev mailing list ([email protected])
Help/Unsubscribe/Update your Subscription:
https://lists.apple.com/mailman/options/webobjects-dev/archive%40mail-archive.com
This email sent to [email protected]
