You were right, it was just a symptom.
I found the bug in the code where a few threads were stuck in a loop,
eating up the cpu.
Once i fixed that, the problem went away.
Thanks for all your help.
On Aug 11, 2009, at 4:52 PM, Chuck Hill wrote:
On Aug 11, 2009, at 1:27 PM, doug andrews wrote:
I did get a thread dump from jconsole.
When ever this condition occurs, i have several threads that look
like this:
Stack trace:
java.lang.Object.wait(Native Method)
java.lang.Object.wait(Object.java:474)
com.webobjects.appserver.WOSessionStore.checkOutSessionWithID
(WOSessionStore.java:191)
com.webobjects.appserver.WOApplication.restoreSessionWithID
(WOApplication.java:1913)
com.webobjects.appserver._private.WOComponentRequestHandler._dispatch
WithPreparedApplication(WOComponentRequestHandler.java:324)
com.webobjects.appserver._private.WOComponentRequestHandler._handleRe
quest(WOComponentRequestHandler.java:369)
com.webobjects.appserver._private.WOComponentRequestHandler.handleReq
uest(WOComponentRequestHandler.java:445)
com.webobjects.appserver.WOApplication.dispatchRequest
(WOApplication.java:1687)
Application.dispatchRequest(Application.java:630)
com.webobjects.appserver._private.WOWorkerThread.runOnce
(WOWorkerThread.java:144)
com.webobjects.appserver._private.WOWorkerThread.run
(WOWorkerThread.java:226)
java.lang.Thread.run(Thread.java:613)
As mentioned before, this is a request for a session that never
got checked back in.
Anyone else run into this problem, or know how do i fix?
This is not a problem, this is just a symptom. You should ignore
these and look for the problem causing this.
As soon as this starts to happen, the cpu usage starts to climb to
capacity, until the app is unreachable.
This sounds like the problem, or at least a symptom closer to the
problem. What are the threads doing that cause this activity? The
traces above may be caused by users clicking a link/button a second
time after the first click returns no response as that thread is
what is using all the CPU. Focus on the thread actively processing
and using CPU. That is where your problem is.
Chuck
My session class does not override terminate or awake anymore,
only sleep, which is:
public void sleep()
{
try
{
makePageSecure = false;
skipSecurityCheck = false;
}
catch (Exception i)
{
i.printStackTrace();
}
finally
{
super.sleep();
}
}
On Aug 5, 2009, at 3:42 PM, Chuck Hill wrote:
It does work with Java 1.5. I have had trouble getting a thread
dump like this from one Leopard server. I have not dug into it
yet. The lack of thread names in jstack on OS X looks like a bug
to me.
If you have GUI access to the server and http access out, you
could try this: http://www.adaptj.com/main/download
jconsole is also worth a look.
Chuck
On Aug 5, 2009, at 11:22 AM, doug andrews wrote:
Does the method recommended on the gvcsitemaker only work with
java 1.4?
I have 1.5, and all that shows up in /tmp/wothreaddump.log is:
Shared archive: sharing disabled for server vm
[2009-8-4 16:34:31 EDT] <main> WOApplication: Renamed previous
WOOutputPath file to /Library/WebObjects/Logs/
LeopardTest-1.20090804163431729
I modified /System/Library/WebObjects/JavaApplications/
wotaskd.woa/Contents/Resources/SpawnOfWotaskd.sh and then
rebooted the whole machine.
Also, my original assumption that this had something to do with
WO 5.4 was wrong.
I can reproduce the error with WO 5.3
On Jul 31, 2009, at 5:52 PM, doug andrews wrote:
Ok, nevermind.
I did not read to the end of your last message.
I will try whats recommended on the gvcsitemaker site.
On Jul 31, 2009, at 5:27 PM, Chuck Hill wrote:
On Jul 31, 2009, at 2:17 PM, doug andrews wrote:
Memory: 4GB 800 Mhz DDR2 FB-DIMM
Database: MySQL 4.1.15
Heap Space: Xmx1024m and Xms1024m
Output from top -u:
PID COMMAND %CPU TIME #TH #PRTS #MREGS RPRVT RSHRD
RSIZE VSIZE
17494 java 387.8% 8:24:25 81 1126 1091 10M
38M 736M 1322M
=8-0
20799 top 4.0% 0:00.79 1 18 29 604K
188K 1196K 18M
1826 Timbuktu H 2.5% 7:21.72 9 113 144 10M
8000K 16M 185M
1931 pmTool 2.1% 27:30.53 1 63 25 612K
1132K 1484K 27M
0 kernel_tas 1.9% 18:15.98 53 2 502 6116K 0
285M 112M
17761 Terminal 1.7% 0:08.43 3 105 188 4468K
9744K 12M 172M
1930 Activity M 1.6% 27:36.16 7 117 252 6616K
20M 15M 203M
58 hwmond 0.2% 17:11.70 3 65 46 2208K 416K
3852K 20M
183 WindowServ 0.2% 2:02.65 7 199 418 10M
18M 26M 164M
31 mDNSRespon 0.0% 0:10.91 2 36 27 572K 192K
1356K 20M
63 emond 0.0% 0:39.01 1 31 22 368K 1140K
1880K 27M
45 Python 0.0% 1:07.24 1 42 443 29M
1248K 33M 55M
44 java 0.0% 0:32.77 32 608 316 39M
16M 50M 322M
43 java 0.0% 0:55.18 32 661 321 37M
16M 49M 323M
17531 java 0.0% 0:13.42 32 603 312 46M
37M 63M 327M
17341 java 0.0% 0:20.98 31 691 309 59M
37M 77M 327M
17339 java 0.0% 0:13.08 30 585 305 42M
37M 58M 326M
17340 java 0.0% 0:14.07 30 590 302 42M
37M 59M 326M
17342 java 0.0% 0:14.69 32 604 310 45M
37M 62M 327M
40 ntpd 0.0% 0:06.40 1 15 19 196K 1
I will look at my session.terminate and sleep methods.
Can't do the kill QUIT right now because it's a live site.
a) at 388% CPU it is not very live :-P
b) kill -QUIT <pid> just does a thread dump, it does not stop
the process
See:
http://www.gvcsitemaker.com/gvc.webobjects/
faq&mode=single&recordID=41413&nextMode=list
Note that if your wotaskd is not configured like this already,
you will need to restart it and the application for kill -QUIT
to provide you with any output.
Chuck
On Jul 31, 2009, at 5:06 PM, Chuck Hill wrote:
On Jul 31, 2009, at 1:47 PM, doug andrews wrote:
This is still an issue for us.
We have a site with the following:
OS 10.5.7
2.8 Ghz Quad-Core Intel Xeon
How much memory?
WebObjects 5.4.3
Java 1.5.0_19
Apache2.2 (We are using the Apache adaptor from Apple
right now, and not the Wonder version)
There is one instance of our app.
There are about 7 sessions running.
Each session has a separate connection to the database.
Which database? Which version? Separate database
connections can consume extra memory. Are you allocating
enough heap space to the JVM?
Within about 5 hours or so, all 4 processors run at close
to 100%, until eventually the app stops responding.
Which process(es) are using this CPU? top -u will quickly
show this. I doubt that you will find it is Java (i.e. your
WO app). You might find it is the database.
We have a few sites that have this problem, and the only
thing i can see in common is running WO5.4.x on Leopard.
I did manage to get the jstack output when this machine's
cpu usage was close to capacity.
Below is the jstack output.
Does anything jump out as obviously wrong to anybody?
Not exactly. It looks idle. It was really slow to respond
at some time in the past and created a lot of worker
threads. Eventually those did finish processing. You have
a bug in your session's sleep() and / or terminate() methods
that is preventing them from being checked in. The bug is
likely either a missing call to super() (unlikely, you'd
notice this right away), or an exception that is thrown.
Wrap the body of each in try...finally:
public void sleep() {
try {
// your buggy code here :-)
}
finally {
super.sleep();
}
}
kill -QUIT gives better info, BTW. Apple's jstack seems to
have gotten worse.
This is just an idle worker thread:
Thread t...@64771: (state = BLOCKED)
- java.net.PlainSocketImpl.accept(java.net.SocketImpl)
@bci=0, line=382 (Interpreted frame)
- java.net.ServerSocket.implAccept(java.net.Socket)
@bci=50, line=450 (Interpreted frame)
- java.net.ServerSocket.accept() @bci=48, line=421
(Interpreted frame)
- com.webobjects.appserver._private.WOWorkerThread.run()
@bci=26, line=210 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=613 (Interpreted frame)
This is a request for a session that never got checked back
in (see bug above):
Thread t...@80131: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Interpreted frame)
- java.lang.Object.wait() @bci=2, line=474 (Interpreted frame)
-
com.webobjects.appserver.WOSessionStore.checkOutSessionWithID(
java.lang.String, com.webobjects.appserver.WORequest)
@bci=48, line=191 (Interpreted frame)
-
com.webobjects.appserver.WOApplication.restoreSessionWithID
(java.lang.String, com.webobjects.appserver.WOContext)
@bci=9, line=1913 (Interpreted frame)
-
com.webobjects.appserver._private.WOComponentRequestHandler._d
ispatchWithPreparedApplication
(com.webobjects.appserver.WOApplication,
com.webobjects.appserver.WOContext,
com.webobjects.foundation.NSDictionary) @bci=55, line=324
(Interpreted frame)
-
com.webobjects.appserver._private.WOComponentRequestHandler._h
andleRequest(com.webobjects.appserver.WORequest) @bci=113,
line=369 (Interpreted frame)
-
com.webobjects.appserver._private.WOComponentRequestHandler.ha
ndleRequest(com.webobjects.appserver.WORequest) @bci=46,
line=445 (Interpreted frame)
- com.webobjects.appserver.WOApplication.dispatchRequest
(com.webobjects.appserver.WORequest) @bci=32, line=1687
(Compiled frame)
- Application.dispatchRequest
(com.webobjects.appserver.WORequest) @bci=2, line=629
(Interpreted frame)
- com.webobjects.appserver._private.WOWorkerThread.runOnce
() @bci=473, line=144 (Compiled frame)
- com.webobjects.appserver._private.WOWorkerThread.run()
@bci=129, line=226 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=613 (Interpreted frame)
These are jstack / OS X bugs (and might possibly be hiding
the problem in your app but I don't think so):
Thread t...@95491: (state = IN_NATIVE)
Error occurred during stack walking:
sun.jvm.hotspot.debugger.UnalignedAddressException: Trying
to read at address: 0xc9e58955 with alignment: 4
at
sun.jvm.hotspot.debugger.DebuggerUtilities.checkAlignment
(DebuggerUtilities.java:40)
at
sun.jvm.hotspot.debugger.macosx.MacOSXDebuggerLocal2.readCInte
ger(MacOSXDebuggerLocal2.java:387)
at sun.jvm.hotspot.debugger.DebuggerBase.readAddressValue
(DebuggerBase.java:425)
at
sun.jvm.hotspot.debugger.macosx.MacOSXDebuggerLocal2.readAddre
ss(MacOSXDebuggerLocal2.java:257)
at
sun.jvm.hotspot.debugger.macosx.MacOSXAddress.getAddressAt
(MacOSXAddress.java:54)
at sun.jvm.hotspot.runtime.x86.X86Frame.getLink
(X86Frame.java:338)
at sun.jvm.hotspot.runtime.x86.X86Frame.sender
(X86Frame.java:218)
at sun.jvm.hotspot.runtime.Frame.sender(Frame.java:184)
at sun.jvm.hotspot.runtime.Frame.realSender(Frame.java:189)
at sun.jvm.hotspot.runtime.VFrame.sender(VFrame.java:102)
at sun.jvm.hotspot.runtime.VFrame.javaSender(VFrame.java:134)
at sun.jvm.hotspot.runtime.JavaThread.getLastJavaVFrameDbg
(JavaThread.java:231)
at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:53)
at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:27)
at sun.jvm.hotspot.tools.JStack.run(JStack.java:41)
at sun.jvm.hotspot.tools.Tool.start(Tool.java:204)
at sun.jvm.hotspot.tools.JStack.main(JStack.java:62)
Thread t...@98051: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Interpreted frame)
- java.lang.ref.ReferenceQueue.remove(long) @bci=44,
line=120 (Interpreted frame)
- java.lang.ref.ReferenceQueue.remove() @bci=2, line=136
(Compiled frame)
Error occurred during stack walking:
java.lang.NullPointerException
at sun.jvm.hotspot.runtime.Frame.addressOfStackSlot
(Frame.java:214)
at sun.jvm.hotspot.runtime.x86.X86Frame.getSenderSP
(X86Frame.java:355)
at sun.jvm.hotspot.runtime.x86.X86Frame.sender
(X86Frame.java:218)
at sun.jvm.hotspot.runtime.Frame.sender(Frame.java:184)
at sun.jvm.hotspot.runtime.Frame.realSender(Frame.java:189)
at sun.jvm.hotspot.runtime.VFrame.sender(VFrame.java:102)
at sun.jvm.hotspot.runtime.VFrame.javaSender(VFrame.java:134)
at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:53)
at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:27)
at sun.jvm.hotspot.tools.JStack.run(JStack.java:41)
at sun.jvm.hotspot.tools.Tool.start(Tool.java:204)
at sun.jvm.hotspot.tools.JStack.main(JStack.java:62)
Thread t...@98311: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Interpreted frame)
- java.lang.Object.wait() @bci=2, line=474 (Compiled frame)
Error occurred during stack walking:
java.lang.NullPointerException
at sun.jvm.hotspot.runtime.Frame.addressOfStackSlot
(Frame.java:214)
at sun.jvm.hotspot.runtime.x86.X86Frame.getSenderSP
(X86Frame.java:355)
at sun.jvm.hotspot.runtime.x86.X86Frame.sender
(X86Frame.java:218)
at sun.jvm.hotspot.runtime.Frame.sender(Frame.java:184)
at sun.jvm.hotspot.runtime.Frame.realSender(Frame.java:189)
at sun.jvm.hotspot.runtime.VFrame.sender(VFrame.java:102)
at sun.jvm.hotspot.runtime.VFrame.javaSender(VFrame.java:134)
at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:53)
at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:27)
at sun.jvm.hotspot.tools.JStack.run(JStack.java:41)
at sun.jvm.hotspot.tools.Tool.start(Tool.java:204)
at sun.jvm.hotspot.tools.JStack.main(JStack.java:62)
Chuck
On Sep 23, 2008, at 5:03 PM, Chuck Hill wrote:
Hi Doug,
On Sep 23, 2008, at 12:19 PM, doug andrews wrote:
Still in need of help-
I took your advice and made a build specifically for this
site using java 1.5 and the latest xcode and wo.
I am still having this problem.
We have one instance of our application that is causing
each processor to spike to over 90 percent of its cpu
capacity.
When this happens, one random user will get the 'instance
not available'.
Have you gotten a thread dump from such a run away instance?
http://www.gvcsitemaker.com/gvc.webobjects/
faq&mode=single&recordID=41413
Is that one specific instance, or just randomly one of
them? If one specific, what is different about it?
This only happens when someone upgrades to WO 5.4.2 or
5.4.3.
Tiger machines running WO 5.3.x have no problems.
Are the machines upgraded to WO 5.4 running a copy of the
app compiled against the same version of WO?
These are intel machines running OS 10.5.x.
Does anyone know of any changes made in WO 5.4.x (or OS
10.5.x) that could cause this?
Not off hand.
Chuck
On Aug 25, 2008, at 3:22 PM, Mike Schrag wrote:
No, they were built using Xcode and WO 5.3.3.
This is a really bad idea ... I would never deploy on a
version that's different than development. It's just
asking for problems. While it MAY work, you're just
playing roulette with your app. Either 1) replace 5.4
with 5.3 on that server, 2) rebuild your app with WO 5.3
embedded, or 3) Setup a proper 5.4.2 build and
development environment and rebuild and test your app in
5.4.2.
ms
--
Chuck Hill Senior Consultant / VP Development
Practical WebObjects - for developers who want to increase their
overall knowledge of WebObjects or who are trying to solve
specific problems.
http://www.global-village.net/products/practical_webobjects
_______________________________________________
Do not post admin requests to the list. They will be ignored.
Webobjects-deploy mailing list (Webobjects-
[email protected])
Help/Unsubscribe/Update your Subscription:
http://lists.apple.com/mailman/options/webobjects-deploy/chill%
40global-village.net
This email sent to [email protected]
--
Chuck Hill Senior Consultant / VP Development
Practical WebObjects - for developers who want to increase their
overall knowledge of WebObjects or who are trying to solve specific
problems.
http://www.global-village.net/products/practical_webobjects
_______________________________________________
Do not post admin requests to the list. They will be ignored.
Webobjects-deploy mailing list ([email protected])
Help/Unsubscribe/Update your Subscription:
http://lists.apple.com/mailman/options/webobjects-deploy/archive%40mail-archive.com
This email sent to [email protected]
