On Jul 31, 2009, at 1:47 PM, doug andrews wrote:
This is still an issue for us.
We have a site with the following:
OS 10.5.7
2.8 Ghz Quad-Core Intel Xeon
How much memory?
WebObjects 5.4.3
Java 1.5.0_19
Apache2.2 (We are using the Apache adaptor from Apple right
now, and not the Wonder version)
There is one instance of our app.
There are about 7 sessions running.
Each session has a separate connection to the database.
Which database? Which version? Separate database connections
can consume extra memory. Are you allocating enough heap space
to the JVM?
Within about 5 hours or so, all 4 processors run at close to
100%, until eventually the app stops responding.
Which process(es) are using this CPU? top -u will quickly show
this. I doubt that you will find it is Java (i.e. your WO app).
You might find it is the database.
We have a few sites that have this problem, and the only thing i
can see in common is running WO5.4.x on Leopard.
I did manage to get the jstack output when this machine's cpu
usage was close to capacity.
Below is the jstack output.
Does anything jump out as obviously wrong to anybody?
Not exactly. It looks idle. It was really slow to respond at
some time in the past and created a lot of worker threads.
Eventually those did finish processing. You have a bug in your
session's sleep() and / or terminate() methods that is preventing
them from being checked in. The bug is likely either a missing
call to super() (unlikely, you'd notice this right away), or an
exception that is thrown. Wrap the body of each in try...finally:
public void sleep() {
try {
// your buggy code here :-)
}
finally {
super.sleep();
}
}
kill -QUIT gives better info, BTW. Apple's jstack seems to have
gotten worse.
This is just an idle worker thread:
Thread t...@64771: (state = BLOCKED)
- java.net.PlainSocketImpl.accept(java.net.SocketImpl) @bci=0,
line=382 (Interpreted frame)
- java.net.ServerSocket.implAccept(java.net.Socket) @bci=50,
line=450 (Interpreted frame)
- java.net.ServerSocket.accept() @bci=48, line=421 (Interpreted
frame)
- com.webobjects.appserver._private.WOWorkerThread.run()
@bci=26, line=210 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=613 (Interpreted frame)
This is a request for a session that never got checked back in
(see bug above):
Thread t...@80131: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Interpreted frame)
- java.lang.Object.wait() @bci=2, line=474 (Interpreted frame)
- com.webobjects.appserver.WOSessionStore.checkOutSessionWithID
(java.lang.String, com.webobjects.appserver.WORequest) @bci=48,
line=191 (Interpreted frame)
- com.webobjects.appserver.WOApplication.restoreSessionWithID
(java.lang.String, com.webobjects.appserver.WOContext) @bci=9,
line=1913 (Interpreted frame)
-
com.webobjects.appserver._private.WOComponentRequestHandler._dispat
chWithPreparedApplication
(com.webobjects.appserver.WOApplication,
com.webobjects.appserver.WOContext,
com.webobjects.foundation.NSDictionary) @bci=55, line=324
(Interpreted frame)
-
com.webobjects.appserver._private.WOComponentRequestHandler._handle
Request(com.webobjects.appserver.WORequest) @bci=113, line=369
(Interpreted frame)
-
com.webobjects.appserver._private.WOComponentRequestHandler.handleR
equest(com.webobjects.appserver.WORequest) @bci=46, line=445
(Interpreted frame)
- com.webobjects.appserver.WOApplication.dispatchRequest
(com.webobjects.appserver.WORequest) @bci=32, line=1687
(Compiled frame)
- Application.dispatchRequest
(com.webobjects.appserver.WORequest) @bci=2, line=629
(Interpreted frame)
- com.webobjects.appserver._private.WOWorkerThread.runOnce()
@bci=473, line=144 (Compiled frame)
- com.webobjects.appserver._private.WOWorkerThread.run()
@bci=129, line=226 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=613 (Interpreted frame)
These are jstack / OS X bugs (and might possibly be hiding the
problem in your app but I don't think so):
Thread t...@95491: (state = IN_NATIVE)
Error occurred during stack walking:
sun.jvm.hotspot.debugger.UnalignedAddressException: Trying to
read at address: 0xc9e58955 with alignment: 4
at sun.jvm.hotspot.debugger.DebuggerUtilities.checkAlignment
(DebuggerUtilities.java:40)
at
sun.jvm.hotspot.debugger.macosx.MacOSXDebuggerLocal2.readCInteger
(MacOSXDebuggerLocal2.java:387)
at sun.jvm.hotspot.debugger.DebuggerBase.readAddressValue
(DebuggerBase.java:425)
at
sun.jvm.hotspot.debugger.macosx.MacOSXDebuggerLocal2.readAddress
(MacOSXDebuggerLocal2.java:257)
at sun.jvm.hotspot.debugger.macosx.MacOSXAddress.getAddressAt
(MacOSXAddress.java:54)
at sun.jvm.hotspot.runtime.x86.X86Frame.getLink(X86Frame.java:338)
at sun.jvm.hotspot.runtime.x86.X86Frame.sender(X86Frame.java:218)
at sun.jvm.hotspot.runtime.Frame.sender(Frame.java:184)
at sun.jvm.hotspot.runtime.Frame.realSender(Frame.java:189)
at sun.jvm.hotspot.runtime.VFrame.sender(VFrame.java:102)
at sun.jvm.hotspot.runtime.VFrame.javaSender(VFrame.java:134)
at sun.jvm.hotspot.runtime.JavaThread.getLastJavaVFrameDbg
(JavaThread.java:231)
at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:53)
at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:27)
at sun.jvm.hotspot.tools.JStack.run(JStack.java:41)
at sun.jvm.hotspot.tools.Tool.start(Tool.java:204)
at sun.jvm.hotspot.tools.JStack.main(JStack.java:62)
Thread t...@98051: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Interpreted frame)
- java.lang.ref.ReferenceQueue.remove(long) @bci=44, line=120
(Interpreted frame)
- java.lang.ref.ReferenceQueue.remove() @bci=2, line=136
(Compiled frame)
Error occurred during stack walking:
java.lang.NullPointerException
at sun.jvm.hotspot.runtime.Frame.addressOfStackSlot(Frame.java:
214)
at sun.jvm.hotspot.runtime.x86.X86Frame.getSenderSP
(X86Frame.java:355)
at sun.jvm.hotspot.runtime.x86.X86Frame.sender(X86Frame.java:218)
at sun.jvm.hotspot.runtime.Frame.sender(Frame.java:184)
at sun.jvm.hotspot.runtime.Frame.realSender(Frame.java:189)
at sun.jvm.hotspot.runtime.VFrame.sender(VFrame.java:102)
at sun.jvm.hotspot.runtime.VFrame.javaSender(VFrame.java:134)
at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:53)
at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:27)
at sun.jvm.hotspot.tools.JStack.run(JStack.java:41)
at sun.jvm.hotspot.tools.Tool.start(Tool.java:204)
at sun.jvm.hotspot.tools.JStack.main(JStack.java:62)
Thread t...@98311: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Interpreted frame)
- java.lang.Object.wait() @bci=2, line=474 (Compiled frame)
Error occurred during stack walking:
java.lang.NullPointerException
at sun.jvm.hotspot.runtime.Frame.addressOfStackSlot(Frame.java:
214)
at sun.jvm.hotspot.runtime.x86.X86Frame.getSenderSP
(X86Frame.java:355)
at sun.jvm.hotspot.runtime.x86.X86Frame.sender(X86Frame.java:218)
at sun.jvm.hotspot.runtime.Frame.sender(Frame.java:184)
at sun.jvm.hotspot.runtime.Frame.realSender(Frame.java:189)
at sun.jvm.hotspot.runtime.VFrame.sender(VFrame.java:102)
at sun.jvm.hotspot.runtime.VFrame.javaSender(VFrame.java:134)
at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:53)
at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:27)
at sun.jvm.hotspot.tools.JStack.run(JStack.java:41)
at sun.jvm.hotspot.tools.Tool.start(Tool.java:204)
at sun.jvm.hotspot.tools.JStack.main(JStack.java:62)
Chuck
On Sep 23, 2008, at 5:03 PM, Chuck Hill wrote:
Hi Doug,
On Sep 23, 2008, at 12:19 PM, doug andrews wrote:
Still in need of help-
I took your advice and made a build specifically for this site
using java 1.5 and the latest xcode and wo.
I am still having this problem.
We have one instance of our application that is causing each
processor to spike to over 90 percent of its cpu capacity.
When this happens, one random user will get the 'instance not
available'.
Have you gotten a thread dump from such a run away instance?
http://www.gvcsitemaker.com/gvc.webobjects/
faq&mode=single&recordID=41413
Is that one specific instance, or just randomly one of them?
If one specific, what is different about it?
This only happens when someone upgrades to WO 5.4.2 or 5.4.3.
Tiger machines running WO 5.3.x have no problems.
Are the machines upgraded to WO 5.4 running a copy of the app
compiled against the same version of WO?
These are intel machines running OS 10.5.x.
Does anyone know of any changes made in WO 5.4.x (or OS
10.5.x) that could cause this?
Not off hand.
Chuck
On Aug 25, 2008, at 3:22 PM, Mike Schrag wrote:
No, they were built using Xcode and WO 5.3.3.
This is a really bad idea ... I would never deploy on a
version that's different than development. It's just asking
for problems. While it MAY work, you're just playing
roulette with your app. Either 1) replace 5.4 with 5.3 on
that server, 2) rebuild your app with WO 5.3 embedded, or 3)
Setup a proper 5.4.2 build and development environment and
rebuild and test your app in 5.4.2.
ms
_______________________________________________
Do not post admin requests to the list. They will be ignored.
Webobjects-deploy mailing list (Webobjects-
[email protected])
Help/Unsubscribe/Update your Subscription:
http://lists.apple.com/mailman/options/webobjects-deploy/
dandrews%40mediaspansoftware.com
This email sent to [email protected]
_______________________________________________
Do not post admin requests to the list. They will be ignored.
Webobjects-deploy mailing list (Webobjects-
[email protected])
Help/Unsubscribe/Update your Subscription:
http://lists.apple.com/mailman/options/webobjects-deploy/chill%
40global-village.net
This email sent to [email protected]
--
Chuck Hill Senior Consultant / VP Development
Practical WebObjects - for developers who want to increase
their overall knowledge of WebObjects or who are trying to
solve specific problems.
http://www.global-village.net/products/practical_webobjects
_______________________________________________
Do not post admin requests to the list. They will be ignored.
Webobjects-deploy mailing list (Webobjects-
[email protected])
Help/Unsubscribe/Update your Subscription:
http://lists.apple.com/mailman/options/webobjects-deploy/chill%
40global-village.net
This email sent to [email protected]
--
Chuck Hill Senior Consultant / VP Development
Practical WebObjects - for developers who want to increase their
overall knowledge of WebObjects or who are trying to solve
specific problems.
http://www.global-village.net/products/practical_webobjects
