Thanks Joe for sharing your findings on the list. Radu
On Fri, 2007-03-23 at 09:12 -0800, [EMAIL PROTECTED] wrote: > Hi, > > I have pasted in at the bottom a previous inquiry to this group > about a long 'hang' we were seeing in the XML Beans > XmlObject::Validate method on WebSphere. It seemed random, and we > couldn't reproduce it by replaying the same message again, when we > found one that did cause the hang. > > Since the inquiry below, we have since isolated the problem with IBM > WebSphere Support, and IBM has issued a fix for the issue. > > The problem was that the IBM JDK that is embedded in WebSphere 5.x > and 6.x was JIT compiling the following class or method, which would > sometimes take 4-5 minutes to complete: > > org.apache.xmlbeans.impl.store.Saver$ValidatorSaver.textIsWhitespace(Saver.java:3985) > > When this occured all of WebSphere comes to a complete halt. All > threads in the container stopped executing, and you couldn't even > get to the WebSphere Admin Console Web Application page. The reason > this was occuring was because while it was taking this 4-5 minutes > to compile, it was holding on to lock called JIT Global Compile > Lock. Every other websphere thread was waiting for this lock while > this 4-5 minute compile was going on...hence WebSphere is dead while > this is happening. > > IBM Issued a fix for this with their JDK 1.4.2 SR7 release, which > can be found here: > http://www-1.ibm.com/support/docview.wss?uid=swg24011133 > > This problem is not specific to any version of xml beans, it occurs > in 1.0.3, 1.0.4, and 2.x, as we have tried all versions, and they > all experienced the issue. > > Once we figured this out via core dumps, stack traces, and lock > dumps, we actually found out that there was another WebSphere > customer out there that was having the exact same issue as us. The > other customer confirmed their problem went away with this SR7. We > applied the update yesterday and are testing it out now. So far we > have not been able to recreate the issue. > > Hope this helps anyone having similar issues. > > Thanks, > Joe > > > Here is the contents of my earlier posts to this thread: > > Radu, thanks for the thoughts. I will investigate some of those > ideas...since we are out of ideas at this point. > > Thanks, > Joe > > > -----Original Message----- > From: Radu Preotiuc-Pietro [mailto:[EMAIL PROTECTED] > Sent: Monday, October 16, 2006 9:56 AM > To: [email protected] > Subject: RE: Wierd "Freeze" in Validate on WebSphere 5.1 > > Well, I got to say I have no idea where in the XMLBeans code this problem > might be. You say it's related to validation and it's not around the > SchemaTypeVisitorImpl.visit() method (which is the "trickiest" > validation method). > > On the other hand, you are saying that you don't see this problem on > Windows, which hints that this is not an "algorithmic" problem, but > something environment-related (the one that we fixed for 1.0.4 was > happening regardless of the environment used if I remember correctly). > > I guess one simple test that you could do is try to validate the same > document using the command-line tool "validate" or even better, your code > running outside of WebSphere. > > One thing which _might_ be happening is that the loading of all the Schema > type information off the classpath during validation triggers some weird > classloading behaviour from WAS (that could also explain why the whole > server is frozen). By the way, XMLBeans uses the context classloader (by > default) to load this type information. > > Radu > > -----Original Message----- > From: Joseph Mihalich [mailto:[EMAIL PROTECTED] > Sent: Tuesday, October 10, 2006 6:15 PM > To: [email protected] > Subject: RE: Wierd "Freeze" in Validate on WebSphere 5.1 > > > Hi all, > > We are using xml beans 1.0.4, latest binary release. We're > having a really horrible problem on WebSphere 5.1 trying to > validate > xml requests coming in on our web services. > > What seems to be randomly, different components (in our > ear's) > deserialize XML received from Axis, will call into the XmlObject > Validate method, and just go in, and sit there for anywhere > between > 2 and 4 minutes. The xml is usually 1K or less, so it's really > small. > And the same XML Document doesn't cause the long delay if I try > to > replay the message. > > During this time that validate is taking so long the entire > WebSphere AppServer is frozen...all execution and logging stops > until this call returns. (Not sure how that can happen, but > that's > another story). > > I noticed a bug logged against 1.0.3 where the exact same > behavior > was noted, and an optimization was put into 1.0.4 to fix it. We > were > on 1.0.3, and upgraded to 1.0.4 and still experienced the > problem. > > I pulled the 1.0.4 source, and added logging around the fix > that was made, and verified that in fact, the "delay" was NOT being > observed in that method, so the problem is somewhere else. > > I attempted to add more logging to try and isolate where the > delay was coming from, but of course, once I did that I could > not > reproduce it...hinting at some kind of timing issue. Then I ran > out > of time, and had to put this problem aside for a while. > > This happens on every single box we install on. This occurs > on > Suse Linux 9.1 Ent, WAS 5.1, IBM JDK 1.4.2. Also, we do NOT see > this > on our windows platform which runs Windows Server 2003, Jboss > 4.0.4, > and Sun JDK 1.4.2. > > Anyway, I was wondering if anyone else has experienced this > issue > on WAS 5.1, and if anyone has any suggestions on what or where > the > problem might be in the xml beans code? > > Thanks for any help you can provide. > > Thanks, > Joe > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [EMAIL PROTECTED] > For additional commands, e-mail: [EMAIL PROTECTED] > Notice: This email message, together with any attachments, may contain information of BEA Systems, Inc., its subsidiaries and affiliated entities, that may be confidential, proprietary, copyrighted and/or legally privileged, and is intended solely for the use of the individual or entity named in this message. If you are not the intended recipient, and have received this message in error, please immediately return this by email and then delete it. --------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
