The problem isn't that EMPTY_DOC is being modified - that would only happen if we appended to the Document itself. Since Document is a factory, we can use it to create fragments that, while their "parent" is the Document, they are not actually attached to it (otherwise you'd get RuntimeExceptions about the Document having more than one root element).
For reasons that I do not understand, the Xerces parser/factory is not stateless, and that is causing the concurrency bugs. The simple act of creating new elements at the same time generates the exception. Either way, we'll have to remove it. Dan "Vinh Nguyen \(vinguye2\)" <[EMAIL PROTECTED]> wrote on 08/31/2007 04:34:28 AM: > Hi Dan, > I've also done some testing just now and am finding this to be a > very serious issue. It really affects the usability of Muse as a whole. > > To test, I created three resources. Each had a loop to generate 100 > notifications with no pause between notifications. So out of 300 > total notifications generated, I had these results: > Test A = 9 notifications dropped > Test B = 24 notifications dropped > > Attached is a file containing the various exceptions which caused > SimpleSubscriptionManager.publish() to fail. As Rich pointed out, > the culprit is using EMPTY_DOC from multiple threads (i.e. each > client request is one thread). > > The XmlUtils.EMPTY_DOC javadocs has this: > "This should NOT be used by callers that want to build new fragments > and attach them to a Document...you should never append children."> > > But in XmlUtils itself and many other classes, the following code > patterns are used: > > Document doc = XmlUtils.EMPTY_DOC; > Element xml = XmlUtils.createElement(doc, qname); > xml.appendChild(node); > > OR: > doc.importNode(node, true); > The doc is a shared object but is being modified, so errors will > occur. So just about all the serializers are affected, including > the EndpointReference class. This means errors will most likely > occur when Muse handles requests/responses from multiple clients, or > when notifications are sent from multiple resources. The latter is > easier to test. > > So far, we've been testing using just one client and one producer > instance, so the problem doesn't occur. But now that we are testing > by using multiple producers, the exceptions are occuring frequently. > > To begin the fix, all serializers and the EndpointReference class > needs to be patched. So instead of: > doc = XmlUtils.EMPTY_DOC; > > We should do: > doc = XmlUtils.createDocument(); > > The overhead of creating a new document is small when compared to > the multi thread issue, which cannot be avoided. But, we can > optimize creating new documents by updating XmlUtils to create the > DocumentBuilderFactory only once. This way, XmlUtils. > createBuilder() doesn't have to create a new factory everytime. > -Vinh > From: Daniel Jemiolo [mailto:[EMAIL PROTECTED] > Sent: Monday, August 27, 2007 7:31 AM > To: [email protected] > Subject: Re: EMPTY_DOC thread stability issues > The use of EMPTY_DOC was an attempt to avoid creating a new Document > every time we wanted to copy or create small fragments of XML. This > happened a lot during the request/response process, so the creation > of these factory objects was not insignificant. Of course, you are > right about the threading issue, so I guess we're out of luck there. > > I am setting aside Friday of this week to go through current JIRA > items and apply all patches that have been submitted, close any > issues that are recommended for closure, and sort unscheduled items > into 2.3 if necessary. Sorry for the delay. > > Dan > > > [image removed] [EMAIL PROTECTED] > > > [EMAIL PROTECTED] > 08/24/2007 04:37 PM > > Please respond to > [email protected] > > [image removed] > To > > [image removed] > [email protected] > > [image removed] > cc > > [image removed] > > [image removed] > Subject > > [image removed] > EMPTY_DOC thread stability issues > > [image removed] > > [image removed] > > > Use of EMPTY_DOC (an instance of an empty DOM Document element within > class XmlUtils) among multiple threads causes unpredictable results. > While testing a messaging service that uses the Apache Muse WS-N > implementation, I noticed that 4-6 messages in a 1000 were being dropped. > Notification messages were being sent at a rate of 20 per second. Where > the messages were dropped, the following stack trace occurred: > > 2007-08-24 15:27:58,031 ERROR [STDERR] Aug 24, 2007 3:27:58 PM > org.apache.muse.util.LoggingUtils logError > INFO: There was an error while processing a request: > > [ID = 'NoMessageContent'] The NotificationMessage XML does not have a > Message element. All messages must have a message payload associated with > them. > > org.apache.muse.ws.notification.impl. > SimpleNotificationMessage.<init>(SimpleNotificationMessage.java:117) > org.apache.muse.ws.notification.impl. > NotificationMessageSerializer.fromXML (NotificationMessageSerializer.java:46) > org.apache.muse.core.serializer.ArraySerializer. > fromXML(ArraySerializer.java:126) > org.apache.muse.ws.notification.impl.NotifyHandler. > fromXML(NotifyHandler.java:62) > org.apache.muse.core.SimpleResource.invoke (SimpleResource.java:368) > org.apache.muse.core.routing.SimpleResourceRouter. > invoke(SimpleResourceRouter.java:290) > org.apache.muse.core.platform.axis2.AxisIsolationLayer. > invoke(AxisIsolationLayer.java:136) > org.apache.muse.core.platform.axis2.AxisIsolationLayer. > handleRequest(AxisIsolationLayer.java:88) > sun.reflect.GeneratedMethodAccessor102.invoke(Unknown Source) > sun.reflect.DelegatingMethodAccessorImpl. > invoke(DelegatingMethodAccessorImpl.java:25) > java.lang.reflect.Method.invoke(Method.java:324) > org.apache.axis2.receivers.RawXMLINOutMessageReceiver. > invokeBusinessLogic(RawXMLINOutMessageReceiver.java:88) > org.apache.axis2.receivers.AbstractInOutSyncMessageReceiver. > receive(AbstractInOutSyncMessageReceiver.java:39) > org.apache.axis2.engine.AxisEngine.receive(AxisEngine.java:493) > org.apache.axis2.transport.http.HTTPTransportUtils. > processHTTPPostRequest(HTTPTransportUtils.java:319) > 22 more... > > All of the messages did in fact have a message payload associated with > them. After further investigation I noticed that > AxisEnvironment.convertToDOM is calling XmlUtils.createElement(QName) > which uses the common EMPTY_DOC Document instance. Sharing EMPTY_DOC > among multiple threads is not safe. > > I've also encountered this issue while attempting to set the producer > reference on a NotificationMessage. The code actually performs a deep > copy of the EndpointReference class which also relies on EMPTY_DOC. I > hacked a work-around within my code that avoided use of EMPTY_DOC and > resolved the issue. > > A grep for EMPTY_DOC against the code base turned up over 70 instances of > its use. That does not count methods that use it that are called by other > methods, so the actual usage is much higher. I'm going to open a JIRA bug > against this issue and submit patches for it. > > Related to that, I've submitted patches for other issues (MUSE-240, > MUSE-241) over a month ago that have not been committed. The patch for > MUSE-240 also resolves MUSE-225. Please let me know the appropriate > process for non-committers to have patches applied to the code base. When > I submitted the JIRA issues they showed up in this mailing list, but is an > explicit email more appropriate? Let me know how best to proceed. > > Thanks! > > Rich Lucente > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [EMAIL PROTECTED] > For additional commands, e-mail: [EMAIL PROTECTED] > > [attachment "EMPTY_DOC_errors.txt" deleted by Daniel Jemiolo/Durham/IBM] > --------------------------------------------------------------------- > To unsubscribe, e-mail: [EMAIL PROTECTED] > For additional commands, e-mail: [EMAIL PROTECTED]
