Hi Dan,
I've also done some testing just now and am finding this to be a very
serious issue. It really affects the usability of Muse as a whole.
To test, I created three resources. Each had a loop to generate 100
notifications with no pause between notifications. So out of 300 total
notifications generated, I had these results:
Test A = 9 notifications dropped
Test B = 24 notifications dropped
Attached is a file containing the various exceptions which caused
SimpleSubscriptionManager.publish() to fail. As Rich pointed out, the
culprit is using EMPTY_DOC from multiple threads (i.e. each client
request is one thread).
The XmlUtils.EMPTY_DOC javadocs has this:
"This should NOT be used by callers that want to build new fragments and
attach them to a Document...you should never append children."
But in XmlUtils itself and many other classes, the following code
patterns are used:
Document doc = XmlUtils.EMPTY_DOC;
Element xml = XmlUtils.createElement(doc, qname);
xml.appendChild(node);
OR:
doc.importNode(node, true);The doc is a shared object but is being modified, so errors will occur. So just about all the serializers are affected, including the EndpointReference class. This means errors will most likely occur when Muse handles requests/responses from multiple clients, or when notifications are sent from multiple resources. The latter is easier to test. So far, we've been testing using just one client and one producer instance, so the problem doesn't occur. But now that we are testing by using multiple producers, the exceptions are occuring frequently. To begin the fix, all serializers and the EndpointReference class needs to be patched. So instead of: doc = XmlUtils.EMPTY_DOC; We should do: doc = XmlUtils.createDocument(); The overhead of creating a new document is small when compared to the multi thread issue, which cannot be avoided. But, we can optimize creating new documents by updating XmlUtils to create the DocumentBuilderFactory only once. This way, XmlUtils.createBuilder() doesn't have to create a new factory everytime. -Vinh ________________________________ From: Daniel Jemiolo [mailto:[EMAIL PROTECTED] Sent: Monday, August 27, 2007 7:31 AM To: [email protected] Subject: Re: EMPTY_DOC thread stability issues The use of EMPTY_DOC was an attempt to avoid creating a new Document every time we wanted to copy or create small fragments of XML. This happened a lot during the request/response process, so the creation of these factory objects was not insignificant. Of course, you are right about the threading issue, so I guess we're out of luck there. I am setting aside Friday of this week to go through current JIRA items and apply all patches that have been submitted, close any issues that are recommended for closure, and sort unscheduled items into 2.3 if necessary. Sorry for the delay. Dan [EMAIL PROTECTED] [EMAIL PROTECTED] 08/24/2007 04:37 PM Please respond to [email protected] To [email protected] cc Subject EMPTY_DOC thread stability issues Use of EMPTY_DOC (an instance of an empty DOM Document element within class XmlUtils) among multiple threads causes unpredictable results. While testing a messaging service that uses the Apache Muse WS-N implementation, I noticed that 4-6 messages in a 1000 were being dropped. Notification messages were being sent at a rate of 20 per second. Where the messages were dropped, the following stack trace occurred: 2007-08-24 15:27:58,031 ERROR [STDERR] Aug 24, 2007 3:27:58 PM org.apache.muse.util.LoggingUtils logError INFO: There was an error while processing a request: [ID = 'NoMessageContent'] The NotificationMessage XML does not have a Message element. All messages must have a message payload associated with them. org.apache.muse.ws.notification.impl.SimpleNotificationMessage.<init>(Si mpleNotificationMessage.java:117) org.apache.muse.ws.notification.impl.NotificationMessageSerializer.fromX ML(NotificationMessageSerializer.java:46) org.apache.muse.core.serializer.ArraySerializer.fromXML(ArraySerializer. java:126) org.apache.muse.ws.notification.impl.NotifyHandler.fromXML(NotifyHandler .java:62) org.apache.muse.core.SimpleResource.invoke(SimpleResource.java:368) org.apache.muse.core.routing.SimpleResourceRouter.invoke(SimpleResourceR outer.java:290) org.apache.muse.core.platform.axis2.AxisIsolationLayer.invoke(AxisIsolat ionLayer.java:136) org.apache.muse.core.platform.axis2.AxisIsolationLayer.handleRequest(Axi sIsolationLayer.java:88) sun.reflect.GeneratedMethodAccessor102.invoke(Unknown Source) sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor Impl.java:25) java.lang.reflect.Method.invoke(Method.java:324) org.apache.axis2.receivers.RawXMLINOutMessageReceiver.invokeBusinessLogi c(RawXMLINOutMessageReceiver.java:88) org.apache.axis2.receivers.AbstractInOutSyncMessageReceiver.receive(Abst ractInOutSyncMessageReceiver.java:39) org.apache.axis2.engine.AxisEngine.receive(AxisEngine.java:493) org.apache.axis2.transport.http.HTTPTransportUtils.processHTTPPostReques t(HTTPTransportUtils.java:319) 22 more... All of the messages did in fact have a message payload associated with them. After further investigation I noticed that AxisEnvironment.convertToDOM is calling XmlUtils.createElement(QName) which uses the common EMPTY_DOC Document instance. Sharing EMPTY_DOC among multiple threads is not safe. I've also encountered this issue while attempting to set the producer reference on a NotificationMessage. The code actually performs a deep copy of the EndpointReference class which also relies on EMPTY_DOC. I hacked a work-around within my code that avoided use of EMPTY_DOC and resolved the issue. A grep for EMPTY_DOC against the code base turned up over 70 instances of its use. That does not count methods that use it that are called by other methods, so the actual usage is much higher. I'm going to open a JIRA bug against this issue and submit patches for it. Related to that, I've submitted patches for other issues (MUSE-240, MUSE-241) over a month ago that have not been committed. The patch for MUSE-240 also resolves MUSE-225. Please let me know the appropriate process for non-committers to have patches applied to the code base. When I submitted the JIRA issues they showed up in this mailing list, but is an explicit email more appropriate? Let me know how best to proceed. Thanks! Rich Lucente --------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
<<graycol.gif>>
<<ecblank.gif>>
17:26:49,730 ERROR [STDERR] java.lang.NullPointerException 17:26:49,730 ERROR [STDERR] at org.apache.xerces.dom.CoreDocumentImpl.importNode(Unknown Source) 17:26:49,730 ERROR [STDERR] at org.apache.xerces.dom.CoreDocumentImpl.importNode(Unknown Source) 17:26:49,730 ERROR [STDERR] at org.apache.muse.ws.addressing.EndpointReference.<init>(EndpointReference.java:186) 17:26:49,730 ERROR [STDERR] at org.apache.muse.ws.notification.impl.SimpleNotificationMessage.setProducerReference(SimpleNotificationMessage.java:209) 17:26:49,730 ERROR [STDERR] at org.apache.muse.ws.notification.impl.SimpleSubscriptionManager.publish(SimpleSubscriptionManager.java:256) 23:35:30,504 ERROR [STDERR] java.lang.NullPointerException 23:35:30,504 ERROR [STDERR] at org.apache.xerces.dom.ParentNode.nodeListItem(Unknown Source) 23:35:30,504 ERROR [STDERR] at org.apache.xerces.dom.ParentNode.item(Unknown Source) 23:35:30,504 ERROR [STDERR] at org.apache.muse.util.xml.XmlUtils.getAllElements(XmlUtils.java:883) 23:35:30,519 ERROR [STDERR] at org.apache.muse.util.xml.XmlUtils.getElement(XmlUtils.java:1095) 23:35:30,519 ERROR [STDERR] at org.apache.muse.util.xml.XmlUtils.getElement(XmlUtils.java:1070) 23:35:30,519 ERROR [STDERR] at org.apache.muse.ws.addressing.EndpointReference.initializeFromXML(EndpointReference.java:549) 23:35:30,519 ERROR [STDERR] at org.apache.muse.ws.addressing.EndpointReference.<init>(EndpointReference.java:192) 23:35:30,519 ERROR [STDERR] at org.apache.muse.ws.notification.impl.SimpleNotificationMessage.setProducerReference(SimpleNotificationMessage.java:209) 23:35:30,519 ERROR [STDERR] at org.apache.muse.ws.notification.impl.SimpleSubscriptionManager.publish(SimpleSubscriptionManager.java:256) 23:35:28,912 ERROR [STDERR] java.lang.NullPointerException 23:35:28,912 ERROR [STDERR] at org.apache.xerces.dom.ParentNode.nodeListItem(Unknown Source) 23:35:28,912 ERROR [STDERR] at org.apache.xerces.dom.ParentNode.item(Unknown Source) 23:35:28,912 ERROR [STDERR] at org.apache.muse.util.xml.XmlUtils.getAllElements(XmlUtils.java:883) 23:35:28,928 ERROR [STDERR] at org.apache.muse.util.xml.XmlUtils.getElement(XmlUtils.java:1095) 23:35:28,928 ERROR [STDERR] at org.apache.muse.util.xml.XmlUtils.getElement(XmlUtils.java:1070) 23:35:28,928 ERROR [STDERR] at org.apache.muse.util.xml.XmlUtils.getElementText(XmlUtils.java:1222) 23:35:28,928 ERROR [STDERR] at org.apache.muse.ws.addressing.EndpointReference.initializeFromXML(EndpointReference.java:530) 23:35:28,928 ERROR [STDERR] at org.apache.muse.ws.addressing.EndpointReference.<init>(EndpointReference.java:192) 23:35:28,928 ERROR [STDERR] at org.apache.muse.ws.notification.impl.SimpleNotificationMessage.setProducerReference(SimpleNotificationMessage.java:209) 23:35:28,928 ERROR [STDERR] at org.apache.muse.ws.notification.impl.SimpleSubscriptionManager.publish(SimpleSubscriptionManager.java:256) 23:35:28,647 ERROR [STDERR] java.lang.RuntimeException: [ID = 'InvalidEPRCreated'] The EndpointReference being copied is invalid. The XML serialization (or deserialization) is broken - it should not be possible to create an invalid EPR. 23:35:28,647 ERROR [STDERR] at org.apache.muse.ws.addressing.EndpointReference.<init>(EndpointReference.java:197) 23:35:28,662 ERROR [STDERR] at org.apache.muse.ws.notification.impl.SimpleNotificationMessage.setProducerReference(SimpleNotificationMessage.java:209) 23:35:28,662 ERROR [STDERR] at org.apache.muse.ws.notification.remote.NotificationConsumerClient.notify(NotificationConsumerClient.java:92) 23:35:28,662 ERROR [STDERR] at org.apache.muse.ws.notification.impl.SimpleSubscriptionManager.publish(SimpleSubscriptionManager.java:267) 23:35:28,662 ERROR [STDERR] ... 23:35:28,662 ERROR [STDERR] at java.lang.Thread.run(Unknown Source) 23:35:28,662 ERROR [STDERR] Caused by: org.apache.muse.ws.addressing.soap.SoapFault: [ID = 'InvalidAddressURI'] Invalid wsa:Address URI: null 23:35:28,662 ERROR [STDERR] at org.apache.muse.ws.addressing.EndpointReference.initializeFromXML(EndpointReference.java:541) 23:35:28,662 ERROR [STDERR] at org.apache.muse.ws.addressing.EndpointReference.<init>(EndpointReference.java:192) 23:35:28,662 ERROR [STDERR] ... 12 more 23:35:28,662 ERROR [STDERR] Caused by: java.lang.NullPointerException 23:35:28,662 ERROR [STDERR] at java.net.URI$Parser.parse(Unknown Source) 23:35:28,662 ERROR [STDERR] at java.net.URI.<init>(Unknown Source) 23:35:28,662 ERROR [STDERR] at org.apache.muse.ws.addressing.EndpointReference.initializeFromXML(EndpointReference.java:534) 23:35:28,662 ERROR [STDERR] ... 13 more 00:16:53,400 ERROR [STDERR] Publish failed: 00:16:53,400 ERROR [STDERR] java.lang.NullPointerException 00:16:53,400 ERROR [STDERR] at org.apache.xerces.dom.ParentNode.nodeListItem(Unknown Source) 00:16:53,400 ERROR [STDERR] at org.apache.xerces.dom.ParentNode.item(Unknown Source) 00:16:53,400 ERROR [STDERR] at org.apache.muse.util.xml.XmlUtils.getAllElements(XmlUtils.java:883) 00:16:53,400 ERROR [STDERR] at org.apache.muse.util.xml.XmlUtils.getAllElements(XmlUtils.java:815) 00:16:53,400 ERROR [STDERR] at org.apache.muse.util.xml.XmlUtils.getAllElements(XmlUtils.java:791) 00:16:53,400 ERROR [STDERR] at org.apache.muse.util.xml.XmlUtils.getAllNamespaces(XmlUtils.java:974) 00:16:53,400 ERROR [STDERR] at org.apache.muse.util.xml.XmlUtils.getAllNamespaces(XmlUtils.java:977) 00:16:53,400 ERROR [STDERR] at org.apache.muse.util.xml.XmlUtils.getAllNamespaces(XmlUtils.java:940) 00:16:53,400 ERROR [STDERR] at org.apache.muse.ws.notification.impl.SimpleNotificationMessage.toXML(SimpleNotificationMessage.java:291) 00:16:53,415 ERROR [STDERR] at org.apache.muse.ws.notification.impl.SimpleNotificationMessage.toXML(SimpleNotificationMessage.java:239) 00:16:53,415 ERROR [STDERR] at org.apache.muse.ws.notification.remote.NotificationConsumerClient.notify(NotificationConsumerClient.java:97) 00:16:53,415 ERROR [STDERR] at org.apache.muse.ws.notification.impl.SimpleSubscriptionManager.publish(SimpleSubscriptionManager.java:267)
--------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
