Re: Really large amounts of data, was Re: XML-RPC for Database Access

Ryan Hoegg Fri, 21 Nov 2003 20:16:29 -0800

I had a similar issue because the server was using HTTP 1.1 and the client was requesting HTTP 1.1 but only accepting HTTP 1.0. HTTP 1.1 uses chunking, which messes up the XML returned by the server if it is interpreted as if it was not chunked.

You can fix it by
a) changing the server to use HTTP 1.0
b) making the client request HTTP 1.0
c) making the client understand HTTP 1.1

I opted for b first, then c. If you use the code in CVS HEAD, you can plug in a CommonsXmlRpcClient, which supports HTTP 1.1

--
Ryan Hoegg
ISIS Networks
http://www.isisnetworks.net

Greg Barnett wrote:

Bruce Van Horn wrote:

about a year and a half ago I built a transaction server that took a series of queries in an xml document via xmlrpc.

It works great as long as you don't try to move really large amounts of data or BLOBs.

I'm currently writing a script that needs to retrieve a large amount of data via XML-RPC. When the server returns a smallish amount of data, everything works correctly. When a large amount of data is returned, it looks like not all of the data is getting sent to the SAX parser. I have tested the server with other clients, and they are able to receive all of the data just fine.

Here is the relevant portion of my code (if it looks a bit odd, it is because it is written in Jython).
############################################################
from org.apache.xmlrpc import XmlRpcClient
from org.apache.xmlrpc import XmlRpc
def xmlrpc(method, params, username, password):
    #XmlRpc.setDriver('org.apache.xerces.parsers.SAXParser')
    XmlRpc.setDebug(true)
    client = XmlRpcClient('https://host/cgi-bin/file.py')
    client.setBasicAuthentication(username, password)
    return client.execute(method, params)
############################################################
The server returns a list of dictionaries. I've had the client successfully retrieve up to about 10,000 dictionaries in the list. Typically, it seems to have problems with more than a few hundred dictionaries.

My original thought was that it was the default SAX parser, but switching to xerces just gave a slightly different error message.

Is there any way to make sure that all of the data gets read by the parser?
Greg Barnett
firstname dot lastname at crowncollege dot edu
This is with the xerces SAX parser:

Created client to url space https://host/cgi-bin/file.py Client calling procedure 'foo.bar' with parameters [spam , eggs] Beginning parsing XML input stream startElement: methodResponse startElement: params startElement: param startElement: value startElement: array startElement: data startElement: value startElement: struct startElement: member startElement: name endElement: name startElement: value startElement: string endElement: string endElement: value endElement: member startElement: member startElement: name [ 2700 lines snipped ] startElement: value startElement: struct startElement: member startElement: name Fatal error parsing XML: org.xml.sax.SAXParseException: The element type "name" must be terminated by the matching end-tag "</name>". org.xml.sax.SAXParseException: The element type "name" must be terminated by the matching end-tag "</name>". at org.apache.xerces.parsers.AbstractSAXParser.parse(Unknown Source) at org.apache.xmlrpc.XmlRpc.parse(XmlRpc.java:419) at org.apache.xmlrpc.XmlRpcClient$Worker.execute(XmlRpcClient.java:436) at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:163) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.lang.reflect.Method.invoke(Unknown Source) at org.python.core.PyReflectedFunction.__call__(PyReflectedFunction.java) at org.python.core.PyMethod.__call__(PyMethod.java) at org.python.core.PyObject.__call__(PyObject.java) at org.python.core.PyInstance.invoke(PyInstance.java) at org.python.pycode._pyx0.xmlrpc$3(h:\cvswork\ccdb\client\JythonClient.py:52)

at org.python.pycode._pyx0.call_function(h:\cvswork\ccdb\client\JythonClient.py)

at org.python.core.PyTableCode.call(PyTableCode.java) at org.python.core.PyTableCode.call(PyTableCode.java) at org.python.core.PyFunction.__call__(PyFunction.java) at org.python.core.PyInstance.invoke(PyInstance.java) at org.python.pycode._pyx0.updateTest$4(h:\cvswork\ccdb\client\JythonClient.py:81)

at org.python.pycode._pyx0.call_function(h:\cvswork\ccdb\client\JythonClient.py)

at org.python.core.PyTableCode.call(PyTableCode.java) at org.python.core.PyTableCode.call(PyTableCode.java) at org.python.core.PyTableCode.call(PyTableCode.java) at org.python.core.PyFunction.__call__(PyFunction.java) at org.python.core.PyMethod.__call__(PyMethod.java) at org.python.core.PyCompoundCallable.__call__(PyCompoundCallable.java) at org.python.core.PyObject.__call__(PyObject.java) at org.python.core.PyObject._jcallexc(PyObject.java) at org.python.core.PyObject._jcall(PyObject.java) at org.python.proxies.java.awt.event.ActionListener$Adapter.actionPerformed(Unknown Source) at javax.swing.AbstractButton.fireActionPerformed(Unknown Source) at javax.swing.AbstractButton$ForwardActionEvents.actionPerformed(Unknown Source) at javax.swing.DefaultButtonModel.fireActionPerformed(Unknown Source) at javax.swing.DefaultButtonModel.setPressed(Unknown Source) at javax.swing.plaf.basic.BasicButtonListener.mouseReleased(Unknown Source) at java.awt.Component.processMouseEvent(Unknown Source) at java.awt.Component.processEvent(Unknown Source) at java.awt.Container.processEvent(Unknown Source) at java.awt.Component.dispatchEventImpl(Unknown Source) at java.awt.Container.dispatchEventImpl(Unknown Source) at java.awt.Component.dispatchEvent(Unknown Source) at java.awt.LightweightDispatcher.retargetMouseEvent(Unknown Source) at java.awt.LightweightDispatcher.processMouseEvent(Unknown Source) at java.awt.LightweightDispatcher.dispatchEvent(Unknown Source) at java.awt.Container.dispatchEventImpl(Unknown Source) at java.awt.Window.dispatchEventImpl(Unknown Source) at java.awt.Component.dispatchEvent(Unknown Source) at java.awt.EventQueue.dispatchEvent(Unknown Source) at java.awt.EventDispatchThread.pumpOneEventForHierarchy(Unknown Source) at java.awt.EventDispatchThread.pumpEventsForHierarchy(Unknown Source) at java.awt.EventDispatchThread.pumpEvents(Unknown Source) at java.awt.EventDispatchThread.pumpEvents(Unknown Source) at java.awt.EventDispatchThread.run(Unknown Source) Traceback (innermost last): (no code object) at line 0

this is with the default parser:

Created client to url space https://host/cgi-bin/file.py Client calling procedure 'foo.bar' with parameters [spam, eggs] Beginning parsing XML input stream startElement: methodResponse startElement: params startElement: param startElement: value startElement: array startElement: data startElement: value startElement: struct startElement: member startElement: name endElement: name startElement: value startElement: string endElement: string endElement: value endElement: member startElement: member startElement: name <snip> endElement: string endElement: value endElement: member startElement: member startElement: name endElement: name startElement: value startElement: string endElement: string endElement: value endElement: member endElement: struct endElement: value startElement: value startElement: struct startElement: member startElement: name Fatal error parsing XML: org.xml.sax.SAXParseException: unexpected end of file found Spent 401 millis parsing Spent 10325 in request

Re: Really large amounts of data, was Re: XML-RPC for Database Access

Reply via email to