Hello,
I have recently completed a series of simple performance measurements for Apache Axis. I wanted to share the results and see if I could get any new insights on the data.
To check out the results you should go to the following URL:
http://www.atl.external.lmco.com/projects/QoS/compare/dist_oo_compare_ipc.html
Here you will see a tool developed here at Lockheed Martin's Advanced Technology Laboratory for comparing various forms of IPC. There are a ton of results for all different types of middleware solutions. You can browse through them by looking at the links in the left frame. If you want to go directly to the various SOAP results please use the filter located at the top of the left frame. You can enter the word 'xml', without the quotes, into the box and hit enter. All of our recent results for web services should appear. It should look something like this:
smp/ web-service-and-xml-rpc/ apache-axis/ v1.1_attachments v1.1_dime v1.1_inline apache-xml-rpc/ v1.2-b1 XSOAP/ v1.2.23
First and foremost, if you click on the "apache-axis/" link you will get our attempt at a detailed explanation of all the conditions in which these tests were conducted. You can then look at the three sets of results my clicking on either, SOAP with attachments, DIME, or simple base64 encoded messages(byte arrays). All of our tests are simple round trip latency tests that involve sending a message of x bytes and waiting for an acknowledgment. We time this exchange for a size of x varying from 4 to 65536 on every power of 2. The results you are looking at should all be 1 million samples for each message size.
I have attached a graph I made real quick using the above tool. It contains a comparison of apache-axis to a couple other middleware's. These include basic tcp, apache implementation of xml-rpc, XSOAP (aka soaprmi), and the TAO orb. We are trying to understand a number of things.
Firstly, where is the large overhead coming from? Obviously the use of XML in the SOAP protocol makes for more verbose messages. In addition, we also considered the creation of a new connection for each RPC(we are currently looking into developing some simple tests to examine this issue). Are these two issues enough to explain this kind of overhead or are we not considering something?
Secondly, if you look closely at the curves for XSOAP and apache-axis base64 messages they are extremely similar. We do not usually see results that are this close when testing two different implementations of any middleware. Would anyone know of a specific reason why these two implementations would be so close?
Finally, you can see that there is a break in performance of axis with attachments and DIME at around 16384 size messages. We are assuming this is because that is when axis begins writing temp files for every attachment to the attachments/ directory. Is there a way to turn this behavior off?
Thanks for reading this long email and I am anxious to here any comments or questions.
David Turner ATL
<<attachment: apache-axis-cmp.png>>