Paul, I work for a major telecom and I've focused on this sort of work for may years. I appreciate the work done here. It is an important topic and I'm glad to see someone working in this area and publishing results. There are serious problems with Web Services performance; I see the results daily, generally after it is too late.
The problem with publishing results, however, is that you take shots from the community and mine are below. I'd love to see them integrated into this work and I'm willing to discuss offline. Bottom Line: My view is the test design does not address the real problem and, therefore, the conclusion that "Web Services are not slow!" is unfounded. Web Services tools are generally used in synchronous mode. As a result, the overriding performance consideration is transaction latency from the client api's perspective. This test does not address that problem. Furthermore, one cannot test transaction latency without using the same WS toolkit on both the server and the client, which is not done in this test. Additional points: 1) We execute latency tests and the results show the performance of all Web Services toolkits to be poor relative to binary protocol based middlewares; by one, two, or three orders of magnitude depending on message size. After that the soap stacks exhaust memory or some other bad thing happens. 2) 4-6k is not a large message. In telecom for example, it is common for messages to contain networks of stuff or for corporate bills to contain massive numbers of line items. These data structures contain deeply nested arrays of arrays. We sometimes see messages where the payload contains a few megabytes and message payloads of several hundred kb are common. At present, we have a corporate limit on SOAP message payload size of a couple hundred kb because the WS toolkits are so poor. When messages are expected to exceed that size we must use a "real" middleware. 3) While your comment regarding messaging systems and MQ is correct, it is off base. The comparison you should consider is CORBA or EJB/IIOP. These frameworks, like Web Services, do the program language marshaling and Web Services is extremely slow relative to them. I encourage you to get JacORB or something, implement your test with it, and compare the results to your Web Services results. The result are shocking, especially as message size increases. 4) With your existing test, it would be interesting to execute the test a few more times but with increasingly larger message sizes, say 40-60k, 400-600k, 4-6M, and analyze the results. I'd love to know 4-6M messages even work. Bryon Donahue --- Paul Fremantle <[EMAIL PROTECTED]> wrote: > Steve > > Please remember that these messages are being parsed > into Java objects and > also echo'ed. So this isn't a good comparison with a > traditional messaging > system (like MQ) where the messages are typically > one-way and the actual > parsing processing of the body isn't considered part > of the test. > > The closest we did to your test was the > echoStrings-5000 test which had a > message size of 200k (therefore 400k throughput per > message) and we got more > that 400req/s which was completely saturating the > 1Gb/s ethernet segment. I > don't think you can really ask more?! > > Paul > > On 1/30/07, Steve Jones <[EMAIL PROTECTED]> > wrote: > > > > Paul, > > > > Its interesting stuff, but I'd certainly say that > "serious work" in XML > > land can often be significantly above 4-6k, and > often (when using industry > > standard schemas) in the 200-500k range and pretty > complex structures. > > Which means around 30 tps from the graph which > isn't very much. One of the > > issues in industry is that there are great reasons > to use industry standard > > schemas but they are pretty evil things as they > aim to cover all cases, this > > makes them very inefficient. > > > > Have you folks done anything around how to scale > B2B and domain to domain > > scenarios with large documents? From my wikipedia > ripping experience I'd > > expect StAX to massively outperform JAXB due to > the large envelope but small > > content pieces of those schemas. > > > > I'd be happy to provide some test schemas that > I've seen cause trouble. > > > > Steve > > > > > > On 30/01/07, Paul Fremantle < [EMAIL PROTECTED]> > wrote: > > > > > A while back we had a discussion on whether > Web Services are slow. > > > > > > Here is some data that I think concludes that > SOAP can scale to high > > > transaction rates (e.g. 300 million transactions > a day). The test > > > isn't a real-world test, but it does show that > the overhead of SOAP > > > processing is minimal with the latest toolkits. > > > > > > Some quotes from the article. > > > ---------- > > > > > > This article shows the latest performance > results of Apache Axis2 vs. > > > Codehaus XFire, both Java implementations. The > results demonstrate > > > that modern Web Services engines can perform at > very high transaction > > > rates. > > > > > > Axis2 using the default ADB binding framework > shows outstanding > > > performance, with consistently better results > than XFire/JAXB or > > > Axis2/JAXB. > > > > > > Using either toolkit, the overhead of using XML > and SOAP is no longer > > > a limiting factor in writing distributed systems > for most applications > > > (with may be the exception of trading floors!). > While these tests do > > > not perform 'real' work, the fact that a XML > messaging system can > > > scale to more than 10 million transactions an > hour on a single > > > quad-core server shows that Web services can be > used for significant > > > systems applications. > > > > > > --------- > > > > > > Read more here: http://wso2.org/library/588 > > > > > > My disclaimer - I co-authored the document and > I'm a committer on the > > > Axis2 and other Apache WS projects. > > > > > > -- > > > Paul Fremantle > > > > > > http://bloglines.com/blog/paulfremantle > > > [EMAIL PROTECTED] <paul%40wso2.com> > > > > > > > > > > > > > -- > Paul Fremantle > VP/Technology, WSO2 and OASIS WS-RX TC Co-chair > > http://bloglines.com/blog/paulfremantle > [EMAIL PROTECTED] > > "Oxygenating the Web Service Platform", www.wso2.com > ____________________________________________________________________________________ Never miss an email again! Yahoo! Toolbar alerts you the instant new Mail arrives. http://tools.search.yahoo.com/toolbar/features/mail/
