Back in 2005 I updated my earlier performance test http://www.sosnoski.com/presents/cleansoap/comparing.html to work with Axis2 and XMLBeans data binding (at the time ADB wouldn't handle the data, and I hadn't implemented the JiBX support). We had some discussion of the results on the mailing list, and I meant to contribute the test data but I don't think I ever did.
The Axis2 changes since then have created the need for more code changes, and I should also add ADB and JiBX versions for a full comparison. I'll try to get that done within the next couple of weeks. The nice thing about this test is that it uses real data of mixed types in a plausible structure, with scalable result set sizes. I think that makes for a more realistic comparison than the <int>1</int><int>2</int><int>3</int>-type data sets. - Dennis Dennis M. Sosnoski SOA and Web Services in Java Training and Consulting http://www.sosnoski.com - http://www.sosnoski.co.nz Seattle, WA +1-425-939-0576 - Wellington, NZ +64-4-298-6117 Sanjiva Weerawarana wrote: > These are great points. Would you be willing to help define an improved > benchmark that will hopefully become an industry wide meaningful > benchmark? > > We used the current benchmark because it came from a university (IU) and > has been used by multiple vendors before. However, its not > representative of all scenarios as Paul noted. > > We'd like to improve the benchmark and make it more useful. Then we will > automate the perf testing so that results can be posted regularly. > > Help from everyone appreciated. Please send schemas & WSDLs .. you can > open a JIRA (issue) against the benchmark on wso2.org and submit it > there (or just mail it to us). > > The benchmark is open source under Apache license. We welcome other > vendors to use it and improve it. > > Thanks, > > Sanjiva. > > On Wed, 2007-01-31 at 07:22 -0800, Donahue Bryon wrote: > >> Paul, >> >> I work for a major telecom and I've focused on this >> sort of work for may years. I appreciate the work done >> here. It is an important topic and I'm glad to see >> someone working in this area and publishing results. >> There are serious problems with Web Services >> performance; I see the results daily, generally after >> it is too late. >> >> The problem with publishing results, however, is that >> you take shots from the community and mine are below. >> I'd love to see them integrated into this work and I'm >> willing to discuss offline. >> >> Bottom Line: My view is the test design does not >> address the real problem and, therefore, the >> conclusion that "Web Services are not slow!" is >> unfounded. >> >> Web Services tools are generally used in synchronous >> mode. As a result, the overriding performance >> consideration is transaction latency from the client >> api's perspective. This test does not address that >> problem. Furthermore, one cannot test transaction >> latency without using the same WS toolkit on both the >> server and the client, which is not done in this test. >> >> >> Additional points: >> >> 1) We execute latency tests and the results show the >> performance of all Web Services toolkits to be poor >> relative to binary protocol based middlewares; by one, >> two, or three orders of magnitude depending on message >> size. After that the soap stacks exhaust memory or >> some other bad thing happens. >> >> 2) 4-6k is not a large message. In telecom for >> example, it is common for messages to contain networks >> of stuff or for corporate bills to contain massive >> numbers of line items. These data structures contain >> deeply nested arrays of arrays. We sometimes see >> messages where the payload contains a few megabytes >> and message payloads of several hundred kb are common. >> At present, we have a corporate limit on SOAP message >> payload size of a couple hundred kb because the WS >> toolkits are so poor. When messages are expected to >> exceed that size we must use a "real" middleware. >> >> 3) While your comment regarding messaging systems and >> MQ is correct, it is off base. The comparison you >> should consider is CORBA or EJB/IIOP. These >> frameworks, like Web Services, do the program language >> marshaling and Web Services is extremely slow relative >> to them. I encourage you to get JacORB or something, >> implement your test with it, and compare the results >> to your Web Services results. The result are shocking, >> especially as message size increases. >> >> 4) With your existing test, it would be interesting to >> execute the test a few more times but with >> increasingly larger message sizes, say 40-60k, >> 400-600k, 4-6M, and analyze the results. I'd love to >> know 4-6M messages even work. >> >> Bryon Donahue >> >> --- Paul Fremantle <[EMAIL PROTECTED]> wrote: >> >> >>> Steve >>> >>> Please remember that these messages are being parsed >>> into Java objects and >>> also echo'ed. So this isn't a good comparison with a >>> traditional messaging >>> system (like MQ) where the messages are typically >>> one-way and the actual >>> parsing processing of the body isn't considered part >>> of the test. >>> >>> The closest we did to your test was the >>> echoStrings-5000 test which had a >>> message size of 200k (therefore 400k throughput per >>> message) and we got more >>> that 400req/s which was completely saturating the >>> 1Gb/s ethernet segment. I >>> don't think you can really ask more?! >>> >>> Paul >>> >>> On 1/30/07, Steve Jones <[EMAIL PROTECTED]> >>> wrote: >>> >>>> Paul, >>>> >>>> Its interesting stuff, but I'd certainly say that >>>> >>> "serious work" in XML >>> >>>> land can often be significantly above 4-6k, and >>>> >>> often (when using industry >>> >>>> standard schemas) in the 200-500k range and pretty >>>> >>> complex structures. >>> >>>> Which means around 30 tps from the graph which >>>> >>> isn't very much. One of the >>> >>>> issues in industry is that there are great reasons >>>> >>> to use industry standard >>> >>>> schemas but they are pretty evil things as they >>>> >>> aim to cover all cases, this >>> >>>> makes them very inefficient. >>>> >>>> Have you folks done anything around how to scale >>>> >>> B2B and domain to domain >>> >>>> scenarios with large documents? From my wikipedia >>>> >>> ripping experience I'd >>> >>>> expect StAX to massively outperform JAXB due to >>>> >>> the large envelope but small >>> >>>> content pieces of those schemas. >>>> >>>> I'd be happy to provide some test schemas that >>>> >>> I've seen cause trouble. >>> >>>> Steve >>>> >>>> >>>> On 30/01/07, Paul Fremantle < [EMAIL PROTECTED]> >>>> >>> wrote: >>> >>>>> A while back we had a discussion on whether >>>>> >>> Web Services are slow. >>> >>>>> Here is some data that I think concludes that >>>>> >>> SOAP can scale to high >>> >>>>> transaction rates (e.g. 300 million transactions >>>>> >>> a day). The test >>> >>>>> isn't a real-world test, but it does show that >>>>> >>> the overhead of SOAP >>> >>>>> processing is minimal with the latest toolkits. >>>>> >>>>> Some quotes from the article. >>>>> ---------- >>>>> >>>>> This article shows the latest performance >>>>> >>> results of Apache Axis2 vs. >>> >>>>> Codehaus XFire, both Java implementations. The >>>>> >>> results demonstrate >>> >>>>> that modern Web Services engines can perform at >>>>> >>> very high transaction >>> >>>>> rates. >>>>> >>>>> Axis2 using the default ADB binding framework >>>>> >>> shows outstanding >>> >>>>> performance, with consistently better results >>>>> >>> than XFire/JAXB or >>> >>>>> Axis2/JAXB. >>>>> >>>>> Using either toolkit, the overhead of using XML >>>>> >>> and SOAP is no longer >>> >>>>> a limiting factor in writing distributed systems >>>>> >>> for most applications >>> >>>>> (with may be the exception of trading floors!). >>>>> >>> While these tests do >>> >>>>> not perform 'real' work, the fact that a XML >>>>> >>> messaging system can >>> >>>>> scale to more than 10 million transactions an >>>>> >>> hour on a single >>> >>>>> quad-core server shows that Web services can be >>>>> >>> used for significant >>> >>>>> systems applications. >>>>> >>>>> --------- >>>>> >>>>> Read more here: http://wso2.org/library/588 >>>>> >>>>> My disclaimer - I co-authored the document and >>>>> >>> I'm a committer on the >>> >>>>> Axis2 and other Apache WS projects. >>>>> >>>>> -- >>>>> Paul Fremantle >>>>> >>>>> http://bloglines.com/blog/paulfremantle >>>>> [EMAIL PROTECTED] <paul%40wso2.com> >>>>> >>>>> >>>> >>>> >>> >>> -- >>> Paul Fremantle >>> VP/Technology, WSO2 and OASIS WS-RX TC Co-chair >>> >>> http://bloglines.com/blog/paulfremantle >>> [EMAIL PROTECTED] >>> >>> "Oxygenating the Web Service Platform", www.wso2.com >>> >>> >> >> >> ____________________________________________________________________________________ >> Never miss an email again! >> Yahoo! Toolbar alerts you the instant new Mail arrives. >> http://tools.search.yahoo.com/toolbar/features/mail/ >> >> >> >> Yahoo! Groups Links >> >> >> >>
