Bryon Thanks for the note. I agree that Web Services are not the answer for all situations. Of course a binary protocol will be faster than an XML protocol - especially if its implemented correctly.
However, I was trying to place this in context. There is a wide spectrum of requirements, and many users do not need sub-millisecond latency and very large datasets. The sort of customers who do more than 100 million transactions a day are fairly few and far between. I don't disagree with you - I just think that your scenarios aren't the only ones. I also agree we didn't do latency testing, because we only looked at server performance. To give you some (server-side) numbers, the latency for 200k in/200k out (strings) was 56ms including the network, so I imagine the total latency would be around 100ms including client ser/deser. All that said, there is still plenty of scope for improvement. We know that we are doing too much character<->byte conversion, and we also know that a binary XML representation could have a significant impact. So there's still scope for growth. Paul On 1/31/07, Donahue Bryon <[EMAIL PROTECTED]> wrote: > Paul, > > I work for a major telecom and I've focused on this > sort of work for may years. I appreciate the work done > here. It is an important topic and I'm glad to see > someone working in this area and publishing results. > There are serious problems with Web Services > performance; I see the results daily, generally after > it is too late. > > The problem with publishing results, however, is that > you take shots from the community and mine are below. > I'd love to see them integrated into this work and I'm > willing to discuss offline. > > Bottom Line: My view is the test design does not > address the real problem and, therefore, the > conclusion that "Web Services are not slow!" is > unfounded. > > Web Services tools are generally used in synchronous > mode. As a result, the overriding performance > consideration is transaction latency from the client > api's perspective. This test does not address that > problem. Furthermore, one cannot test transaction > latency without using the same WS toolkit on both the > server and the client, which is not done in this test. > > > Additional points: > > 1) We execute latency tests and the results show the > performance of all Web Services toolkits to be poor > relative to binary protocol based middlewares; by one, > two, or three orders of magnitude depending on message > size. After that the soap stacks exhaust memory or > some other bad thing happens. > > 2) 4-6k is not a large message. In telecom for > example, it is common for messages to contain networks > of stuff or for corporate bills to contain massive > numbers of line items. These data structures contain > deeply nested arrays of arrays. We sometimes see > messages where the payload contains a few megabytes > and message payloads of several hundred kb are common. > At present, we have a corporate limit on SOAP message > payload size of a couple hundred kb because the WS > toolkits are so poor. When messages are expected to > exceed that size we must use a "real" middleware. > > 3) While your comment regarding messaging systems and > MQ is correct, it is off base. The comparison you > should consider is CORBA or EJB/IIOP. These > frameworks, like Web Services, do the program language > marshaling and Web Services is extremely slow relative > to them. I encourage you to get JacORB or something, > implement your test with it, and compare the results > to your Web Services results. The result are shocking, > especially as message size increases. > > 4) With your existing test, it would be interesting to > execute the test a few more times but with > increasingly larger message sizes, say 40-60k, > 400-600k, 4-6M, and analyze the results. I'd love to > know 4-6M messages even work. > > Bryon Donahue > > --- Paul Fremantle <[EMAIL PROTECTED]> wrote: > > > Steve > > > > Please remember that these messages are being parsed > > into Java objects and > > also echo'ed. So this isn't a good comparison with a > > traditional messaging > > system (like MQ) where the messages are typically > > one-way and the actual > > parsing processing of the body isn't considered part > > of the test. > > > > The closest we did to your test was the > > echoStrings-5000 test which had a > > message size of 200k (therefore 400k throughput per > > message) and we got more > > that 400req/s which was completely saturating the > > 1Gb/s ethernet segment. I > > don't think you can really ask more?! > > > > Paul > > > > On 1/30/07, Steve Jones <[EMAIL PROTECTED]> > > wrote: > > > > > > Paul, > > > > > > Its interesting stuff, but I'd certainly say that > > "serious work" in XML > > > land can often be significantly above 4-6k, and > > often (when using industry > > > standard schemas) in the 200-500k range and pretty > > complex structures. > > > Which means around 30 tps from the graph which > > isn't very much. One of the > > > issues in industry is that there are great reasons > > to use industry standard > > > schemas but they are pretty evil things as they > > aim to cover all cases, this > > > makes them very inefficient. > > > > > > Have you folks done anything around how to scale > > B2B and domain to domain > > > scenarios with large documents? From my wikipedia > > ripping experience I'd > > > expect StAX to massively outperform JAXB due to > > the large envelope but small > > > content pieces of those schemas. > > > > > > I'd be happy to provide some test schemas that > > I've seen cause trouble. > > > > > > Steve > > > > > > > > > On 30/01/07, Paul Fremantle < [EMAIL PROTECTED]> > > wrote: > > > > > > > A while back we had a discussion on whether > > Web Services are slow. > > > > > > > > Here is some data that I think concludes that > > SOAP can scale to high > > > > transaction rates (e.g. 300 million transactions > > a day). The test > > > > isn't a real-world test, but it does show that > > the overhead of SOAP > > > > processing is minimal with the latest toolkits. > > > > > > > > Some quotes from the article. > > > > ---------- > > > > > > > > This article shows the latest performance > > results of Apache Axis2 vs. > > > > Codehaus XFire, both Java implementations. The > > results demonstrate > > > > that modern Web Services engines can perform at > > very high transaction > > > > rates. > > > > > > > > Axis2 using the default ADB binding framework > > shows outstanding > > > > performance, with consistently better results > > than XFire/JAXB or > > > > Axis2/JAXB. > > > > > > > > Using either toolkit, the overhead of using XML > > and SOAP is no longer > > > > a limiting factor in writing distributed systems > > for most applications > > > > (with may be the exception of trading floors!). > > While these tests do > > > > not perform 'real' work, the fact that a XML > > messaging system can > > > > scale to more than 10 million transactions an > > hour on a single > > > > quad-core server shows that Web services can be > > used for significant > > > > systems applications. > > > > > > > > --------- > > > > > > > > Read more here: http://wso2.org/library/588 > > > > > > > > My disclaimer - I co-authored the document and > > I'm a committer on the > > > > Axis2 and other Apache WS projects. > > > > > > > > -- > > > > Paul Fremantle > > > > > > > > http://bloglines.com/blog/paulfremantle > > > > [EMAIL PROTECTED] <paul%40wso2.com> > > > > > > > > > > > > > > > > > > > > > -- > > Paul Fremantle > > VP/Technology, WSO2 and OASIS WS-RX TC Co-chair > > > > http://bloglines.com/blog/paulfremantle > > [EMAIL PROTECTED] > > > > "Oxygenating the Web Service Platform", www.wso2.com > > > > > > > ____________________________________________________________________________________ > Never miss an email again! > Yahoo! Toolbar alerts you the instant new Mail arrives. > http://tools.search.yahoo.com/toolbar/features/mail/ > > > > Yahoo! Groups Links > > > > -- Paul Fremantle VP/Technology, WSO2 and OASIS WS-RX TC Co-chair http://bloglines.com/blog/paulfremantle [EMAIL PROTECTED] "Oxygenating the Web Service Platform", www.wso2.com
