To answer Mike's comment first. What I was expecting to see was a slow
incremental increase in response time with load up to some point where the
server saturates when the response time sky-rockets, probably with some
instability near the saturation point. In terms of your equation "a + b *
load", to illustrate the shape of the graph I'd expect, b would be small say
around 0.05 and constant, at saturation b would increase significantly say
to something like 20 and b would be unstable near the saturation point. What
I'm seeing is b approximately equal to 1 maybe 1.1 near the end of the run.
I would have expected the test condition to allow for stabilization between
increments. I started with 10 users and was adding another 10 at 60 second
intervals for 10 thru 500 users the ramp up took 49 minutes, if you graph
the users with time you get a staircase, the response time graph looks like
a plank laid on the stairs. Towards the end I was seeing more than linear
growth. Another test, albeit under slightly different conditions, with a
slower ramp of 10 users every 300 seconds to a max of 200 showed a similar
response graph however I can't rule out a different cause in that case.
To answer Toshi's comment, the configuration I was testing loop a response
back in the method proxy, the test payload is an echo request so I was able
to simply instantiate our response object and move the payload without
entering our application framework. The only non-Axis/infrastructure
processing is instantiate, move object reference, return. I'll double check
the instantiate doesn't involve a synchronize though.
The thread dump as requested, I've trimmed it to the lock holder and one on
the waiters, the other two were at the same location in code.
"Thread-23" daemon prio=5 tid=0x03650408 nid=0xfd8 runnable
[43df000..43dfd94]
at
org.apache.xerces.jaxp.DocumentBuilderFactoryImpl.newDocumentBuilder(Unknown
Source)
at org.apache.axis.utils.XMLUtils.newDocument(XMLUtils.java:318)
- locked <0x10ac8688> (a
org.apache.xerces.jaxp.DocumentBuilderFactoryImpl)
at
org.apache.axis.message.MessageElement.getAsDocument(MessageElement.java:688
)
at
org.apache.axis.message.MessageElement.getAsDOM(MessageElement.java:714)
at
org.apache.axis.providers.java.MsgProvider.processMessage(MsgProvider.java:1
44)
at
org.apache.axis.providers.java.JavaProvider.invoke(JavaProvider.java:333)
at
org.apache.axis.strategies.InvocationStrategy.visit(InvocationStrategy.java:
71)
at org.apache.axis.SimpleChain.doVisiting(SimpleChain.java:150)
at org.apache.axis.SimpleChain.invoke(SimpleChain.java:120)
at
org.apache.axis.handlers.soap.SOAPService.invoke(SOAPService.java:481)
at org.apache.axis.server.AxisServer.invoke(AxisServer.java:323)
at
org.apache.axis.transport.http.AxisServlet.doPost(AxisServlet.java:854)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:760)
at
org.apache.axis.transport.http.AxisServletBase.service(AxisServletBase.java:
339)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Application
FilterChain.java:247)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterCh
ain.java:193)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.ja
va:256)
at
org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invok
eNext(StandardPipeline.java:643)
at
org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)
at
org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.ja
va:191)
at
org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invok
eNext(StandardPipeline.java:643)
at
org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)
at
org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
at
org.apache.catalina.core.StandardContext.invoke(StandardContext.java:2415)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:180
)
at
org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invok
eNext(StandardPipeline.java:643)
at
org.apache.catalina.valves.ErrorDispatcherValve.invoke(ErrorDispatcherValve.
java:171)
at
org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invok
eNext(StandardPipeline.java:641)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:172
)
at
org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invok
eNext(StandardPipeline.java:641)
at
org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)
at
org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java
:174)
at
org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invok
eNext(StandardPipeline.java:643)
at
org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)
at
org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
at
org.apache.coyote.tomcat4.CoyoteAdapter.service(CoyoteAdapter.java:223)
at
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:594)
at
org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.processConne
ction(Http11Protocol.java:392)
at
org.apache.tomcat.util.net.TcpWorkerThread.runIt(PoolTcpEndpoint.java:565)
at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.jav
a:619)
at java.lang.Thread.run(Thread.java:534)
"Thread-22" daemon prio=5 tid=0x035d4930 nid=0x734 waiting for monitor entry
[439f000..439fd94]
at org.apache.axis.utils.XMLUtils.newDocument(XMLUtils.java:317)
- waiting to lock <0x10ac8688> (a
org.apache.xerces.jaxp.DocumentBuilderFactoryImpl)
at
org.apache.axis.message.MessageElement.getAsDocument(MessageElement.java:688
)
at
org.apache.axis.message.MessageElement.getAsDOM(MessageElement.java:714)
at
org.apache.axis.providers.java.MsgProvider.processMessage(MsgProvider.java:1
44)
at
org.apache.axis.providers.java.JavaProvider.invoke(JavaProvider.java:333)
at
org.apache.axis.strategies.InvocationStrategy.visit(InvocationStrategy.java:
71)
at org.apache.axis.SimpleChain.doVisiting(SimpleChain.java:150)
at org.apache.axis.SimpleChain.invoke(SimpleChain.java:120)
at
org.apache.axis.handlers.soap.SOAPService.invoke(SOAPService.java:481)
at org.apache.axis.server.AxisServer.invoke(AxisServer.java:323)
at
org.apache.axis.transport.http.AxisServlet.doPost(AxisServlet.java:854)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:760)
at
org.apache.axis.transport.http.AxisServletBase.service(AxisServletBase.java:
339)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Application
FilterChain.java:247)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterCh
ain.java:193)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.ja
va:256)
at
org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invok
eNext(StandardPipeline.java:643)
at
org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)
at
org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.ja
va:191)
at
org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invok
eNext(StandardPipeline.java:643)
at
org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)
at
org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
at
org.apache.catalina.core.StandardContext.invoke(StandardContext.java:2415)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:180
)
at
org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invok
eNext(StandardPipeline.java:643)
at
org.apache.catalina.valves.ErrorDispatcherValve.invoke(ErrorDispatcherValve.
java:171)
at
org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invok
eNext(StandardPipeline.java:641)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:172
)
at
org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invok
eNext(StandardPipeline.java:641)
at
org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)
at
org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java
:174)
at
org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invok
eNext(StandardPipeline.java:643)
at
org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)
at
org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
at
org.apache.coyote.tomcat4.CoyoteAdapter.service(CoyoteAdapter.java:223)
at
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:594)
at
org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.processConne
ction(Http11Protocol.java:392)
at
org.apache.tomcat.util.net.TcpWorkerThread.runIt(PoolTcpEndpoint.java:565)
at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.jav
a:619)
at java.lang.Thread.run(Thread.java:534)
Thanks
Ray Sutton
Systems Engineer
303-397-5292
-----Original Message-----
From: Sutton, Ray [mailto:[EMAIL PROTECTED]
Sent: Thursday, October 02, 2003 5:09 PM
To: '[EMAIL PROTECTED]'
Subject: Axis Response Time appears linear with load
Hello,
I've been tasked (for my sins, which must have been huge!) with performance
testing an application framework that sits on top of Tomcat and uses Axis as
the soap server. My initial testing slams the server as hard as possible
with 10 - 500 users stepping up 10 ever 60 seconds, each running an
identical script in an endless loop. The purpose being to identify any
points where synchronization was occurring.� I'm seeing a response time that
is almost linear with load (97% correlation) which implies, too me, a
synchronization issue. I started to localize the problem by isolating
components. Currently I have the soap method proxy turning the response
around without calling the application framework, so the only components
under test is Axis, ignoring the infrastructure which is a 4CPU 2Gb blade
server running Tomcat & the 1.4.1 JVM on RedHat Advanced Server.
I'm still seeing the linear response time change with increasing load.
Further investigation with JProbe (under XP) pointed the finger at
javax.xml.parsers.DocumentBuilderFactory. A JVM level thread dump with 10
users hitting the server showed org.apache.xerces.DocumentBuilderFactoryImpl
holding a lock and 3 other threads waiting on the lock.��
That's the background, now the question.
Has anybody encountered a similar problem and could give me some pointers on
possible configuration options or other any other suggestion that might help
remove this bottleneck.
If there's a well documented solution to this then, firstly I'd love a
pointer to it as a 30min search on Google failed to find anything that
seemed relevant.� Secondly I apologize for wasting people's time,
web-services/soap is a new area for me & I'm under some time pressure on
this task so it's quite possible I've missed the obvious.
Thanks
Ray Sutton
Systems Engineer
The information in this electronic mail message is sender's business
Confidential and may be legally privileged.� It is intended solely for the
addressee(s).� Access to this Internet electronic mail message by anyone
else is unauthorized.� If you are not the intended recipient, any
disclosure, copying, distribution or any action taken or omitted to be taken
in reliance on it is prohibited and may be unlawful.
The sender believes that this E-mail and any attachments were free of any
virus, worm, Trojan horse, and/or malicious code when sent. This message and
its attachments could have been infected during� transmission. By reading
the message and opening any attachments, the recipient accepts full
responsibility for taking protective and remedial action about viruses and
other defects. Galileo International is not liable for any loss or damage
arising in any way from this message or its attachments.
The information in this electronic mail message is sender's business
Confidential and may be legally privileged. It is intended solely for the
addressee(s). Access to this Internet electronic mail message by anyone
else is unauthorized. If you are not the intended recipient, any
disclosure, copying, distribution or any action taken or omitted to be taken
in reliance on it is prohibited and may be unlawful.
The sender believes that this E-mail and any attachments were free of any
virus, worm, Trojan horse, and/or malicious code when sent. This message and
its attachments could have been infected during transmission. By reading
the message and opening any attachments, the recipient accepts full
responsibility for taking protective and remedial action about viruses and
other defects. Galileo International is not liable for any loss or damage
arising in any way from this message or its attachments.
