Hi all,
I am doing some performance data collection on tomcat7 on a 8 processor
high end machine with CentOS release 6.7 (Final) operating system and i am
observing slowness with subsequent runs of the same jmeter test. Here are
the version and specfication details followed by the endpoint source and
the problem statement finally.
##############################################
Tomcat7 Version ==>
##############################################
Server version: Apache Tomcat/7.0.64
Server built: Aug 19 2015 17:18:06 UTC
Server number:* 7.0.64.0*
OS Name: Linux
OS Version: 2.6.32-573.3.1.el6.x86_64
Architecture: amd64
JVM Version: 1.7.0_79-b15
JVM Vendor: Oracle Corporation
##############################################
##############################################
Connector configurations
##############################################
<Connector port="8443" protocol="org.apache.coyote.http11.
*Http11NioProtocol*" SSLEnabled="true"
maxThreads="150" scheme="https" secure="true"
keystoreType="PKCS12"
keystoreFile="XXXXXX"
keystorePass="password" clientAuth="false"
sslEnabledProtocols="TLSv1,TLSv1.1,TLSv1.2" />
##############################################
##############################################
Machine Specifications
##############################################
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 8
On-line CPU(s) list: 0-7
Thread(s) per core: 2
Core(s) per socket: 4
Socket(s): 1
NUMA node(s): 1
Vendor ID: GenuineIntel
CPU family: 6
Model: 58
Stepping: 9
CPU MHz: 1600.000
BogoMIPS: 7000.55
Virtualization: VT-x
L1d cache: 32K
L1i cache: 32K
L2 cache: 256K
L3 cache: 8192K
NUMA node0 CPU(s): 0-7
##############################################
##############################################
Source code of the endpoint being performance tested
##############################################
import javax.ws.rs.Consumes;
import javax.ws.rs.POST;
import javax.ws.rs.Path;
import javax.ws.rs.Produces;
import javax.ws.rs.core.MediaType;
import javax.ws.rs.core.Response;
@Path("/null")
public class NullEndPoint {
public static final String UTF_8 = "UTF-8";
public static final String JSON_CONTENT_TYPE = MediaType.APPLICATION_JSON +
"; charset=" + UTF_8;
@POST
@Consumes(JSON_CONTENT_TYPE)
@Produces(JSON_CONTENT_TYPE)
public Response verify(String params) {
return Response.ok("{ \"status\":\"success\"}").build();
}
}
##############################################
##############################################
Jmeter Test Plan - 60 users test for 1200 seconds duration
##############################################
<?xml version="1.0" encoding="UTF-8"?>
<jmeterTestPlan version="1.2" properties="2.4" jmeter="2.9 r1437961">
<hashTree>
<TestPlan guiclass="TestPlanGui" testclass="TestPlan"
testname="SampleTest" enabled="true">
<stringProp name="TestPlan.comments"></stringProp>
<boolProp name="TestPlan.functional_mode">false</boolProp>
<boolProp name="TestPlan.serialize_threadgroups">false</boolProp>
<elementProp name="TestPlan.user_defined_variables"
elementType="Arguments" guiclass="ArgumentsPanel" testclass="Arguments"
testname="User Defined Variables" enabled="true">
<collectionProp name="Arguments.arguments"/>
</elementProp>
<stringProp name="TestPlan.user_define_classpath"></stringProp>
</TestPlan>
<hashTree>
<kg.apc.jmeter.threads.SteppingThreadGroup
guiclass="kg.apc.jmeter.threads.SteppingThreadGroupGui"
testclass="kg.apc.jmeter.threads.SteppingThreadGroup" testname="MU - Reg"
enabled="true">
<stringProp name="ThreadGroup.on_sample_error">continue</stringProp>
<stringProp
name="ThreadGroup.num_threads">${__P(users,60)}</stringProp>
<stringProp name="Threads initial delay">0</stringProp>
<stringProp name="Start users count">10</stringProp>
<stringProp name="Start users count burst">0</stringProp>
<stringProp name="Start users period">30</stringProp>
<stringProp name="Stop users count">10</stringProp>
<stringProp name="Stop users period">1</stringProp>
<stringProp name="flighttime">${__P(duration,1200)}</stringProp>
<stringProp name="rampUp">5</stringProp>
<elementProp name="ThreadGroup.main_controller"
elementType="LoopController" guiclass="LoopControlPanel"
testclass="LoopController" testname="Loop Controller" enabled="true">
<boolProp name="LoopController.continue_forever">false</boolProp>
<intProp name="LoopController.loops">-1</intProp>
</elementProp>
</kg.apc.jmeter.threads.SteppingThreadGroup>
<hashTree>
<HeaderManager guiclass="HeaderPanel" testclass="HeaderManager"
testname="HTTP Header Manager" enabled="true">
<collectionProp name="HeaderManager.headers">
<elementProp name="" elementType="Header">
<stringProp name="Header.name">Content-Type</stringProp>
<stringProp name="Header.value">application/json</stringProp>
</elementProp>
</collectionProp>
</HeaderManager>
<hashTree/>
<Arguments guiclass="ArgumentsPanel" testclass="Arguments"
testname="User Defined Variables" enabled="true">
<collectionProp name="Arguments.arguments">
<elementProp name="userNameUUID" elementType="Argument">
<stringProp name="Argument.name">userNameUUID</stringProp>
<stringProp name="Argument.value">user_${__UUID</stringProp>
<stringProp name="Argument.metadata">=</stringProp>
</elementProp>
<elementProp name="host" elementType="Argument">
<stringProp name="Argument.name">host</stringProp>
<stringProp
name="Argument.value">${__P(myhost,XXX.XX.XX.XX)}</stringProp>
<stringProp name="Argument.metadata">=</stringProp>
</elementProp>
<elementProp name="port" elementType="Argument">
<stringProp name="Argument.name">port</stringProp>
<stringProp name="Argument.value">8443</stringProp>
<stringProp name="Argument.metadata">=</stringProp>
</elementProp>
<elementProp name="protocol" elementType="Argument">
<stringProp name="Argument.name">protocol</stringProp>
<stringProp name="Argument.value">https</stringProp>
<stringProp name="Argument.metadata">=</stringProp>
</elementProp>
<elementProp name="pathPrefix" elementType="Argument">
<stringProp name="Argument.name">pathPrefix</stringProp>
<stringProp name="Argument.value">/null</stringProp>
<stringProp name="Argument.metadata">=</stringProp>
</elementProp>
</collectionProp>
</Arguments>
<hashTree/>
<HTTPSamplerProxy guiclass="HttpTestSampleGui"
testclass="HTTPSamplerProxy" testname="Null Request" enabled="true">
<boolProp name="HTTPSampler.postBodyRaw">true</boolProp>
<elementProp name="HTTPsampler.Arguments" elementType="Arguments">
<collectionProp name="Arguments.arguments">
<elementProp name="" elementType="HTTPArgument">
<boolProp name="HTTPArgument.always_encode">false</boolProp>
<stringProp name="Argument.value">{
}</stringProp>
<stringProp name="Argument.metadata">=</stringProp>
</elementProp>
</collectionProp>
</elementProp>
<stringProp name="HTTPSampler.domain">${host}</stringProp>
<stringProp name="HTTPSampler.port">${port}</stringProp>
<stringProp name="HTTPSampler.connect_timeout"></stringProp>
<stringProp name="HTTPSampler.response_timeout"></stringProp>
<stringProp name="HTTPSampler.protocol">${protocol}</stringProp>
<stringProp name="HTTPSampler.contentEncoding"></stringProp>
<stringProp name="HTTPSampler.path">${pathPrefix}</stringProp>
<stringProp name="HTTPSampler.method">POST</stringProp>
<boolProp name="HTTPSampler.follow_redirects">true</boolProp>
<boolProp name="HTTPSampler.auto_redirects">false</boolProp>
<boolProp name="HTTPSampler.use_keepalive">true</boolProp>
<boolProp name="HTTPSampler.DO_MULTIPART_POST">false</boolProp>
<boolProp name="HTTPSampler.monitor">false</boolProp>
<stringProp name="HTTPSampler.embedded_url_re"></stringProp>
</HTTPSamplerProxy>
<hashTree/>
<ResultCollector guiclass="ViewResultsFullVisualizer"
testclass="ResultCollector" testname="View Results Tree" enabled="true">
<boolProp name="ResultCollector.error_logging">false</boolProp>
<objProp>
<name>saveConfig</name>
<value class="SampleSaveConfiguration">
<time>true</time>
<latency>true</latency>
<timestamp>true</timestamp>
<success>true</success>
<label>true</label>
<code>true</code>
<message>true</message>
<threadName>true</threadName>
<dataType>true</dataType>
<encoding>false</encoding>
<assertions>true</assertions>
<subresults>true</subresults>
<responseData>false</responseData>
<samplerData>false</samplerData>
<xml>false</xml>
<fieldNames>false</fieldNames>
<responseHeaders>false</responseHeaders>
<requestHeaders>false</requestHeaders>
<responseDataOnError>false</responseDataOnError>
<saveAssertionResultsFailureMessage>false</saveAssertionResultsFailureMessage>
<assertionsResultsToSave>0</assertionsResultsToSave>
<bytes>true</bytes>
</value>
</objProp>
<stringProp name="filename"></stringProp>
</ResultCollector>
<hashTree/>
</hashTree>
</hashTree>
</hashTree>
</jmeterTestPlan>
##############################################
##############################################
On tomcat7 number of requests fullfilled per second
##############################################
* RUN1 ** ==> ** 38128.97704918033 runs/sec*
*RUN2 ==> ** 19040.35947712418 runs/sec*
*RUN3 ==> ** 19043.7908496732** runs/sec*
* RUN4 ==> ** 19001.71568627451 runs/sec*
##############################################
##############################################
*PROBLEM*
##############################################
It can be clearly seen that the performance degrades with multiple runs of
the same jmeter test by almost 50% and then on the 3rd or 4th run it seems
to stabliize. This issue is not reproducible on tomcat6 on the same
machine. It shows up only on tomcat7 onwards. Also tomcat8 is showing
similar symptoms. The slowness shows up only in the next subsequent run
against the server without tomcat restart.
Also the slowness issue is more obvious when the endpoint is doing
cryptographic validations(more processing).
Please let us know what change in tomcat7 is showing this change. Can some
configuration help in fixing this. We are suspecting that the threading
model in tomcat7 onwards is the cause of this.
##############################################
Regards,
Dimple
