Hi Stephen, 1. CPU spike correlates with growth of number of threads. Why did number of threads grow significantly at 10:55AM (from ~1K to ~10K)? 2. What actions are performed during the test? Do you use any extra libs (your own or third party) in your test via BeanShell or JSR223? 3. Concerning SSL: can your web pages download any resources (like js, images, etc) from third party servers via SSL? It could be if you use 'Retreive All Embedded Resources' option in HTTP Sampler. To ensure, you can inspect actual traffic using such tools as Fiddler.
----- VR -----Original Message----- From: Stephen Townshend [mailto:[email protected]] Sent: Tuesday, May 13, 2014 7:41 AM To: [email protected] Subject: RE: createSSLContext CPU issue Hi all, I'm currently on a performance testing assignment for a client who has a public website sitting within a virtual infrastructure. I've been given a virtual server (running Windows Server 2008) to run my tests from with access to the system under test. The issue is that after a period of time running my load tests the CPU suddenly spikes from ~50% or less up to 100% and never recovers. Next what happens is that every test I run after that failure will also consume CPU at a much higher rate than before, quickly become overloaded again and eventually Jmeter will crash or I have to stop the test. I've been using VisualVM to analyse the Jmeter JVM during my test runs. My questions are (relating to the detail below): - Taking a thread dump there is a whole lot of activity around SSL connections but the test I'm running has no HTTPS requests at all - they're all plain HTTP - so why are any SSL connections being made at all? - Why is it taking up all the CPU? - I'm also seeing thousands of CLOSE_WAIT connections when a test is running, they continue to exist even once the JVM has been closed down - why is that? I've never had to install certificates before even when testing HTTPS traffic. JVM monitoring of a test which suddenly goes to 100% CPU: http://i.imgur.com/lmrfaoH.png And an example of a follow up test where it quickly hits 100% again: http://i.imgur.com/929YQXh.png A CPU sample in VisualVM when it starts failing showing it taking a lot of time on createSSLContext(): http://i.imgur.com/YOjBeqg.png - Thread dump of JVM when test is failing - it looks like it is reading certificate files? "Thread-16959" prio=6 tid=0x0000000013b2a800 nid=0x8fb4 runnable [0x000000009ef8e000] java.lang.Thread.State: RUNNABLE at java.io.FileInputStream.read0(Native Method) at java.io.FileInputStream.read(Unknown Source) at java.io.DataInputStream.readInt(Unknown Source) at sun.security.provider.JavaKeyStore.engineLoad(Unknown Source) - locked <0x00000007fa3193c8> (a java.util.Hashtable) at sun.security.provider.JavaKeyStore$JKS.engineLoad(Unknown Source) at java.security.KeyStore.load(Unknown Source) at sun.security.ssl.TrustManagerFactoryImpl.getCacertsKeyStore(Unknown Source) at sun.security.ssl.TrustManagerFactoryImpl.engineInit(Unknown Source) at javax.net.ssl.TrustManagerFactory.init(Unknown Source) at org.apache.http.conn.ssl.SSLSocketFactory.createSSLContext(SSLSocketFactory.java:229) at org.apache.http.conn.ssl.SSLSocketFactory.createDefaultSSLContext(SSLSocketFactory.java:358) at org.apache.http.conn.ssl.SSLSocketFactory.getSocketFactory(SSLSocketFactory.java:175) at org.apache.http.impl.conn.SchemeRegistryFactory.createDefault(SchemeRegistryFactory.java:49) at org.apache.http.impl.client.AbstractHttpClient.createClientConnectionManager(AbstractHttpClient.java:306) at org.apache.http.impl.client.AbstractHttpClient.getConnectionManager(AbstractHttpClient.java:466) - locked <0x00000007fa318398> (a org.apache.jmeter.protocol.http.sampler.HTTPHC4Impl$4) at org.apache.jmeter.protocol.http.sampler.HTTPHC4Impl.setupClient(HTTPHC4Impl.java:495) at org.apache.jmeter.protocol.http.sampler.HTTPHC4Impl.sample(HTTPHC4Impl.java:230) at org.apache.jmeter.protocol.http.sampler.HTTPSamplerProxy.sample(HTTPSamplerProxy.java:62) at org.apache.jmeter.protocol.http.sampler.HTTPSamplerBase$ASyncSample.call(HTTPSamplerBase.java:1804) at org.apache.jmeter.protocol.http.sampler.HTTPSamplerBase$ASyncSample.call(HTTPSamplerBase.java:1772) at java.util.concurrent.FutureTask.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at org.apache.jmeter.protocol.http.sampler.HTTPSamplerBase$1$1.run(HTTPSamplerBase.java:1240) at java.lang.Thread.run(Unknown Source) I'm also seeing thousands of CLOSE_WAIT connections when a test is running, they continue to exist even once the JVM has been closed down. Update: Looking at process monitor it is reading the file C:\Program Files\Java\jdk1.7.0_55\jre\lib\security\cacerts thousands of times, might be related. Stephen Townshend, Performance Engineer, Equinox Limited, http://www.equinox.co.nz/ Address Level 5, Shortland Chambers, 70 Shortland Street, PO Box 3903, Auckland 1140, New Zealand Contact [email protected]<mailto:[email protected]>, Phone: +64 9 307 9450, DDI: +64 9 307 9007, Mobile: +64 22 6051 423 Please consider the environment before printing this e-mail. ___________________________________________________________________________________ This e-mail message and any attachments contain information that is confidential and may be subject to legal privilege. If you are not the intended recipient, you must not peruse, use, pass on or copy this message or any attachments. If you have received this e-mail in error, please notify us by return e-mail and erase all copies of this message including any attachments. Thank you. --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
