@Stephen, There has been a fix in 2.10 on an issue: https://issues.apache.org/bugzilla/show_bug.cgi?id=55023
That's why I am asking you. On Tue, May 13, 2014 at 10:46 AM, Ryabtsev Vladimir <[email protected]>wrote: > > Hi Stephen, > > 1. CPU spike correlates with growth of number of threads. Why did number > of threads grow significantly at 10:55AM (from ~1K to ~10K)? > 2. What actions are performed during the test? Do you use any extra libs > (your own or third party) in your test via BeanShell or JSR223? > 3. Concerning SSL: can your web pages download any resources (like js, > images, etc) from third party servers via SSL? It could be if you use > 'Retreive All Embedded Resources' option in HTTP Sampler. To ensure, you > can inspect actual traffic using such tools as Fiddler. > > ----- > VR > > > -----Original Message----- > From: Stephen Townshend [mailto:[email protected]] > Sent: Tuesday, May 13, 2014 7:41 AM > To: [email protected] > Subject: RE: createSSLContext CPU issue > > Hi all, > > I'm currently on a performance testing assignment for a client who has a > public website sitting within a virtual infrastructure. I've been given a > virtual server (running Windows Server 2008) to run my tests from with > access to the system under test. > > The issue is that after a period of time running my load tests the CPU > suddenly spikes from ~50% or less up to 100% and never recovers. Next what > happens is that every test I run after that failure will also consume CPU > at a much higher rate than before, quickly become overloaded again and > eventually Jmeter will crash or I have to stop the test. > > I've been using VisualVM to analyse the Jmeter JVM during my test runs. > > My questions are (relating to the detail below): > > - Taking a thread dump there is a whole lot of activity around > SSL connections but the test I'm running has no HTTPS requests at all - > they're all plain HTTP - so why are any SSL connections being made at all? > > - Why is it taking up all the CPU? > > - I'm also seeing thousands of CLOSE_WAIT connections when a test > is running, they continue to exist even once the JVM has been closed down - > why is that? > > I've never had to install certificates before even when testing HTTPS > traffic. > > JVM monitoring of a test which suddenly goes to 100% CPU: > > http://i.imgur.com/lmrfaoH.png > > And an example of a follow up test where it quickly hits 100% again: > > http://i.imgur.com/929YQXh.png > > A CPU sample in VisualVM when it starts failing showing it taking a lot of > time on createSSLContext(): > > http://i.imgur.com/YOjBeqg.png > > - > Thread dump of JVM when test is failing - it looks like it is reading > certificate files? > > "Thread-16959" prio=6 tid=0x0000000013b2a800 nid=0x8fb4 runnable > [0x000000009ef8e000] > java.lang.Thread.State: RUNNABLE > at java.io.FileInputStream.read0(Native Method) > at java.io.FileInputStream.read(Unknown Source) > at java.io.DataInputStream.readInt(Unknown Source) > at sun.security.provider.JavaKeyStore.engineLoad(Unknown Source) > - locked <0x00000007fa3193c8> (a java.util.Hashtable) > at sun.security.provider.JavaKeyStore$JKS.engineLoad(Unknown > Source) > at java.security.KeyStore.load(Unknown Source) > at > sun.security.ssl.TrustManagerFactoryImpl.getCacertsKeyStore(Unknown Source) > at sun.security.ssl.TrustManagerFactoryImpl.engineInit(Unknown > Source) > at javax.net.ssl.TrustManagerFactory.init(Unknown Source) > at > org.apache.http.conn.ssl.SSLSocketFactory.createSSLContext(SSLSocketFactory.java:229) > at > org.apache.http.conn.ssl.SSLSocketFactory.createDefaultSSLContext(SSLSocketFactory.java:358) > at > org.apache.http.conn.ssl.SSLSocketFactory.getSocketFactory(SSLSocketFactory.java:175) > at > org.apache.http.impl.conn.SchemeRegistryFactory.createDefault(SchemeRegistryFactory.java:49) > at > org.apache.http.impl.client.AbstractHttpClient.createClientConnectionManager(AbstractHttpClient.java:306) > at > org.apache.http.impl.client.AbstractHttpClient.getConnectionManager(AbstractHttpClient.java:466) > - locked <0x00000007fa318398> (a > org.apache.jmeter.protocol.http.sampler.HTTPHC4Impl$4) > at > org.apache.jmeter.protocol.http.sampler.HTTPHC4Impl.setupClient(HTTPHC4Impl.java:495) > at > org.apache.jmeter.protocol.http.sampler.HTTPHC4Impl.sample(HTTPHC4Impl.java:230) > at > org.apache.jmeter.protocol.http.sampler.HTTPSamplerProxy.sample(HTTPSamplerProxy.java:62) > at > org.apache.jmeter.protocol.http.sampler.HTTPSamplerBase$ASyncSample.call(HTTPSamplerBase.java:1804) > at > org.apache.jmeter.protocol.http.sampler.HTTPSamplerBase$ASyncSample.call(HTTPSamplerBase.java:1772) > at java.util.concurrent.FutureTask.run(Unknown Source) > at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown > Source) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown > Source) > at > org.apache.jmeter.protocol.http.sampler.HTTPSamplerBase$1$1.run(HTTPSamplerBase.java:1240) > at java.lang.Thread.run(Unknown Source) > > I'm also seeing thousands of CLOSE_WAIT connections when a test is > running, they continue to exist even once the JVM has been closed down. > > Update: Looking at process monitor it is reading the file C:\Program > Files\Java\jdk1.7.0_55\jre\lib\security\cacerts thousands of times, might > be related. > > Stephen Townshend, Performance Engineer, Equinox Limited, > http://www.equinox.co.nz/ Address Level 5, Shortland Chambers, 70 > Shortland Street, PO Box 3903, Auckland 1140, New Zealand Contact > [email protected]<mailto:[email protected]>, > Phone: +64 9 307 9450, DDI: +64 9 307 9007, Mobile: +64 22 6051 423 > > Please consider the environment before printing this e-mail. > > ___________________________________________________________________________________ > This e-mail message and any attachments contain information that is > confidential and may be subject to legal privilege. If you are not the > intended recipient, you must not peruse, use, pass on or copy this message > or any attachments. If you have received this e-mail in error, please > notify us by return e-mail and erase all copies of this message including > any attachments. Thank you. > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > > -- Cordialement. Philippe Mouawad.
