Hello, I confirm issue with HttpClient 4. Penalty is in init of Httpclient object: httpClient.getConnectionManager() which initialize an SSLContext even if you don't use https
I asked a question on mailing list. Thanks for report Regards On Tue, May 13, 2014 at 10:39 PM, Stephen Townshend < [email protected]> wrote: > > Hi all, > > It looks like we've managed to avoid the issue. Switching my HTTP Request > Defaults to use HTTP3.1 instead of the HTTP4.1 implementation was the key. > > The way we got to this was first noticing calls to "SSLcreateContext" > while monitoring the JVM. We managed to get a stack trace and noticed a lot > of java.io.FileInputStream.read requests (related to SSL certificates). > Wanting to know what file it was requesting we used Windows Process Monitor > to monitor the java.exe process and what we saw during a test run was the > file "C:\Program Files\Java\jre7\lib\security\cacerts" was being accessed > thousands upon thousands of times constantly. > > My colleague happened to notice in the stack trace it referencing the > HTTP4.1 implementation in some of the method calls so at a whim we tried > HTTP3.1 instead, and in an instant everything improved. That cacerts file > was no longer being requested, and during a test run CPU usage went from > ~60% into a crash down to 15% and finishing the full hour. Running "netstat > -a" also shows a marked difference - the connections my test made were > sitting mostly in the "TIME_WAIT" state which is what we want to see, only > a couple sitting in "CLOSE_WAIT". > > So we've found a work-around but I still think there's a bug here - it > concerns me that it was checking certificates even when all requests were > simple HTTP. > > Answers to your questions: > - There is no possibility it was HTTPS requests to external resources. > Every HTTP request in my test plan has the filter " .*${MAIN_SERVER}.*" on > the field "Embedded URLs must match" (where MAIN_SERVER is variable which > holds the location of my test environment). > - I am currently on Jmeter 2.9 with Java 7u55 64-bit. I was previously > running Jmeter 2.11 with Java 8u5 64-bit in which I saw the same behaviour. > I chose 2.9 because I'd used it last year successfully on a similar project. > - I have the "JmeterPlugins" extensions pack, the one that has the > "jp@gcUltimate Thread Group" but just to be sure yesterday before we fixed the > issue I reverted to the default thread group and removed the extension, but > the same behaviour occurred. > - The only actions are HTTP Requests, Regular Expression Extractors, a > Transaction Sampler, and I was originally using BeanShell PreProcessors but > I since removed all of them and replaced them with ${__javaScript(...)} > expressions because they were causing a memory leak. I also use Constant > Throughput Timers and Uniform Random Timers to control the throughput. > > Because I haven't seen this before and I have used the HTTP 4.1 > implementation on multiple assignments in the past it looks like it is > specifically related to the combination of Jmeter and the server I am using > as a load generator. If I can provide more information I'm more than happy, > but for now I'm just relieved we actually found a work-around. > > -----Original Message----- > From: Ryabtsev Vladimir [mailto:[email protected]] > Sent: Tuesday, 13 May 2014 8:46 p.m. > To: JMeter Users List > Subject: RE: createSSLContext CPU issue > > Hi Stephen, > > 1. CPU spike correlates with growth of number of threads. Why did number > of threads grow significantly at 10:55AM (from ~1K to ~10K)? > 2. What actions are performed during the test? Do you use any extra libs > (your own or third party) in your test via BeanShell or JSR223? > 3. Concerning SSL: can your web pages download any resources (like js, > images, etc) from third party servers via SSL? It could be if you use > 'Retreive All Embedded Resources' option in HTTP Sampler. To ensure, you > can inspect actual traffic using such tools as Fiddler. > > ----- > VR > > > -----Original Message----- > From: Stephen Townshend [mailto:[email protected]] > Sent: Tuesday, May 13, 2014 7:41 AM > To: [email protected] > Subject: RE: createSSLContext CPU issue > > Hi all, > > I'm currently on a performance testing assignment for a client who has a > public website sitting within a virtual infrastructure. I've been given a > virtual server (running Windows Server 2008) to run my tests from with > access to the system under test. > > The issue is that after a period of time running my load tests the CPU > suddenly spikes from ~50% or less up to 100% and never recovers. Next what > happens is that every test I run after that failure will also consume CPU > at a much higher rate than before, quickly become overloaded again and > eventually Jmeter will crash or I have to stop the test. > > I've been using VisualVM to analyse the Jmeter JVM during my test runs. > > My questions are (relating to the detail below): > > - Taking a thread dump there is a whole lot of activity around > SSL connections but the test I'm running has no HTTPS requests at all - > they're all plain HTTP - so why are any SSL connections being made at all? > > - Why is it taking up all the CPU? > > - I'm also seeing thousands of CLOSE_WAIT connections when a test > is running, they continue to exist even once the JVM has been closed down - > why is that? > > I've never had to install certificates before even when testing HTTPS > traffic. > > JVM monitoring of a test which suddenly goes to 100% CPU: > > http://i.imgur.com/lmrfaoH.png > > And an example of a follow up test where it quickly hits 100% again: > > http://i.imgur.com/929YQXh.png > > A CPU sample in VisualVM when it starts failing showing it taking a lot of > time on createSSLContext(): > > http://i.imgur.com/YOjBeqg.png > > - > Thread dump of JVM when test is failing - it looks like it is reading > certificate files? > > "Thread-16959" prio=6 tid=0x0000000013b2a800 nid=0x8fb4 runnable > [0x000000009ef8e000] > java.lang.Thread.State: RUNNABLE > at java.io.FileInputStream.read0(Native Method) > at java.io.FileInputStream.read(Unknown Source) > at java.io.DataInputStream.readInt(Unknown Source) > at sun.security.provider.JavaKeyStore.engineLoad(Unknown Source) > - locked <0x00000007fa3193c8> (a java.util.Hashtable) > at sun.security.provider.JavaKeyStore$JKS.engineLoad(Unknown > Source) > at java.security.KeyStore.load(Unknown Source) > at > sun.security.ssl.TrustManagerFactoryImpl.getCacertsKeyStore(Unknown Source) > at sun.security.ssl.TrustManagerFactoryImpl.engineInit(Unknown > Source) > at javax.net.ssl.TrustManagerFactory.init(Unknown Source) > at > org.apache.http.conn.ssl.SSLSocketFactory.createSSLContext(SSLSocketFactory.java:229) > at > org.apache.http.conn.ssl.SSLSocketFactory.createDefaultSSLContext(SSLSocketFactory.java:358) > at > org.apache.http.conn.ssl.SSLSocketFactory.getSocketFactory(SSLSocketFactory.java:175) > at > org.apache.http.impl.conn.SchemeRegistryFactory.createDefault(SchemeRegistryFactory.java:49) > at > org.apache.http.impl.client.AbstractHttpClient.createClientConnectionManager(AbstractHttpClient.java:306) > at > org.apache.http.impl.client.AbstractHttpClient.getConnectionManager(AbstractHttpClient.java:466) > - locked <0x00000007fa318398> (a > org.apache.jmeter.protocol.http.sampler.HTTPHC4Impl$4) > at > org.apache.jmeter.protocol.http.sampler.HTTPHC4Impl.setupClient(HTTPHC4Impl.java:495) > at > org.apache.jmeter.protocol.http.sampler.HTTPHC4Impl.sample(HTTPHC4Impl.java:230) > at > org.apache.jmeter.protocol.http.sampler.HTTPSamplerProxy.sample(HTTPSamplerProxy.java:62) > at > org.apache.jmeter.protocol.http.sampler.HTTPSamplerBase$ASyncSample.call(HTTPSamplerBase.java:1804) > at > org.apache.jmeter.protocol.http.sampler.HTTPSamplerBase$ASyncSample.call(HTTPSamplerBase.java:1772) > at java.util.concurrent.FutureTask.run(Unknown Source) > at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown > Source) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown > Source) > at > org.apache.jmeter.protocol.http.sampler.HTTPSamplerBase$1$1.run(HTTPSamplerBase.java:1240) > at java.lang.Thread.run(Unknown Source) > > I'm also seeing thousands of CLOSE_WAIT connections when a test is > running, they continue to exist even once the JVM has been closed down. > > Update: Looking at process monitor it is reading the file C:\Program > Files\Java\jdk1.7.0_55\jre\lib\security\cacerts thousands of times, might > be related. > > Stephen Townshend, Performance Engineer, Equinox Limited, > http://www.equinox.co.nz/ Address Level 5, Shortland Chambers, 70 > Shortland Street, PO Box 3903, Auckland 1140, New Zealand Contact > [email protected]<mailto:[email protected]>, > Phone: +64 9 307 9450, DDI: +64 9 307 9007, Mobile: +64 22 6051 423 > > Please consider the environment before printing this e-mail. > > ___________________________________________________________________________________ > This e-mail message and any attachments contain information that is > confidential and may be subject to legal privilege. If you are not the > intended recipient, you must not peruse, use, pass on or copy this message > or any attachments. If you have received this e-mail in error, please > notify us by return e-mail and erase all copies of this message including > any attachments. Thank you. > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > > -- Cordialement. Philippe Mouawad.
