The 1 microsecond is hardcoded into the HTTPHC4Impl.java HTTP Sampler
implementation in JMeter (org.apache.jmeter.protocol.http.sampler)
private void closeCurrentConnections(
Map<HttpClientKey, MutableTriple<CloseableHttpClient, AuthState,
PoolingHttpClientConnectionManager>> mapHttpClientPerHttpClientKey) {
for (MutableTriple<CloseableHttpClient, AuthState,
PoolingHttpClientConnectionManager> triple :
mapHttpClientPerHttpClientKey.values()) {
PoolingHttpClientConnectionManager poolingHttpClientConnectionManager =
triple.getRight();
poolingHttpClientConnectionManager.closeExpiredConnections();
poolingHttpClientConnectionManager.closeIdleConnections(1L,
TimeUnit.MICROSECONDS);
}
}
The server is sending Connection:Keep-Alive
Yes it is using SSL, which is why the connection is taking so long.
Good pick on the connection pool – with debug logging on the bigger test, I can
see it also never gets to more than 1 of 2, so that’s a factor in
parallelisation as all the data is going to the same host.
It’s actually just a Splunk HEC endpoint, so there’s no real smarts in that
server other than accepting data it’s given and it’s intended to have
persistent connections.
I’ve traditionally used Java samplers to do my test logic, and I have had full
control over the HTTP sampler, but this is the first time I’m using the
standard HTTP Sampler in anger.
Thanks for the pointers.
From: [email protected] <[email protected]>
Date: Wednesday, 7 July 2021 at 8:13 am
To: [email protected] <[email protected]>
Subject: RE: JMeter closing connections
In your log output I see this:
Closing connections idle longer than 1 MICROSECONDS
That seems a little extreme. (:
It’s also possible the server is sending the “Connection: close” header for its
own reasons. If so, I would raise that with the server people.
Finally, does the server require mutual authentication? There is a quirk of
Apache HTTP Client that will cause it to not reuse connections unless you jump
through the right hoops. I’m working from memory, but there is a
disableConnectionState method on one of the builder classes. If you don’t call
it, mutual auth connections won’t be reused unless you specify the client cert
subject in your pool request. Hopefully the JMeter devs have taken care of
this.
You also have this, which I would expect to cause a lot of waiting at high
volume:
route allocated: 1 of 2; total allocated: 1 of 20
That means your pool is only 2 connections per host. That is the default and
almost always too small. I don’t know how to increase it through the UI.
From: Bowesman, Antony
<[email protected]<mailto:[email protected]>>
Date: Tuesday, Jul 06, 2021, 4:42 PM
To: JMeter List <[email protected]<mailto:[email protected]>>
Subject: JMeter closing connections
I am noticing that the outgoing port number is changing during a simple test
and it appears that the socket is getting closed.
It’s JMetert 5.4.1
HTTP Sampler has keep alives set and I can see the following in the log.
Sampling rate is once every 4 seconds, so this shows the end of one sample
(16:43:16) and the start of the next (16:43:20), so it appears to be closing
the connections before it starts the sample
2021-07-06T16:43:12,927 C=org.apache.http.wire M="http-outgoing-0 >>
"Connection: keep-alive[\r][\n]""
…
2021-07-06T16:43:16,578 C=org.apache.http.headers M="http-outgoing-1 <<
Connection: Keep-Alive"
2021-07-06T16:43:16,578 C=execchain.MainClientExec M="Connection can be kept
alive indefinitely"
2021-07-06T16:43:16,579 C=conn.PoolingHttpClientConnectionManager M="Connection
[id: 1][route:
{s}->https://urldefense.com/v3/__https://x.y.z:8088**Astate__;XVs!!F9svGWnIaVPGSwU!_Dw1VcCv0yCT6pGo_yfMjSxANBY_EEQtN98gFbr1b23AATNlc2eRMDT2l6UJ3cp5JP8KAOI$
: metrics 2-1] can be kept alive indefinitely"
2021-07-06T16:43:16,579 C=conn.DefaultManagedHttpClientConnection
M="http-outgoing-1: set socket timeout to 0"
2021-07-06T16:43:16,579 C=conn.PoolingHttpClientConnectionManager M="Connection
released: [id: 1][route:
{s}->https://urldefense.com/v3/__https://x.y.z:8088**Astate__;XVs!!F9svGWnIaVPGSwU!_Dw1VcCv0yCT6pGo_yfMjSxANBY_EEQtN98gFbr1b23AATNlc2eRMDT2l6UJ3cp5JP8KAOI$
: metrics 2-1][total available: 1; route allocated: 1 of 2; total allocated: 1
of 20]"
2021-07-06T16:43:20,379 C=conn.PoolingHttpClientConnectionManager M="Closing
expired connections"
2021-07-06T16:43:20,380 C=conn.PoolingHttpClientConnectionManager M="Closing
connections idle longer than 1 MICROSECONDS"
2021-07-06T16:43:20,380 C=conn.DefaultManagedHttpClientConnection
M="http-outgoing-1: Close connection"
2021-07-06T16:43:20,380 C=conn.PoolingHttpClientConnectionManager M="Connection
request: [route:
{s}->https://urldefense.com/v3/__https://x.y.z:8088**Astate__;XVs!!F9svGWnIaVPGSwU!_Dw1VcCv0yCT6pGo_yfMjSxANBY_EEQtN98gFbr1b23AATNlc2eRMDT2l6UJ3cp5JP8KAOI$
: metrics 2-1][total available: 0; route allocated: 0 of 2; total allocated: 0
of 20]"
2021-07-06T16:43:20,381 C=conn.PoolingHttpClientConnectionManager M="Connection
leased: [id: 2][route: {s}->https:// x.y.z:8088][total available: 0; route
allocated: 1 of 2; total allocated: 1 of 20]"
2021-07-06T16:43:20,381 C=execchain.MainClientExec M="Opening connection
{s}->https:// x.y.z:8088"
2021-07-06T16:43:20,381 C=apache.http.conn.ssl.SSLConnectionSocketFactory
M="Connecting socket to x.y.z /n.n.n.n:8088 with timeout 0"
I have a 30 minutes TTL set with
httpclient4.time_to_live=1800000
Problem is that during the real test, where I am sending ~3000 request/min, the
connection time on the jmeter reported metrics goes up to 2 seconds after about
90 seconds and then sits at that during the test and the count of ESTABLISHED
connections varies in the plateau stage of the test where is should be constant.
I have tried various settings of
httpclient4.idletimeout
with the default of 0. 10,000 and -1. In each case it changes the message above
(can be kept alive XXX) to an appropriate value matching the setting, so I know
it’s got the config.
Any ideas?
This e-mail and any attachments to it are confidential. You must not use,
disclose or act on the e-mail if you are not the intended recipient. If you
have received this e-mail in error, please let us know by contacting the sender
and deleting the original e-mail. Liability limited by a scheme approved under
Professional Standards Legislation. Deloitte refers to a Deloitte member firm,
one of its related entities, or Deloitte Touche Tohmatsu Limited (“DTTL”). Each
Deloitte member firm is a separate legal entity and a member of DTTL. DTTL does
not provide services to clients. Please see
https://urldefense.com/v3/__http://www.deloitte.com/about__;!!F9svGWnIaVPGSwU!_Dw1VcCv0yCT6pGo_yfMjSxANBY_EEQtN98gFbr1b23AATNlc2eRMDT2l6UJ3cp5CJtg3b0$<https://urldefense.com/v3/__http:/www.deloitte.com/about__;!!F9svGWnIaVPGSwU!_Dw1VcCv0yCT6pGo_yfMjSxANBY_EEQtN98gFbr1b23AATNlc2eRMDT2l6UJ3cp5CJtg3b0$>
to learn more. Nothing in this e-mail, nor any related attachments or
communications or services, have any capacity to bind any other entity under
the ‘Deloitte’ network of member firms (including those operating in Australia).
This e-mail and any attachments to it are confidential. You must not use,
disclose or act on the e-mail if you are not the intended recipient. If you
have received this e-mail in error, please let us know by contacting the sender
and deleting the original e-mail. Liability limited by a scheme approved under
Professional Standards Legislation. Deloitte refers to a Deloitte member firm,
one of its related entities, or Deloitte Touche Tohmatsu Limited (“DTTL”). Each
Deloitte member firm is a separate legal entity and a member of DTTL. DTTL does
not provide services to clients. Please see www.deloitte.com/about to learn
more. Nothing in this e-mail, nor any related attachments or communications or
services, have any capacity to bind any other entity under the ‘Deloitte’
network of member firms (including those operating in Australia).
