Re: HttpClient with a web service: Consistent data center switch-over
On Thu, 2016-05-05 at 07:12 -0700, Aaron Curley wrote: > On 5/3/2016 9:49 AM, Oleg Kalnichevski wrote: > > On Tue, 2016-05-03 at 09:15 -0700, Aaron Curley wrote: > >> On 5/3/2016 8:22 AM, Oleg Kalnichevski wrote: > >>> On Mon, 2016-05-02 at 21:28 -0700, Aaron Curley wrote: > Hello all, > > I'm using HttpClient version 4.5.1 to make high-volume calls to an > internal highly-available web service that is hosted at multiple > geographic locations. On average, the web service calls need to be > executed rather quickly; therefore, we are presently making heavy use of > HttpClient's connection pooling & reuse behavior (via > PoolingHttpClientConnectionManager) to alleviate the overhead (TCP and > TLS) incurred when setting up a new connection. Our DNS records direct > clients of the service to the (geographically) nearest data center. > Should a data center experience problems, we update our DNS records to > redirect our clients to our remaining "good" site(s). > > In times of an outage, HttpClient appears to have automatically > failed-over to our alternate sites (IPs) pretty timely and consistently; > however, upon service restoration at the primary data-center for a > particular client node, HttpClient does not appear to reliably switch > back to calling our primary site. (In such instances, I have been able > to confirm that our DNS records ARE switching back in a timely manner. > The TTLs are set to an appropriately short value.) > > My best guess is that the lack of consistent "switch back" to the > primary site is caused by HttpClient's connection pooling & reuse > behavior. Because of our relatively high (and consistent) request > volume, creation of NEW connections is likely a rarity once a particular > client node is operating for a while; instead, as previously noted, I > would generally expect that connections in the pool be re-used whenever > possible. Any re-used connections will still be established with the > alternate site(s), therefore, the client nodes communicating with > alternate sites would generally never (or only VERY gradually) switch > back to communicating with the primary site. This lines up with what I > have observed; "switching back" seems to only happen once request > throughput drops sufficiently to allow most of our client nodes' pooled > connections to "time out" and be closed due to inactivity (e.g. during > overnight hours). > > > I believe a reasonably "standard" way to solve this problem would be to > configure a maximum 'lifetime' of a connection in the connection pool > (e.g. 1 hour). This lifetime would be enforced regardless of whether or > not the connection is idle or can otherwise be re-used. On first > glance, the HttpClientBuilder.setConnectionTimeToLive() method seemed > ideal for this, but upon further investigation and review of the > HttpClient code base, this method appears to configure the maximum TTL > without introducing any element of randomness into each connection's > TTL. As a result, I'm concerned that if I enable the built-in TTL > feature, my clients are likely to experience regular performance > "spikes" at the configured TTL interval. (This would be caused when > most/all of the pooled connections expire simultaneously, since they > were mostly all created at once, at application start-up.) > > > Instead of using the built-in TTL limit setting, I considered overriding > the time to live using the available callbacks > (ConnectionKeepAliveStrategy, ConnectionReuseStrategy). Unfortunately, > this approach appears to be infeasible because the parameters to those > callbacks do not have access to the underlying connection object; there > is no "key" that can be used to look up a connection's lifetime (i.e. in > a ConcurrentHashMap) such that a decision could be made about whether to > close or retain the connection. I also took a look at the various > places that I could try to override the default connection pooling > behavior (e.g. MainClientExec, HttpClientConnectionManager). > HttpClientConnectionManager appears to be the best bet (specifically, > HttpClientConnectionManager.releaseConnection() would have to be > overridden), but this would require duplication of the existing > releaseConnection() code with slight modifications in the overriding > class. This seems brittle. > > > Can anyone think of a way that I could implement this (cleanly) with > HttpClient 4.x? Maybe I missed something? > > If not, I would be happy to open a JIRA for a possible HttpClient > enhancement to enable such a feature. If people are op
Re: NoHttpResponseException received at startup
On Thu, 2016-05-05 at 14:46 +, Pete Keyes wrote: > Update for the elapsed time for these calls that fail strictly at startup. > The stair-step 5/25/500... was strictly due to a pause we insert between > retries. All 4 tries fail within 5ms. > > Given the configuration settings for CloseableHttpClient a failure within > 5ms seems unlikely. The trip to the F5 VIP and back would barely complete > within 5ms. > > > Any thoughts on how to stabilize this a bit at startup? Or glean more > information about why apache-hc would fail with NoHttpResponseException so > quickly. The best course of action would be to capture IP packets with a traffic analyzer such as Wireshark and to try to figure out why TCP connections get reset. Oleg > Stack trace for the error: > org.apache.http.NoHttpResponseException: > crm-asp-prod.starbucks.net:443 failed to respond > > org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpRe > sponseParser.java:143) > > org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpRe > sponseParser.java:57) > > org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.j > ava:261) > > org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(Def > aultBHttpClientConnection.java:165) > > org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java: > 167) > > org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestE > xecutor.java:272) > > org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.ja > va:124) > > org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:2 > 71) > > org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184) > org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88) > > org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) > > org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient > .java:184) > > org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient > .java:82) > > > > ‹ > Pete Keyes > > > > > > > On 5/4/16, 11:13 AM, "Pete Keyes" wrote: > > >We have an odd issue wherein at Tomcat(TomEE) startup apache-hc execution > >requests throw a NoHttpResponseException within less than 1 second. > >After a few minutes this behavior completely stops and isn¹t seen again > >until the next time we stop and restart Tomcat(TomEE). > > > >Basic apache-hc configuration information: > > - single static CloseableHttpClient instance initialized at container > >startup; > > - all requests managed via PoolingHttpClientConnectionManager > > - connection timeout 6000ms > > - read timeout 6000ms > > - we use 4 retries > > - the typical timing for each > >HttpClient.execute(HttpUriRequest,HttpContext) during this startup period > >is that they fail very fast: > > try-1: ~ 5ms > > try-2: ~250ms > > try-3: ~500ms > > try-5: ~700ms > >We measure the elapsed milli-seconds with respect to ³execute()². It > >seems very odd that execute() fails so fast given the apache-hc settings. > > It isn¹t every execution request that fails. But, a very high > >percentage of them do exhibit this behavior during the initial ~5 minutes > >after Tomcat(TomEE) begins dispatching inbound requests. > > > >The inbound HTTP request volume inbound is ~70 RPS(requests per second). > > > >After ~5 minutes this type of error never repeats. We have logging in > >place that verifies all the apache-hc components are fully instantiated > >prior to any request being processed. > > > >Version information: > > apache-hc: 4.5.2 > > java : 1.8r60 > > tomcat : 7.0.68 > > tomee: 1.7.4 > > > > > >Any ideas at all given the minimal information that I have to provide are > >greatly appreciated. > > > >‹ > >Pete > > > > > - > To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org > For additional commands, e-mail: httpclient-users-h...@hc.apache.org > - To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org For additional commands, e-mail: httpclient-users-h...@hc.apache.org
Re: HttpClient with a web service: Consistent data center switch-over
On 5/6/2016 5:15 AM, Oleg Kalnichevski wrote: On Thu, 2016-05-05 at 07:12 -0700, Aaron Curley wrote: On 5/3/2016 9:49 AM, Oleg Kalnichevski wrote: On Tue, 2016-05-03 at 09:15 -0700, Aaron Curley wrote: On 5/3/2016 8:22 AM, Oleg Kalnichevski wrote: On Mon, 2016-05-02 at 21:28 -0700, Aaron Curley wrote: Hello all, I'm using HttpClient version 4.5.1 to make high-volume calls to an internal highly-available web service that is hosted at multiple geographic locations. On average, the web service calls need to be executed rather quickly; therefore, we are presently making heavy use of HttpClient's connection pooling & reuse behavior (via PoolingHttpClientConnectionManager) to alleviate the overhead (TCP and TLS) incurred when setting up a new connection. Our DNS records direct clients of the service to the (geographically) nearest data center. Should a data center experience problems, we update our DNS records to redirect our clients to our remaining "good" site(s). In times of an outage, HttpClient appears to have automatically failed-over to our alternate sites (IPs) pretty timely and consistently; however, upon service restoration at the primary data-center for a particular client node, HttpClient does not appear to reliably switch back to calling our primary site. (In such instances, I have been able to confirm that our DNS records ARE switching back in a timely manner. The TTLs are set to an appropriately short value.) My best guess is that the lack of consistent "switch back" to the primary site is caused by HttpClient's connection pooling & reuse behavior. Because of our relatively high (and consistent) request volume, creation of NEW connections is likely a rarity once a particular client node is operating for a while; instead, as previously noted, I would generally expect that connections in the pool be re-used whenever possible. Any re-used connections will still be established with the alternate site(s), therefore, the client nodes communicating with alternate sites would generally never (or only VERY gradually) switch back to communicating with the primary site. This lines up with what I have observed; "switching back" seems to only happen once request throughput drops sufficiently to allow most of our client nodes' pooled connections to "time out" and be closed due to inactivity (e.g. during overnight hours). I believe a reasonably "standard" way to solve this problem would be to configure a maximum 'lifetime' of a connection in the connection pool (e.g. 1 hour). This lifetime would be enforced regardless of whether or not the connection is idle or can otherwise be re-used. On first glance, the HttpClientBuilder.setConnectionTimeToLive() method seemed ideal for this, but upon further investigation and review of the HttpClient code base, this method appears to configure the maximum TTL without introducing any element of randomness into each connection's TTL. As a result, I'm concerned that if I enable the built-in TTL feature, my clients are likely to experience regular performance "spikes" at the configured TTL interval. (This would be caused when most/all of the pooled connections expire simultaneously, since they were mostly all created at once, at application start-up.) Instead of using the built-in TTL limit setting, I considered overriding the time to live using the available callbacks (ConnectionKeepAliveStrategy, ConnectionReuseStrategy). Unfortunately, this approach appears to be infeasible because the parameters to those callbacks do not have access to the underlying connection object; there is no "key" that can be used to look up a connection's lifetime (i.e. in a ConcurrentHashMap) such that a decision could be made about whether to close or retain the connection. I also took a look at the various places that I could try to override the default connection pooling behavior (e.g. MainClientExec, HttpClientConnectionManager). HttpClientConnectionManager appears to be the best bet (specifically, HttpClientConnectionManager.releaseConnection() would have to be overridden), but this would require duplication of the existing releaseConnection() code with slight modifications in the overriding class. This seems brittle. Can anyone think of a way that I could implement this (cleanly) with HttpClient 4.x? Maybe I missed something? If not, I would be happy to open a JIRA for a possible HttpClient enhancement to enable such a feature. If people are open to this idea, I was generally thinking that adding a more generic callback might be the best approach (since my use case may not be other people's use cases), but I could also be convinced to make the enhancement request specifically support a connection expiration "window" for the TTL feature instead of a specific "hard-limit". Any thoughts on this? Thanks in advance (and sorry for the long email)! Hi Aaron PoolingHttpClientConnectionManager does not pro-actively evic
Re: NoHttpResponseException received at startup
Thanks Oleg. The keyword “reset” is enough to help us point networking in a plausible direction. - since this is a symptom associated with a broker connection this confirms why each retry always fails within the same 5ms elapsed time; - we currently allow a retry using the same execution context; - obviously a fruitless waste of resource time; — Pete On 5/6/16, 5:24 AM, "Oleg Kalnichevski" wrote: >On Thu, 2016-05-05 at 14:46 +, Pete Keyes wrote: >> Update for the elapsed time for these calls that fail strictly at >>startup. >> The stair-step 5/25/500... was strictly due to a pause we insert >>between >> retries. All 4 tries fail within 5ms. >> >> Given the configuration settings for CloseableHttpClient a failure >>within >> 5ms seems unlikely. The trip to the F5 VIP and back would barely >>complete >> within 5ms. >> >> >> Any thoughts on how to stabilize this a bit at startup? Or glean more >> information about why apache-hc would fail with NoHttpResponseException >>so >> quickly. > >The best course of action would be to capture IP packets with a traffic >analyzer such as Wireshark and to try to figure out why TCP connections >get reset. > >Oleg > > > >> Stack trace for the error: >> org.apache.http.NoHttpResponseException: >> crm-asp-prod.starbucks.net:443 failed to respond >> >> >>org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttp >>Re >> sponseParser.java:143) >> >> >>org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttp >>Re >> sponseParser.java:57) >> >> >>org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser >>.j >> ava:261) >> >> >>org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(D >>ef >> aultBHttpClientConnection.java:165) >> >> >>org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.jav >>a: >> 167) >> >> >>org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpReques >>tE >> xecutor.java:272) >> >> >>org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor. >>ja >> va:124) >> >> >>org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java >>:2 >> 71) >> >> >>org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184 >>) >> org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88) >> >> >>org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110 >>) >> >> >>org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClie >>nt >> .java:184) >> >> >>org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClie >>nt >> .java:82) >> >> >> >> ‹ >> Pete Keyes >> >> >> >> >> >> >> On 5/4/16, 11:13 AM, "Pete Keyes" wrote: >> >> >We have an odd issue wherein at Tomcat(TomEE) startup apache-hc >>execution >> >requests throw a NoHttpResponseException within less than 1 second. >> >After a few minutes this behavior completely stops and isn¹t seen again >> >until the next time we stop and restart Tomcat(TomEE). >> > >> >Basic apache-hc configuration information: >> > - single static CloseableHttpClient instance initialized at container >> >startup; >> > - all requests managed via PoolingHttpClientConnectionManager >> > - connection timeout 6000ms >> > - read timeout 6000ms >> > - we use 4 retries >> > - the typical timing for each >> >HttpClient.execute(HttpUriRequest,HttpContext) during this startup >>period >> >is that they fail very fast: >> > try-1: ~ 5ms >> > try-2: ~250ms >> > try-3: ~500ms >> > try-5: ~700ms >> >We measure the elapsed milli-seconds with respect to ³execute()². It >> >seems very odd that execute() fails so fast given the apache-hc >>settings. >> > It isn¹t every execution request that fails. But, a very high >> >percentage of them do exhibit this behavior during the initial ~5 >>minutes >> >after Tomcat(TomEE) begins dispatching inbound requests. >> > >> >The inbound HTTP request volume inbound is ~70 RPS(requests per >>second). >> > >> >After ~5 minutes this type of error never repeats. We have logging in >> >place that verifies all the apache-hc components are fully instantiated >> >prior to any request being processed. >> > >> >Version information: >> > apache-hc: 4.5.2 >> > java : 1.8r60 >> > tomcat : 7.0.68 >> > tomee: 1.7.4 >> > >> > >> >Any ideas at all given the minimal information that I have to provide >>are >> >greatly appreciated. >> > >> >‹ >> >Pete >> > >> >> >> - >> To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org >> For additional commands, e-mail: httpclient-users-h...@hc.apache.org >> > > > >- >To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org >For additional commands, e-mail: httpclient-users-h...@hc.apache.org >
Re: HttpClient with a web service: Consistent data center switch-over
On Fri, 2016-05-06 at 07:17 -0700, Aaron Curley wrote: > On 5/6/2016 5:15 AM, Oleg Kalnichevski wrote: > > On Thu, 2016-05-05 at 07:12 -0700, Aaron Curley wrote: > >> On 5/3/2016 9:49 AM, Oleg Kalnichevski wrote: > >>> On Tue, 2016-05-03 at 09:15 -0700, Aaron Curley wrote: > On 5/3/2016 8:22 AM, Oleg Kalnichevski wrote: > > On Mon, 2016-05-02 at 21:28 -0700, Aaron Curley wrote: > >> Hello all, > >> > >> I'm using HttpClient version 4.5.1 to make high-volume calls to an > >> internal highly-available web service that is hosted at multiple > >> geographic locations. On average, the web service calls need to be > >> executed rather quickly; therefore, we are presently making heavy use > >> of HttpClient's connection pooling & reuse behavior (via > >> PoolingHttpClientConnectionManager) to alleviate the overhead (TCP and > >> TLS) incurred when setting up a new connection. Our DNS records > >> direct clients of the service to the (geographically) nearest data > >> center. Should a data center experience problems, we update our DNS > >> records to redirect our clients to our remaining "good" site(s). > >> > >> In times of an outage, HttpClient appears to have automatically > >> failed-over to our alternate sites (IPs) pretty timely and > >> consistently; however, upon service restoration at the primary > >> data-center for a particular client node, HttpClient does not appear > >> to reliably switch back to calling our primary site. (In such > >> instances, I have been able to confirm that our DNS records ARE > >> switching back in a timely manner. The TTLs are set to an > >> appropriately short value.) > >> > >> My best guess is that the lack of consistent "switch back" to the > >> primary site is caused by HttpClient's connection pooling & reuse > >> behavior. Because of our relatively high (and consistent) request > >> volume, creation of NEW connections is likely a rarity once a > >> particular client node is operating for a while; instead, as > >> previously noted, I would generally expect that connections in the > >> pool be re-used whenever possible. Any re-used connections will still > >> be established with the alternate site(s), therefore, the client nodes > >> communicating with alternate sites would generally never (or only VERY > >> gradually) switch back to communicating with the primary site. This > >> lines up with what I have observed; "switching back" seems to only > >> happen once request throughput drops sufficiently to allow most of our > >> client nodes' pooled connections to "time out" and be closed due to > >> inactivity (e.g. during overnight hours). > >> > >> > >> I believe a reasonably "standard" way to solve this problem would be > >> to configure a maximum 'lifetime' of a connection in the connection > >> pool (e.g. 1 hour). This lifetime would be enforced regardless of > >> whether or not the connection is idle or can otherwise be re-used. On > >> first glance, the HttpClientBuilder.setConnectionTimeToLive() method > >> seemed ideal for this, but upon further investigation and review of > >> the HttpClient code base, this method appears to configure the maximum > >> TTL without introducing any element of randomness into each > >> connection's TTL. As a result, I'm concerned that if I enable the > >> built-in TTL feature, my clients are likely to experience regular > >> performance "spikes" at the configured TTL interval. (This would be > >> caused when most/all of the pooled connections expire simultaneously, > >> since they were mostly all created at once, at application start-up.) > >> > >> > >> Instead of using the built-in TTL limit setting, I considered > >> overriding the time to live using the available callbacks > >> (ConnectionKeepAliveStrategy, ConnectionReuseStrategy). > >> Unfortunately, this approach appears to be infeasible because the > >> parameters to those callbacks do not have access to the underlying > >> connection object; there is no "key" that can be used to look up a > >> connection's lifetime (i.e. in a ConcurrentHashMap) such that a > >> decision could be made about whether to close or retain the > >> connection. I also took a look at the various places that I could try > >> to override the default connection pooling behavior (e.g. > >> MainClientExec, HttpClientConnectionManager). > >> HttpClientConnectionManager appears to be the best bet (specifically, > >> HttpClientConnectionManager.releaseConnection() would have to be > >> overridden), but this would require duplication of the existing > >> releaseConnection() code with slight modifications in the overriding > >> class. This seems brittle. > >> > >>>