Re: Migration from HC4 to HC5 : few questions

[Oleg Kalnichevski]

On Wed, 2020-06-10 at 19:11 +0200, Philippe Mouawad wrote:
> Hello,
> I am looking into migrating Apache JMeter project from HC4 to HC5 to
> benefit from the new features for Classic  HttpClient like the new
> lax
> connection pool and performance improvements.
> In the future we would hopefully switch to Async.
> 
> First thanks for this documentation, very useful ! :
> 
https://ok2c.github.io/httpclient-migration-guide/migration-to-classic.html
> 
> Few questions:
> 
>1. I don't see an equivalent of
> RequestConfig.Builder#setLocalAddress ?
>How do we get this feature ?

 RequestConfig.Builder#setLocalAddress was a mistake. This parameter
has nothing to do with request execution.

Please use custom HttpRoutePlanner instead.


>2. Is there an equivalent of LaxRedirectStrategy?  or
>DefaultRedirectStrategy is now the equivalent which seems to be
> the case ?

DefaultRedirectStrategy should now cover all common cases. Lax strategy
implementation is now superfluous. 


>3.
> 
>PoolingHttpClientConnectionManager does not expose anymore the
>HttpClientConnectionOperator, it's a protected constructor. In
> JMeter we
>use this to compute connect time. It looks like we would need to
> extend
>PoolingHttpClientConnectionManager, is there another better way to
> keep
>this ?


No, this is the intended way. 


>4.
> 
>Regarding bandwidth measurement per request, we used this approach
> based
>on your SO answer (
>
> https://stackoverflow.com/questions/26166469/measure-bandwidth-usage-with-apache-httpcomponents-httpclient
>):
> 
> 
>-
>   
> https://github.com/apache/jmeter/blob/master/src/protocol/http/src/main/java/org/apache/jmeter/protocol/http/sampler/HTTPHC4Impl.java#L448
>   - Is it still ok using
> 
>   preProcess/postProcess by redefining them in an extension of
>   HttpRequestExecutor and access endpointdetails through:
>   -
> 
>   HttpCoreContext.*adapt*(context).getEndpointDetails();
> 
>   But I don't see any more reset(), so how should we proceed ?
> 
> 

There is no #reset method. We could add but I am not sure if you really
need it at all. If you need the total bytes sent / received and the
total number of requests / responses those details are provided by
HttpConnectionMetrics interface. 

Hope this helps

Oleg



-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

[ANNOUNCEMENT] HttpComponents Core 5.0.1 GA released

[Oleg Kalnichevski]

The Apache HttpComponents project is pleased to announce 5.0.1 GA
release of HttpComponents Core. 

This maintenance release improves handling of early response messages
by the classic client protocol handler and fixes a number of minor
defects.

Download - 
Release notes - 

HttpComponents site - 

About HttpComponents Core

HttpCore is a set of HTTP/1.1 and HTTP/2 transport components that can
be used to build custom client and server side HTTP services with a
minimal footprint


-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

Re: Server-side mid-air connection close on upload

[Oleg Kalnichevski]

On Tue, 2020-06-02 at 11:36 +0200, Oleg Kalnichevski wrote:
> On Tue, 2020-06-02 at 11:09 +0200, Michael Osipov wrote:
> > Am 2020-06-02 um 00:09 schrieb Michael Osipov:
> > > Am 2020-06-01 um 15:11 schrieb Oleg Kalnichevski:
> > > > On Sun, 2020-05-31 at 23:13 +0200, Michael Osipov wrote:
> > > > > Am 2020-05-30 um 22:53 schrieb Oleg Kalnichevski:
> > > > > > On Sat, 2020-05-30 at 21:58 +0200, Michael Osipov wrote:
> > > > > > ...
> > > > > > > > Now we need to decide if this is as good as it gets or
> > > > > > > > we
> > > > > > > > want
> > > > > > > > to
> > > > > > > > put
> > > > > > > > more research into it.
> > > > > > > 
> > > > > > > I read liked you idea of the chunks. 2048 seems too low
> > > > > > > for
> > > > > > > me
> > > > > > > because
> > > > > > > most well write with a buffer of at least 4 KiB.
> > > > > > > 
> > > > > > > So I think we can try to things to miminize overhead:
> > > > > > > 
> > > > > > > * Make this feature optional, just like we do with
> > > > > > > Expectation.
> > > > > > > * Try to increase the step chunk size, e.g., 64 KiB or
> > > > > > > greater.
> > > > > > > With
> > > > > > > that we don't peek on each and every invocation.
> > > > > > > 
> > > > > > 
> > > > > > Yes, we may consider making it optional / disabled by
> > > > > > default
> > > > > 
> > > > > Via RequestConfig?
> > > > > 
> > > > > > We can give step chunking another go but it is less
> > > > > > important
> > > > > > now
> > > > > > as I
> > > > > > have inserted the check into the low level socket stream
> > > > > > whose
> > > > > > write
> > > > > > operations should be already quite optimized by the
> > > > > > protocol
> > > > > > layer.
> > > > > 
> > > > > But wouldn't that still reduce the check count because we'd
> > > > > hit
> > > > > the
> > > > > wait
> > > > > with SSLSocket? I am bit confused by your statement.
> > > > > 
> > > > 
> > > > It might but what I am trying to say is that even now there
> > > > would
> > > > likely be only one check on every large data chunk eliminating
> > > > the
> > > > worst case scenario of the application layer writing one or two
> > > > bytes
> > > > at a time each triggering a check and incurring a potential hit
> > > > of 50
> > > > ms.
> > > 
> > > I have tried the modified patch. Works. Picked up the the chunk
> > > patch 
> > > and raised to 64 KiB. Also in consideration that
> > > AbstractHttpEntity
> > > uses 
> > > a default buffer size of 4 KiB and TCP max packet size of 64 KiB.
> > > Then tried with to different connections to the server at work:
> > > 
> > > * VPN connection: Performs great for plain and TLS, after 128 KiB
> > > (two 
> > > probes) in almost all cases 401 is seen. Works also w/o chunks
> > > great.
> > > * Via ZScaler app which routes the traffic through a HTTP proxy
> > > to
> > > the 
> > > corporate network. It takes way more probes to detect the 401.
> > > But
> > > still 
> > > performs acceptable. Chunks made also virtually no difference. At
> > > some 
> > > point it is even worse with broken connections (plain HTTP).
> > > HTTPS 
> > > performs much worse.
> > > 
> > > So at least for the test options I have here, I tend to agree
> > > with
> > > you. 
> > > We either drop the step chunks or need to reduce the size to a
> > > sane
> > > value.
> > 
> > I have now reduced the chunk size which performs nicely. I hope
> > that
> > it 
> > will not be necessary and no one will write byte-by-byte. See
> > attachment.
> > 
> 
> Shall I commit the patch to my branch or do you want to fork my
> branch,
> tweak it and submit a PR against the main project repository?
> 
> > My last question would be: How can we wire the RequestConfig
> > parameter 
> > to the DefaultBHttpClientConnection class? Http1Config?
> > 
> 
> We cannot use RequestConfig at the connection level. It will have to
> be
> Http1Config.
> 

By the way, if we employ chunk counting I think we might as well always
leave the check on. I guess its overhead should now be negligible.

Oleg  



-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

Re: Server-side mid-air connection close on upload

[Oleg Kalnichevski]

On Tue, 2020-06-02 at 11:09 +0200, Michael Osipov wrote:
> Am 2020-06-02 um 00:09 schrieb Michael Osipov:
> > Am 2020-06-01 um 15:11 schrieb Oleg Kalnichevski:
> > > On Sun, 2020-05-31 at 23:13 +0200, Michael Osipov wrote:
> > > > Am 2020-05-30 um 22:53 schrieb Oleg Kalnichevski:
> > > > > On Sat, 2020-05-30 at 21:58 +0200, Michael Osipov wrote:
> > > > > ...
> > > > > > > Now we need to decide if this is as good as it gets or we
> > > > > > > want
> > > > > > > to
> > > > > > > put
> > > > > > > more research into it.
> > > > > > 
> > > > > > I read liked you idea of the chunks. 2048 seems too low for
> > > > > > me
> > > > > > because
> > > > > > most well write with a buffer of at least 4 KiB.
> > > > > > 
> > > > > > So I think we can try to things to miminize overhead:
> > > > > > 
> > > > > > * Make this feature optional, just like we do with
> > > > > > Expectation.
> > > > > > * Try to increase the step chunk size, e.g., 64 KiB or
> > > > > > greater.
> > > > > > With
> > > > > > that we don't peek on each and every invocation.
> > > > > > 
> > > > > 
> > > > > Yes, we may consider making it optional / disabled by default
> > > > 
> > > > Via RequestConfig?
> > > > 
> > > > > We can give step chunking another go but it is less important
> > > > > now
> > > > > as I
> > > > > have inserted the check into the low level socket stream
> > > > > whose
> > > > > write
> > > > > operations should be already quite optimized by the protocol
> > > > > layer.
> > > > 
> > > > But wouldn't that still reduce the check count because we'd hit
> > > > the
> > > > wait
> > > > with SSLSocket? I am bit confused by your statement.
> > > > 
> > > 
> > > It might but what I am trying to say is that even now there would
> > > likely be only one check on every large data chunk eliminating
> > > the
> > > worst case scenario of the application layer writing one or two
> > > bytes
> > > at a time each triggering a check and incurring a potential hit
> > > of 50
> > > ms.
> > 
> > I have tried the modified patch. Works. Picked up the the chunk
> > patch 
> > and raised to 64 KiB. Also in consideration that AbstractHttpEntity
> > uses 
> > a default buffer size of 4 KiB and TCP max packet size of 64 KiB.
> > Then tried with to different connections to the server at work:
> > 
> > * VPN connection: Performs great for plain and TLS, after 128 KiB
> > (two 
> > probes) in almost all cases 401 is seen. Works also w/o chunks
> > great.
> > * Via ZScaler app which routes the traffic through a HTTP proxy to
> > the 
> > corporate network. It takes way more probes to detect the 401. But
> > still 
> > performs acceptable. Chunks made also virtually no difference. At
> > some 
> > point it is even worse with broken connections (plain HTTP). HTTPS 
> > performs much worse.
> > 
> > So at least for the test options I have here, I tend to agree with
> > you. 
> > We either drop the step chunks or need to reduce the size to a sane
> > value.
> 
> I have now reduced the chunk size which performs nicely. I hope that
> it 
> will not be necessary and no one will write byte-by-byte. See
> attachment.
> 

Shall I commit the patch to my branch or do you want to fork my branch,
tweak it and submit a PR against the main project repository?

> My last question would be: How can we wire the RequestConfig
> parameter 
> to the DefaultBHttpClientConnection class? Http1Config?
> 

We cannot use RequestConfig at the connection level. It will have to be
Http1Config.

Oleg



-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

Re: Server-side mid-air connection close on upload

[Oleg Kalnichevski]

On Sun, 2020-05-31 at 23:13 +0200, Michael Osipov wrote:
> Am 2020-05-30 um 22:53 schrieb Oleg Kalnichevski:
> > On Sat, 2020-05-30 at 21:58 +0200, Michael Osipov wrote:
> > ...
> > > > Now we need to decide if this is as good as it gets or we want
> > > > to
> > > > put
> > > > more research into it.
> > > 
> > > I read liked you idea of the chunks. 2048 seems too low for me
> > > because
> > > most well write with a buffer of at least 4 KiB.
> > > 
> > > So I think we can try to things to miminize overhead:
> > > 
> > > * Make this feature optional, just like we do with Expectation.
> > > * Try to increase the step chunk size, e.g., 64 KiB or greater.
> > > With
> > > that we don't peek on each and every invocation.
> > > 
> > 
> > Yes, we may consider making it optional / disabled by default
> 
> Via RequestConfig?
> 
> > We can give step chunking another go but it is less important now
> > as I
> > have inserted the check into the low level socket stream whose
> > write
> > operations should be already quite optimized by the protocol layer.
> 
> But wouldn't that still reduce the check count because we'd hit the
> wait 
> with SSLSocket? I am bit confused by your statement.
> 

It might but what I am trying to say is that even now there would
likely be only one check on every large data chunk eliminating the
worst case scenario of the application layer writing one or two bytes
at a time each triggering a check and incurring a potential hit of 50
ms.

Oleg


-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

Re: Server-side mid-air connection close on upload

[Oleg Kalnichevski]

On Sat, 2020-05-30 at 21:58 +0200, Michael Osipov wrote:
> 

...

> > #isDataAvailable is not cheap even with 1 ms timeout. Socket
> > timeout
> > granularity is far from being 1 ms. In fact it is pretty close to
> > 50
> > ms. If we are not careful we can easily end up with 50 ms overhead
> > for
> > every write operation.
> 
> Where did you get the 50 ms? Is that a Java thing? OS thing? If OS
> which 
> OS did you try, reference for this value?
> 

Past experience with the stale connection check. It may not always be
50 ms depending on JRE / OS / Java version / other factors but it is
always substantially more than 1 ms. I presume this is a general
shortcoming of the classic i/o in Java. 

> > I am _hoping_ that SSL input stream would not actually read
> > anything
> > from the underlying socket as long as there is unencrypted data
> > stuck
> > in its buffers.
> 
> 

...

> > Makes sense. Please see the updates in my branch.
> 
> Will pick this up on Monday as soon as I get access to our
> infrastructure.
> 
> > Now we need to decide if this is as good as it gets or we want to
> > put
> > more research into it.
> 
> I read liked you idea of the chunks. 2048 seems too low for me
> because 
> most well write with a buffer of at least 4 KiB.
> 
> So I think we can try to things to miminize overhead:
> 
> * Make this feature optional, just like we do with Expectation.
> * Try to increase the step chunk size, e.g., 64 KiB or greater. With 
> that we don't peek on each and every invocation.
> 

Yes, we may consider making it optional / disabled by default

We can give step chunking another go but it is less important now as I
have inserted the check into the low level socket stream whose write
operations should be already quite optimized by the protocol layer.

Oleg



-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

Re: Server-side mid-air connection close on upload

[Oleg Kalnichevski]

On Fri, 2020-05-29 at 00:46 +0200, Michael Osipov wrote:
> Am 2020-05-28 um 22:51 schrieb Oleg Kalnichevski:
> > On Thu, 2020-05-28 at 22:01 +0200, Michael Osipov wrote:
> > > Am 2020-05-28 um 10:40 schrieb Oleg Kalnichevski:
> > > > On Thu, 2020-05-28 at 00:02 +0200, Michael Osipov wrote:
> > > > > 
> > > > 
> > > > 
> > > > ...
> > > > 
> > > > > > Please try out this patch. This should now give us proper
> > > > > > out
> > > > > > of
> > > > > > sequence response check with minimal overhead. The check
> > > > > > will
> > > > > > now
> > > > > > be
> > > > > > performed for chunks of 2048 bytes instead of each and
> > > > > > every
> > > > > > write
> > > > > > operations.
> > > > > > 
> > > > > > 
> > > > 
> > > > 
> > 
> > 
https://github.com/ok2c/httpcomponents-core/commit/7baa51f873da759d4e81973d4ca0a16f4199c51b
> > > > > 
> > > > > This one does not work at all. It does even break the final
> > > > > (authenticated) request. See for yourself:
> > > > > http://home.apache.org/~michaelo/issues/early-response/try-3/
> > > > > 
> > > > 
> > > > Please confirm this one works, at least for plain http:
> > > > 
> > > > 
> > 
> > 
https://github.com/ok2c/httpcomponents-core/commit/fb115a562921d8249a6bd8f3a5f5c4d6dba76949
> > > 
> > > Good news, this looks very decent for HTTP. HTTPS always fails.
> > > I am quite amazed that the SSL socket stream (AppInputStream) has
> > > such a
> > > different behavior. Other buffers internally which make
> > > #available()
> > > fail? There is an internal byte buffer of 4 KiB.
> > > 
> > > Output: 
> > > http://home.apache.org/~michaelo/issues/early-response/try-4/
> > > 
> > 
> > 
> > Great. Please pull one more commit from the same branch. It
> > _should_
> > fix the out of sequence response check with SSL connections.
> 
> Eureka, it works!
> 
> Can you explain why SSLSocket requires different handling? Moreover,
> why 
> not use "isDataAvailable(Timeout.ONE_MILLISECOND)" all the time? 
> Avoiding filling the buffer?
> 

#isDataAvailable is not cheap even with 1 ms timeout. Socket timeout
granularity is far from being 1 ms. In fact it is pretty close to 50
ms. If we are not careful we can easily end up with 50 ms overhead for
every write operation. 

I am _hoping_ that SSL input stream would not actually read anything
from the underlying socket as long as there is unencrypted data stuck
in its buffers.  


> If the different handling is necessary, here is a small proposal:
> > diff --git
> > a/httpcore5/src/main/java/org/apache/hc/core5/http/impl/io/DefaultB
> > HttpClientConnection.java
> > b/httpcore5/src/main/java/org/apache/hc/core5/http/impl/io/DefaultB
> > HttpClientConnection.java
> > index 37d750fc2..050da1864 100644
> > ---
> > a/httpcore5/src/main/java/org/apache/hc/core5/http/impl/io/DefaultB
> > HttpClientConnection.java
> > +++
> > b/httpcore5/src/main/java/org/apache/hc/core5/http/impl/io/DefaultB
> > HttpClientConnection.java
> > @@ -165,8 +165,4 @@ void checkForEarlyResponse() throws IOException
> > {
> > -if (ssl) {
> > -if
> > (isDataAvailable(Timeout.ONE_MILLISECOND)) {
> > -throw new
> > ResponseOutOfOrderException();
> > -}
> > -} else {
> > -if (socketInputStream.available() > 0)
> > {
> > -throw new
> > ResponseOutOfOrderException();
> > -}
> > +final boolean dataAvailable = ssl ?
> > isDataAvailable(Timeout.ONE_MILLISECOND) :
> > +(socketInputStream.available() >
> > 0);
> > +if (dataAvailable) {
> > +throw new
> > ResponseOutOfOrderException();
> 

Makes sense. Please see the updates in my branch.

Now we need to decide if this is as good as it gets or we want to put
more research into it.

Oleg



-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

Re: Server-side mid-air connection close on upload

[Oleg Kalnichevski]

On Thu, 2020-05-28 at 22:01 +0200, Michael Osipov wrote:
> Am 2020-05-28 um 10:40 schrieb Oleg Kalnichevski:
> > On Thu, 2020-05-28 at 00:02 +0200, Michael Osipov wrote:
> > > 
> > 
> > 
> > ...
> > 
> > > > Please try out this patch. This should now give us proper out
> > > > of
> > > > sequence response check with minimal overhead. The check will
> > > > now
> > > > be
> > > > performed for chunks of 2048 bytes instead of each and every
> > > > write
> > > > operations.
> > > > 
> > > > 
> > 
> > 
https://github.com/ok2c/httpcomponents-core/commit/7baa51f873da759d4e81973d4ca0a16f4199c51b
> > > 
> > > This one does not work at all. It does even break the final
> > > (authenticated) request. See for yourself:
> > > http://home.apache.org/~michaelo/issues/early-response/try-3/
> > > 
> > 
> > Please confirm this one works, at least for plain http:
> > 
> > 
https://github.com/ok2c/httpcomponents-core/commit/fb115a562921d8249a6bd8f3a5f5c4d6dba76949
> 
> Good news, this looks very decent for HTTP. HTTPS always fails.
> I am quite amazed that the SSL socket stream (AppInputStream) has
> such a 
> different behavior. Other buffers internally which make #available() 
> fail? There is an internal byte buffer of 4 KiB.
> 
> Output: http://home.apache.org/~michaelo/issues/early-response/try-4/
> 


Great. Please pull one more commit from the same branch. It _should_
fix the out of sequence response check with SSL connections.

Oleg



-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

Re: Server-side mid-air connection close on upload

[Oleg Kalnichevski]

On Thu, 2020-05-28 at 00:02 +0200, Michael Osipov wrote:
> 


...

> > Please try out this patch. This should now give us proper out of
> > sequence response check with minimal overhead. The check will now
> > be
> > performed for chunks of 2048 bytes instead of each and every write
> > operations.
> > 
> > 
https://github.com/ok2c/httpcomponents-core/commit/7baa51f873da759d4e81973d4ca0a16f4199c51b
> 
> This one does not work at all. It does even break the final 
> (authenticated) request. See for yourself: 
> http://home.apache.org/~michaelo/issues/early-response/try-3/
> 

Please confirm this one works, at least for plain http:

https://github.com/ok2c/httpcomponents-core/commit/fb115a562921d8249a6bd8f3a5f5c4d6dba76949

Oleg



-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

Re: Server-side mid-air connection close on upload

[Oleg Kalnichevski]

On Wed, 2020-05-27 at 22:14 +0200, Oleg Kalnichevski wrote:
> On Wed, 2020-05-27 at 21:10 +0200, Michael Osipov wrote:
> > Am 2020-05-27 um 19:13 schrieb Oleg Kalnichevski:
> > > On Wed, 2020-05-27 at 18:26 +0200, Michael Osipov wrote:
> > > > 
> > > 
> > > ...
> > > 
> > > 
> > > 
> > > 
> 
> 
https://github.com/ok2c/httpcomponents-core/commit/732a99563e73529e0359625fda54351b843b2e34
> > > > 
> > > > This patch does not work (reliably):
> > > > 
> > > > * I have never seen EarlyResponseException
> > > > * When run over HTTPS I see early responses somtimes, sometimes
> > > > I
> > > > get
> > > > SSLException
> > > > * Via HTTP I never see an early response, but only
> > > > SocketException
> > > > 
> > > > Buffers again? Note that curl reads the physical response in
> > > > any
> > > > case.
> > > > 
> > > > See output here:
> > > > http://home.apache.org/~michaelo/issues/early-response/try-1/
> > > > 
> > > 
> > > I do not want to give up yet. Could you please try out this
> > > patch?
> > > 
> > > 
> 
> 
https://github.com/ok2c/httpcomponents-core/commit/8d7b2e3d38b4d696d45cde16c52dba278be15438
> > 
> > Apologies, I made a mistake, since take 1 was 5.1-SNAPSHOT, I
> > assumed 
> > that take2 has this version too, noticed too late that it is 
> > 5.0.1-SNAPSHOT actually.
> > 
> > I have redone the tests, and raised to 50 iterations. Write buffer
> > in 
> > HttpEntity is 16 KiB.
> > 
> > 732a99563e73529e0359625fda54351b843b2e34:
> > * Performs great on HTTP, fails really early
> > * Does not work with TLS 1.3 at all
> > 
> > 8d7b2e3d38b4d696d45cde16c52dba278be15438:
> > * Works with HTTP, but takes a lot of payload to detect early
> > response
> > * HTTPS is a bit better, it takes around 400 kB.
> > 
> > Output is here: 
> > http://home.apache.org/~michaelo/issues/early-response/try-2/
> > 
> > Michael
> 
> I do not think it really matters if it is 0 kb or 400 kb as long as
> the
> server does not reset the connection and the request gets correctly
> retried. 
> 
> I will improve the patch further. Please bear with me.
> 

Please try out this patch. This should now give us proper out of
sequence response check with minimal overhead. The check will now be
performed for chunks of 2048 bytes instead of each and every write
operations.

https://github.com/ok2c/httpcomponents-core/commit/7baa51f873da759d4e81973d4ca0a16f4199c51b

Oleg


-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

Re: Server-side mid-air connection close on upload

[Oleg Kalnichevski]

On Wed, 2020-05-27 at 21:10 +0200, Michael Osipov wrote:
> Am 2020-05-27 um 19:13 schrieb Oleg Kalnichevski:
> > On Wed, 2020-05-27 at 18:26 +0200, Michael Osipov wrote:
> > > 
> > 
> > ...
> > 
> > 
> > 
> > 
https://github.com/ok2c/httpcomponents-core/commit/732a99563e73529e0359625fda54351b843b2e34
> > > 
> > > This patch does not work (reliably):
> > > 
> > > * I have never seen EarlyResponseException
> > > * When run over HTTPS I see early responses somtimes, sometimes I
> > > get
> > > SSLException
> > > * Via HTTP I never see an early response, but only
> > > SocketException
> > > 
> > > Buffers again? Note that curl reads the physical response in any
> > > case.
> > > 
> > > See output here:
> > > http://home.apache.org/~michaelo/issues/early-response/try-1/
> > > 
> > 
> > I do not want to give up yet. Could you please try out this patch?
> > 
> > 
https://github.com/ok2c/httpcomponents-core/commit/8d7b2e3d38b4d696d45cde16c52dba278be15438
> 
> Apologies, I made a mistake, since take 1 was 5.1-SNAPSHOT, I
> assumed 
> that take2 has this version too, noticed too late that it is 
> 5.0.1-SNAPSHOT actually.
> 
> I have redone the tests, and raised to 50 iterations. Write buffer
> in 
> HttpEntity is 16 KiB.
> 
> 732a99563e73529e0359625fda54351b843b2e34:
> * Performs great on HTTP, fails really early
> * Does not work with TLS 1.3 at all
> 
> 8d7b2e3d38b4d696d45cde16c52dba278be15438:
> * Works with HTTP, but takes a lot of payload to detect early
> response
> * HTTPS is a bit better, it takes around 400 kB.
> 
> Output is here: 
> http://home.apache.org/~michaelo/issues/early-response/try-2/
> 
> Michael

I do not think it really matters if it is 0 kb or 400 kb as long as the
server does not reset the connection and the request gets correctly
retried. 

I will improve the patch further. Please bear with me.

Oleg



-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

Re: Server-side mid-air connection close on upload

[Oleg Kalnichevski]

On Wed, 2020-05-27 at 18:26 +0200, Michael Osipov wrote:
> 

...



https://github.com/ok2c/httpcomponents-core/commit/732a99563e73529e0359625fda54351b843b2e34
> 
> This patch does not work (reliably):
> 
> * I have never seen EarlyResponseException
> * When run over HTTPS I see early responses somtimes, sometimes I
> get 
> SSLException
> * Via HTTP I never see an early response, but only SocketException
> 
> Buffers again? Note that curl reads the physical response in any
> case.
> 
> See output here: 
> http://home.apache.org/~michaelo/issues/early-response/try-1/
> 

I do not want to give up yet. Could you please try out this patch?

https://github.com/ok2c/httpcomponents-core/commit/8d7b2e3d38b4d696d45cde16c52dba278be15438

Oleg



-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

Re: Server-side mid-air connection close on upload

[Oleg Kalnichevski]

On Wed, 2020-05-27 at 14:32 +0200, Oleg Kalnichevski wrote:
> 


...

> 
https://github.com/apache/httpcomponents-client/commit/a554aadabafe26ae5412b26a311ffa105e623cc2
> > 
> > Tried with various timeouts, 5 ms, 15 ms, 25 ms, 50 ms, 500 ms.
> > Connections are correctly closed when:
> > * early response is received
> > * early response did not arrive in time and the server reached its 
> > swallow limit (worked already)
> > 
> > I ran multiple requests in a loop as well as a subsequent one with 
> > authentication header.
> > 
> > I am happily accepting this patch!
> > 
> 
> Great, but I still do not quite like the patch, though 
> 
> 1. There can be extra latency for normal request / response exchanges
> due to the early response check.
> 
> 2. The early response check is performed once and never retried. An
> out
> of sequence response can still be sent as a later point.  
> 
> I working on a better fix. Please bear with me.
> 

Micheal

Could you please try out this patch?

https://github.com/ok2c/httpcomponents-core/commit/732a99563e73529e0359625fda54351b843b2e34

Oleg



-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

Re: Server-side mid-air connection close on upload

[Oleg Kalnichevski]

On Wed, 2020-05-27 at 14:23 +0200, Michael Osipov wrote:
> Am 2020-05-27 um 12:14 schrieb Oleg Kalnichevski:
> > On Wed, 2020-05-27 at 00:01 +0200, Michael Osipov wrote:
> > > Am 2020-05-26 um 23:09 schrieb Oleg Kalnichevski:
> > > > On Tue, 2020-05-26 at 22:28 +0200, Michael Osipov wrote:
> > > > > Am 2020-05-26 um 20:20 schrieb Oleg Kalnichevski:
> > > > > > On Tue, 2020-05-26 at 17:58 +, Bernd Eckenfels wrote:
> > > > > > > Michael, this looks a bit like the packets in between
> > > > > > > have
> > > > > > > been
> > > > > > > TLS
> > > > > > > handshakes which have not been carried out because the
> > > > > > > engine
> > > > > > > was
> > > > > > > not
> > > > > > > kicked off. Maybe a starHandshake() would help? Or can
> > > > > > > you
> > > > > > > share
> > > > > > > the
> > > > > > > full traces? Did you try http as well?
> > > > > > > 
> > > > > > 
> > > > > > Michael,
> > > > > > 
> > > > > > I can only second what Bernd has said. Try to simplify the
> > > > > > setup
> > > > > > and
> > > > > > see if you can get things to work with plain HTTP first.
> > > > > > 
> > > > > > Also feel free to tweak my code as you see fit.
> > > > > 
> > > > > I followed Bernd's advise and disabled TLS. Although I
> > > > > already
> > > > > had a
> > > > > clue what is going wrong. Attached is a patch on top of
> > > > > Oleg's
> > > > > branch
> > > > > which makes it work unencrypted and encrypted. I will explain
> > > > > in
> > > > > detail
> > > > > why the changes are necessary.
> > > > > 
> > > > > > -public static final Timeout
> > > > > > DEFAULT_WAIT_FOR_EARLY_RESPONSE =
> > > > > > Timeout.ofMilliseconds(5);
> > > > > > +public static final Timeout
> > > > > > DEFAULT_WAIT_FOR_EARLY_RESPONSE =
> > > > > > Timeout.ofMilliseconds(50);
> > > > > 
> > > > > unless client and server are physically next to each other 5
> > > > > ms
> > > > > are
> > > > > virtually impossible. I have tried with our server at work. I
> > > > > am
> > > > > connected with my laptop via VPN to the corporate network, I
> > > > > don't
> > > > > know
> > > > > where the peering point of our VPN provider is, but the
> > > > > server is
> > > > > physically 10 km away from me and with TLS 1.3 it takes 28 ms
> > > > > to
> > > > > produce
> > > > > the 401.
> > > > > 
> > > > > Maybe both timeouts (expect and early) should be configurable
> > > > > via
> > > > > RequestConfig?
> > > > > 
> > > > > > +conn.flush();
> > > > 
> > > > Michael
> > > > 
> > > > You are absolutely right. Not flushing the request head was the
> > > > cause
> > > > of the problem.
> > > > 
> > > > Could you please try out a slightly different take on your
> > > > original
> > > > patch, though?
> > > > 
> > > > 
> > 
> > 
https://github.com/ok2c/httpcomponents-core/commit/39f6d69a87586c147dc080431d45d6d48acb2c80
> > > > 
> > > > HttpRequestExecutor ought not be flushing all request message
> > > > heads
> > > > indiscriminately and should still try to pack small payload
> > > > messages
> > > > into a single IP frame for performance reasons.
> > > 
> > > Makes sense!
> > > 
> > > > I also would not overload RequestConfig with too many options
> > > > but
> > > > we
> > > > can discuss that later.
> > > 
> > > Agreed.
> > > 
> > > > > This is why it did not work before is that headers were stuck
> > > > > in
> > > > > the
> > > > > local buffer and were flushed only when the body was sent. So
> > > > > the
&g

Re: Server-side mid-air connection close on upload

[Oleg Kalnichevski]

On Wed, 2020-05-27 at 00:01 +0200, Michael Osipov wrote:
> Am 2020-05-26 um 23:09 schrieb Oleg Kalnichevski:
> > On Tue, 2020-05-26 at 22:28 +0200, Michael Osipov wrote:
> > > Am 2020-05-26 um 20:20 schrieb Oleg Kalnichevski:
> > > > On Tue, 2020-05-26 at 17:58 +, Bernd Eckenfels wrote:
> > > > > Michael, this looks a bit like the packets in between have
> > > > > been
> > > > > TLS
> > > > > handshakes which have not been carried out because the engine
> > > > > was
> > > > > not
> > > > > kicked off. Maybe a starHandshake() would help? Or can you
> > > > > share
> > > > > the
> > > > > full traces? Did you try http as well?
> > > > > 
> > > > 
> > > > Michael,
> > > > 
> > > > I can only second what Bernd has said. Try to simplify the
> > > > setup
> > > > and
> > > > see if you can get things to work with plain HTTP first.
> > > > 
> > > > Also feel free to tweak my code as you see fit.
> > > 
> > > I followed Bernd's advise and disabled TLS. Although I already
> > > had a
> > > clue what is going wrong. Attached is a patch on top of Oleg's
> > > branch
> > > which makes it work unencrypted and encrypted. I will explain in
> > > detail
> > > why the changes are necessary.
> > > 
> > > > -public static final Timeout
> > > > DEFAULT_WAIT_FOR_EARLY_RESPONSE =
> > > > Timeout.ofMilliseconds(5);
> > > > +public static final Timeout
> > > > DEFAULT_WAIT_FOR_EARLY_RESPONSE =
> > > > Timeout.ofMilliseconds(50);
> > > 
> > > unless client and server are physically next to each other 5 ms
> > > are
> > > virtually impossible. I have tried with our server at work. I am
> > > connected with my laptop via VPN to the corporate network, I
> > > don't
> > > know
> > > where the peering point of our VPN provider is, but the server is
> > > physically 10 km away from me and with TLS 1.3 it takes 28 ms to
> > > produce
> > > the 401.
> > > 
> > > Maybe both timeouts (expect and early) should be configurable via
> > > RequestConfig?
> > > 
> > > > +conn.flush();
> > 
> > Michael
> > 
> > You are absolutely right. Not flushing the request head was the
> > cause
> > of the problem.
> > 
> > Could you please try out a slightly different take on your original
> > patch, though?
> > 
> > 
https://github.com/ok2c/httpcomponents-core/commit/39f6d69a87586c147dc080431d45d6d48acb2c80
> > 
> > HttpRequestExecutor ought not be flushing all request message heads
> > indiscriminately and should still try to pack small payload
> > messages
> > into a single IP frame for performance reasons.
> 
> Makes sense!
> 
> > I also would not overload RequestConfig with too many options but
> > we
> > can discuss that later.
> 
> Agreed.
> 
> > > This is why it did not work before is that headers were stuck in
> > > the
> > > local buffer and were flushed only when the body was sent. So the
> > > server
> > > never saw the headers before the body. The wait was pointless.
> > > 
> > > Please also note that even the buffer for headers might be large
> > > due
> > > to
> > > some authnz tokens. E.g., JWT or SPNEGO, cookies, etc.
> > > 
> > > > +response =
> > > > conn.receiveResponseHeader();
> > > 
> > > One needs to read the headers because the client shall be able to
> > > see
> > > the status code and if fast enough the response body. In my case
> > > the
> > > error page from Tomcat.
> > > 
> > > RFC 7230, section 6.5 says:
> > > > A client sending a message body SHOULD monitor the network
> > > > connection
> > > > for an error response while it is transmitting the
> > > > request.  If
> > > > the
> > > > client sees a response that indicates the server does not
> > > > wish
> > > > to
> > > > receive the message body and is closing the connection, the
> > > > client
> > > > SHOULD immediately cease transmitting the body and close
> > > > its
> > > > side of
> > > > the connection.
> > > 
> > > I have issued another request and HttpClient reuses the
> > > connection
> > > because Tomcat forgot to send "Connection: close".
> > 
> > Hmm. That sounds weird. HttpClient must terminate the connection if
> > it
> > failed to submit the entire message body declared in `Content-
> > Length`
> > header. The #terminateRequest should have marked the connection as
> > `inconsistent`.
> > 
> > Could you please send me a wire log of the session?
> 
> So far, the patch works and I can access the first 401 response.
> 

This should fix the problem with incorrect connection re-use. Please
review:

https://github.com/apache/httpcomponents-client/commit/a554aadabafe26ae5412b26a311ffa105e623cc2

Oleg 



-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

Re: Server-side mid-air connection close on upload

[Oleg Kalnichevski]

On Wed, 2020-05-27 at 11:48 +0200, Michael Osipov wrote:
> Am 2020-05-27 um 11:39 schrieb Oleg Kalnichevski:
> > On Wed, 2020-05-27 at 00:01 +0200, Michael Osipov wrote:
> > > 
> > 
> > 
> > ...
> > 
> > > So far, the patch works and I can access the first 401 response.
> > > 
> > 
> > I found a pretty nasty bug in HttpClient that leads to re-use of
> > connections after request termination. I am working on a fix.
> 
> Awesome! Excited to try. I guess this is also present in 4.5.x.
> 

No, it is 5.x only.

Cheers

Oleg



-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

Re: Server-side mid-air connection close on upload

[Oleg Kalnichevski]

On Wed, 2020-05-27 at 00:01 +0200, Michael Osipov wrote:
> 


...

> So far, the patch works and I can access the first 401 response.
> 

I found a pretty nasty bug in HttpClient that leads to re-use of
connections after request termination. I am working on a fix.

Oleg



-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

Re: Server-side mid-air connection close on upload

[Oleg Kalnichevski]

On Tue, 2020-05-26 at 22:28 +0200, Michael Osipov wrote:
> Am 2020-05-26 um 20:20 schrieb Oleg Kalnichevski:
> > On Tue, 2020-05-26 at 17:58 +, Bernd Eckenfels wrote:
> > > Michael, this looks a bit like the packets in between have been
> > > TLS
> > > handshakes which have not been carried out because the engine was
> > > not
> > > kicked off. Maybe a starHandshake() would help? Or can you share
> > > the
> > > full traces? Did you try http as well?
> > > 
> > 
> > Michael,
> > 
> > I can only second what Bernd has said. Try to simplify the setup
> > and
> > see if you can get things to work with plain HTTP first.
> > 
> > Also feel free to tweak my code as you see fit.
> 
> I followed Bernd's advise and disabled TLS. Although I already had a 
> clue what is going wrong. Attached is a patch on top of Oleg's
> branch 
> which makes it work unencrypted and encrypted. I will explain in
> detail 
> why the changes are necessary.
> 
> > -public static final Timeout DEFAULT_WAIT_FOR_EARLY_RESPONSE =
> > Timeout.ofMilliseconds(5);
> > +public static final Timeout DEFAULT_WAIT_FOR_EARLY_RESPONSE =
> > Timeout.ofMilliseconds(50);
> 
> unless client and server are physically next to each other 5 ms are 
> virtually impossible. I have tried with our server at work. I am 
> connected with my laptop via VPN to the corporate network, I don't
> know 
> where the peering point of our VPN provider is, but the server is 
> physically 10 km away from me and with TLS 1.3 it takes 28 ms to
> produce 
> the 401.
> 
> Maybe both timeouts (expect and early) should be configurable via 
> RequestConfig?
> 
> > +conn.flush();
> 

Michael

You are absolutely right. Not flushing the request head was the cause
of the problem.

Could you please try out a slightly different take on your original
patch, though?

https://github.com/ok2c/httpcomponents-core/commit/39f6d69a87586c147dc080431d45d6d48acb2c80

HttpRequestExecutor ought not be flushing all request message heads
indiscriminately and should still try to pack small payload messages
into a single IP frame for performance reasons.

I also would not overload RequestConfig with too many options but we
can discuss that later.


> This is why it did not work before is that headers were stuck in the 
> local buffer and were flushed only when the body was sent. So the
> server 
> never saw the headers before the body. The wait was pointless.
> 
> Please also note that even the buffer for headers might be large due
> to 
> some authnz tokens. E.g., JWT or SPNEGO, cookies, etc.
> 
> > +response = conn.receiveResponseHeader();
> 
> One needs to read the headers because the client shall be able to
> see 
> the status code and if fast enough the response body. In my case the 
> error page from Tomcat.
> 
> RFC 7230, section 6.5 says:
> >A client sending a message body SHOULD monitor the network
> > connection
> >for an error response while it is transmitting the request.  If
> > the
> >client sees a response that indicates the server does not wish
> > to
> >receive the message body and is closing the connection, the
> > client
> >SHOULD immediately cease transmitting the body and close its
> > side of
> >the connection.
> 
> I have issued another request and HttpClient reuses the connection 
> because Tomcat forgot to send "Connection: close".

Hmm. That sounds weird. HttpClient must terminate the connection if it
failed to submit the entire message body declared in `Content-Length`
header. The #terminateRequest should have marked the connection as
`inconsistent`.

Could you please send me a wire log of the session?

Oleg


>  At some point the 
> second request will fail and no additional 401 is responded (of
> course. 
> I consider the missing "Connection: close" to be a bug in Tomcat [1].
> So HttpClent behaves correctly. I assume that then a close would be 
> send, response#close() will do the right thing and subsequent
> requests 
> will use new connections.
> 
> WDYT?
> 
> Michael
> 
> [1] 
> https://www.mail-archive.com/users@tomcat.apache.org/msg135255.html
> -
> To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
> For additional commands, e-mail: httpclient-users-h...@hc.apache.org


-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

Re: Server-side mid-air connection close on upload

[Oleg Kalnichevski]

On Tue, 2020-05-26 at 17:58 +, Bernd Eckenfels wrote:
> Michael, this looks a bit like the packets in between have been TLS
> handshakes which have not been carried out because the engine was not
> kicked off. Maybe a starHandshake() would help? Or can you share the
> full traces? Did you try http as well?
> 

Michael,

I can only second what Bernd has said. Try to simplify the setup and
see if you can get things to work with plain HTTP first.

Also feel free to tweak my code as you see fit.

Oleg


> Bernd
> --
> http://bernd.eckenfels.net
> 
> Von: Michael Osipov 
> Gesendet: Dienstag, Mai 26, 2020 7:53 PM
> An: HttpClient User Discussion
> Betreff: Re: Server-side mid-air connection close on upload
> 
> Am 2020-05-26 um 09:14 schrieb Oleg Kalnichevski:
> > On Mon, 2020-05-25 at 11:56 +0200, Michael Osipov wrote:
> > > 
> > 
> > ...
> > 
> > 
> > > Where long denotes the min size of the request body and Timeout
> > > how
> > > long
> > > we want to wait for the early response.
> > > 
> > > Let me know if you have somthing to test, setup is there at work.
> > > 
> > 
> > Hi Michael
> > 
> > Please try out this branch in your environment
> > 
> > 
https://github.com/ok2c/httpcomponents-core/tree/early_response_blocking_io
> 
> I did try this, with 5 ms and 50 ms. Both do not work. The early
> response is not seen. Here are two lines from Wireshark:
> 
> > No.   TimeSource  Destination ProtocolLength  Info
> > 222   0.072434139.22.137.23   147.54.64.149   TLSv1.3
> > 343 [TLS segment of a reassembled PDU]
> > 239   0.025213147.54.64.149   139.22.137.23   HTTP678  
> >HTTP/1.1 401   (text/html)
> > No.   TimeSource  Destination ProtocolLength  Info
> > 153   0.130427139.22.137.23   147.54.64.149   TLSv1.3
> > 343 [TLS segment of a reassembled PDU]
> > 176   0.026806147.54.64.149   139.22.137.23   HTTP678  
> >HTTP/1.1 401   (text/html)
> 
> times are relative to the previous packet.
> 
> When I set to 5 s the POST paket with the sole headers shows 5
> seconds
> compared to the previous packet "New Session Ticket". I don't believe
> that this needs 5 s:
> 
> > No.   TimeSource  Destination ProtocolLength  Info
> > 1373  *REF*   147.54.64.149   139.22.137.23   TLSv1.3 133 New
> > Session Ticket
> > 1374  0.23139.22.137.23   147.54.64.149   TCP 54   
> >53316 → 1 [ACK] Seq=506 Ack=7157 Win=261888 Len=0
> > 1375  0.44139.22.137.23   147.54.64.149   TCP 54   
> >[TCP Dup ACK 1374#1] 53316 → 1 [ACK] Seq=506 Ack=7157
> > Win=261888 Len=0
> > 2905  5.104702139.22.137.23   147.54.64.149   TLSv1.3
> > 343 [TLS segment of a reassembled PDU]
> > 2906  5.104793139.22.137.23   147.54.64.149   TCP 343  
> >[TCP Retransmission] 53316 → 1 [PSH, ACK] Seq=506 Ack=7157
> > Win=261888 Len=289
> > 2925  5.130667147.54.64.149   139.22.137.23   HTTP678  
> >HTTP/1.1 401   (text/html)
> 
> I am either incorrectly interpreting the output or something is wrong
> here.
> In any case it still tried to send off the entire body.
> 
> Is there anything else I can try?
> 
> 
> -
> To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
> For additional commands, e-mail: httpclient-users-h...@hc.apache.org
> 


-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

Re: Server-side mid-air connection close on upload

[Oleg Kalnichevski]

On Mon, 2020-05-25 at 11:56 +0200, Michael Osipov wrote:
> 

...


> Where long denotes the min size of the request body and Timeout how
> long 
> we want to wait for the early response.
> 
> Let me know if you have somthing to test, setup is there at work.
> 

Hi Michael 

Please try out this branch in your environment 

https://github.com/ok2c/httpcomponents-core/tree/early_response_blocking_io

Here is the change-set:
https://github.com/ok2c/httpcomponents-core/commit/e64bab5dc41518cfeb05e2ac8174a5c3ffd246c7
 

Oleg



-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

Re: Server-side mid-air connection close on upload

[Oleg Kalnichevski]

On Mon, 2020-05-25 at 11:45 +0200, Michael Osipov wrote:
> Am 2020-05-25 um 09:47 schrieb Oleg Kalnichevski:
> > On Sat, 2020-05-23 at 01:10 +0200, Michael Osipov wrote:
> > > > 
> > 
> > ...
> > 
> > > Am 2020-05-22 um 22:18 schrieb Oleg Kalnichevski:
> > > > Theoretically we could enhance the classic client-side protocol
> > > > handler
> > > > to check if the length of the request entity is known and is
> > > > greater,
> > > > say, than 1MB, send the request head first, stop and wait for
> > > > an
> > > > early
> > > > response, but this could potentially kill performance of entity
> > > > enclosing requests processed in proper request-response
> > > > sequence.
> > > > This
> > > > approach has been considered on several occasions but always
> > > > decided
> > > > against.
> > > 
> > > So basically, if I understand your idea correctly, we could have
> > > either
> > > at client or request level a #setEarlyResponseThreshold(long,
> > > Timeout)
> > > (in bytes, how long to wait for an early response) which checks
> > > for
> > > "Content-Length" and sends headers first, tries to read, no
> > > response,
> > > continues sending else process early response -- all in the same
> > > thread?!
> > > 
> > 
> > I am not sure I understand the question. How many threads would you
> > like?
> 
> My wording was bad. The basic idea is to read from the socket
> (possible 
> early response) before invoking 
> org.apache.hc.core5.http.HttpEntity.writeTo(OutputStream). Forget
> about 
> the threads.
> 

Got you now. This is an option but this extra read will add to the
overall response processing time and can be potentially very expensive
for small payload messages. The idea to use SocketInputStream#available
might actually solve the problem. I am going to look into it.

Cheers

Oleg


> > > Michael
> > > 
> > > [1] https://stackoverflow.com/a/9176445/696632
> > > [2] https://stackoverflow.com/a/14483857/696632
> > > 
> > > ---
> > > --
> > > To unsubscribe, e-mail: 
> > > httpclient-users-unsubscr...@hc.apache.org
> > > For additional commands, e-mail: 
> > > httpclient-users-h...@hc.apache.org
> > > 
> > 
> > 
> > -
> > 
> > To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
> > For additional commands, e-mail: 
> > httpclient-users-h...@hc.apache.org
> > 
> 
> 
> -
> To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
> For additional commands, e-mail: httpclient-users-h...@hc.apache.org
> 


-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

Re: Server-side mid-air connection close on upload

[Oleg Kalnichevski]

On Sat, 2020-05-23 at 01:10 +0200, Michael Osipov wrote:
> > 

...

> Am 2020-05-22 um 22:18 schrieb Oleg Kalnichevski:
> > Theoretically we could enhance the classic client-side protocol
> > handler
> > to check if the length of the request entity is known and is
> > greater,
> > say, than 1MB, send the request head first, stop and wait for an
> > early
> > response, but this could potentially kill performance of entity
> > enclosing requests processed in proper request-response sequence.
> > This
> > approach has been considered on several occasions but always
> > decided
> > against.
> 
> So basically, if I understand your idea correctly, we could have
> either 
> at client or request level a #setEarlyResponseThreshold(long,
> Timeout) 
> (in bytes, how long to wait for an early response) which checks for 
> "Content-Length" and sends headers first, tries to read, no
> response, 
> continues sending else process early response -- all in the same
> thread?!
> 

I am not sure I understand the question. How many threads would you
like?

Oleg


> Michael
> 
> [1] https://stackoverflow.com/a/9176445/696632
> [2] https://stackoverflow.com/a/14483857/696632
> 
> -
> To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
> For additional commands, e-mail: httpclient-users-h...@hc.apache.org
> 


-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

Re: Server-side mid-air connection close on upload

[Oleg Kalnichevski]

On Fri, 2020-05-22 at 20:56 +, Bernd Eckenfels wrote:
> Hello,
> 
> the blocking code could use a different reader thread, but maybe that
> would be too much overhead? (Having a single polling thread might
> have a lower footprint but could create concurrency stress)
> 
> The question is if the sync code could use available() after sending
> some buffers (not only the header but enough to fill a send buffer,
> say x*32kb? (configurable)). This might block but it also might catch
> early answers/closes.
> 

Hi Bernd

Using SocketInputStream#available() to test for an out of sequence
response is actually an interesting idea. I will experiment with it in
the coming days. Thank you!

Oleg 


> Gruss
> Bernd
> --
> http://bernd.eckenfels.net
> ____
> Von: Oleg Kalnichevski 
> Gesendet: Friday, May 22, 2020 10:18:02 PM
> An: HttpClient User Discussion 
> Betreff: Re: Server-side mid-air connection close on upload
> 
> On Fri, 2020-05-22 at 19:25 +0200, Michael Osipov wrote:
> > Hi folks,
> > 
> > I believe that this is a wellknown limitation in blocking I/O, but
> > would
> > like to know how to solve/workaround this issue. The source
> > discussion
> > is here [1].
> > 
> > Bascially, I am posting a large file (~7 MB) to Tomcat 8.5.54 where
> > the
> > resource needs to be authenticated:
> > 
> > > 69 [main] DEBUG
> > > org.apache.hc.client5.http.impl.classic.InternalHttpClient - ex-
> > > 0001: preparing request execution
> > > 145 [main] DEBUG
> > > org.apache.hc.client5.http.protocol.RequestAddCookies - Cookie
> > > spec
> > > selected: strict
> > > 150 [main] DEBUG
> > > org.apache.hc.client5.http.protocol.RequestAuthCache - Auth cache
> > > not set in the context
> > > 150 [main] DEBUG
> > > org.apache.hc.client5.http.impl.classic.ProtocolExec - ex-
> > > 0001:
> > > target auth state: UNCHALLENGED
> > > 151 [main] DEBUG
> > > org.apache.hc.client5.http.impl.classic.ProtocolExec - ex-
> > > 0001:
> > > proxy auth state: UNCHALLENGED
> > > 151 [main] DEBUG
> > > org.apache.hc.client5.http.impl.classic.ConnectExec - ex-
> > > 0001:
> > > acquiring connection with route {s}->https://:1
> > > 151 [main] DEBUG
> > > org.apache.hc.client5.http.impl.classic.InternalHttpClient - ex-
> > > 0001: acquiring endpoint (3 MINUTES)
> > > 152 [main] DEBUG
> > > org.apache.hc.client5.http.impl.io.PoolingHttpClientConnectionMan
> > > ag
> > > er - ex-0001: endpoint lease request (3 MINUTES) [route: {s}-
> > > >
> > > https://:1][total available: 0; route allocated: 0
> > > of
> > > 5; total allocated: 0 of 25]
> > > 175 [main] DEBUG
> > > org.apache.hc.client5.http.impl.io.PoolingHttpClientConnectionMan
> > > ag
> > > er - ex-0001: endpoint leased [route: {s}->
> > > https://:1][total available: 0; route allocated: 1
> > > of
> > > 5; total allocated: 1 of 25]
> > > 183 [main] DEBUG
> > > org.apache.hc.client5.http.impl.io.PoolingHttpClientConnectionMan
> > > ag
> > > er - ex-0001: acquired ep-
> > > 183 [main] DEBUG
> > > org.apache.hc.client5.http.impl.classic.InternalHttpClient - ex-
> > > 0001: acquired endpoint ep-
> > > 183 [main] DEBUG
> > > org.apache.hc.client5.http.impl.classic.ConnectExec - ex-
> > > 0001:
> > > opening connection {s}->https://:1
> > > 184 [main] DEBUG
> > > org.apache.hc.client5.http.impl.classic.InternalHttpClient - ep-
> > > : connecting endpoint (3 MINUTES)
> > > 184 [main] DEBUG
> > > org.apache.hc.client5.http.impl.io.PoolingHttpClientConnectionMan
> > > ag
> > > er - ep-: connecting endpoint to https://:1
> > > (3 MINUTES)
> > > 201 [main] DEBUG
> > > org.apache.hc.client5.http.impl.io.DefaultHttpClientConnectionOpe
> > > ra
> > > tor - http-outgoing-0: connecting to /:1
> > > 201 [main] DEBUG
> > > org.apache.hc.client5.http.ssl.SSLConnectionSocketFactory -
> > > Connecting socket to /:1 with timeout 3 MINUTES
> > > 237 [main] DEBUG
> > > org.apache.hc.client5.http.ssl.SSLConnectionSocketFactory -
> > > Enabled
> > > protocols: [TLSv1.3, TLSv1.2]
> > > 237 [main] DEBUG
> > > org.apache.hc.client5.http.ssl.SSLConnectionSocketFactory -
> > &g

Re: Server-side mid-air connection close on upload

[Oleg Kalnichevski]

On Fri, 2020-05-22 at 19:25 +0200, Michael Osipov wrote:
> Hi folks,
> 
> I believe that this is a wellknown limitation in blocking I/O, but
> would 
> like to know how to solve/workaround this issue. The source
> discussion 
> is here [1].
> 
> Bascially, I am posting a large file (~7 MB) to Tomcat 8.5.54 where
> the 
> resource needs to be authenticated:
> 
> > 69 [main] DEBUG
> > org.apache.hc.client5.http.impl.classic.InternalHttpClient - ex-
> > 0001: preparing request execution
> > 145 [main] DEBUG
> > org.apache.hc.client5.http.protocol.RequestAddCookies - Cookie spec
> > selected: strict
> > 150 [main] DEBUG
> > org.apache.hc.client5.http.protocol.RequestAuthCache - Auth cache
> > not set in the context
> > 150 [main] DEBUG
> > org.apache.hc.client5.http.impl.classic.ProtocolExec - ex-0001: 
> > target auth state: UNCHALLENGED
> > 151 [main] DEBUG
> > org.apache.hc.client5.http.impl.classic.ProtocolExec - ex-0001: 
> > proxy auth state: UNCHALLENGED
> > 151 [main] DEBUG
> > org.apache.hc.client5.http.impl.classic.ConnectExec - ex-0001:
> > acquiring connection with route {s}->https://:1
> > 151 [main] DEBUG
> > org.apache.hc.client5.http.impl.classic.InternalHttpClient - ex-
> > 0001: acquiring endpoint (3 MINUTES)
> > 152 [main] DEBUG
> > org.apache.hc.client5.http.impl.io.PoolingHttpClientConnectionManag
> > er - ex-0001: endpoint lease request (3 MINUTES) [route: {s}->
> > https://:1][total available: 0; route allocated: 0 of
> > 5; total allocated: 0 of 25]
> > 175 [main] DEBUG
> > org.apache.hc.client5.http.impl.io.PoolingHttpClientConnectionManag
> > er - ex-0001: endpoint leased [route: {s}->
> > https://:1][total available: 0; route allocated: 1 of
> > 5; total allocated: 1 of 25]
> > 183 [main] DEBUG
> > org.apache.hc.client5.http.impl.io.PoolingHttpClientConnectionManag
> > er - ex-0001: acquired ep-
> > 183 [main] DEBUG
> > org.apache.hc.client5.http.impl.classic.InternalHttpClient - ex-
> > 0001: acquired endpoint ep-
> > 183 [main] DEBUG
> > org.apache.hc.client5.http.impl.classic.ConnectExec - ex-0001:
> > opening connection {s}->https://:1
> > 184 [main] DEBUG
> > org.apache.hc.client5.http.impl.classic.InternalHttpClient - ep-
> > : connecting endpoint (3 MINUTES)
> > 184 [main] DEBUG
> > org.apache.hc.client5.http.impl.io.PoolingHttpClientConnectionManag
> > er - ep-: connecting endpoint to https://:1
> > (3 MINUTES)
> > 201 [main] DEBUG
> > org.apache.hc.client5.http.impl.io.DefaultHttpClientConnectionOpera
> > tor - http-outgoing-0: connecting to /:1
> > 201 [main] DEBUG
> > org.apache.hc.client5.http.ssl.SSLConnectionSocketFactory -
> > Connecting socket to /:1 with timeout 3 MINUTES
> > 237 [main] DEBUG
> > org.apache.hc.client5.http.ssl.SSLConnectionSocketFactory - Enabled
> > protocols: [TLSv1.3, TLSv1.2]
> > 237 [main] DEBUG
> > org.apache.hc.client5.http.ssl.SSLConnectionSocketFactory - Enabled
> > cipher suites:[TLS_AES_256_GCM_SHA384, TLS_AES_128_GCM_SHA256,
> > TLS_CHACHA20_POLY1305_SHA256,
> > TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,
> > TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,
> > TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256,
> > TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,
> > TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256,
> > TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,
> > TLS_DHE_RSA_WITH_AES_256_GCM_SHA384,
> > TLS_DHE_RSA_WITH_CHACHA20_POLY1305_SHA256,
> > TLS_DHE_DSS_WITH_AES_256_GCM_SHA384,
> > TLS_DHE_RSA_WITH_AES_128_GCM_SHA256,
> > TLS_DHE_DSS_WITH_AES_128_GCM_SHA256,
> > TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384,
> > TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384,
> > TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256,
> > TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256,
> > TLS_DHE_RSA_WITH_AES_256_CBC_SHA256,
> > TLS_DHE_DSS_WITH_AES_256_CBC_SHA256,
> > TLS_DHE_RSA_WITH_AES_128_CBC_SHA256,
> > TLS_DHE_DSS_WITH_AES_128_CBC_SHA256,
> > TLS_ECDH_ECDSA_WITH_AES_256_GCM_SHA384,
> > TLS_ECDH_RSA_WITH_AES_256_GCM_SHA384,
> > TLS_ECDH_ECDSA_WITH_AES_128_GCM_SHA256,
> > TLS_ECDH_RSA_WITH_AES_128_GCM_SHA256,
> > TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA384,
> > TLS_ECDH_RSA_WITH_AES_256_CBC_SHA384,
> > TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA256,
> > TLS_ECDH_RSA_WITH_AES_128_CBC_SHA256,
> > TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA,
> > TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,
> > TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA,
> > TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,
> > TLS_DHE_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_DSS_WITH_AES_256_CBC_SHA,
> > TLS_DHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_DSS_WITH_AES_128_CBC_SHA,
> > TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA,
> > TLS_ECDH_RSA_WITH_AES_256_CBC_SHA,
> > TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA,
> > TLS_ECDH_RSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_256_GCM_SHA384,
> > TLS_RSA_WITH_AES_128_GCM_SHA256, TLS_RSA_WITH_AES_256_CBC_SHA256,
> > TLS_RSA_WITH_AES_128_CBC_SHA256, TLS_RSA_WITH_AES_256_CBC_SHA,
> > TLS_RSA_WITH_AES_128_CBC_SHA, TLS_EMPTY_RENEGOTIATION_INFO_SCSV]
> > 237 [main] DEBUG
> > 

Re: SNI for http core v5 server (non-async)

[Oleg Kalnichevski]

On Wed, 2020-05-20 at 15:52 -0400, C c wrote:
> It seems that DefaultTlsSetupHandler is final and I can't extend it.
> One way is for me to change the source so that I can extend it.
> 
> Is there another suggested way I can adjust that class for SNI ?
> 
> 

The class literally has two lines of code. What is the point of
extending it?

Oleg


> 
> On Wed, May 20, 2020 at 7:05 AM Oleg Kalnichevski 
> wrote:
> 
> > On Tue, 2020-05-19 at 21:25 -0400, C c wrote:
> > > I looked at emails and docs and can't find a trail on getting
> > > started
> > > w/
> > > SNI.
> > > 
> > > In netty, I can set up SNI to look up what domain is being
> > > requested
> > > so I
> > > can use a cert to the non-async server so I can serve multiple
> > > domains with
> > > 'Let's Encrypt'.
> > > 
> > > How do I do SNI in HTTP core v5 non-async server?
> > > 
> > > Vic
> > 
> > Hi Vic
> > 
> > SNI extension APIs are available in Java as of version 11 only.
> > HttpCore is currently Java 1.7 compatible and therefore does not
> > provide any SNI specific functionality. You will likely need to
> > implement a custom TLS setup handler instead of
> > `DefaultTlsSetupHandler` in order to customize TLS context
> > negotiation.
> > 
> > Hope this helps
> > 
> > Oleg
> > 
> > 
> > -
> > 
> > To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
> > For additional commands, e-mail: 
> > httpclient-users-h...@hc.apache.org
> > 
> > 


-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

Re: SNI for http core v5 server (non-async)

[Oleg Kalnichevski]

On Tue, 2020-05-19 at 21:25 -0400, C c wrote:
> I looked at emails and docs and can't find a trail on getting started
> w/
> SNI.
> 
> In netty, I can set up SNI to look up what domain is being requested
> so I
> can use a cert to the non-async server so I can serve multiple
> domains with
> 'Let's Encrypt'.
> 
> How do I do SNI in HTTP core v5 non-async server?
> 
> Vic

Hi Vic

SNI extension APIs are available in Java as of version 11 only.
HttpCore is currently Java 1.7 compatible and therefore does not
provide any SNI specific functionality. You will likely need to
implement a custom TLS setup handler instead of
`DefaultTlsSetupHandler` in order to customize TLS context negotiation.

Hope this helps

Oleg


-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

Re: After 100,000's of requests, I start getting status code 400 bad request for all requests

[Oleg Kalnichevski]

On Tue, 2020-05-19 at 11:39 -0500, Nicholas DiPiazza wrote:
> Yep Kerberos is an option. If I get desperate.
> 

Kerberos support is even worse. One ought to be using HTTP basic over
TLS or digest.

Oleg

> 
> 
> On Tue, May 19, 2020 at 10:28 AM Oleg Kalnichevski 
> wrote:
> 
> > On Tue, 2020-05-19 at 08:56 -0500, Nicholas DiPiazza wrote:
> > > Yeah I agree Oleg. But I'm at a loss as to why I am able to stop
> > > &
> > > start
> > > the program and then it starts working again fine.
> > > 
> > > There is NTLM authentication involved. Perhaps the auth gets into
> > > a
> > > stale
> > > state and starts causing 400 errors?
> > > 
> > 
> > Hmm. Then, client side NTLM state corruption is more likely. The
> > trouble is there is no longer a single person on the project who
> > has
> > both capacity and inclination to work on NTLM.
> > 
> > Does the server support any other authentication schemes?
> > 
> > Oleg
> > 
> > 
> > 
> > > On Tue, May 19, 2020 at 8:06 AM Oleg Kalnichevski <
> > > ol...@apache.org>
> > > wrote:
> > > 
> > > > On Mon, 2020-05-18 at 20:24 -0500, Nicholas DiPiazza wrote:
> > > > > Hey HTTP client users list!
> > > > > 
> > > > > I've got some strange behavior going on and could use some
> > > > > help
> > > > > sorting it
> > > > > out.
> > > > > 
> > > > > I created a reproducer project of the code that I am using to
> > > > > access
> > > > > HTTP
> > > > > client:
> > > > > https://github.com/nddipiazza/reproduce-400-errors
> > > > > 
> > > > > See the Reproduce400Errors class:
> > > > > 
> > > > 
> > > > 
> > 
> > 
https://github.com/nddipiazza/reproduce-400-errors/blob/master/src/main/java/Reproduce400Errors.java
> > > > > 
> > > > > I use this class to download approximately 9 million
> > > > > documents
> > > > > from
> > > > > SharePoint. And for some reason, after about 4 million docs
> > > > > are
> > > > > downloaded,
> > > > > it gets stuck in a state where all requests fail with a 400
> > > > > status
> > > > > code.
> > > > > 
> > > > > The code is pretty simple. Pool of HTTP connections. Nothing
> > > > > crazy.
> > > > > 
> > > > > If I stop and start the program, it will continue just fine.
> > > > > 
> > > > > Has anyone ever heard of a situation like this? How can I fix
> > > > > this
> > > > > issue?
> > > > 
> > > > I strongly suspect this to be a server side issue given the
> > > > symptoms of
> > > > the problem.
> > > > 
> > > > Oleg
> > > > 
> > > > 
> > > > -
> > > > 
> > > > 
> > > > To unsubscribe, e-mail: 
> > > > httpclient-users-unsubscr...@hc.apache.org
> > > > For additional commands, e-mail:
> > > > httpclient-users-h...@hc.apache.org
> > > > 
> > > > 
> > 
> > 
> > -
> > 
> > To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
> > For additional commands, e-mail: 
> > httpclient-users-h...@hc.apache.org
> > 
> > 


-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

Re: After 100,000's of requests, I start getting status code 400 bad request for all requests

[Oleg Kalnichevski]

On Tue, 2020-05-19 at 08:56 -0500, Nicholas DiPiazza wrote:
> Yeah I agree Oleg. But I'm at a loss as to why I am able to stop &
> start
> the program and then it starts working again fine.
> 
> There is NTLM authentication involved. Perhaps the auth gets into a
> stale
> state and starts causing 400 errors?
> 

Hmm. Then, client side NTLM state corruption is more likely. The
trouble is there is no longer a single person on the project who has
both capacity and inclination to work on NTLM. 

Does the server support any other authentication schemes?

Oleg



> On Tue, May 19, 2020 at 8:06 AM Oleg Kalnichevski 
> wrote:
> 
> > On Mon, 2020-05-18 at 20:24 -0500, Nicholas DiPiazza wrote:
> > > Hey HTTP client users list!
> > > 
> > > I've got some strange behavior going on and could use some help
> > > sorting it
> > > out.
> > > 
> > > I created a reproducer project of the code that I am using to
> > > access
> > > HTTP
> > > client:
> > > https://github.com/nddipiazza/reproduce-400-errors
> > > 
> > > See the Reproduce400Errors class:
> > > 
> > 
> > 
https://github.com/nddipiazza/reproduce-400-errors/blob/master/src/main/java/Reproduce400Errors.java
> > > 
> > > I use this class to download approximately 9 million documents
> > > from
> > > SharePoint. And for some reason, after about 4 million docs are
> > > downloaded,
> > > it gets stuck in a state where all requests fail with a 400
> > > status
> > > code.
> > > 
> > > The code is pretty simple. Pool of HTTP connections. Nothing
> > > crazy.
> > > 
> > > If I stop and start the program, it will continue just fine.
> > > 
> > > Has anyone ever heard of a situation like this? How can I fix
> > > this
> > > issue?
> > 
> > I strongly suspect this to be a server side issue given the
> > symptoms of
> > the problem.
> > 
> > Oleg
> > 
> > 
> > -
> > 
> > To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
> > For additional commands, e-mail: 
> > httpclient-users-h...@hc.apache.org
> > 
> > 


-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

Re: After 100,000's of requests, I start getting status code 400 bad request for all requests

[Oleg Kalnichevski]

On Mon, 2020-05-18 at 20:24 -0500, Nicholas DiPiazza wrote:
> Hey HTTP client users list!
> 
> I've got some strange behavior going on and could use some help
> sorting it
> out.
> 
> I created a reproducer project of the code that I am using to access
> HTTP
> client:
> https://github.com/nddipiazza/reproduce-400-errors
> 
> See the Reproduce400Errors class:
> 
https://github.com/nddipiazza/reproduce-400-errors/blob/master/src/main/java/Reproduce400Errors.java
> 
> I use this class to download approximately 9 million documents from
> SharePoint. And for some reason, after about 4 million docs are
> downloaded,
> it gets stuck in a state where all requests fail with a 400 status
> code.
> 
> The code is pretty simple. Pool of HTTP connections. Nothing crazy.
> 
> If I stop and start the program, it will continue just fine.
> 
> Has anyone ever heard of a situation like this? How can I fix this
> issue?

I strongly suspect this to be a server side issue given the symptoms of
the problem.

Oleg


-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

Re: HTTP/2 connection gets closed and won't be reopened

[Oleg Kalnichevski]

On Thu, 2020-05-14 at 13:59 +0200, u...@natune.net.INVALID wrote:
> > I need to see the original exception with a full stack trace that
> > caused the i/o reactor to shut down and preferably the complete
> > session
> > log leading to that condition.
> > 
> > Oleg
> 
> I was unable to get HttpClient5 logging to work. 

I am not sure what I can do without a log or a reproducer I could run
locally in my development environment.

Oleg


> I used this config in 
> /WEB-INF/log4j2.xml. The log files get created, but they contain 0
> output.
> 
> 
>
>  
>
>  
>  
>
>  
>
>
>  
>
>  
>  
>
>  
>  
>
>  
>
> 
> 
> I am able though to reproduce the error by shutting down the tomcat9 
> server that this httpclient is connecting to. After starting the
> server 
> up again, things work normal for a couple of minutes. Then I get
> this 
> exception (the cause is the strange part, )
> 
> ...: Unable to connect to API Gateway
>  at com.rst.ResponseHolder.onCompleted(ResponseHolder.java:15)
>  at 
> org.asynchttpclient.AsyncCompletionHandler.onCompleted(AsyncCompletio
> nHandler.java:66)
>  at 
> com.gold.ThrottleRequestFilter$AsyncHandlerWrapper.onCompleted(Thrott
> leRequestFilter.java:75)
>  at 
> org.asynchttpclient.netty.NettyResponseFuture.loadContent(NettyRespon
> seFuture.java:222)
>  at 
> org.asynchttpclient.netty.NettyResponseFuture.done(NettyResponseFutur
> e.java:257)
>  at 
> org.asynchttpclient.netty.handler.AsyncHttpClientHandler.finishUpdate
> (AsyncHttpClientHandler.java:241)
>  at 
> org.asynchttpclient.netty.handler.HttpHandler.handleChunk(HttpHandler
> .java:114)
>  at 
> org.asynchttpclient.netty.handler.HttpHandler.handleRead(HttpHandler.
> java:143)
>  at 
> org.asynchttpclient.netty.handler.AsyncHttpClientHandler.channelRead(
> AsyncHttpClientHandler.java:78)
>  at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(Abst
> ractChannelHandlerContext.java:379)
>  at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(Abst
> ractChannelHandlerContext.java:365)
>  at 
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(Abstra
> ctChannelHandlerContext.java:357)
>  at 
> io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToM
> essageDecoder.java:102)
>  at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(Abst
> ractChannelHandlerContext.java:379)
>  at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(Abst
> ractChannelHandlerContext.java:365)
>  at 
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(Abstra
> ctChannelHandlerContext.java:357)
>  at 
> io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandle
> rContext.fireChannelRead(CombinedChannelDuplexHandler.java:436)
>  at 
> io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMes
> sageDecoder.java:321)
>  at 
> io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessage
> Decoder.java:295)
>  at 
> io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedCha
> nnelDuplexHandler.java:251)
>  at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(Abst
> ractChannelHandlerContext.java:379)
>  at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(Abst
> ractChannelHandlerContext.java:365)
>  at 
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(Abstra
> ctChannelHandlerContext.java:357)
>  at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1486)
>  at 
> io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1
> 235)
>  at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1282)
>  at 
> io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProte
> ction(ByteToMessageDecoder.java:498)
>  at 
> io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageD
> ecoder.java:437)
>  at 
> io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessage
> Decoder.java:276)
>  at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(Abst
> ractChannelHandlerContext.java:379)
>  at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(Abst
> ractChannelHandlerContext.java:365)
>  at 
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(Abstra
> ctChannelHandlerContext.java:357)
>  at 
> io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(Defau
> ltChannelPipeline.java:1410)
>  at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(Abst
> ractChannelHandlerContext.java:379)
>  at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(Abst
> ractChannelHandlerContext.java:365)
>  at 
> io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChanne
> lPipeline.java:919)
>  at 
> 

Re: HTTP/2 connection gets closed and won't be reopened

[Oleg Kalnichevski]

On Thu, 2020-05-14 at 10:53 +0200, u...@natune.net.INVALID wrote:
> > > While using HTTP/2 only client, after approx. 12 hours the
> > > connection
> > > is
> > > broken and is not established anymore. Every couple of minutes
> > > async
> > > HTTP/2 requests are sent to a Tomcat9 server. There are no errors
> > > whatsoever but after a couple of hours every request fails until
> > > the
> > > application running HttpClient 5.0 is restarted.
> > > 
> > > Would an evictIdleConnections() call help in this case?
> > > Let's suppose Tomcat would close the connection after some time,
> > > what
> > > is
> > > Httclient 5.0's default reaction? Is it re-established
> > > transparently?
> > > 
> > 
> > Yes, it should, I am afraid I would either need a reproducer or a
> > full
> > execution log of the session that exhibits the problem.
> 
> 
> I haven't had the chance yet to enable full logging yet, but after
> the 
> first "Connection closed" Exception, and a couple of requests going 
> through, the cause for failed connection establishment is an "I/O 
> reactor has been shut down" Exception. That's what I could find out
> up 
> to now.
> 

I need to see the original exception with a full stack trace that
caused the i/o reactor to shut down and preferably the complete session
log leading to that condition.

Oleg


> 
> br,
> 
> Can
> 
> 
> -
> To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
> For additional commands, e-mail: httpclient-users-h...@hc.apache.org
> 


-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

Re: HTTP/2 connection gets closed and won't be reopened

[Oleg Kalnichevski]

On Wed, 2020-05-13 at 19:23 +0200, u...@natune.net.INVALID wrote:
> Hello,
> 
> While using HTTP/2 only client, after approx. 12 hours the connection
> is 
> broken and is not established anymore. Every couple of minutes async 
> HTTP/2 requests are sent to a Tomcat9 server. There are no errors 
> whatsoever but after a couple of hours every request fails until the 
> application running HttpClient 5.0 is restarted.
> 
> Would an evictIdleConnections() call help in this case?
> Let's suppose Tomcat would close the connection after some time, what
> is 
> Httclient 5.0's default reaction? Is it re-established transparently?
> 

Yes, it should, I am afraid I would either need a reproducer or a full
execution log of the session that exhibits the problem.

Oleg


> 
> br,
> 
> Can
> 
> 
> Client: Apache Httpclient 5.0
> Used Code for the client:
> 
> final IOReactorConfig ioReactorConfig = IOReactorConfig.custom()
>  .setSoTimeout(Timeout.ofSeconds(3))
>  .build();
> 
> httpTx = HttpAsyncClients.customHttp2()
>  .setIOReactorConfig(ioReactorConfig)
> .setDefaultRequestConfig(RequestConfig.copy(RequestConfig.DEFAULT)
>  .setMaxRedirects(10)
>  .setConnectTimeout(Timeout.ofSeconds(1))
> .setConnectionRequestTimeout(Timeout.ofMilliseconds(500)) //when 
> requesting from conn manager
>  .setConnectionKeepAlive(TimeValue.ofMinutes(10))
>  .build())
>  .build();
> 
> 
> -
> To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
> For additional commands, e-mail: httpclient-users-h...@hc.apache.org
> 


-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

Re: How to HttpRequest. getIPAddress() ?

[Oleg Kalnichevski]

On Sat, 2020-05-09 at 19:48 -0400, C c wrote:
> Hi, I'm sure this has been asked before, but the mail search found no
> results.
> I'm using latest version 5.
> How do I get the IP address other than the headers?

IP address is not a property of the request. It is a property of the
endpoint used to execute the request. Endpoint details can be obtained
from the execution context:

http://hc.apache.org/httpcomponents-core-5.0.x/httpcore5/apidocs/org/apache/hc/core5/http/EndpointDetails.html
http://hc.apache.org/httpcomponents-core-5.0.x/httpcore5/apidocs/org/apache/hc/core5/http/protocol/HttpCoreContext.html#getEndpointDetails()

Oleg


-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

Re: Testing support for HttpClient.

[Oleg Kalnichevski]

On Tue, 2020-05-05 at 21:51 +0200, Paweł Adamski wrote:
> Hi Oleg
> 
> Thanks for your response.
> I'm definitely interested in long term cooperation. I always wanted
> to join
> some open source project and I hope now I will have a chance for
> that.
> 
> Regarding the current testing support that you mentioned, I wanted to
> play
> with it but I couldn't find any manual for it. Do you know if
> anything is
> available?
> 

Hi Paweł

The idea of showing that bit of code was to give you an example how
things can end up neglected even being a part of a large and active
project. You do not have to look at it if you do not feel like it.
Obviously the code never got documented or updated since its initial
contribution.

Please also consider downside of your project no longer being
independent. It will have to share the same release cycle with
HttpClient and conform to the project guidelines and requirements. You
will no longer be able to release it whenever you like. You will not be
able to break APIs whenever you feel like. You will have to co-exist
with some unpleasant people like myself and tolerate their opinion.   

What might be a reasonable thing to do first is to subscribe to the dev
list, start following the discussions on that list, review pull
requests, participate in release votes and help resolve issues.

If you think you are OK with all that you are welcome to contribute
your entire project as a pull request at Github. You will have to port
it to 5.0 APIs first though. There will be no new features in the 4.5.x
code line.

Oleg  


> Regards
> Paweł Adamski
> 
> 
> niedz., 3 maj 2020 o 17:42 Oleg Kalnichevski 
> napisał(a):
> 
> > On Sun, 2020-05-03 at 16:39 +0200, Paweł Adamski wrote:
> > > Hi
> > > My name is Paweł. I'm an author of the library for easy mocking
> > > HttpClient
> > > ( https://github.com/PawelAdamski/HttpClientMock ). I would like
> > > to
> > > ask
> > > about your opinion wherever it would be a good idea to include it
> > > as
> > > an
> > > official library for testing code using HttpClient?
> > > I see that a lot of other libraries and frameworks already do
> > > that.
> > > For
> > > example, every Spring project has an additional component that
> > > helps
> > > to
> > > write test code (e.g. @WebMvcTest, @DataJpaTest,
> > > @SpringBootTest).
> > > 
> > > My project has an API similar to the Mockito. First, you have to
> > > set
> > > up how
> > > the HttpClient should behave, and then you can replay that
> > > behavior.
> > > There
> > > is also a possibility to verify if calls have been really made.
> > > 
> > > If you think it is worth making my project an official mocking
> > > library for
> > > HttpClient, I would appreciate all information about the next
> > > steps I
> > > should take to make it happen.
> > > 
> > > I'm looking forward to hearing from you.
> > > Regards
> > > Paweł Adamski
> > > 
> > 
> > Hi Paweł
> > 
> > There is already some sort of testing support module in HttpCore
> > and
> > HttpClient which I imagine could host mocking and assert support
> > classes as well.
> > 
> > There is a problem however exemplified by the same testing module.
> > It
> > contains a testing framework contributed some while ago by an
> > external
> > contributor [1]. The original contributor lost any further interest
> > in
> > that code pretty much immediately after it had been accepted into
> > the
> > project and walked away.
> > 
> > Now I have no idea what we ought to do about that testing code. I
> > have
> > no idea if there are any external projects using it. It appears to
> > be
> > dead weight for the project.
> > 
> > My first question is quite simple. How likely you are going to
> > stick
> > around after having your library folded into HttpClient?
> > 
> > Oleg
> > 
> > [1]
> > 
> > 
> > 
https://github.com/apache/httpcomponents-core/tree/master/httpcore5-testing/src/main/java/org/apache/hc/core5/testing/framework
> > 
> > 
> > 
> > 
> > -
> > 
> > To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
> > For additional commands, e-mail: 
> > httpclient-users-h...@hc.apache.org
> > 
> > 


-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

Re: Testing support for HttpClient.

[Oleg Kalnichevski]

On Sun, 2020-05-03 at 16:39 +0200, Paweł Adamski wrote:
> Hi
> My name is Paweł. I'm an author of the library for easy mocking
> HttpClient
> ( https://github.com/PawelAdamski/HttpClientMock ). I would like to
> ask
> about your opinion wherever it would be a good idea to include it as
> an
> official library for testing code using HttpClient?
> I see that a lot of other libraries and frameworks already do that.
> For
> example, every Spring project has an additional component that helps
> to
> write test code (e.g. @WebMvcTest, @DataJpaTest, @SpringBootTest).
> 
> My project has an API similar to the Mockito. First, you have to set
> up how
> the HttpClient should behave, and then you can replay that behavior.
> There
> is also a possibility to verify if calls have been really made.
> 
> If you think it is worth making my project an official mocking
> library for
> HttpClient, I would appreciate all information about the next steps I
> should take to make it happen.
> 
> I'm looking forward to hearing from you.
> Regards
> Paweł Adamski
> 

Hi Paweł

There is already some sort of testing support module in HttpCore and
HttpClient which I imagine could host mocking and assert support
classes as well. 

There is a problem however exemplified by the same testing module. It
contains a testing framework contributed some while ago by an external
contributor [1]. The original contributor lost any further interest in
that code pretty much immediately after it had been accepted into the
project and walked away. 

Now I have no idea what we ought to do about that testing code. I have
no idea if there are any external projects using it. It appears to be
dead weight for the project. 

My first question is quite simple. How likely you are going to stick
around after having your library folded into HttpClient?

Oleg

[1]

https://github.com/apache/httpcomponents-core/tree/master/httpcore5-testing/src/main/java/org/apache/hc/core5/testing/framework
 



-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

[ANNOUNCEMENT] HttpComponents Client 4.5.12 GA Released

[Oleg Kalnichevski]

The Apache HttpComponents project is pleased to announce 4.5.12 GA
release of HttpComponents HttpClient.

This is a maintenance release that fixes a regression introduced by the
previous release that caused rejection of certificates with non-
standard domains.

Download - 
Release notes - <


https://www.apache.org/dist/httpcomponents/httpclient/RELEASE_NOTES-4.5.x.txt
> 

HttpComponents site - 

About HttpComponents HttpClient

The Hyper-Text Transfer Protocol (HTTP) is perhaps the most significant
protocol used on the Internet today. Web services, network-enabled
appliances and the growth of network computing continue to expand the
role of the HTTP protocol beyond user-driven web browsers, while
increasing the number of applications that require HTTP support.

Designed for extension while providing robust support for the base HTTP
protocol, HttpClient may be of interest to anyone building HTTP-aware
client applications such as web browsers, web service clients, or
systems that leverage or extend the HTTP protocol for distributed
communication.



-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

Re: Control behavior of Caching based on response

[Oleg Kalnichevski]

On Thu, 2020-02-27 at 14:01 -0800, Novice User wrote:
> Hi,
> 
> We’re using HTTPClient 4.5.11 with caching enabled to make HTTP GET
> calls to our provider.
> 
> The provider returns JSON object and as JSON can miss some keys in
> some cases, we want to treat it as bad data and avoid it being
> cached.
> 
> There does not seem to be any hook for us to intercept the response
> body and inspect it and prevent it from being cached.
> 
> It seems HTTP Client has HttpCacheInvalidator but it would not work
> on GET requests, and also I would like something to avoid storing it
> in cache in first place if response is bad, rather than doing a post
> processing of invalidation.
> 
> Does HTTP client has any ability for us to intercept the response
> body before it could be saved in cache? 
> 
> Thanks,
> Akash

You could subclass CachingHttpClientBuilder, override #decorateMainExec
method and insert an extra interceptor between MainExec and
CachingExec.

Oleg 


-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

Re: use a custom AbstractConnPool

[Oleg Kalnichevski]

On Tue, 2020-02-25 at 17:24 -0800, Clément Guillaume wrote:
> Hello,
> 
> Is it possible to pass a custom AbstractConnPool to
> PoolingHttpClientConnectionManager?
> I would like to use something like the default CPool with some
> custom onLease, onRelease and onReuse.
> 
> Thank you,
> 
> Clement Guillaume


No, it is not. HttpClient 5.0 however provides ConnPoolListener with
#onLease and #onRelease event methods that might do what you want.

Oleg


-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

[ANNOUNCEMENT] HttpComponents Client 5.0 GA released

[Oleg Kalnichevski]

The Apache HttpComponents project is pleased to announce 5.0 GA
release of HttpComponents Client. 

This is the first stable (GA) release of HttpClient 5.0.

Notable changes and features included in the 5.0 series are:

* Support for the HTTP/2 protocol and conformance to requirements and
recommendations of the latest HTTP/2 protocol specification documents
(RFC 7540, RFC 7541.)

  Supported features:

** HPACK header compression
** Stream multiplexing (client and server)
** Flow control
** Response push
** Message trailers
** Expect-continue handshake
** Connection validation (ping)
** Application-layer protocol negotiation (ALPN)
** TLS 1.2 security features

* Improved conformance to requirements and recommendations of the
latest HTTP/1.1 protocol specification documents (RFC 7230, RFC 7231)

* New connection pool implementation with lax connection limit
guarantees and better performance under higher concurrency due to
absence of a global pool lock.

* Support for Reactive Streams API [http://www.reactive-streams.org/]

* Package name space changed to 'org.apache.hc.client5'.

* Maven group id changed to 'org.apache.httpcomponents.client5'.

HttpClient 5.0 releases can be co-located with earlier major versions
on the same classpath due to the change in package names and Maven
module coordinates.

Download - 
Release notes - <
https://www.apache.org/dist/httpcomponents/httpclient/RELEASE_NOTES-5.0.x.txt
>

HttpComponents site - 

About HttpComponents HttpClient

The Hyper-Text Transfer Protocol (HTTP) is perhaps the most significant
protocol used on the Internet today. Web services, network-enabled
appliances and the growth of network computing continue to expand the
role of the HTTP protocol beyond user-driven web browsers, while
increasing the number of applications that require HTTP support.

Although the java.net package provides basic functionality for
accessing resources via HTTP, it doesn't provide the full flexibility
or functionality needed by many applications. HttpClient seeks to fill
this void by providing an efficient, up-to-date, and feature-rich
package implementing the client side of the most recent HTTP standards
and recommendations.

Designed for extension while providing robust support for the base HTTP
protocol, HttpClient may be of interest to anyone building HTTP-aware
client applications such as web browsers, web service clients, or
systems that leverage or extend the HTTP protocol for distributed
communication.



-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

Re: Possible bug in RedirectExec in HttpClient 5.0-beta7

[Oleg Kalnichevski]

On Thu, 2020-02-20 at 20:36 +0100, Michael Osipov wrote:
> Am 2020-02-20 um 10:50 schrieb Oleg Kalnichevski:
> > On Wed, 2020-02-19 at 21:19 +0100, Michael Osipov wrote:
> > > Am 2020-02-18 um 22:29 schrieb Oleg Kalnichevski:
> > > > On Tue, 2020-02-18 at 20:35 +0100, Michael Osipov wrote:
> > > > > Am 2020-02-18 um 10:07 schrieb Oleg Kalnichevski:
> > > > > > On Mon, 2020-02-17 at 21:28 +0100, Michael Osipov wrote:
> > > > > > > Folks,
> > > > > > > 
> > > > > > > I have continued to fiddle a bit more with the redirector
> > > > > > > in
> > > > > > > HttpClient
> > > > > > > 5.0. I have modified my servlet code to respond with 307
> > > > > > > from
> > > > > > > within
> > > > > > > a
> > > > > > > Tomcat valve which immediately kicks in after "Expect:
> > > > > > > 100-
> > > > > > > continue".
> > > > > > > 
> > > > > > > There seems to be a bug in RedirectExec:
> > > > > > > 
> > > > > > 
> > > > > > ...
> > > > > > 
> > > > > > > The client is sending the body "I am a checksum" also 100
> > > > > > > has
> > > > > > > not
> > > > > > > been
> > > > > > > issued, but 307. I have expected the client to follow the
> > > > > > > redirect
> > > > > > > and
> > > > > > > retry. The request fails at the end because the HTTP
> > > > > > > entity
> > > > > > > has
> > > > > > > already
> > > > > > > been consumed.
> > > > > > > 
> > > > > > > Ideas?
> > > > > > > 
> > > > > > > Michael
> > > > > > > 
> > > > > > 
> > > > > > Hi Micheal
> > > > > > 
> > > > > > I presume you no longer remember this discussion:
> > > > > > 
> > > > > > https://issues.apache.org/jira/browse/HTTPCORE-411
> > > > > > 
> > > > > > HttpClient 5.0 behaves differently compared to 4.x with
> > > > > > regards
> > > > > > to
> > > > > > the
> > > > > > expect-continue handshake. What you are seeing is what is
> > > > > > believed
> > > > > > to
> > > > > > be RFC 7230 conformant behavior (not that I am very happy
> > > > > > about
> > > > > > it).
> > > > > > 
> > > > > > What we could do, though, in this particular case is to
> > > > > > disregard
> > > > > > the
> > > > > > spec requirement given that the origin server has signaled
> > > > > > its
> > > > > > intention to close the connection after the redirect by
> > > > > > sending
> > > > > > `Connection: close` response header.
> > > > > 
> > > > > Nuts, I re-read the issue, but it was about chunked encoding.
> > > > > I
> > > > > did
> > > > > not
> > > > > use chunked TE. I have provided a content-length. Morever,
> > > > > the
> > > > > subject
> > > > > of that issue refers to RFC 2616.
> > > > > 
> > > > > I don't understand why the body has to be send before the
> > > > > first
> > > > > status
> > > > > code is received. rfc7231#section-5.1.1 clearly says:
> > > > > 
> > > > > > 
> > > > > >  o  A client that sends a 100-continue expectation is
> > > > > > not
> > > > > > required to
> > > > > > wait for any specific length of time; such a client
> > > > > > MAY
> > > > > > proceed to
> > > > > > send the message body even if it has not yet
> > > > > > received a
> > > > > > response.
> > > > > > Furthermore, since 100 (Continue) responses cannot
> > > > > > be
> > > > > > sent
> > > > > > through
> > > > &

Re: Fail non-repeatable requests when following redirects

[Oleg Kalnichevski]

On Thu, 2020-02-20 at 19:46 +0100, Michael Osipov wrote:
> Am 2020-02-19 um 13:42 schrieb Oleg Kalnichevski:
> > ...
> > 
> > > > Hi Michael
> > > > 
> > > > I can take a look. Is there a test case I could run locally to
> > > > reproduce the defect?
> > > 
> > > I will try to skim this out of Wagon. Meanwhile I have attached
> > > the
> > > failure with Wagon. You'll see that HttpClient follows the
> > > redirect,
> > > but
> > > is stuck (checksum files which from from input streams) and
> > > Tomcat
> > > fails
> > > with a timeout.
> > > 
> > > Michael
> > 
> > Please review
> > 
> > https://github.com/apache/httpcomponents-client/pull/213
> 
> I have a left a comment on the PR. It causes, unfortunately, a 
> regression :-(
> 

I am aware. I will look into it. But I need to improve our test
coverage before proceeding.

Oleg 



-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

Re: Possible bug in RedirectExec in HttpClient 5.0-beta7

[Oleg Kalnichevski]

On Wed, 2020-02-19 at 21:19 +0100, Michael Osipov wrote:
> Am 2020-02-18 um 22:29 schrieb Oleg Kalnichevski:
> > On Tue, 2020-02-18 at 20:35 +0100, Michael Osipov wrote:
> > > Am 2020-02-18 um 10:07 schrieb Oleg Kalnichevski:
> > > > On Mon, 2020-02-17 at 21:28 +0100, Michael Osipov wrote:
> > > > > Folks,
> > > > > 
> > > > > I have continued to fiddle a bit more with the redirector in
> > > > > HttpClient
> > > > > 5.0. I have modified my servlet code to respond with 307 from
> > > > > within
> > > > > a
> > > > > Tomcat valve which immediately kicks in after "Expect: 100-
> > > > > continue".
> > > > > 
> > > > > There seems to be a bug in RedirectExec:
> > > > > 
> > > > 
> > > > ...
> > > > 
> > > > > The client is sending the body "I am a checksum" also 100 has
> > > > > not
> > > > > been
> > > > > issued, but 307. I have expected the client to follow the
> > > > > redirect
> > > > > and
> > > > > retry. The request fails at the end because the HTTP entity
> > > > > has
> > > > > already
> > > > > been consumed.
> > > > > 
> > > > > Ideas?
> > > > > 
> > > > > Michael
> > > > > 
> > > > 
> > > > Hi Micheal
> > > > 
> > > > I presume you no longer remember this discussion:
> > > > 
> > > > https://issues.apache.org/jira/browse/HTTPCORE-411
> > > > 
> > > > HttpClient 5.0 behaves differently compared to 4.x with regards
> > > > to
> > > > the
> > > > expect-continue handshake. What you are seeing is what is
> > > > believed
> > > > to
> > > > be RFC 7230 conformant behavior (not that I am very happy about
> > > > it).
> > > > 
> > > > What we could do, though, in this particular case is to
> > > > disregard
> > > > the
> > > > spec requirement given that the origin server has signaled its
> > > > intention to close the connection after the redirect by sending
> > > > `Connection: close` response header.
> > > 
> > > Nuts, I re-read the issue, but it was about chunked encoding. I
> > > did
> > > not
> > > use chunked TE. I have provided a content-length. Morever, the
> > > subject
> > > of that issue refers to RFC 2616.
> > > 
> > > I don't understand why the body has to be send before the first
> > > status
> > > code is received. rfc7231#section-5.1.1 clearly says:
> > > 
> > > > 
> > > > o  A client that sends a 100-continue expectation is not
> > > > required to
> > > >wait for any specific length of time; such a client MAY
> > > > proceed to
> > > >send the message body even if it has not yet received a
> > > > response.
> > > >Furthermore, since 100 (Continue) responses cannot be
> > > > sent
> > > > through
> > > >an HTTP/1.0 intermediary, such a client SHOULD NOT wait
> > > > for
> > > > an
> > > >indefinite period before sending the message body.
> > > 
> > > 
> > > It says MAY. So we don't have to send the request body before the
> > > 100
> > > status.
> > > 
> > > Can you point to the spot of the spec which caused the change? I
> > > don't
> > > get it.
> > > I believe that we still can send the body AFTER 100 and are still
> > > compliant. Otherwise expect continue is pointless from my POV and
> > > has
> > > the very same effect as not using it.
> > > 
> > > Michael
> > > 
> > 
> > Michael
> > 
> > 
> > The client MUST send the request body matching the request head
> > with
> > `Content-Length` header in response to 307 status in order to
> > comply
> > with the spec. After that the entity is spent and cannot be
> > repeated.
> 
> Please call me stupid, but I would like to read the *MUST* in the
> RFC 
> for that. I still cannot wrap my head around that making this
> feature 
> mostly useless.
> 

I have been saying all along in the course of HTTPCORE-411 argument.
There is no clear statement in RFC 7230 with regards to handling of
request entity during the expect-continue handshake. There is however a
clarification from Roy T Fielding quoted in the ticket.

Based on the current interpretation of RFC 7230 the expect-continue
handshake is pretty much useless for `Content-Length` delimited
requests. 

> Can we pick up your idea:
>  >>> What we could do, though, in this particular case is to
> disregard
>  >>> the
>  >>> spec requirement given that the origin server has signaled its
>  >>> intention to close the connection after the redirect by sending
>  >>> `Connection: close` response header.
> 
> and react on the close *before* sending the body even if we have 
> provided a content length?
> 

We can look into that in 5.1. 

Oleg


-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

Re: Fail non-repeatable requests when following redirects

[Oleg Kalnichevski]

...

> > Hi Michael
> > 
> > I can take a look. Is there a test case I could run locally to
> > reproduce the defect?
> 
> I will try to skim this out of Wagon. Meanwhile I have attached the 
> failure with Wagon. You'll see that HttpClient follows the redirect,
> but 
> is stuck (checksum files which from from input streams) and Tomcat
> fails 
> with a timeout.
> 
> Michael

Please review 

https://github.com/apache/httpcomponents-client/pull/213

Oleg




-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

Re: Fail non-repeatable requests when following redirects

[Oleg Kalnichevski]

On Tue, 2020-02-18 at 19:48 +0100, Michael Osipov wrote:
> > > 

...


> Am 2020-02-18 um 10:50 schrieb Oleg Kalnichevski:
> > On Mon, 2020-02-17 at 15:29 +0100, Michael Osipov wrote:
> Sure, I have uploaded webapp.war. Please deploy it as usual on
> Tomcat 
> and put it as JAR on Tomcat's classpath because the integrated valve 
> must be on Tomcat's classpath.
> 
> The perform a PUT to /webapp/repo/
> 

This line in Test2 gives me grief:

private static final Path MAVEN_CENTRAL = 
Paths.get("C:\\Users\\mosipov\\.m2\\repository");

But I think this is enough to see the problem and debug HttpClient.

Oleg



-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

Re: Possible bug in RedirectExec in HttpClient 5.0-beta7

[Oleg Kalnichevski]

On Tue, 2020-02-18 at 20:35 +0100, Michael Osipov wrote:
> Am 2020-02-18 um 10:07 schrieb Oleg Kalnichevski:
> > On Mon, 2020-02-17 at 21:28 +0100, Michael Osipov wrote:
> > > Folks,
> > > 
> > > I have continued to fiddle a bit more with the redirector in
> > > HttpClient
> > > 5.0. I have modified my servlet code to respond with 307 from
> > > within
> > > a
> > > Tomcat valve which immediately kicks in after "Expect: 100-
> > > continue".
> > > 
> > > There seems to be a bug in RedirectExec:
> > > 
> > 
> > ...
> > 
> > > The client is sending the body "I am a checksum" also 100 has not
> > > been
> > > issued, but 307. I have expected the client to follow the
> > > redirect
> > > and
> > > retry. The request fails at the end because the HTTP entity has
> > > already
> > > been consumed.
> > > 
> > > Ideas?
> > > 
> > > Michael
> > > 
> > 
> > Hi Micheal
> > 
> > I presume you no longer remember this discussion:
> > 
> > https://issues.apache.org/jira/browse/HTTPCORE-411
> > 
> > HttpClient 5.0 behaves differently compared to 4.x with regards to
> > the
> > expect-continue handshake. What you are seeing is what is believed
> > to
> > be RFC 7230 conformant behavior (not that I am very happy about
> > it).
> > 
> > What we could do, though, in this particular case is to disregard
> > the
> > spec requirement given that the origin server has signaled its
> > intention to close the connection after the redirect by sending
> > `Connection: close` response header.
> 
> Nuts, I re-read the issue, but it was about chunked encoding. I did
> not 
> use chunked TE. I have provided a content-length. Morever, the
> subject 
> of that issue refers to RFC 2616.
> 
> I don't understand why the body has to be send before the first
> status 
> code is received. rfc7231#section-5.1.1 clearly says:
> 
> > 
> >o  A client that sends a 100-continue expectation is not
> > required to
> >   wait for any specific length of time; such a client MAY
> > proceed to
> >   send the message body even if it has not yet received a
> > response.
> >   Furthermore, since 100 (Continue) responses cannot be sent
> > through
> >   an HTTP/1.0 intermediary, such a client SHOULD NOT wait for
> > an
> >   indefinite period before sending the message body.
> 
> 
> It says MAY. So we don't have to send the request body before the
> 100 
> status.
> 
> Can you point to the spot of the spec which caused the change? I
> don't 
> get it.
> I believe that we still can send the body AFTER 100 and are still 
> compliant. Otherwise expect continue is pointless from my POV and
> has 
> the very same effect as not using it.
> 
> Michael
> 

Michael


The client MUST send the request body matching the request head with
`Content-Length` header in response to 307 status in order to comply
with the spec. After that the entity is spent and cannot be repeated.

Hope this helps

Oleg 


-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

Re: Fail non-repeatable requests when following redirects

[Oleg Kalnichevski]

On Mon, 2020-02-17 at 15:29 +0100, Michael Osipov wrote:
> Am 2020-02-17 um 14:11 schrieb Oleg Kalnichevski:
> > On Sun, 2020-02-16 at 23:30 +0100, Michael Osipov wrote:
> > > Folks,
> > > 
> > > I am working again on some stuff in Maven Wagon, WAGON-570. After
> > > some
> > > tests I do not understand why this does not fail:
> > > 
> > 
> > ...
> > > The supplied entity is non-repeatable, DefaultRedirectStrategy
> > > has
> > > been
> > > instructed to retry the PUT for this case.
> > > (
> > > 
https://github.com/apache/maven-wagon/commit/58a646776fa4811090063360e80280e21287e92b
> > > )
> > > 
> > > For some strange reason the PUT is retried although the entity is
> > > unusable. The peer (server) fails with request timeout. I would
> > > expect
> > > this to behave like the RetryHandler. If it cannot be retried,
> > > fail.
> > > 
> > > Michael
> > 
> > Hi Michael
> > 
> > I can take a look. Is there a test case I could run locally to
> > reproduce the defect?
> 
> Alright, I have now a reduced test case here: 
> https://home.apache.org/~michaelo/issues/httpclient-redirect/
> 
> The ZIP file contains a sample code RedirectMain.java which
> reproduces 
> Wagon's behavior. Test2.java is a simple servlet running on tomcat 
> 8.5.51 here and accepting GET and PUT requests. out.log contains the 
> output of HttpClient and the failure.
> 
> My expectation would be HttpClient to fail with an expecption that
> the 
> request cannot be retried because the input stream has already been 
> consumed.
> 
> Michael

> Michael

Could you also pack your server side code into a WAR, please? I have no
idea what I need to do about web-fragment.xml

Oleg  




-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

Re: Possible bug in RedirectExec in HttpClient 5.0-beta7

[Oleg Kalnichevski]

On Mon, 2020-02-17 at 21:28 +0100, Michael Osipov wrote:
> Folks,
> 
> I have continued to fiddle a bit more with the redirector in
> HttpClient 
> 5.0. I have modified my servlet code to respond with 307 from within
> a 
> Tomcat valve which immediately kicks in after "Expect: 100-continue".
> 
> There seems to be a bug in RedirectExec:
> 

...

> The client is sending the body "I am a checksum" also 100 has not
> been 
> issued, but 307. I have expected the client to follow the redirect
> and 
> retry. The request fails at the end because the HTTP entity has
> already 
> been consumed.
> 
> Ideas?
> 
> Michael
> 

Hi Micheal

I presume you no longer remember this discussion:

https://issues.apache.org/jira/browse/HTTPCORE-411

HttpClient 5.0 behaves differently compared to 4.x with regards to the
expect-continue handshake. What you are seeing is what is believed to
be RFC 7230 conformant behavior (not that I am very happy about it). 

What we could do, though, in this particular case is to disregard the
spec requirement given that the origin server has signaled its
intention to close the connection after the redirect by sending
`Connection: close` response header. 

Oleg



-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

[ANNOUNCEMENT] HttpComponents Core 5.0 GA Released

[Oleg Kalnichevski]

The Apache HttpComponents project is pleased to announce 5.0 GA release
of HttpComponents Core.

This is the first stable (GA) release of HttpCore 5.0.

Notable changes and features included in the 5.0 series:

* Support for HTTP/2 protocol and conformance to requirements and
recommendations of the latest HTTP/2 protocol specification (RFC 7540,
RFC 7541)

  Supported features:

** HPACK header compression
** stream multiplexing (client and server)
** flow control
** response push (client and server)
** message trailers
** expect-continue handshake
** connection validation (ping)
** application-layer protocol negotiation (ALPN) on Java 9+
** TLS 1.2 security features

   Features out of scope for 5.0 release:

** padding of outgoing frames
** stream priority
** plain connection HTTP/1.1 upgrade
** CONNECT method

* Improved conformance to requirements and recommendations of the
latest HTTP/1.1 protocol specification (RFC 7230, RFC 7231)

* New asynchronous HTTP transport APIs consistent for both HTTP/1.1 and
HTTP/2 transport.

* Redesigned I/O reactor APIs and improved NIO based reactor
implementation for a greater performance and scalability.

* Support for server-side request filters for classic and asynchronous
server implementations. Request filters could be used to implement
cross-cutting protocol aspects such as the 'expect-continue'
handshaking and user authentication / authorization.

* Support for Reactive Streams API [http://www.reactive-streams.org/]

* Redesigned connection pool implementation with strict connection
limit guarantees. The connection pool is expected to have a better
performance under higher concurrency due to reduced global pool lock
contention.

* New connection pool implementation with lax connection limit
guarantees and better performance under higher concurrency due to
absence of a global pool lock.

* Package name space changed to 'org.apache.hc.core5'

* Maven group id changed to 'org.apache.httpcomponents.core5'


Download -

Release notes -

HttpComponents site -


About HttpComponents Core

HttpCore is a set of low level HTTP transport components that can be
used to build custom client and server side HTTP services with a
minimal footprint. HttpCore supports two I/O models: a blocking I/O
model based on the classic Java I/O and a non-blocking, event driven
I/O model based on Java NIO.



-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

Re: Fail non-repeatable requests when following redirects

[Oleg Kalnichevski]

On Sun, 2020-02-16 at 23:30 +0100, Michael Osipov wrote:
> Folks,
> 
> I am working again on some stuff in Maven Wagon, WAGON-570. After
> some 
> tests I do not understand why this does not fail:
> 
> > main] DEBUG org.apache.http.headers - http-outgoing-0 >> PUT /test-
> > secured-put-resource HTTP/1.1
> > [main] DEBUG org.apache.http.headers - http-outgoing-0 >> Cache-
> > control: no-cache
> > [main] DEBUG org.apache.http.headers - http-outgoing-0 >> Cache-
> > store: no-store
> > [main] DEBUG org.apache.http.headers - http-outgoing-0 >> Pragma:
> > no-cache
> > [main] DEBUG org.apache.http.headers - http-outgoing-0 >> Content-
> > Length: 14
> > [main] DEBUG org.apache.http.headers - http-outgoing-0 >> Host:
> > localhost:49248
> > [main] DEBUG org.apache.http.headers - http-outgoing-0 >>
> > Connection: Keep-Alive
> > [main] DEBUG org.apache.http.headers - http-outgoing-0 >> User-
> > Agent: Apache-HttpClient/4.5.11 (Java/1.7.0_252)
> > [main] DEBUG org.apache.http.headers - http-outgoing-0 >> Accept-
> > Encoding: gzip,deflate
> > #[main] DEBUG org.apache.http.headers - http-outgoing-0 << HTTP/1.1
> > 303 See Other
> > [main] DEBUG org.apache.http.headers - http-outgoing-0 << Date:
> > Sun, 16 Feb 2020 22:26:39 GMT
> > [main] DEBUG org.apache.http.headers - http-outgoing-0 << Location:
> > http://localhost:49243/test-secured-put-resource
> > [main] DEBUG org.apache.http.headers - http-outgoing-0 << Content-
> > Length: 0
> > [main] DEBUG org.apache.http.headers - http-outgoing-0 << Server:
> > Jetty(9.2.24.v20180105)
> > [main] DEBUG org.apache.http.impl.execchain.MainClientExec -
> > Connection can be kept alive indefinitely
> > [main] DEBUG
> > org.apache.http.impl.conn.PoolingHttpClientConnectionManager -
> > Connection [id: 0][route: {}->http://localhost:49248] can be kept
> > alive indefinitely
> > [main] DEBUG
> > org.apache.http.impl.conn.DefaultManagedHttpClientConnection -
> > http-outgoing-0: set socket timeout to 0
> > [main] DEBUG
> > org.apache.http.impl.conn.PoolingHttpClientConnectionManager -
> > Connection released: [id: 0][route: {}->
> > http://localhost:49248][total available: 1; route allocated: 1 of
> > 20; total allocated: 1 of 40]
> > [main] DEBUG org.apache.http.impl.execchain.RedirectExec -
> > Resetting target auth state
> > [main] DEBUG org.apache.http.impl.execchain.RedirectExec -
> > Redirecting to 'http://localhost:49243/test-secured-put-resource'
> > via {}->http://localhost:49243
> > [main] DEBUG org.apache.http.client.protocol.RequestAddCookies -
> > CookieSpec selected: compatibility
> > [main] DEBUG
> > org.apache.http.impl.conn.PoolingHttpClientConnectionManager -
> > Connection request: [route: {}->http://localhost:49243][total
> > available: 1; route allocated: 0 of 20; total allocated: 1 of 40]
> > [main] DEBUG
> > org.apache.http.impl.conn.PoolingHttpClientConnectionManager -
> > Connection leased: [id: 1][route: {}->http://localhost:49243][total
> >  available: 1; route allocated: 1 of 20; total allocated: 2 of 40]
> > [main] DEBUG org.apache.http.impl.execchain.MainClientExec -
> > Opening connection {}->http://localhost:49243
> > [main] DEBUG
> > org.apache.http.impl.conn.DefaultHttpClientConnectionOperator -
> > Connecting to localhost/127.0.0.1:49243
> > [main] DEBUG
> > org.apache.http.impl.conn.DefaultHttpClientConnectionOperator -
> > Connection established 127.0.0.1:49254<->127.0.0.1:49243
> > [main] DEBUG
> > org.apache.http.impl.conn.DefaultManagedHttpClientConnection -
> > http-outgoing-1: set socket timeout to 180
> > [main] DEBUG org.apache.http.impl.execchain.MainClientExec -
> > Executing request PUT /test-secured-put-resource HTTP/1.1
> > [main] DEBUG org.apache.http.impl.execchain.MainClientExec - Target
> > auth state: UNCHALLENGED
> > [main] DEBUG org.apache.http.impl.execchain.MainClientExec - Proxy
> > auth state: UNCHALLENGED
> > [main] DEBUG org.apache.http.headers - http-outgoing-1 >> PUT
> > /test-secured-put-resource HTTP/1.1
> > [main] DEBUG org.apache.http.headers - http-outgoing-1 >> Cache-
> > control: no-cache
> > [main] DEBUG org.apache.http.headers - http-outgoing-1 >> Cache-
> > store: no-store
> > [main] DEBUG org.apache.http.headers - http-outgoing-1 >> Pragma:
> > no-cache
> > [main] DEBUG org.apache.http.headers - http-outgoing-1 >> Content-
> > Length: 14
> > [main] DEBUG org.apache.http.headers - http-outgoing-1 >> Host:
> > localhost:49243
> > [main] DEBUG org.apache.http.headers - http-outgoing-1 >>
> > Connection: Keep-Alive
> > [main] DEBUG org.apache.http.headers - http-outgoing-1 >> User-
> > Agent: Apache-HttpClient/4.5.11 (Java/1.7.0_252)
> > [main] DEBUG org.apache.http.headers - http-outgoing-1 >> Accept-
> > Encoding: gzip,deflate
> > [qtp582804106-15] WARN org.eclipse.jetty.server.HttpChannel -
> > /test-secured-put-resource
> > java.io.IOException: java.util.concurrent.TimeoutException: Idle
> > timeout expired: 30021/3 ms
> > at
> > 

Re: SocketTimeoutExceptions When Communicating with a Load Balancer

[Oleg Kalnichevski]

On Mon, 2020-02-10 at 23:25 -0500, Evan J wrote:
>  Hi,
> 
> (looks like I'd sent this to a wrong user group originally)
> 
> We deploy an application B (which is basically a backend application
> serving a web application) to a cluster of application servers (JBoss
> EAP
> 7.2 -- 8 instances). These instances send HTTP requests to a set of
> gateway
> applications, call them application Gs (10 instances), where the
> requests
> go through an F5 loadbalancer that sits between them.
> 
> Instances of application B are Spring Boot applications (2.1.x) that
> have
> been configured with Apache HttpClient 4.5.5 and HttpCore 4.4.9. The
> configuration is almost identical to this github code:
> 
> 
https://github.com/spring-framework-guru/sfg-blog-posts/tree/master/resttemplate/src/main/java/guru/springframework/resttemplate/config
> 
> The only exception is that RestTemplateConfig#restTemplate() is
> configured
> by creation of RestTemplate and passing
> HttpComponentsClientHttpRequestFactory to its constructor rather than
> using
> Spring Boot's RestTemplateBuilder().
> 
> The keep alive configuration in application B is basically defined
> something like:
> 
> 
https://github.com/spring-framework-guru/sfg-blog-posts/blob/0152fb0c4acf08d019128ca38c3dd2523871c43c/resttemplate/src/main/java/guru/springframework/resttemplate/config/ApacheHttpClientConfig.java#L52
> 
> When a load test is executed in a single stack environment where the
> load
> balancer is omitted (one application B -> one application G), the
> requests
> and responses are processed and validated accordingly (each request
> should
> correspond to a right response and be sent to the web application
> front
> end).
> 
> However, when we run the same load test in a distributed environment
> with a
> load balancer between application B's and application G's, we see a
> lot of
> SocketTimeoutExceptions being logged, and we notice them very quickly
> (about 5% of total of responses in application B throw that
> exception).
> 
> The code structure is very straightforward:
> 
> try {
>   // RestTemplate call
> } catch (RestClientException exception) {
>   Exception rootCause = ExceptionUtils.getRootCause(exception); //
> Apache
> Commons lib
>   if (rootCause != null) {
> if
> (SocketTimeoutException.getClass().getName().equals(rootCause.getClas
> s().getName())
> {
> // or even if (rootCause instanceof SocketTimeoutException) {
>   // Log for socket timeout
> }
> if
> (ConnectTimeoutException.getClass().getName().equals(rootCause.getCla
> ss().getName())
> {
> // or even if (rootCause instanceof  ConnectTimeoutException ) {
>   // Log for connection timeout
> }
>   }
> }
> 
> Application B's keep alive has been set to 20 seconds while the
> socket
> timeout has been set to 10 seconds by default (connection timeout to
> 1
> second).
> 
> After placing timer to log how long it takes for an exception is
> thrown, we
> saw, the time that it took from the moment RestTemplate is invoked
> till the
> exception is thrown was slightly above 1 second, e.g. 1030 ms, 1045
> ms,
> 1020 ms, etc.. This led us to increase the connection timeouts from 1
> second to 2 seconds, and afterward, we didn't get any timeout
> exception of
> any sort under the similar load.
> 
> My question is, why is that the majority of exceptions that are being
> thrown have SocketTimeoutExceptions type as opposed to
> ConnectTimeoutExceptions which, based on the timeout adjustment
> mentioned
> above, appears to be the latter (Connect) vs. socket (read) timeout?
> Note
> that I said the majority of time, as I've seen a few
> ConnectTimeoutExceptions as well, but almost 99% of the failed ones
> are
> SocketTimeoutExceptions.
> 
> Also, in our logs, we log the "rootCause's" class name to avoid
> ambiguity,
> but as I mentioned, they are being logged as SocketTimeoutExceptions
> class
> names.
> 
> What is Apache Components library doing under the hood that signals
> the
> underlying JDK code to throw SocketTimeoutExceptions rather than
> ConnectTimeoutException

Connection management code in HttpClient re-throws
SocketTimeoutExceptions thrown while connecting a socket to its remote
endpoint as ConnectTimeoutException to help distinguish connect timeout
from socket (read operation) timeout. SocketTimeoutExceptions are
always thrown by the JRE, not by HttpClient.

Oleg



-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

Re: Log any remote request made by httpClient 4.5.x

[Oleg Kalnichevski]

On Mon, 2020-01-27 at 16:58 +0100, Tamás Cservenák wrote:
> Oleg,
> 
> wire is "too much" for my use case.
> 
> I basically want an "access log"-like log, but for upstream.
> so method, URL (so, full URL or at scheme + host + port + path) and
> roundtrip time
> (time elapsed when the response line arrived). so to say like "access
> log"
> but on client side.
> 
> This log will most probably be post-processed as well (splunk,
> kibana) for
> later analysis. So,
> the size (contents) and output would need to be configurable by me as
> well.
> 

Disable logging for categories you do not want.

Refer to the `Enable context logging for connection management /
request execution` section as an example

Oleg


> 
> TIA
> T
> 
> On Mon, Jan 27, 2020 at 4:44 PM Oleg Kalnichevski 
> wrote:
> 
> > On Mon, 2020-01-27 at 14:14 +0100, Tamás Cservenák wrote:
> > > Howdy,
> > > 
> > > I'd like to log ANY remote request httpClient 4.5.x does
> > > (stable),
> > > what is
> > > the best way to active that? (like "upstream" request log, so
> > > method
> > > + URL
> > > [schema://hostname/path]).
> > > 
> > > I'd like to cover cases even like httpClient CONNECT requests,
> > > httpClient
> > > retries (if configured to do so) and redirection following (if
> > > configured
> > > to do so).
> > > 
> > > Any existing class or best approach? Any help appreciated.
> > > 
> > 
> > Tamás
> > 
> > Any particular reason for not using HttpClient context and wire
> > logging?
> > 
> > http://hc.apache.org/httpcomponents-client-4.5.x/logging.html
> > 
> > Oleg
> > 
> > 
> > 
> > -
> > 
> > To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
> > For additional commands, e-mail: 
> > httpclient-users-h...@hc.apache.org
> > 
> > 


-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

Re: Log any remote request made by httpClient 4.5.x

[Oleg Kalnichevski]

On Mon, 2020-01-27 at 14:14 +0100, Tamás Cservenák wrote:
> Howdy,
> 
> I'd like to log ANY remote request httpClient 4.5.x does (stable),
> what is
> the best way to active that? (like "upstream" request log, so method
> + URL
> [schema://hostname/path]).
> 
> I'd like to cover cases even like httpClient CONNECT requests,
> httpClient
> retries (if configured to do so) and redirection following (if
> configured
> to do so).
> 
> Any existing class or best approach? Any help appreciated.
> 

Tamás

Any particular reason for not using HttpClient context and wire
logging?

http://hc.apache.org/httpcomponents-client-4.5.x/logging.html

Oleg



-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

[ANNOUNCEMENT] HttpComponents Client 5.0 beta7 (GA candidate) released

[Oleg Kalnichevski]

The Apache HttpComponents project is pleased to announce 5.0-beta7
release of HttpComponents Client. 

This BETA release upgrades HttpCore to the latest version  and
addresses a number of issues found since the previous BETA release.

Notable changes and features in the 5.0 series:
---

* Support for the HTTP/2 protocol and conformance to requirements and
recommendations of the latest HTTP/2 protocol specification documents
(RFC 7540, RFC 7541.)

  Supported features:

** HPACK header compression
** Stream multiplexing (client and server)
** Flow control
** Response push
** Message trailers
** Expect-continue handshake
** Connection validation (ping)
** Application-layer protocol negotiation (ALPN)
** TLS 1.2 security features

* Improved conformance to requirements and recommendations of the
latest HTTP/1.1 protocol specification documents (RFC 7230, RFC 7231.)

* New connection pool implementation with lax connection limit
guarantees and better performance under higher concurrency due to
absence of a global pool lock.

* Package name space changed to 'org.apache.hc.client5'.

* Maven group id changed to 'org.apache.httpcomponents.client5'.

HttpClient 5.0 releases can be co-located with earlier major versions
on the same classpath due to the change in package names and Maven
module coordinates.

Download - 
Release notes - <

https://www.apache.org/dist/httpcomponents/httpclient/RELEASE_NOTES-5.0.x.txt
> 

HttpComponents site - 

About HttpComponents HttpClient

The Hyper-Text Transfer Protocol (HTTP) is perhaps the most significant
protocol used on the Internet today. Web services, network-enabled
appliances and the growth of network computing continue to expand the
role of the HTTP protocol beyond user-driven web browsers, while
increasing the number of applications that require HTTP support.

Although the java.net package provides basic functionality for
accessing resources via HTTP, it doesn't provide the full flexibility
or functionality needed by many applications. HttpClient seeks to fill
this void by providing an efficient, up-to-date, and feature-rich
package implementing the client side of the most recent HTTP standards
and recommendations.

Designed for extension while providing robust support for the base HTTP
protocol, HttpClient may be of interest to anyone building HTTP-aware
client applications such as web browsers, web service clients, or
systems that leverage or extend the HTTP protocol for distributed
communication.



-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

Re: DefaultHostnameVerifier for private domains

[Oleg Kalnichevski]

On Mon, 2020-01-20 at 11:49 +, Daniel wrote:
> Hi folks 
> 
> I believe the recent change to
> https://github.com/apache/httpcomponents-client/pull/198 broke
> validation
> for private domains.
> The domain type is now forced to ICANN. In my scenario this causes a
> valid
> cert check for foo.bar to fail because bar is not part of the
> PublicSuffixMatcher rules forcing the getDomainRoot to return a null
> object
> and thus failing validation in the matchDomainRoot method.

Could you please provide us with the exact domain name, the cert CN and
alternative subject names, or better yet add a test case that passes
with 4.5.10 and fails with 4.5.11?

https://github.com/apache/httpcomponents-client/blob/4.5.x/httpclient/src/test/java/org/apache/http/conn/ssl/TestDefaultHostnameVerifier.java

Oleg


-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

[ANNOUNCEMENT] HttpComponents Client 4.5.11 GA Released

[Oleg Kalnichevski]

The Apache HttpComponents project is pleased to announce 4.5.11 GA
release of HttpComponents HttpClient.

This is a maintenance release that fixes a number defects discovered
since 4.5.10 and upgrades HttpCore dependency to version 4.4.13.

Download - 
Release notes - <

https://www.apache.org/dist/httpcomponents/httpclient/RELEASE_NOTES-4.5.x.txt
> 

HttpComponents site - 

About HttpComponents HttpClient

The Hyper-Text Transfer Protocol (HTTP) is perhaps the most significant
protocol used on the Internet today. Web services, network-enabled
appliances and the growth of network computing continue to expand the
role of the HTTP protocol beyond user-driven web browsers, while
increasing the number of applications that require HTTP support.

Designed for extension while providing robust support for the base HTTP
protocol, HttpClient may be of interest to anyone building HTTP-aware
client applications such as web browsers, web service clients, or
systems that leverage or extend the HTTP protocol for distributed
communication.



-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

[ANNOUNCEMENT] HttpComponents Core 4.4.13 Released

[Oleg Kalnichevski]

The Apache HttpComponents project is pleased to announce 4.4.13 GA
release of HttpComponents Core.

This is a maintenance release that corrects a number of defects
discovered since release 4.4.12.

Please note that as of 4.4 HttpCore requires Java 1.6 or newer.

Download -

Release notes -

HttpComponents site -


About HttpComponents Core

HttpCore is a set of low level HTTP transport components that can be
used to build custom client and server side HTTP services with a
minimal footprint. HttpCore supports two I/O models: a blocking I/O
model based on the classic Java I/O and a non-blocking, event driven
I/O model based on Java NIO.



-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

[ANNOUNCEMENT] HttpComponents Core 5.0 beta11 (GA candidate) released

[Oleg Kalnichevski]

The Apache HttpComponents project is pleased to announce 5.0-beta11 (GA
candidate) release of HttpComponents Core. 

This BETA improves handling of illegal or invalid requests on the
server side and fixes a number of defects in HTTP/2 protocol code found
since the last release.

IMPORTANT: This release is expected to be the last BETA version. If no
major design flaws are found the actual 5.0 API will be frozen and the
next version will be promoted to GA.


Notable changes and features included in the 5.0 series:

* Support for HTTP/2 protocol and conformance to requirements and
recommendations of the latest HTTP/2 protocol specification (RFC 7540,
RFC 7541)

  Supported features:

** HPACK header compression
** stream multiplexing (client and server)
** flow control
** response push (client and server)
** message trailers
** expect-continue handshake
** connection validation (ping)
** application-layer protocol negotiation (ALPN) on Java 9+
** TLS 1.2 security features

   Features out of scope for 5.0 release:

** padding of outgoing frames
** stream priority
** plain connection HTTP/1.1 upgrade
** CONNECT method

* Improved conformance to requirements and recommendations of the
latest HTTP/1.1 protocol specification (RFC 7230, RFC 7231)

* New asynchronous HTTP transport APIs consistent for both HTTP/1.1 and
HTTP/2 transport.

* Redesigned I/O reactor APIs and improved NIO based reactor
implementation for a greater performance and scalability.

* Support for server side request filters for classic and asynchronous
server implementations. Request filters could be used to implement
cross-cutting protocol aspects such as the 'expect-continue'
handshaking and user authentication / authorization.

* Redesigned connection pool implementation with strict connection
limit guarantees. The connection pool is expected to have a better
performance under higher concurrency due to reduced global pool lock
contention.

* New connection pool implementation with lax connection limit
guarantees and better performance under higher concurrency due to
absence of a global pool lock.

* Package name space changed to 'org.apache.hc.core5'

* Maven group id changed to 'org.apache.httpcomponents.core5'

HttpCore 5.0 releases can be co-located with earlier versions.

The 5.0 APIs are considered feature complete and are not expected to
undergo any major changes anymore. The focus of development is now
shifting to API polish, code stabilization and documentation
improvements.

Download - 
Release notes - 
HttpComponents site - 

About HttpComponents Core

HttpCore is a set of HTTP/1.1 and HTTP/2 transport components that can
be used to build custom client and server side HTTP services with a
minimal footprint



-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

Re: Socket reuse question

[Oleg Kalnichevski]

On Wed, 2019-12-11 at 13:38 +0100, Tamás Barta wrote:
> Hi Oleg,
> 
> If you read my example carefully, you can see that I create only one
> HttpClient.
> 

Fair enough. 

Please carefully read the 'Connections in TIME_WAIT State' section of
this document:

https://cwiki.apache.org/confluence/display/HttpComponents/FrequentlyAskedConnectionManagementQuestions

Oleg

> Regards, Tamás
> 
> On Wed, Dec 11, 2019 at 1:31 PM Oleg Kalnichevski 
> wrote:
> 
> > On Wed, 2019-12-11 at 10:50 +0100, Tamás Barta wrote:
> > > Hi!
> > > 
> > > I'm using httpclient-4.5.2.redhat-2.jar and I have a question I
> > > hope
> > > somebody can explain for me.
> > > I use this simple sample code to connect to a server a few times:
> > > 
> > > try (CloseableHttpClient client = HttpClients.custom().build()) {
> > 
> > Why on earth would you want to create a new HttpClient instance for
> > each and every request?
> > 
> > Oleg
> > 
> > 
> > >   for (int i=0; i<20; i++) {
> > > try (CloseableHttpResponse response = client.execute(new
> > > HttpGet("
> > > https://www.amazon.com;))) {
> > >   System.out.println("Executed: " + getStatusCode(response));
> > > }
> > >   }
> > > }
> > > 
> > > I use "netstat" to check the outgoing connections. For almost
> > > every
> > > url I
> > > tried netstat connections only increase by 1 and it is very well.
> > > 
> > > But when I use url "https://www.google-analytics.com/collect;
> > > then
> > > every
> > > get request creates a new connection and I can see 20 new
> > > TIME_WAIT
> > > rows in
> > > netstat.
> > > 
> > > I tried everything the find out what makes the difference, but I
> > > can't. I
> > > wonder how can I achieve that GA connections would be reused too.
> > > If
> > > I turn
> > > logging on there is no difference in the logs for both urls.
> > > 
> > > Thanks, Tamás
> > 
> > 
> > -
> > 
> > To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
> > For additional commands, e-mail: 
> > httpclient-users-h...@hc.apache.org
> > 
> > 


-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

Re: Socket reuse question

[Oleg Kalnichevski]

On Wed, 2019-12-11 at 10:50 +0100, Tamás Barta wrote:
> Hi!
> 
> I'm using httpclient-4.5.2.redhat-2.jar and I have a question I hope
> somebody can explain for me.
> I use this simple sample code to connect to a server a few times:
> 
> try (CloseableHttpClient client = HttpClients.custom().build()) {

Why on earth would you want to create a new HttpClient instance for
each and every request?

Oleg


>   for (int i=0; i<20; i++) {
> try (CloseableHttpResponse response = client.execute(new
> HttpGet("
> https://www.amazon.com;))) {
>   System.out.println("Executed: " + getStatusCode(response));
> }
>   }
> }
> 
> I use "netstat" to check the outgoing connections. For almost every
> url I
> tried netstat connections only increase by 1 and it is very well.
> 
> But when I use url "https://www.google-analytics.com/collect; then
> every
> get request creates a new connection and I can see 20 new TIME_WAIT
> rows in
> netstat.
> 
> I tried everything the find out what makes the difference, but I
> can't. I
> wonder how can I achieve that GA connections would be reused too. If
> I turn
> logging on there is no difference in the logs for both urls.
> 
> Thanks, Tamás


-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

Re: SSL failure when using new Async APIs in 5.0-beta6

[Oleg Kalnichevski]

On Tue, 2019-12-10 at 11:06 +0100, Nicolas de Pomereu wrote:
> Hi,
> I still have the problem with the new master version.
> (
> https://github.com/apache/httpcomponents-client/commit/f3c418c50a6ebba4f7f89d568a542c5599956310
> )
> Enclosed  class call fails with javax.net.ssl.SSLHandshakeException:
> General SSLEngine problem
> (USE_WEBSITE_WITH_IP_DIFFERENT_THAN_INNET_ADDRESS  = true)
> Regards,
> Nicolas de Pomereu
> 

You have not updated your code to use the #execute method with an
explicit HttpHost parameter, have you?

Oleg


> <<...>> <<...>>
> -Original Message-
> From: Oleg Kalnichevski 
> Sent: Thursday, December 5, 2019 4:56 PM
> To: HttpClient User Discussion 
> Subject: Re: SSL failure when using new Async APIs in 5.0-beta6
> On Wed, 2019-12-04 at 15:07 +0100, Oleg Kalnichevski wrote:
> >
> > > On Tue, 2019-12-03 at 15:35 +0100, Nicolas de Pomereu wrote:
> > > > Hi,
> > > >
> > > > With HttpClient 5.0-beta6  Classic APIs, HttpClient call will
> work
> > > > using HttpHost with following code, even if initHost IP is
> > > > different from
> > > > destServerNameIP:
> > > >
> > > >   initHost = "www.acme.com";
> > > >   destServerNameIP = "100.100.100.100";
> > > >   destScheme = "https";
> > > >   port = 443;
> > > >
> > > >   InetAddress inetAddress =
> > > > InetAddress.getByName(destServerNameIP);
> > > >   final HttpHost target = new HttpHost(destScheme, inetAddress,
> > > > initHost, port);
> > > >
> > > > With HttpClient 5.0-beta6  new Async APIs for HTTP/2, 
> HttpClient
> > > > will fail with a SSLEngineException failure if initHost IP is
> > > > different from destServerNameIP.
> > > >
> > > > Is this a new feature or a bug please?
> > > >
> >
> > I found out that async versions of HttpClient presently do not take
> > InetAddress parameter of HttpHost into account.
> >
> > Feel free to raise a JIRA if you want to track resolution of this
> > defect.
> >
> > I am seriously considering removing InetAddress attribute from
> > HttpHost. It should have never been added to the class in the first
> > place. Though I will see if there is an alternative fix that does
> not
> > involve any changes to the HttpHost as the first step.
> >
> > Oleg
> >
> The problem fixed in master with
> 
https://github.com/apache/httpcomponents-client/commit/f3c418c50a6ebba4f7f89d568a542c5599956310
> Oleg
> 
> 
> -
> To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
> For additional commands, e-mail: httpclient-users-h...@hc.apache.org
> -
> To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
> For additional commands, e-mail: httpclient-users-h...@hc.apache.org


-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

Re: SSL failure when using new Async APIs in 5.0-beta6

[Oleg Kalnichevski]

On Wed, 2019-12-04 at 15:07 +0100, Oleg Kalnichevski wrote:
> 
> > On Tue, 2019-12-03 at 15:35 +0100, Nicolas de Pomereu wrote:
> > > Hi,
> > > 
> > > With HttpClient 5.0-beta6  Classic APIs, HttpClient call will
> > > work
> > > using
> > > HttpHost with following code, even if initHost IP is different
> > > from
> > > destServerNameIP:
> > > 
> > >   initHost = "www.acme.com";
> > >   destServerNameIP = "100.100.100.100";
> > >   destScheme = "https";
> > >   port = 443;
> > > 
> > >   InetAddress inetAddress =
> > > InetAddress.getByName(destServerNameIP);
> > >   final HttpHost target = new HttpHost(destScheme, inetAddress,
> > > initHost,
> > > port);
> > > 
> > > With HttpClient 5.0-beta6  new Async APIs for HTTP/2,  HttpClient
> > > will fail
> > > with a SSLEngineException failure if initHost IP is different
> > > from
> > > destServerNameIP.
> > > 
> > > Is this a new feature or a bug please?
> > > 
> 
> I found out that async versions of HttpClient presently do not take
> InetAddress parameter of HttpHost into account.
> 
> Feel free to raise a JIRA if you want to track resolution of this
> defect.
> 
> I am seriously considering removing InetAddress attribute from
> HttpHost. It should have never been added to the class in the first
> place. Though I will see if there is an alternative fix that does not
> involve any changes to the HttpHost as the first step.
> 
> Oleg
> 

The problem fixed in master with 

https://github.com/apache/httpcomponents-client/commit/f3c418c50a6ebba4f7f89d568a542c5599956310

Oleg



-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

Re: SSL failure when using new Async APIs in 5.0-beta6

[Oleg Kalnichevski]

On Tue, 2019-12-03 at 16:45 +0100, Oleg Kalnichevski wrote:
> What I immediately see if that the server sends a different (broken)
> SSL certificate chain during the TLS handshake with
> RawSSLAsyncRegularTest example. At this point I _cannot_ confirm that
> as HttpClient's fault but I will take a closer look tomorrow.
> 
> Oleg
> 
> 
> On Tue, 2019-12-03 at 15:35 +0100, Nicolas de Pomereu wrote:
> > Hi,
> > 
> > With HttpClient 5.0-beta6  Classic APIs, HttpClient call will work
> > using
> > HttpHost with following code, even if initHost IP is different from
> > destServerNameIP:
> > 
> > initHost = "www.acme.com";
> > destServerNameIP = "100.100.100.100";
> > destScheme = "https";
> > port = 443;
> > 
> > InetAddress inetAddress =
> > InetAddress.getByName(destServerNameIP);
> > final HttpHost target = new HttpHost(destScheme, inetAddress,
> > initHost,
> > port);
> > 
> > With HttpClient 5.0-beta6  new Async APIs for HTTP/2,  HttpClient
> > will fail
> > with a SSLEngineException failure if initHost IP is different from
> > destServerNameIP.
> > 
> > Is this a new feature or a bug please?
> > 

I found out that async versions of HttpClient presently do not take
InetAddress parameter of HttpHost into account.

Feel free to raise a JIRA if you want to track resolution of this
defect.

I am seriously considering removing InetAddress attribute from
HttpHost. It should have never been added to the class in the first
place. Though I will see if there is an alternative fix that does not
involve any changes to the HttpHost as the first step.

Oleg



-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

Re: SSL failure when using new Async APIs in 5.0-beta6

[Oleg Kalnichevski]

What I immediately see if that the server sends a different (broken)
SSL certificate chain during the TLS handshake with
RawSSLAsyncRegularTest example. At this point I _cannot_ confirm that
as HttpClient's fault but I will take a closer look tomorrow.

Oleg


On Tue, 2019-12-03 at 15:35 +0100, Nicolas de Pomereu wrote:
> Hi,
> 
> With HttpClient 5.0-beta6  Classic APIs, HttpClient call will work
> using
> HttpHost with following code, even if initHost IP is different from
> destServerNameIP:
> 
>   initHost = "www.acme.com";
>   destServerNameIP = "100.100.100.100";
>   destScheme = "https";
>   port = 443;
> 
>   InetAddress inetAddress =
> InetAddress.getByName(destServerNameIP);
>   final HttpHost target = new HttpHost(destScheme, inetAddress,
> initHost,
> port);
> 
> With HttpClient 5.0-beta6  new Async APIs for HTTP/2,  HttpClient
> will fail
> with a SSLEngineException failure if initHost IP is different from
> destServerNameIP.
> 
> Is this a new feature or a bug please?
> 
> How can I call HttpClient in SSL with Async & HTTP/2 new APIs using a
> HttpHost with the 4 parameters HttpHost constructor?
> (We need it because we are writing a reverse proxy).
> 
> Please see two real enclosed examples that can be run to test
> implementation
> details.
> Runs have be tested with Windows 10 / Java jre1.8.0_211  and Linux
> and
> OpenJDK 10.0.2.
> 
> Regards,
> Nicolas
> -
> To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
> For additional commands, e-mail: httpclient-users-h...@hc.apache.org


-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

[ANNOUNCEMENT] HttpComponents Core 5.0 beta10 (GA candidate) released

[Oleg Kalnichevski]

The Apache HttpComponents project is pleased to announce 5.0-beta10 (GA
candidate) release of HttpComponents Core. 

   A. This BETA fixes a bug in the HTTP/2 setting handshake implementation
  and a performance regression in the HTTP/1.1 protocol handler.

IMPORTANT: This release is expected to be the last BETA version. If no
major design flaws are found the actual 5.0 API will be frozen and the
next version will be promoted to GA.


Notable changes and features included in the 5.0 series:

* Support for HTTP/2 protocol and conformance to requirements and
recommendations of the latest HTTP/2 protocol specification (RFC 7540,
RFC 7541)

  Supported features:

** HPACK header compression
** stream multiplexing (client and server)
** flow control
** response push (client and server)
** message trailers
** expect-continue handshake
** connection validation (ping)
** application-layer protocol negotiation (ALPN) on Java 9+
** TLS 1.2 security features

   Features out of scope for 5.0 release:

** padding of outgoing frames
** stream priority
** plain connection HTTP/1.1 upgrade
** CONNECT method

* Improved conformance to requirements and recommendations of the
latest HTTP/1.1 protocol specification (RFC 7230, RFC 7231)

* New asynchronous HTTP transport APIs consistent for both HTTP/1.1 and
HTTP/2 transport.

* Redesigned I/O reactor APIs and improved NIO based reactor
implementation for a greater performance and scalability.

* Support for server side request filters for classic and asynchronous
server implementations. Request filters could be used to implement
cross-cutting protocol aspects such as the 'expect-continue'
handshaking and user authentication / authorization.

* Redesigned connection pool implementation with strict connection
limit guarantees. The connection pool is expected to have a better
performance under higher concurrency due to reduced global pool lock
contention.

* New connection pool implementation with lax connection limit
guarantees and better performance under higher concurrency due to
absence of a global pool lock.

* Package name space changed to 'org.apache.hc.core5'

* Maven group id changed to 'org.apache.httpcomponents.core5'

HttpCore 5.0 releases can be co-located with earlier versions.

The 5.0 APIs are considered feature complete and are not expected to
undergo any major changes anymore. The focus of development is now
shifting to API polish, code stabilization and documentation
improvements.

Download - 
Release notes - 
HttpComponents site - 

About HttpComponents Core

HttpCore is a set of HTTP/1.1 and HTTP/2 transport components that can
be used to build custom client and server side HTTP services with a
minimal footprint



-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

Re: SO_SNDBUF/SO_RCVBUF

[Oleg Kalnichevski]

On Tue, 2019-10-22 at 11:49 -0400, Elliotte Rusty Harold wrote:
> Where does HttpClient get its default values of SO_SNDBUF and
> SO_RCVBUF from? Are these set i the code somewhere, and if so where?
> 
> Or are they inherited from the JDK or the host OS?
> 
> Thanks.
> 

By default HttpCore and HttpClient do not modify the default JRE / OS
SO_SNDBUF and SO_RCVBUF settings.

---
socket.setSoTimeout(this.sconfig.getSoTimeout());
if
(this.sconfig.getSndBufSize() > 0) {
socket.setSendBufferSize(this.s
config.getSndBufSize());
}
if (this.sconfig.getRcvBufSize() > 0) {
soc
ket.setReceiveBufferSize(this.sconfig.getRcvBufSize());
}
---

Oleg


-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

Re: Advise on HttpEntity#isRepeatable()

[Oleg Kalnichevski]

On Tue, 2019-10-22 at 16:17 +0200, Michael Osipov wrote:
> Folks,
> 
> I am currently working on WAGON-568 and would like better understand
> how 
> HttpEntity#isReproducible() should be properly handled.
> 
> In AbstractHttpClientWagon#put() [1] on specific status codes the 
> request (PUT) is retried manually with the same HttpEntity [2]
> without:
> 
> * consuming the given response entity,
> * calling fireTransferError() (Wagon internal),
> * knowing that the entity is repeatable
> 
> This may apply if expect/continue did not work properly.
> 
>  From what I understand, this approach is broken. I have either to
> fix 
> the above or employ a modified version of DefaultRedirectStrategy.
> 
> Is my understanding correct?
> 

Hi Michael

No, I do not think your current approach is wrong. The contract of
HttpEntity is such that if HttpEntity#isRepeatable returns true the
HttpEntity#writeTo method is expected to produce the same entity
representation upon each invocation. HttpClient should be able to
automatically re-try the request without any intervention as long as
the HttpEntity contract is correctly implemented.

> Anyway, I have always signal an exception when a already consumed 
> request body cannot be replayed?!
> 

HttpClient automatically throws NonRepeatableRequestException if a
method re-execution is impossible due to the closed entity being non-
repeatable. I do not think Wagon needs to provide any additional
handling logic unless there are some Wagon specific requirements I am
not aware of. 

Hope this helps.

Oleg

> Michael
> 
> [1] 
> 
https://github.com/apache/maven-wagon/blob/master/wagon-providers/wagon-http-shared/src/main/java/org/apache/maven/wagon/shared/http/AbstractHttpClientWagon.java#L674
> [2] 
> 
https://github.com/apache/maven-wagon/blob/master/wagon-providers/wagon-http-shared/src/main/java/org/apache/maven/wagon/shared/http/AbstractHttpClientWagon.java#L739-L743
> 
> -
> To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
> For additional commands, e-mail: httpclient-users-h...@hc.apache.org
> 


-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

[ANNOUNCEMENT] HttpComponents Client 5.0 beta6 (GA candidate) released

[Oleg Kalnichevski]

The Apache HttpComponents project is pleased to announce 5.0-beta6
release of HttpComponents Client. 

This BETA release picks up the latest fixes and performance
improvements from HttpCore and addresses a number of issues found since
the previous BETA release.

IMPORTANT: This release is expected to be the last BETA version. If no
major design flaws are found the actual 5.0 API will be frozen and the
next version will be promoted to GA.


Notable changes and features in the 5.0 series:
---

* Support for the HTTP/2 protocol and conformance to requirements and
recommendations of the latest HTTP/2 protocol specification documents
(RFC 7540, RFC 7541.)

  Supported features:

** HPACK header compression
** Stream multiplexing (client and server)
** Flow control
** Response push
** Message trailers
** Expect-continue handshake
** Connection validation (ping)
** Application-layer protocol negotiation (ALPN)
** TLS 1.2 security features

* Improved conformance to requirements and recommendations of the
latest HTTP/1.1 protocol specification documents (RFC 7230, RFC 7231.)

* New connection pool implementation with lax connection limit
guarantees and better performance under higher concurrency due to
absence of a global pool lock.

* Package name space changed to 'org.apache.hc.client5'.

* Maven group id changed to 'org.apache.httpcomponents.client5'.

HttpClient 5.0 releases can be co-located with earlier major versions
on the same classpath due to the change in package names and Maven
module coordinates.

The 5.0 APIs are considered feature complete and are not expected to
undergo any major changes anymore. The focus of development is now
shifting to API polish, code stabilization and documentation
improvements.

Download - 
Release notes - <



https://www.apache.org/dist/httpcomponents/httpclient/RELEASE_NOTES-5.0.x.txt
> 

HttpComponents site - 

About HttpComponents HttpClient

The Hyper-Text Transfer Protocol (HTTP) is perhaps the most significant
protocol used on the Internet today. Web services, network-enabled
appliances and the growth of network computing continue to expand the
role of the HTTP protocol beyond user-driven web browsers, while
increasing the number of applications that require HTTP support.

Although the java.net package provides basic functionality for
accessing resources via HTTP, it doesn't provide the full flexibility
or functionality needed by many applications. HttpClient seeks to fill
this void by providing an efficient, up-to-date, and feature-rich
package implementing the client side of the most recent HTTP standards
and recommendations.

Designed for extension while providing robust support for the base HTTP
protocol, HttpClient may be of interest to anyone building HTTP-aware
client applications such as web browsers, web service clients, or
systems that leverage or extend the HTTP protocol for distributed
communication.



-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

[ANNOUNCEMENT] HttpComponents Core 5.0 beta9 (GA candidate) released

[Oleg Kalnichevski]

The Apache HttpComponents project is pleased to announce 5.0-beta9 (GA
candidate) release of HttpComponents Core. 

This BETA fixes a number of defects found since the last release,
improves behavior of the lax (concurrent) connection pools (special
thanks to Linton Miller), simplifies and improves input event handling
of SSL/TLS sessions and the HTTP/1.1 protocol event handler.

IMPORTANT: This release is expected to be the last BETA version. If no
major design flaws are found the actual 5.0 API will be frozen and the
next version will be promoted to GA.


Notable changes and features included in the 5.0 series:

* Support for HTTP/2 protocol and conformance to requirements and
recommendations of the latest HTTP/2 protocol specification (RFC 7540,
RFC 7541)

  Supported features:

** HPACK header compression
** stream multiplexing (client and server)
** flow control
** response push (client and server)
** message trailers
** expect-continue handshake
** connection validation (ping)
** application-layer protocol negotiation (ALPN) on Java 9+
** TLS 1.2 security features

   Features out of scope for 5.0 release:

** padding of outgoing frames
** stream priority
** plain connection HTTP/1.1 upgrade
** CONNECT method

* Improved conformance to requirements and recommendations of the
latest HTTP/1.1 protocol specification (RFC 7230, RFC 7231)

* New asynchronous HTTP transport APIs consistent for both HTTP/1.1 and
HTTP/2 transport.

* Redesigned I/O reactor APIs and improved NIO based reactor
implementation for a greater performance and scalability.

* Support for server side request filters for classic and asynchronous
server implementations. Request filters could be used to implement
cross-cutting protocol aspects such as the 'expect-continue'
handshaking and user authentication / authorization.

* Redesigned connection pool implementation with strict connection
limit guarantees. The connection pool is expected to have a better
performance under higher concurrency due to reduced global pool lock
contention.

* New connection pool implementation with lax connection limit
guarantees and better performance under higher concurrency due to
absence of a global pool lock.

* Package name space changed to 'org.apache.hc.core5'

* Maven group id changed to 'org.apache.httpcomponents.core5'

HttpCore 5.0 releases can be co-located with earlier versions.

The 5.0 APIs are considered feature complete and are not expected to
undergo any major changes anymore. The focus of development is now
shifting to API polish, code stabilization and documentation
improvements.

Download - 
Release notes - 
HttpComponents site - 

About HttpComponents Core

HttpCore is a set of HTTP/1.1 and HTTP/2 transport components that can
be used to build custom client and server side HTTP services with a
minimal footprint



-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

HttpCore / HttpClient 5.0 compatibility testing

[Oleg Kalnichevski]

Folks

I have been testing HttpCore 5.0 and HttpClient 5.0 for compatibility
with common HTTP servers such as Apache HTTPD and Ngnix for quite some
time using Docker containers. I also recently added httpbin.org
container to the list of target test environments. The image
definitions can be found here [1] and the compatibility tests here
[2]. 

Usually the intent of such tests is to ensure compatibility with non-
trivial functionality such as HTTP/2 message multiplexing and server
push, and HTTP/1.1 message pipelining and out of sequence responses.

If you want us to regularly test for compatibility with other web
platforms, be it Jetty or Apache Tomcat or anything else, please
consider contributing Docker images and adding to the existing
compatibility test suits.

This would be a perfect way to make a major contribution to the project
and would be a huge win for the community.
 
My Docker skills are rudimentary and probably the way I go about Docker
based tests is likely very silly. 

Anyone who would be willing to help us get those tests to run with
Jenkins as a regular job would make a tremendous contribution. 

Oleg

[1] 
https://github.com/apache/httpcomponents-core/tree/master/httpcore5-testing/docker
[2] 
https://github.com/apache/httpcomponents-core/blob/master/httpcore5-testing/src/test/java/org/apache/hc/core5/testing/compatibility/http2/H2CompatibilityTest.java


-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

Re: max total connections

[Oleg Kalnichevski]

On Mon, 2019-09-23 at 10:39 +, Братухин Никита Романович wrote:
> Hey there, recently we have been using this beautiful lib for our
> internal microservices communication, but I have some question about
> max connection count.
> What will happen when max count of connections will be used?
> For example, if  PoolingHttpClientConnectionManager has maximum 200
> of connections
> cm.setMaxTotal(200);
> and all of them are used, what will happen then client should execute
> another request, which will require 201 connection?

Доброго вам дня.

The thread executing the request will get blocked until a connection
becomes available in the pool or `connectionRequestTimeout` expires.


> Will it throw error or will some of the active connections closed (
> for example some LRU/LFU logic) ?

Several outcomes are possible:

1. If a persistent connection gets released back to the pool manager it
will be re-used and served to a pending connection lease request for
the same route.

2. If a closed connection for any route or a persistent connection for
a different route gets released back to the pool the pool manager will
discard the old connection and open a new connection for the next
pending connection request. 

3. `ConnectionPoolTimeoutException` is thrown if
the `connectionRequestTimeout` expires with no connection released back
to the pool manager.

Hope this helps

Oleg


> УВЕДОМЛЕНИЕ О КОНФИДЕНЦИАЛЬНОСТИ: Это электронное сообщение и любые
> документы, приложенные к нему, содержат конфиденциальную информацию.
> Настоящим уведомляем Вас о том, что если это сообщение не
> предназначено Вам, использование, копирование, распространение
> информации, содержащейся в настоящем сообщении, а также осуществление
> любых действий на основе этой информации, строго запрещено. Если Вы
> получили это сообщение по ошибке, пожалуйста, сообщите об этом
> отправителю по электронной почте и удалите это сообщение.
> CONFIDENTIALITY NOTICE: This email and any files attached to it are
> confidential. If you are not the intended recipient you are notified
> that using, copying, distributing or taking any action in reliance on
> the contents of this information is strictly prohibited. If you have
> received this email in error please notify the sender and delete this
> email.

-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

[ANNOUNCEMENT] HttpComponents Client 4.5.10 GA Released

[Oleg Kalnichevski]

The Apache HttpComponents project is pleased to announce 4.5.10 GA
release of HttpComponents HttpClient.

This is a maintenance release that fixes a number defects discovered
since 4.5.9.

Download - 
Release notes - <
https://www.apache.org/dist/httpcomponents/httpclient/RELEASE_NOTES-4.5.x.txt
>
HttpComponents site - 

About HttpComponents HttpClient

The Hyper-Text Transfer Protocol (HTTP) is perhaps the most significant
protocol used on the Internet today. Web services, network-enabled
appliances and the growth of network computing continue to expand the
role of the HTTP protocol beyond user-driven web browsers, while
increasing the number of applications that require HTTP support.

Designed for extension while providing robust support for the base HTTP
protocol, HttpClient may be of interest to anyone building HTTP-aware
client applications such as web browsers, web service clients, or
systems that leverage or extend the HTTP protocol for distributed
communication.



-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

[ANNOUNCEMENT] HttpComponents Core 4.4.12 Released

[Oleg Kalnichevski]

The Apache HttpComponents project is pleased to announce 4.4.12 GA
release of HttpComponents Core.

This is a maintenance release that corrects a number of defects
discovered since release 4.4.11.

Please note that as of 4.4 HttpCore requires Java 1.6 or newer.

IMPORTANT: Users of HttpCore 4.x GA releases are strongly encouraged to
evaluate new HttpCore 5.0 APIs and give the project developers
feedback, share critique or propose changes.

Download -

Release notes -

HttpComponents site -


About HttpComponents Core

HttpCore is a set of low level HTTP transport components that can be
used to build custom client and server side HTTP services with a
minimal footprint. HttpCore supports two I/O models: a blocking I/O
model based on the classic Java I/O and a non-blocking, event driven
I/O model based on Java NIO.



-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

Re: Issue sending https to an http endpoint

[Oleg Kalnichevski]

On Tue, 2019-08-27 at 11:27 +0200, Joan grupoventus wrote:
> Hi Oleg,
> 
> I uploaded the httpcore-5b9 we generated to the git. Is it possible
> you download it and try just one test with async Http5 to check if it
> works for you?
> We have checked the jar and it seems to be correct, so if you are
> using the same version I don't understand why it works for you and
> not for us.
> 
> Thanks,
> Joan.
> 

Could you please try to reproduce the issue inside a docker container?

Oleg



> -Original Message-
> From: Oleg Kalnichevski  
> Sent: Friday, August 23, 2019 5:32 PM
> To: HttpClient User Discussion 
> Subject: Re: Issue sending https to an http endpoint
> 
> On Fri, 2019-08-23 at 17:24 +0200, Joan grupoventus wrote:
> > Hi Oleg,
> > 
> > Is this the right link to build the httpcore-b9? 
> > https://github.com/apache/httpcomponents-core
> > 
> 
> Yes, it is. 
> 
> Oleg
> 
> > We checked the version of our jar and it seems to be correctly. Is 
> > there something else we have to build? We are already using the
> > latest 
> > httpclient (httpclient5-5.0-beta5)
> > 
> > Thanks,
> > Joan.
> > 
> > -Original Message-
> > From: Oleg Kalnichevski 
> > Sent: Friday, August 23, 2019 9:49 AM
> > To: HttpClient User Discussion 
> > Subject: Re: Issue sending https to an http endpoint
> > 
> > On Thu, 2019-08-22 at 23:12 +0200, Joan grupoventus wrote:
> > > Hi Oleg,
> > > 
> > > Well, not sure what's happening.  In "our" git
> > > (https://github.com/joanbalaguero/HttpClient)  I have
> > > uploaded  4 
> > > files that connect by https to the problematic url.
> > > 
> > > The "TestSync" is using a sync http5 client and works, ends with
> > > an 
> > > exception "javax.net.ssl.SSLException: Unrecognized SSL message, 
> > > plaintext connection?"
> > > The " TestAsync41ResponseConsumer.java" is using the async http
> > > 4.1.3
> > > client with an "HttpAsyncResponseConsumer" and works, ends with
> > > an 
> > > exception
> > > "javax.net.ssl.SSLException: Unrecognized SSL message, plaintext 
> > > connection?"
> > > 
> > > The "TestAsync5FutureCallback" and "TestAsync5ResponseConsumer"
> > > are
> > > using an async http5 client (httpclient5-5-beta5 and 
> > > httpcore5-5-beta9), the first one with a 
> > > "FutureCallback" and the second with a 
> > > "AsyncResponseConsumer". Both hang forever.
> > > 
> > > So if this is not a bug in http5, what's wrong with these two
> > > last 
> > > samples?
> > > The code there is minimal. 
> > > 
> > 
> > Joan,
> > 
> > I see nothing wrong with those samples and they all work for me 
> > (HttpClient 5.0b5 + HttpCore 5.0b9-SNAPSHOT).
> > 
> > Oleg
> > ---
> > 2019-08-23 09:46:30,669 DEBUG
> > [main][org.apache.hc.client5.http.impl.async.InternalHttpAsyncClien
> > t]
> > ex-0001: preparing request execution
> > 2019-08-23 09:46:30,676 DEBUG
> > [main][org.apache.hc.client5.http.protocol.RequestAddCookies]
> > CookieSpec selected: standard
> > 2019-08-23 09:46:30,680 DEBUG
> > [main][org.apache.hc.client5.http.protocol.RequestAuthCache] Auth 
> > cache not set in the context
> > 2019-08-23 09:46:30,680 DEBUG
> > [main][org.apache.hc.client5.http.impl.async.AsyncProtocolExec] ex-
> > 0001: target auth state: UNCHALLENGED
> > 2019-08-23 09:46:30,681 DEBUG
> > [main][org.apache.hc.client5.http.impl.async.AsyncProtocolExec] ex-
> > 0001: proxy auth state: UNCHALLENGED
> > 2019-08-23 09:46:30,683 DEBUG
> > [main][org.apache.hc.client5.http.impl.async.AsyncConnectExec] ex-
> > 0001: acquiring connection with route {s}->
> > https://54.38.179.182:8083
> > 2019-08-23 09:46:30,684 DEBUG
> > [main][org.apache.hc.client5.http.impl.async.InternalHttpAsyncClien
> > t]
> > ex-0001: acquiring endpoint (3 MINUTES)
> > 2019-08-23 09:46:30,685 DEBUG
> > [main][org.apache.hc.client5.http.impl.nio.PoolingAsyncClientConnec
> > ti
> > onManager] ex-0001: endpoint lease request (3 MINUTES) [route:
> > {s}->https://54.38.179.182:8083][total kept alive: 0; route
> > allocated: 0 of 5; total allocated: 0 of 25]
> > 2019-08-23 09:46:30,689 DEBUG
> > [main][org.apache.hc.client5.http.impl.nio.PoolingAsyncClientConnec
> > ti
> > onManager] ex-00

Re: Issue sending https to an http endpoint

[Oleg Kalnichevski]

On Fri, 2019-08-23 at 17:24 +0200, Joan grupoventus wrote:
> Hi Oleg,
> 
> Is this the right link to build the httpcore-b9? 
> https://github.com/apache/httpcomponents-core
> 

Yes, it is. 

Oleg

> We checked the version of our jar and it seems to be correctly. Is
> there something else we have to build? We are already using the
> latest httpclient (httpclient5-5.0-beta5)
> 
> Thanks,
> Joan.
> 
> -Original Message-
> From: Oleg Kalnichevski  
> Sent: Friday, August 23, 2019 9:49 AM
> To: HttpClient User Discussion 
> Subject: Re: Issue sending https to an http endpoint
> 
> On Thu, 2019-08-22 at 23:12 +0200, Joan grupoventus wrote:
> > Hi Oleg,
> > 
> > Well, not sure what's happening.  In "our" git
> > (https://github.com/joanbalaguero/HttpClient)  I have uploaded  4 
> > files that connect by https to the problematic url.
> > 
> > The "TestSync" is using a sync http5 client and works, ends with
> > an 
> > exception "javax.net.ssl.SSLException: Unrecognized SSL message, 
> > plaintext connection?"
> > The " TestAsync41ResponseConsumer.java" is using the async http
> > 4.1.3 
> > client with an "HttpAsyncResponseConsumer" and works, ends with an 
> > exception
> > "javax.net.ssl.SSLException: Unrecognized SSL message, plaintext 
> > connection?"
> > 
> > The "TestAsync5FutureCallback" and "TestAsync5ResponseConsumer"
> > are 
> > using an async http5 client (httpclient5-5-beta5 and 
> > httpcore5-5-beta9), the first one with a 
> > "FutureCallback" and the second with a 
> > "AsyncResponseConsumer". Both hang forever.
> > 
> > So if this is not a bug in http5, what's wrong with these two last 
> > samples?
> > The code there is minimal. 
> > 
> 
> Joan,
> 
> I see nothing wrong with those samples and they all work for me
> (HttpClient 5.0b5 + HttpCore 5.0b9-SNAPSHOT).
> 
> Oleg
> ---
> 2019-08-23 09:46:30,669 DEBUG
> [main][org.apache.hc.client5.http.impl.async.InternalHttpAsyncClient]
> ex-0001: preparing request execution
> 2019-08-23 09:46:30,676 DEBUG
> [main][org.apache.hc.client5.http.protocol.RequestAddCookies]
> CookieSpec selected: standard
> 2019-08-23 09:46:30,680 DEBUG
> [main][org.apache.hc.client5.http.protocol.RequestAuthCache] Auth
> cache not set in the context
> 2019-08-23 09:46:30,680 DEBUG
> [main][org.apache.hc.client5.http.impl.async.AsyncProtocolExec] ex-
> 0001: target auth state: UNCHALLENGED
> 2019-08-23 09:46:30,681 DEBUG
> [main][org.apache.hc.client5.http.impl.async.AsyncProtocolExec] ex-
> 0001: proxy auth state: UNCHALLENGED
> 2019-08-23 09:46:30,683 DEBUG
> [main][org.apache.hc.client5.http.impl.async.AsyncConnectExec] ex-
> 0001: acquiring connection with route {s}->
> https://54.38.179.182:8083
> 2019-08-23 09:46:30,684 DEBUG
> [main][org.apache.hc.client5.http.impl.async.InternalHttpAsyncClient]
> ex-0001: acquiring endpoint (3 MINUTES)
> 2019-08-23 09:46:30,685 DEBUG
> [main][org.apache.hc.client5.http.impl.nio.PoolingAsyncClientConnecti
> onManager] ex-0001: endpoint lease request (3 MINUTES) [route:
> {s}->https://54.38.179.182:8083][total kept alive: 0; route
> allocated: 0 of 5; total allocated: 0 of 25]
> 2019-08-23 09:46:30,689 DEBUG
> [main][org.apache.hc.client5.http.impl.nio.PoolingAsyncClientConnecti
> onManager] ex-0001: endpoint leased [route: {s}->
> https://54.38.179.182:8083][total kept alive: 0; route allocated: 1
> of 5; total allocated: 1 of 25]
> 2019-08-23 09:46:30,690 DEBUG
> [main][org.apache.hc.client5.http.impl.nio.PoolingAsyncClientConnecti
> onManager] ex-0001: acquired ep-
> 2019-08-23 09:46:30,690 DEBUG
> [main][org.apache.hc.client5.http.impl.async.InternalHttpAsyncClient]
> ex-0001: acquired endpoint ep-
> 2019-08-23 09:46:30,691 DEBUG
> [main][org.apache.hc.client5.http.impl.async.InternalHttpAsyncClient]
> ep-: connecting endpoint (3 MINUTES)
> 2019-08-23 09:46:30,692 DEBUG
> [main][org.apache.hc.client5.http.impl.nio.PoolingAsyncClientConnecti
> onManager] ep-: connecting endpoint to 
> https://54.38.179.182:8083 (3 MINUTES)
> 2019-08-23 09:46:30,692 DEBUG
> [main][org.apache.hc.client5.http.impl.nio.MultihomeIOSessionRequeste
> r] https://54.38.179.182:8083: resolving remote address
> 2019-08-23 09:46:30,692 DEBUG
> [main][org.apache.hc.client5.http.impl.nio.MultihomeIOSessionRequeste
> r] https://54.38.179.182:8083: resolved to [/54.38.179.182]
> 2019-08-23 09:46:30,693 DEBUG
> [main][org.apache.hc.client5.http.impl.nio.MultihomeIOSessionRequeste
>

Re: Issue sending https to an http endpoint

[Oleg Kalnichevski]

ava:313)
at 
org.apache.hc.core5.reactor.ssl.SSLIOSession.isAppInputReady(SSLIOSession.java:529)
at 
org.apache.hc.core5.reactor.InternalDataChannel.onIOEvent(InternalDataChannel.java:122)
at 
org.apache.hc.core5.reactor.InternalChannel.handleIOEvent(InternalChannel.java:51)
at 
org.apache.hc.core5.reactor.SingleCoreIOReactor.processEvents(SingleCoreIOReactor.java:177)
at 
org.apache.hc.core5.reactor.SingleCoreIOReactor.doExecute(SingleCoreIOReactor.java:127)
at 
org.apache.hc.core5.reactor.AbstractSingleCoreIOReactor.execute(AbstractSingleCoreIOReactor.java:82)
at 
org.apache.hc.core5.reactor.IOReactorWorker.run(IOReactorWorker.java:44)
at java.lang.Thread.run(Thread.java:748)
2019-08-23 09:46:31,169 DEBUG 
[main][org.apache.hc.client5.http.impl.async.InternalHttpAsyncClient] Shutdown 
GRACEFUL
2019-08-23 09:46:31,180 DEBUG 
[main][org.apache.hc.client5.http.impl.nio.PoolingAsyncClientConnectionManager] 
Shutdown connection pool GRACEFUL
2019-08-23 09:46:31,181 DEBUG 
[main][org.apache.hc.client5.http.impl.nio.PoolingAsyncClientConnectionManager] 
Connection pool shut down


> Thanks,
> Joan.
> 
> 
> -Original Message-
> From: Oleg Kalnichevski  
> Sent: Thursday, August 22, 2019 12:05 PM
> To: HttpClient User Discussion 
> Subject: Re: Issue sending https to an http endpoint
> 
> On Wed, 2019-08-21 at 22:24 +0200, Joan grupoventus wrote:
> > Hi Oleg,
> > 
> > Tried with httpcore5-5.0-beta9.jar. The same result.
> > 
> 
> I am sorry, Joan, but I am unable to reproduce the problem with
> HttpClient
> 5.0-beta5 and the latest HttpCore 5.0-beta9-SNAPSHOT. It works as
> expected
> for me.
> 
> Oleg 
> 
> ---
> 2019-08-22 12:02:43,978 DEBUG
> [main][org.apache.hc.client5.http.impl.async.InternalHttpAsyncClient]
> ex-0001: preparing request execution
> 2019-08-22 12:02:43,986 DEBUG
> [main][org.apache.hc.client5.http.protocol.RequestAddCookies]
> CookieSpec
> selected: standard
> 2019-08-22 12:02:43,990 DEBUG
> [main][org.apache.hc.client5.http.protocol.RequestAuthCache] Auth
> cache not
> set in the context
> 2019-08-22 12:02:43,991 DEBUG
> [main][org.apache.hc.client5.http.impl.async.AsyncProtocolExec] ex-
> 0001:
> target auth state: UNCHALLENGED
> 2019-08-22 12:02:43,991 DEBUG
> [main][org.apache.hc.client5.http.impl.async.AsyncProtocolExec] ex-
> 0001:
> proxy auth state: UNCHALLENGED
> 2019-08-22 12:02:43,993 DEBUG
> [main][org.apache.hc.client5.http.impl.async.AsyncConnectExec] ex-
> 0001:
> acquiring connection with route {s}->https://localhost:8080
> 2019-08-22 12:02:43,993 DEBUG
> [main][org.apache.hc.client5.http.impl.async.InternalHttpAsyncClient]
> ex-0001: acquiring endpoint (3 MINUTES)
> 2019-08-22 12:02:43,995 DEBUG
> [main][org.apache.hc.client5.http.impl.nio.PoolingAsyncClientConnecti
> onManag
> er] ex-0001: endpoint lease request (3 MINUTES) [route:
> {s}->https://localhost:8080][total kept alive: 0; route allocated: 0
> of 5;
> total allocated: 0 of 25]
> 2019-08-22 12:02:43,998 DEBUG
> [main][org.apache.hc.client5.http.impl.nio.PoolingAsyncClientConnecti
> onManag
> er] ex-0001: endpoint leased [route: {s}->
> https://localhost:8080][total
> kept alive: 0; route allocated: 1 of 5; total allocated: 1 of 25]
> 2019-08-22 12:02:43,998 DEBUG
> [main][org.apache.hc.client5.http.impl.nio.PoolingAsyncClientConnecti
> onManag
> er] ex-0001: acquired ep-
> 2019-08-22 12:02:43,999 DEBUG
> [main][org.apache.hc.client5.http.impl.async.InternalHttpAsyncClient]
> ex-0001: acquired endpoint ep-
> 2019-08-22 12:02:43,999 DEBUG
> [main][org.apache.hc.client5.http.impl.async.InternalHttpAsyncClient]
> ep-: connecting endpoint (3 MINUTES)
> 2019-08-22 12:02:44,000 DEBUG
> [main][org.apache.hc.client5.http.impl.nio.PoolingAsyncClientConnecti
> onManag
> er] ep-: connecting endpoint to https://localhost:8080 (3
> MINUTES)
> 2019-08-22 12:02:44,000 DEBUG
> [main][org.apache.hc.client5.http.impl.nio.MultihomeIOSessionRequeste
> r]
> https://localhost:8080: resolving remote address
> 2019-08-22 12:02:44,001 DEBUG
> [main][org.apache.hc.client5.http.impl.nio.MultihomeIOSessionRequeste
> r]
> https://localhost:8080: resolved to [localhost/127.0.0.1]
> 2019-08-22 12:02:44,001 DEBUG
> [main][org.apache.hc.client5.http.impl.nio.MultihomeIOSessionRequeste
> r]
> https://localhost:8080: connecting null to localhost/127.0.0.1:8080
> (3
> MINUTES)
> 2019-08-22 12:02:44,023 DEBUG
> [httpclient-dispatch-
> 1][org.apache.hc.client5.http.impl.nio.MultihomeIOSessi
> onRequester] https://localhost:8080: connected i/o-
> /127.0.0.1:50902->localhost/127.0.0.1:8080
> 20

Re: Issue sending https to an http endpoint

[Oleg Kalnichevski]

_WITH_AES_256_CBC_SHA384, TLS_RSA_WITH_AES_256_CBC_SHA256, 
TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDH_RSA_WITH_AES_256_CBC_SHA384, 
TLS_DHE_RSA_WITH_AES_256_CBC_SHA256, TLS_DHE_DSS_WITH_AES_256_CBC_SHA256, 
TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, 
TLS_RSA_WITH_AES_256_CBC_SHA, TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA, 
TLS_ECDH_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_RSA_WITH_AES_256_CBC_SHA, 
TLS_DHE_DSS_WITH_AES_256_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, 
TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_RSA_WITH_AES_128_CBC_SHA256, 
TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDH_RSA_WITH_AES_128_CBC_SHA256, 
TLS_DHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_DSS_WITH_AES_128_CBC_SHA256, 
TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, 
TLS_RSA_WITH_AES_128_CBC_SHA, TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA, 
TLS_ECDH_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_CBC_SHA, 
TLS_DHE_DSS_WITH_AES_128_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, 
TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, 
TLS_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDH_ECDSA_WITH_AES_256_GCM_SHA384, 
TLS_ECDH_RSA_WITH_AES_256_GCM_SHA384, TLS_DHE_RSA_WITH_AES_256_GCM_SHA384, 
TLS_DHE_DSS_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, 
TLS_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDH_ECDSA_WITH_AES_128_GCM_SHA256, 
TLS_ECDH_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_RSA_WITH_AES_128_GCM_SHA256, 
TLS_DHE_DSS_WITH_AES_128_GCM_SHA256, TLS_EMPTY_RENEGOTIATION_INFO_SCSV]
2019-08-22 12:02:44,083 DEBUG 
[httpclient-dispatch-1][org.apache.hc.client5.http.impl.async.HttpAsyncMainClientExec]
 ex-0001: execution failed: Unrecognized SSL message, plaintext connection?
2019-08-22 12:02:44,084 DEBUG 
[httpclient-dispatch-1][org.apache.hc.client5.http.impl.async.InternalHttpAsyncClient]
 ex-0001: request failed: Unrecognized SSL message, plaintext connection?
2019-08-22 12:02:44,084 DEBUG 
[httpclient-dispatch-1][org.apache.hc.client5.http.impl.nio.PoolingAsyncClientConnectionManager]
 ep-: close IMMEDIATE
2019-08-22 12:02:44,084 DEBUG 
[httpclient-dispatch-1][org.apache.hc.client5.http.impl.nio.DefaultManagedAsyncClientConnection]
 i/o-: Shutdown connection IMMEDIATE
2019-08-22 12:02:44,084 DEBUG 
[httpclient-dispatch-1][org.apache.hc.client5.http.impl.async.InternalHttpAsyncClient]
 ep-: endpoint closed
2019-08-22 12:02:44,084 DEBUG 
[httpclient-dispatch-1][org.apache.hc.client5.http.impl.async.InternalHttpAsyncClient]
 ep-: discarding endpoint
2019-08-22 12:02:44,084 DEBUG 
[httpclient-dispatch-1][org.apache.hc.client5.http.impl.nio.PoolingAsyncClientConnectionManager]
 ep-: releasing endpoint
2019-08-22 12:02:44,084 DEBUG 
[httpclient-dispatch-1][org.apache.hc.client5.http.impl.nio.PoolingAsyncClientConnectionManager]
 ep-: connection released [route: {s}->https://localhost:8080][total 
kept alive: 0; route allocated: 0 of 5; total allocated: 0 of 25]
NOK: HTTP/1.1 https://localhost:8080: OPTIONS * -> (Unrecognized SSL message, 
plaintext connection?)

> Thanks,
> Joan.
> 
> -Original Message-
> From: Oleg Kalnichevski  
> Sent: Monday, August 19, 2019 9:15 AM
> To: HttpClient User Discussion 
> Subject: Re: Issue sending https to an http endpoint
> 
> On Sun, 2019-08-18 at 21:05 +0200, Joan grupoventus wrote:
> > Hello,
> > 
> >   
> > 
> > A mistake made by one of our clients has brought us a possible
> > issue 
> > to our attention. We have been able to reproduce it on our test 
> > environment.
> > 
> >   
> > 
> > The client has an endpoint that only listens on http connections
> > on 
> > ports 8080,8081,8082 and 8083. By mistake all traffic from our
> > gateway 
> > to their backend has been sent by https instead of http, and this 
> > provoked the pool to have all connections allocated (pool
> > exhausted).
> > 
> 
> Hi Joan
> 
> I think this issue has already been resolved in HttpCore master.
> Could you please try to reproduce the issue the latest HttpCore
> snapshot? You will need to build HttpCore from source to do so.
> 
> https://github.com/apache/httpcomponents-core
> 
> Oleg
> 
> >   
> > 
> > We have tried to send https requests to an http endpoint using
> > post 
> > methods through a “  > INF%5C/lib%5C/httpclient5-5.0-beta5.jar%3Corg> org.  > javadoc:%E2%98%82=vpfw/web%5C/WEB-INF%5C/lib%5C/httpclient5-5.0-
> > beta5.jar%3Corg.apache> apache.  > javadoc:%E2%98%82=vpfw/web%5C/WEB-INF%5C/lib%5C/httpclient5-5.0-
> > beta5.jar%3Corg.apache.hc> hc.  > javadoc:%E2%98%82=vpfw/web%5C/WEB-INF%5C/lib%5C/httpclient5-5.0-
> > beta5.jar%3Corg.apache.hc.client5> client5.  > javadoc:%E2%98%82=vpfw/web%5C/WEB-INF%5C/lib%5C/httpclient5-5.0-
> >

Re: Issue sending https to an http endpoint

[Oleg Kalnichevski]

On Sun, 2019-08-18 at 21:05 +0200, Joan grupoventus wrote:
> Hello,
> 
>  �
> 
> A mistake made by one of our clients has brought us a possible issue
> to our attention. We have been able to reproduce it on our test
> environment.
> 
>  �
> 
> The client has an endpoint that only listens on http connections on
> ports 8080,8081,8082 and 8083. By mistake all traffic from our
> gateway to their backend has been sent by https instead of http, and
> this provoked the pool to have all connections allocated (pool
> exhausted).
> 

Hi Joan

I think this issue has already been resolved in HttpCore master. Could
you please try to reproduce the issue the latest HttpCore snapshot? You
will need to build HttpCore from source to do so.

https://github.com/apache/httpcomponents-core

Oleg

>  �
> 
> We have tried to send https requests to an http endpoint using post
> methods through a “  INF%5C/lib%5C/httpclient5-5.0-beta5.jar%3Corg> org.  javadoc:%E2%98%82=vpfw/web%5C/WEB-INF%5C/lib%5C/httpclient5-5.0-
> beta5.jar%3Corg.apache> apache.  javadoc:%E2%98%82=vpfw/web%5C/WEB-INF%5C/lib%5C/httpclient5-5.0-
> beta5.jar%3Corg.apache.hc> hc.  javadoc:%E2%98%82=vpfw/web%5C/WEB-INF%5C/lib%5C/httpclient5-5.0-
> beta5.jar%3Corg.apache.hc.client5> client5.  javadoc:%E2%98%82=vpfw/web%5C/WEB-INF%5C/lib%5C/httpclient5-5.0-
> beta5.jar%3Corg.apache.hc.client5.http> http.  javadoc:%E2%98%82=vpfw/web%5C/WEB-INF%5C/lib%5C/httpclient5-5.0-
> beta5.jar%3Corg.apache.hc.client5.http.impl> impl.  javadoc:%E2%98%82=vpfw/web%5C/WEB-INF%5C/lib%5C/httpclient5-5.0-
> beta5.jar%3Corg.apache.hc.client5.http.impl.io>
> io.PoolingHttpClientConnectionManager”. Just send the request
> synchronously and await the response … and the error we immediately
> get is: javax.net.ssl.SSLException: Unrecognized SSL message,
> plaintext connection?
> 
>  �
> 
> That’s ok. Now we have tried the real situation, sending a request
> asynchronously to the same endpoint using a “  javadoc:%E2%98%82=vpfw/web%5C/WEB-INF%5C/lib%5C/httpclient5-5.0-
> beta5.jar%3Corg> org.  INF%5C/lib%5C/httpclient5-5.0-beta5.jar%3Corg.apache> apache.
>  5.0-beta5.jar%3Corg.apache.hc> hc.  javadoc:%E2%98%82=vpfw/web%5C/WEB-INF%5C/lib%5C/httpclient5-5.0-
> beta5.jar%3Corg.apache.hc.client5> client5.  javadoc:%E2%98%82=vpfw/web%5C/WEB-INF%5C/lib%5C/httpclient5-5.0-
> beta5.jar%3Corg.apache.hc.client5.http> http.  javadoc:%E2%98%82=vpfw/web%5C/WEB-INF%5C/lib%5C/httpclient5-5.0-
> beta5.jar%3Corg.apache.hc.client5.http.impl> impl.  javadoc:%E2%98%82=vpfw/web%5C/WEB-INF%5C/lib%5C/httpclient5-5.0-
> beta5.jar%3Corg.apache.hc.client5.http.impl.nio>
> nio.PoolingAsyncClientConnectionManager” and trying to manage the
> response in a “  INF%5C/lib%5C/httpcore5-5.0-beta8.jar%3Corg> org.  javadoc:%E2%98%82=vpfw/web%5C/WEB-INF%5C/lib%5C/httpcore5-5.0-
> beta8.jar%3Corg.apache> apache.  javadoc:%E2%98%82=vpfw/web%5C/WEB-INF%5C/lib%5C/httpcore5-5.0-
> beta8.jar%3Corg.apache.hc> hc.  javadoc:%E2%98%82=vpfw/web%5C/WEB-INF%5C/lib%5C/httpcore5-5.0-
> beta8.jar%3Corg.apache.hc.core5> core5.  javadoc:%E2%98%82=vpfw/web%5C/WEB-INF%5C/lib%5C/httpcore5-5.0-
> beta8.jar%3Corg.apache.hc.core5.http> http.  javadoc:%E2%98%82=vpfw/web%5C/WEB-INF%5C/lib%5C/httpcore5-5.0-
> beta8.jar%3Corg.apache.hc.core5.http.nio> nio.AsyncResponseConsumer”.
> The result is the request never finishes (the thread is only released
> when the container asynchronous timeout is reached) and the
> connection is never returned to the pool. 
> 
>  �
> 
> This is the trace for the first request sent at 20:30:51 that
> finished after 120s when the container asynchronous timeout was
> reached:
> 
>  �
> 
> App trace:
> 
> 20:30:51.379,120419,response.noResponse,20:32:51.798
> 
>  �
> 
> Http5 trace:
> 
> 20:30:51.381 � � � ex-0004: preparing request execution
> 
> 20:30:51.381 � � � ex-0004: target auth state: UNCHALLENGED
> 
> 20:30:51.381 � � � ex-0004: proxy auth state: UNCHALLENGED
> 
> 20:30:51.381 � � � ex-0004: acquiring connection with route {s}->
> https://54.38.179.182:8080
> 
> 20:30:51.381 � � � ex-0004: acquiring endpoint (1 MILLISECONDS)
> 
> 20:30:51.382 � � � ex-0004: endpoint lease request (1
> MILLISECONDS) [route: {s}->https://54.38.179.182:8080][total kept
> alive: 0; route allocated: 0 of 50; total allocated: 0 of 50]
> 
> 20:30:51.382 � � � ex-0004: endpoint leased [route: {s}->
> https://54.38.179.182:8080][total kept alive: 0; route allocated: 1
> of 50; total allocated: 1 of 50]
> 
> 20:30:51.382 � � � ex-0004: acquired ep-0003
> 
> 20:30:51.382 � � � ex-0004: acquired endpoint ep-0003
> 
> 20:30:51.382 � � � ep-0003: connecting endpoint (2,000
> MILLISECONDS)
> 
> 20:30:51.382 � � � ep-0003: connecting endpoint to 
> https://54.38.179.182:8080 (2,000 MILLISECONDS)
> 
> 20:30:51.382 � � � https://54.38.179.182:8080: resolving remote
> address
> 
> 20:30:51.382 � � � https://54.38.179.182:8080: resolved to
> [/54.38.179.182]
> 
> 20:30:51.382 � � � 

Re: AsyncRequestBuilder deprecated

[Oleg Kalnichevski]

On Thu, 2019-08-01 at 10:34 +0200, Joan grupoventus wrote:
> Hello,
> 
>  �
> 
> It seems I should use the
> ‘org.apache.hc.core5.http.nio.support.AsyncRequestBuilder’ class.
> Replacing the import works except for the ‘setConfig’ method that we
> use to replace the connection/response timeout for each request sent
> to the backend. What should I use now to replace these 2 values at
> request level?
> 

Please use HttpClientContext#setRequestConfig instead.

Oleg

>  �
> 
> Thanks,
> 
> Joan.
> 
>  �
> 
> From: Joan grupoventus  
> Sent: Thursday, August 1, 2019 10:20 AM
> To: 'HttpClient User Discussion' 
> Subject: AsyncRequestBuilder deprecated
> 
>  �
> 
> Hello,
> 
>  �
> 
> With the latest HttpClient5 release the AsyncRequestBuilder class is
> deprecated. Checking the api docs it’s telling to use
> ‘AsyncRequestBuilder’
> 
>  �
> 
> org.apache.hc.client5.http.async.methods
> 
> 
> Class AsyncRequestBuilder
> 
> 
> * <
> https://docs.oracle.com/javase/7/docs/api/java/lang/Object.html?is-external=true
> > java.lang.Object
> 
> *org.apache.hc.client5.http.async.methods.AsyncRequestBuilder
> 
> Deprecated.  
> 
> Use  AsyncRequestBuilder
> 
>  �
> 
> What class should I use now?
> 
>  �
> 
> Thanks,
> 
>  �
> 
> Joan.
> 


-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

Re: How to use SSL CONNECT to the proxy with Apache HttpClient?

[Oleg Kalnichevski]

On Wed, 2019-07-31 at 11:32 +, Jochen Wiedmann wrote:
> 
> On 2019/07/31 10:48:37, Oleg Kalnichevski  wrote: 
> 
> > I am confused now. HttpClient always uses plain HTTP to connect to
> > the
> > first proxy hop. It upgrades to TLS only once the connection to the
> > request target has been fully established. Naturally all CONNECT
> > messages get transmitted in clear text on the wire.
> 
> Well, what I am observing, does sound like a CONNECT, which is done
> in okain HTTP, possibly followed by an "upgrade to TLS" (assuming
> that would mean the SSL negotiation with the remote server within the
> established plain text tunnel), doesn't it?
> 

It does. It is precisely what HTTP agents are expected to do per 
RFC2817:

https://tools.ietf.org/html/rfc2817

Oleg


-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

Re: How to use SSL CONNECT to the proxy with Apache HttpClient?

[Oleg Kalnichevski]

On Wed, 2019-07-31 at 10:29 +, Jochen Wiedmann wrote:
> 
> On 2019/07/31 10:06:40, Oleg Kalnichevski  wrote: 
> 
> > This is depends on the connection route defined by the route
> > planner.
> > If the connection route in marked as secure HttpClient will attempt
> > to
> > establish a tunnel through the proxy (by way of CONNECT request)
> > instead of passing the request the the proxy for execution. All
> > connection routes that use HTTPS scheme are considered secure by
> > default. 
> > 
> > Make sure the request you are trying to execute has been configured
> > to
> > use HTTPS.
> 
> Thanks very much, Oleg. In retrospective, there is one thing that
> puzzles me, and which you might be able to explain. I understand,
> that I can onfigure the HttpClient to use SSL for connecting to the
> proxy. But, if that is what SoapUI does: Why can I read the http
> headers of the CONNECT in plain text?
> 

I am confused now. HttpClient always uses plain HTTP to connect to the
first proxy hop. It upgrades to TLS only once the connection to the
request target has been fully established. Naturally all CONNECT
messages get transmitted in clear text on the wire.

Oleg   


> Thanks,
> 
> Jochen
> 
> 
> -
> To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
> For additional commands, e-mail: httpclient-users-h...@hc.apache.org
> 


-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

Re: How to use SSL CONNECT to the proxy with Apache HttpClient?

[Oleg Kalnichevski]

On Wed, 2019-07-31 at 11:31 +0200, Jochen Wiedmann wrote:
> I've got an application, which is sending a SOAP request to an
> external server via a corporate proxy, which I do not control.
> Sending
> the request fails with a message "HTTP/1.1 500 handshakefailed".
> However, sending another request via SoapUI through the same proxy
> works just fine.
> 
> In order to find a difference, I detected a surprising fact: Th
> Wireshark trace shows, that SoapUI is doing an HTTP CONNECT, rather
> than an HTTP POST. The complete headers look like this:
> 
> CONNECT proxy.company.com:443 HTTP/1.1
> Host: reverseproxy.othercompany.com
> Proxy-Connection: Keep-Alive
> User-Agent: Apache-HttpClient/4.1.1 (java 1.5)
> 
> The proxy servers response is
> 
> HTTP/1.0 Connection established.
> 
> The rest of the connection is mainly unreadable, except for some
> words, which make me believe, that an SSL Certificate is being passed
> through from the remote server to my client.
> 
> My impression is, that SoapUI establishes an SSL connection to the
> proxy server, although I did not configure it to do so. (In fact, the
> SoapUI preferences do not provide that option, AFAICT.) Perhaps
> SoapUI
> does so, because the remote URL is an https URL?
> 
> My question is: How can I configure the Apache HttpClient to do the
> same? Obviously, SoapUI does that, because the headers show, that it
> is using Apache HttpClient, too, albeit a different version.
> Currently, my proxy related code looks like this:
> 
> CredentialsProvider provider = new BasicCredentialsProvider();
> UsernamePasswordCredentials upc = new
>  UsernamePasswordCredentials(getUserName(), getPassword());
> if (getProxyUrl() != null  &&  getProxyUserName() != null) {
> provider.setCredentials(new
> AuthScope(getProxyUrl().getHost(),
> 
>  getProxyUrl().getPort()),
> new UsernamePasswordCredentials(getProxyUserName(),
> 
> getProxyPassword()));
> }
> provider.setCredentials(AuthScope.ANY, upc);
> 
> 
> Thanks in advance,
> 
> Jochen

Jochen

This is depends on the connection route defined by the route planner.
If the connection route in marked as secure HttpClient will attempt to
establish a tunnel through the proxy (by way of CONNECT request)
instead of passing the request the the proxy for execution. All
connection routes that use HTTPS scheme are considered secure by
default. 

Make sure the request you are trying to execute has been configured to
use HTTPS.

Hope this helps.

Oleg



-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

Re: org.apache.hc.core5.http.NoHttpResponseException: localhost:8112 failed to respond

[Oleg Kalnichevski]

urrent.BasicFuture.cancel(BasicFuture.java:
> > 154)
> > at
> > org.apache.hc.core5.concurrent.ComplexFuture.cancel(ComplexFuture.j
> > ava:93)
> > at
> > org.apache.hc.core5.concurrent.BasicFuture.cancel(BasicFuture.java:
> > 161)
> > at
> > org.apache.hc.client5.http.impl.nio.MultihomeIOSessionRequester$1$1
> > .cancelled(MultihomeIOSessionRequester.java:147)
> > at
> > org.apache.hc.core5.concurrent.BasicFuture.cancel(BasicFuture.java:
> > 154)
> > at
> > org.apache.hc.core5.concurrent.BasicFuture.cancel(BasicFuture.java:
> > 161)
> > at
> > org.apache.hc.core5.reactor.IOSessionRequest.cancel(IOSessionReques
> > t.java:83)
> > at
> > org.apache.hc.core5.reactor.SingleCoreIOReactor.closePendingConnect
> > ionRequests(SingleCoreIOReactor.java:366)
> > at
> > org.apache.hc.core5.reactor.SingleCoreIOReactor.doTerminate(SingleC
> > oreIOReactor.java:101)
> > at
> > org.apache.hc.core5.reactor.AbstractSingleCoreIOReactor.execute(Abs
> > tractSingleCoreIOReactor.java:89)
> > at
> > org.apache.hc.core5.reactor.IOReactorWorker.run(IOReactorWorker.jav
> > a:44)
> > 
> > Not always - most of the times it works fine, it is just sometimes.
> > 
> > 
> > На пн, 22.07.2019 г. в 12:43 ч. Oleg Kalnichevski  > >
> > написа:
> > 
> > > On Mon, 2019-07-22 at 12:29 +0300, Petar Tahchiev wrote:
> > > > Hello,
> > > > I have a Tomcat server running on port 8112 via HTTPS. The
> > > > Tomcat has
> > > > http2
> > > > enabled and is running with OpenJDK11.
> > > > Now when I try to connect with HTTPClient 5.0-beta5 I get this
> > > > exception:
> > > > ===
> > > > 2019-07-22 12:22:57,699 [https-jsse-nio-8443-exec-2] INFO :
> > > > Recoverable I/O
> > > > exception (org.apache.hc.core5.http.NoHttpResponseException)
> > > > caught
> > > > when
> > > > processing request to {s}->https://localhost:8112
> > > > 2019-07-22 12:22:57,709 [https-jsse-nio-8443-exec-2] INFO :
> > > > Recoverable I/O
> > > > exception (org.apache.hc.core5.http.NoHttpResponseException)
> > > > caught
> > > > when
> > > > processing request to {s}->https://localhost:8112
> > > > 2019-07-22 12:22:57,719 [https-jsse-nio-8443-exec-2] INFO :
> > > > Recoverable I/O
> > > > exception (org.apache.hc.core5.http.NoHttpResponseException)
> > > > caught
> > > > when
> > > > processing request to {s}->https://localhost:8112
> > > > 2019-07-22 12:22:57,732 [https-jsse-nio-8443-exec-2] ERROR:
> > > > localhost:8112
> > > > failed to respond
> > > > org.apache.hc.core5.http.NoHttpResponseException:
> > > > localhost:8112
> > > > failed to
> > > > respond
> > > > at
> > > > org.apache.hc.core5.http.impl.io.DefaultHttpResponseParser.crea
> > > > teConn
> > > > ectionClosedException(DefaultHttpResponseParser.java:87)
> > > > at
> > > > org.apache.hc.core5.http.impl.io.AbstractMessageParser.parse(Ab
> > > > stract
> > > > MessageParser.java:243)
> > > > at
> > > > org.apache.hc.core5.http.impl.io.AbstractMessageParser.parse(Ab
> > > > stract
> > > > MessageParser.java:53)
> > > > at
> > > > org.apache.hc.core5.http.impl.io.DefaultBHttpClientConnection.r
> > > > eceive
> > > > ResponseHeader(DefaultBHttpClientConnection.java:187)
> > > > at
> > > > org.apache.hc.core5.http.impl.io.HttpRequestExecutor.execute(Ht
> > > > tpRequ
> > > > estExecutor.java:181)
> > > > at
> > > > org.apache.hc.core5.http.impl.io.HttpRequestExecutor.execute(Ht
> > > > tpRequ
> > > > estExecutor.java:224)
> > > > at
> > > > org.apache.hc.client5.http.impl.io.PoolingHttpClientConnectionM
> > > > anager
> > > > $InternalConnectionEndpoint.execute(PoolingHttpClientConnection
> > > > Manage
> > > > r.java:596)
> > > > at
> > > > org.apache.hc.client5.http.impl.classic.InternalExecRuntime.exe
> > > > cute(I
> > > > nternalExecRuntime.java:220)
> > > > at
> > > > org.apache.hc.client5.http.impl.classic.MainClientExec.execute(
> > > > MainCl
> > > > ientExec.java:107)
> > > > at
> > > 

Re: org.apache.hc.core5.http.NoHttpResponseException: localhost:8112 failed to respond

[Oleg Kalnichevski]

On Mon, 2019-07-22 at 12:29 +0300, Petar Tahchiev wrote:
> Hello,
> I have a Tomcat server running on port 8112 via HTTPS. The Tomcat has
> http2
> enabled and is running with OpenJDK11.
> Now when I try to connect with HTTPClient 5.0-beta5 I get this
> exception:
> ===
> 2019-07-22 12:22:57,699 [https-jsse-nio-8443-exec-2] INFO :
> Recoverable I/O
> exception (org.apache.hc.core5.http.NoHttpResponseException) caught
> when
> processing request to {s}->https://localhost:8112
> 2019-07-22 12:22:57,709 [https-jsse-nio-8443-exec-2] INFO :
> Recoverable I/O
> exception (org.apache.hc.core5.http.NoHttpResponseException) caught
> when
> processing request to {s}->https://localhost:8112
> 2019-07-22 12:22:57,719 [https-jsse-nio-8443-exec-2] INFO :
> Recoverable I/O
> exception (org.apache.hc.core5.http.NoHttpResponseException) caught
> when
> processing request to {s}->https://localhost:8112
> 2019-07-22 12:22:57,732 [https-jsse-nio-8443-exec-2] ERROR:
> localhost:8112
> failed to respond
> org.apache.hc.core5.http.NoHttpResponseException: localhost:8112
> failed to
> respond
> at
> org.apache.hc.core5.http.impl.io.DefaultHttpResponseParser.createConn
> ectionClosedException(DefaultHttpResponseParser.java:87)
> at
> org.apache.hc.core5.http.impl.io.AbstractMessageParser.parse(Abstract
> MessageParser.java:243)
> at
> org.apache.hc.core5.http.impl.io.AbstractMessageParser.parse(Abstract
> MessageParser.java:53)
> at
> org.apache.hc.core5.http.impl.io.DefaultBHttpClientConnection.receive
> ResponseHeader(DefaultBHttpClientConnection.java:187)
> at
> org.apache.hc.core5.http.impl.io.HttpRequestExecutor.execute(HttpRequ
> estExecutor.java:181)
> at
> org.apache.hc.core5.http.impl.io.HttpRequestExecutor.execute(HttpRequ
> estExecutor.java:224)
> at
> org.apache.hc.client5.http.impl.io.PoolingHttpClientConnectionManager
> $InternalConnectionEndpoint.execute(PoolingHttpClientConnectionManage
> r.java:596)
> at
> org.apache.hc.client5.http.impl.classic.InternalExecRuntime.execute(I
> nternalExecRuntime.java:220)
> at
> org.apache.hc.client5.http.impl.classic.MainClientExec.execute(MainCl
> ientExec.java:107)
> at
> org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(Exec
> ChainElement.java:51)
> at
> org.apache.hc.client5.http.impl.classic.ExecChainElement$1.proceed(Ex
> ecChainElement.java:57)
> at
> org.apache.hc.client5.http.impl.classic.ConnectExec.execute(ConnectEx
> ec.java:181)
> at
> org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(Exec
> ChainElement.java:51)
> at
> org.apache.hc.client5.http.impl.classic.ExecChainElement$1.proceed(Ex
> ecChainElement.java:57)
> at
> org.apache.hc.client5.http.impl.classic.ProtocolExec.execute(Protocol
> Exec.java:165)
> at
> org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(Exec
> ChainElement.java:51)
> at
> org.apache.hc.client5.http.impl.classic.ExecChainElement$1.proceed(Ex
> ecChainElement.java:57)
> at
> org.apache.hc.client5.http.impl.classic.RetryExec.execute(RetryExec.j
> ava:88)
> at
> org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(Exec
> ChainElement.java:51)
> at
> org.apache.hc.client5.http.impl.classic.ExecChainElement$1.proceed(Ex
> ecChainElement.java:57)
> at
> org.apache.hc.client5.http.impl.classic.RedirectExec.execute(Redirect
> Exec.java:116)
> at
> org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(Exec
> ChainElement.java:51)
> at
> org.apache.hc.client5.http.impl.classic.ExecChainElement$1.proceed(Ex
> ecChainElement.java:57)
> at
> org.apache.hc.client5.http.impl.classic.ContentCompressionExec.execut
> e(ContentCompressionExec.java:125)
> at
> org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(Exec
> ChainElement.java:51)
> at
> org.apache.hc.client5.http.impl.classic.InternalHttpClient.doExecute(
> InternalHttpClient.java:175)
> at
> org.apache.hc.client5.http.impl.classic.CloseableHttpClient.execute(C
> loseableHttpClient.java:77)
> at
> com.nemesis.console.backend.storefront.DefaultRestAuthenticationProvi
> der.authenticate(DefaultRestAuthenticationProvider.java:116)
> ===
> 
>  - If I change the URL from 
> https://localhost:8112/storefront/rest/auth to
> https://some-website-with-valid-certificate.com/ it all works fine.
>  - If I change the Tomcat server to work with HTTP1.1 it all works
> fine.
>  Here is my code:
> ===
> 
> try {
> // Trust standard CA and those trusted by our custom strategy
> final SSLContext sslcontext =
> SSLContexts.custom().loadTrustMaterial(new TrustStrategy() {
> 
> @Override
> public boolean isTrusted(final X509Certificate[] chain, final
> String authType) throws CertificateException {
> return true;
> }
> 
> }).build();
> 
> int timeout = 5;
> 
> RequestConfig config =
> RequestConfig.custom().setResponseTimeout(timeout,
> TimeUnit.SECONDS).setConnectTimeout(timeout,
> TimeUnit.SECONDS).build();
> 
> // Allow TLSv1.2 

[ANNOUNCEMENT] HttpComponents Client 5.0 beta5 (GA candidate) released

[Oleg Kalnichevski]

The Apache HttpComponents project is pleased to announce 5.0-beta5
release of HttpComponents Client. 

This BETA release picks up the latest fixes and performance
improvements from HttpCore and addresses a number of issues found since
the previous BETA release.

IMPORTANT: This release is expected to be the last BETA version. If no
major design flaws are found the actual 5.0 API will be frozen and the
next version will be promoted to GA.


Notable changes and features in the 5.0 series:
---

* Support for the HTTP/2 protocol and conformance to requirements and
recommendations of the latest HTTP/2 protocol specification documents
(RFC 7540, RFC 7541.)

  Supported features:

** HPACK header compression
** Stream multiplexing (client and server)
** Flow control
** Response push
** Message trailers
** Expect-continue handshake
** Connection validation (ping)
** Application-layer protocol negotiation (ALPN)
** TLS 1.2 security features

* Improved conformance to requirements and recommendations of the
latest HTTP/1.1 protocol specification documents (RFC 7230, RFC 7231.)

* New connection pool implementation with lax connection limit
guarantees and better performance under higher concurrency due to
absence of a global pool lock.

* Package name space changed to 'org.apache.hc.client5'.

* Maven group id changed to 'org.apache.httpcomponents.client5'.

HttpClient 5.0 releases can be co-located with earlier major versions
on the same classpath due to the change in package names and Maven
module coordinates.

The 5.0 APIs are considered feature complete and are not expected to
undergo any major changes anymore. The focus of development is now
shifting to API polish, code stabilization and documentation
improvements.

Download - 
Release notes - <


https://www.apache.org/dist/httpcomponents/httpclient/RELEASE_NOTES-5.0.x.txt
> 

HttpComponents site - 

About HttpComponents HttpClient

The Hyper-Text Transfer Protocol (HTTP) is perhaps the most significant
protocol used on the Internet today. Web services, network-enabled
appliances and the growth of network computing continue to expand the
role of the HTTP protocol beyond user-driven web browsers, while
increasing the number of applications that require HTTP support.

Although the java.net package provides basic functionality for
accessing resources via HTTP, it doesn't provide the full flexibility
or functionality needed by many applications. HttpClient seeks to fill
this void by providing an efficient, up-to-date, and feature-rich
package implementing the client side of the most recent HTTP standards
and recommendations.

Designed for extension while providing robust support for the base HTTP
protocol, HttpClient may be of interest to anyone building HTTP-aware
client applications such as web browsers, web service clients, or
systems that leverage or extend the HTTP protocol for distributed
communication.



-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

Re: Changes in handling Cyrillic characters since 4.5.8

[Oleg Kalnichevski]

On Wed, 2019-07-17 at 15:23 +0600, Denis Malyshkin wrote:
> Hello,
> 
> After upgrade to HttpClient version 4.5.8+ we encountered that
> requests
> with Cyrillic characters are broken. Below is the simple test to
> expose the
> issue with HttpClient version 4.5.8:
> ===
> public void cyrillicSymbolsExtraTest() throws Exception {
>   String urlStr = "http://google.com/кириллица-2019/?q=кириллица-2019
> ";
>   URL url = new URL(urlStr);
>   HttpUriRequest req = new HttpGet(url.toString());
> 
>   // Prints "
> 
http://google.com/%D0%BA%D0%B8%D1%80%D0%B8%D0%BB%D0%BB%D0%B8%D1%86%D0%B0-2019/?q=%D0%BA%D0%B8%D1%80%D0%B8%D0%BB%D0%BB%D0%B8%D1%86%D0%B0-2019
> "
>   System.out.println(req.getRequestLine().getUri());
> 
>   HttpClientContext context = HttpClientContext.create();
>   HttpClient client = HttpClients.custom().build();
>   HttpResponse resp = client.execute(req, context);
> 
>   Assert.assertEquals(req.getRequestLine().getUri(), "
> http://google.com; +
> context.getRequest().getRequestLine().getUri());
> // Expected :
> 
http://google.com/%D0%BA%D0%B8%D1%80%D0%B8%D0%BB%D0%BB%D0%B8%D1%86%D0%B0-2019/?q=%D0%BA%D0%B8%D1%80%D0%B8%D0%BB%D0%BB%D0%B8%D1%86%D0%B0-2019
> // Actual   :
> 
http://google.com/:8@8;;8F0-2019/?q=%D0%BA%D0%B8%D1%80%D0%B8%D0%BB%D0%BB%D0%B8%D1%86%D0%B0-2019
> }
> ===
> 
> With HttpClient 4.5.7 the test is passed correctly.
> 
> Yes, I know that non-ASCII codes aren't allowed in URLs. But I worry
> about
> the next things in the listed above behavior:
> 
> 1. req.getRequestLine().getUri() returns the correctly URL-Encoded
> URI, but
> the request is sent to an address with an incorrect path -- "
> http://google.com/:8@8;;8F0-2019/;.
> 
> 2. If the URL is incorrect it seems very weird to me to send the
> request to
> a broken URL instead of returning an error.
> 
> 3. There is an inconsistency between the encoding of the URL path
> part and
> the URL query part -- the path part becomes broken while the query
> part is
> correctly URL-encoded.
> 

This is a classic case of "garbage in - garbage out" rule. Please do
not use invalid characters in URI components.

Oleg


-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

Re: NTLM Authentication fails

[Oleg Kalnichevski]

On Tue, 2019-07-16 at 16:21 +0200, Kirill wrote:
> Dear HtpClient Users Group,
> 
> 
> Is it possible to make NTLM Authentication using only HttpClient
> library
> (without htppclient-win library) which will be working independently
> on
> whether it is  running on Windows or Linux?
> 

Of course, it is. It is the default mode of operation supported by
HttpClient and recommended by the project.

Cheers

Oleg

> Best regards,
> Kirill
> 
> 
> 
> *Von:* Oleg Kalnichevski 
> 
> *Gesendet:* Mittwoch, 10. Juli 2019 15:49
> *An:* HttpClient User Discussion
> *Betreff:* Re: NTLM Authentication fails
> 
> 
> 
> On Mon, 2019-07-08 at 09:27 +, Kirill Rajbhandary wrote:
> > Dear HttpClient Support List,
> > 
> > I found out that when trying to make NTLM Authentication using
> > httpclient-win-4.5.6.jar library it relies on
> > CurrentWindowsCredentials instead of using credentials provided in
> > WindowsCredentialsProvider which it seems to be incorrect for the
> > case when web-container (Tomcat in my case) is running as a service
> > under another "Local System" user on Windows machine. It retrieves
> > incorrect username which is not authorized to pass NTLM
> > authentication and gets 401 Unauthorized Error. Besides, if web
> > container (Tomcat in my case) is running inside Docker Linux
> > Container it does not work at all because the user specified inside
> > Docker Container is completely different from the Windows one. I
> > suppose that in WindowsNegotiateScheme.authenticate() method the
> > below implementation should not rely on CurrentWindowsCredentials
> > and
> > throw Exception but have to use the Credentials specified in
> > WindowsCredentialsProvider.
> > 
> > if (clientCred == null) {
> > // ?? We don't use the credentials, should we allow anything?
> > if (!(credentials instanceof CurrentWindowsCredentials)) {
> > throw new InvalidCredentialsException(
> > "Credentials cannot be used for " + getSchemeName() + "
> > authentication: "
> > + credentials.getClass().getName());
> > }
> > 
> > 
> > Also WindowsCredentialsProvider should not use instance of
> > CurrentWindowsCredentials in case of AuthSchemes.NTLM but use
> > provider.getCredentials(authscope) one:
> > 
> > public Credentials getCredentials(final AuthScope authscope) {
> > final String scheme = authscope.getScheme();
> > if (AuthSchemes.NTLM.equalsIgnoreCase(scheme) ||
> > AuthSchemes.SPNEGO.equalsIgnoreCase(scheme)) {
> > return CurrentWindowsCredentials.INSTANCE;
> > } else {
> > return provider.getCredentials(authscope);
> > }
> > }
> > 
> > Besides, if user provides the credentials of another user which is
> > different from the user logged in to Windows system, httpclient-win
> > API should not try to get information about currently logged user
> > via
> > CurrentWindowsCredentials class but has to use those credentials
> > provided in WindowsCredentialsProvider if there are provided. If
> > the
> > credentials are not provided, then probably makes sense to get user
> > using CurrentWindowsCredentials.
> > 
> > Here is the code snippet how NTLM authentication was used in my
> > case
> > via httpclient-4.4.0.jar and httpclient-win-4.5.6.jar libraries:
> > 
> > HttpClientBuilder clientbuilder = HttpClients.custom();
> > Registry authSchemeRegistry =
> > RegistryBuilder.create()
> > .register(AuthSchemes.NTLM, new WindowsNTLMSchemeFactory(null))
> > .build();
> > CredentialsProvider windowsCredentialsProvider = new
> > WindowsCredentialsProvider(new SystemDefaultCredentialsProvider());
> > windowsCredentialsProvider.setCredentials(AuthScope.ANY, new
> > NTCredentials("username, "password", "workstation", "domain"));
> > clientbuilder.setDefaultCredentialsProvider(windowsCredentialsProvi
> > de
> > r);
> > clientbuilder.setDefaultAuthSchemeRegistry(authSchemeRegistry);
> > 
> > RequestConfig.Builder requestBuilder = RequestConfig.custom();
> > requestBuilder =
> > requestBuilder.setConnectTimeout(connectionTimeout);
> > requestBuilder =
> > requestBuilder.setConnectionRequestTimeout(connectionTimeout);
> > clientbuilder.setDefaultRequestConfig(requestBuilder.build());
> > client = clientbuilder.build();
> > 
> > HttpGet get = new HttpGet("http://test.url/ntlm;);
> > CloseableHttpResponse response = client.execute(get);
> > 
> > Could you please advise a workaround for the issue and make the
> >

Re: [HttpClient 4.5.5 ]Proxy-Authorization header missing while requesting HTTPS url

[Oleg Kalnichevski]

On Mon, 2019-07-15 at 15:02 +0530, Kaushal Kothari wrote:
> Just for clarification,I want to access the HTTPS url via proxy
> server and proxy server has the basic authentication set.
> 
> In order to authenticate,"Proxy-Authorization Basic "
> header should be sent along with CONNECT request but which is
> missing.
> 

It is missing because HttpClient has not been correctly configured to
respond the the authentication challenge.

you might have seen in last mail attached screenshot.If you are
> unable to see then let me know how can I send it to you again? 
>  
> So I created the POST request using the HttpPost class which is from
> the HttpClient API. HttpClient doesn't generate the "Proxy-
> Authorization Basic " header so added the same and pass
> it to HttpClient to extract it from the POST request.
> 

Why are you adding `Proxy-Authorization` to the POST request? This
makes no sense.

Please see HttpClient documentation, in particular, `Proxy
authentication` code example:

http://hc.apache.org/httpcomponents-client-4.5.x/examples.html

Oleg


>  request = new HttpPost();   //please refer screenshot in last mail
> for details in request object
>  request.addHeader("Proxy-Authorization","Basic "+encodedAuth);
>  httpResponse = httpClientBuilder.execute(request);
> 
> 
> 
> On Fri, Jul 12, 2019 at 5:55 PM Kaushal Kothari <
> kaushalkothari2...@gmail.com> wrote:
> > Hi Team,
> > 
> > I am trying to do POC using the HttpClient 4.5.5 library and want
> > to access HTTPS url masked through a http proxy server with basic
> > authentication
> > and its failing.
> > 
> > The authentication proxy header should be sent as part of the
> > connect but it is not happening.
> > 
> > I have also added the header "Proxy-Authorization Basic
> > " to POST request as it requires and passing request
> > object to httpclient through execute() method.
> > 
> > Apart from that preemptive authentication is also set.
> > 
> > 
> > 
> > 
> > But over the network, "Proxy-Authorization Basic  "
> > header is missing.
> > 
> > I tried same scenario to access the HTTP request everything works
> > fine.
> > 
> > On the fly [No "Proxy-Authorization" header]:
> > 
> > 
> > REQUEST:
> > 
> > CONNECT localhost:4080 HTTP/1.1
> > Host: 127.0.0.1:9090
> > User-Agent: Apache-HttpClient/4.5.5 (Java/1.8.0_191)
> > 
> > RESPONSE:
> > 
> > "HTTP/1.1 407 Proxy Authentication Required
> > Server: Proxy
> > Proxy-Authenticate: Basic realm=" Authorization"
> > Cache-control: no-cache
> > Connection: Close
> > Proxy-Connection: Close
> > Content-Length: 0"
> > 
> > Could you please help me on this issue? and let me know if you
> > require any further information.
> > 
> > -- 
> > Thanks,
> > Kaushal.


-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

[ANNOUNCEMENT] HttpComponents Core 5.0 beta8 (GA candidate) released

[Oleg Kalnichevski]

The Apache HttpComponents project is pleased to announce 5.0-beta8 (GA
candidate) release of HttpComponents Core. 

This BETA fixes a number of defects found since the last release and
adds several convenience factory and builder classes, mainly for TLS
configuration and HTTP message construction.

As of this version all server and requester implementations exclude
weak TLS protocol versions and ciphers.

IMPORTANT: This release is expected to be the last BETA version. If no
major design flaws are found the actual 5.0 API will be frozen and the
next version will be promoted to GA.


Notable changes and features included in the 5.0 series:

* Support for HTTP/2 protocol and conformance to requirements and
recommendations of the latest HTTP/2 protocol specification (RFC 7540,
RFC 7541)

  Supported features:

** HPACK header compression
** stream multiplexing (client and server)
** flow control
** response push (client and server)
** message trailers
** expect-continue handshake
** connection validation (ping)
** application-layer protocol negotiation (ALPN) on Java 9+
** TLS 1.2 security features

   Features out of scope for 5.0 release:

** padding of outgoing frames
** stream priority
** plain connection HTTP/1.1 upgrade
** CONNECT method

* Improved conformance to requirements and recommendations of the
latest HTTP/1.1 protocol specification (RFC 7230, RFC 7231)

* New asynchronous HTTP transport APIs consistent for both HTTP/1.1 and
HTTP/2 transport.

* Redesigned I/O reactor APIs and improved NIO based reactor
implementation for a greater performance and scalability.

* Support for server side request filters for classic and asynchronous
server implementations. Request filters could be used to implement
cross-cutting protocol aspects such as the 'expect-continue'
handshaking and user authentication / authorization.

* Redesigned connection pool implementation with strict connection
limit guarantees. The connection pool is expected to have a better
performance under higher concurrency due to reduced global pool lock
contention.

* New connection pool implementation with lax connection limit
guarantees and better performance under higher concurrency due to
absence of a global pool lock.

* Package name space changed to 'org.apache.hc.core5'

* Maven group id changed to 'org.apache.httpcomponents.core5'

HttpCore 5.0 releases can be co-located with earlier versions.

The 5.0 APIs are considered feature complete and are not expected to
undergo any major changes anymore. The focus of development is now
shifting to API polish, code stabilization and documentation
improvements.

Download - 
Release notes - 
HttpComponents site - 

About HttpComponents Core

HttpCore is a set of HTTP/1.1 and HTTP/2 transport components that can
be used to build custom client and server side HTTP services with a
minimal footprint



-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

Re: [HttpClient 4.5.5 ]Proxy-Authorization header missing while requesting HTTPS url

[Oleg Kalnichevski]

On Fri, 2019-07-12 at 17:55 +0530, Kaushal Kothari wrote:
> Hi Team,
> 
> I am trying to do POC using the HttpClient 4.5.5 library and want to
> access HTTPS url masked through a http proxy server with basic
> authentication
> and its failing.
> 
> The authentication proxy header should be sent as part of the connect
> but it is not happening.
> 
> I have also added the header "Proxy-Authorization Basic
> " to POST request as it requires and passing request
> object to httpclient through execute() method.
> 
> Apart from that preemptive authentication is also set.
> 
> 
> 
> 
> But over the network, "Proxy-Authorization Basic  "
> header is missing.
> 
> I tried same scenario to access the HTTP request everything works
> fine.
> 
> On the fly [No "Proxy-Authorization" header]:
> 
> 
> REQUEST:
> 
> CONNECT localhost:4080 HTTP/1.1
> Host: 127.0.0.1:9090
> User-Agent: Apache-HttpClient/4.5.5 (Java/1.8.0_191)
> 
> RESPONSE:
> 
> "HTTP/1.1 407 Proxy Authentication Required
> Server: Proxy
> Proxy-Authenticate: Basic realm=" Authorization"
> Cache-control: no-cache
> Connection: Close
> Proxy-Connection: Close
> Content-Length: 0"
> 
> Could you please help me on this issue? and let me know if you
> require any further information.
> 
> -- 
> Thanks,
> Kaushal.


Hi Kaushal

I am not sure I understand what your expectations are. You are manually
adding the Proxy-Authorization header to the wrong request message
instead of using the standard authentication API and letting HttpClient
generate the header for you.

Oleg


-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

Re: TLS 1.3 with Java 11

[Oleg Kalnichevski]

On Fri, 2019-07-12 at 16:07 +0900, Koji Kawamura wrote:
> Hi Oleg,
> 
> Thanks for your quick response as always.
> 
> After I upgraded JDK to 11.0.3, TLS 1.3 started working even with
> HttpAsyncClient 4.1.3 and HttpCore 4.5.3.
> Glad to know that the JDK issue has been fixed and back-ported.
> https://bugs.openjdk.java.net/browse/JDK-8212885
> 

Hi Koji

That is curious. I had to tweak non-blocking SSL code in HttpCore quite
a bit in order to get it to work with Java 11.0.2. But if 11.0.3 works
without any of those tweaks, it is even better. 

Cheers

Oleg


> Best,
> Koji
> 
> On Thu, Jul 11, 2019 at 7:01 PM Oleg Kalnichevski 
> wrote:
> > 
> > On Thu, 2019-07-11 at 16:19 +0900, Koji Kawamura wrote:
> > > Hello,
> > > 
> > > I found a comment on in the 5.0-BETA4 release note:
> > > 
> > > * Resolved compatibility issues with TLS 1.3 engine shipped with
> > > Java
> > > 11
> > >   Contributed by Oleg Kalnichevski 
> > > 
> > 
> > 
https://archive.apache.org/dist/httpcomponents/httpcore/RELEASE_NOTES-5.0.x.txt
> > > 
> > > Is there any JIRA specifically addressing this issue?
> > > I could only find this one, but the release note comment sounds
> > > like
> > > there are others.
> > > https://issues.apache.org/jira/browse/HTTPCLIENT-1967
> > > 
> > 
> > The only one that truly matters is this one
> > 
> > https://bugs.openjdk.java.net/browse/JDK-8212885
> > 
> > 
> > > Also, I'd like to know if TLS 1.3 with Java 11 is (or will be?)
> > > supported by HttpAsyncClient.
> > > 
> > 
> > HttpAsyncClient 4.1.4 should be fully TLSv1.3 / Java11 compatible
> > but
> > one needs to upgrade its HttpCore dependency to version 4.4.11.
> > 
> > Cheers
> > 
> > Oleg
> > 
> > 
> > 
> > -
> > 
> > To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
> > For additional commands, e-mail: 
> > httpclient-users-h...@hc.apache.org
> > 
> 
> -
> To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
> For additional commands, e-mail: httpclient-users-h...@hc.apache.org
> 


-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

Re: TLS 1.3 with Java 11

[Oleg Kalnichevski]

On Thu, 2019-07-11 at 16:19 +0900, Koji Kawamura wrote:
> Hello,
> 
> I found a comment on in the 5.0-BETA4 release note:
> 
> * Resolved compatibility issues with TLS 1.3 engine shipped with Java
> 11
>   Contributed by Oleg Kalnichevski 
> 
https://archive.apache.org/dist/httpcomponents/httpcore/RELEASE_NOTES-5.0.x.txt
> 
> Is there any JIRA specifically addressing this issue?
> I could only find this one, but the release note comment sounds like
> there are others.
> https://issues.apache.org/jira/browse/HTTPCLIENT-1967
> 

The only one that truly matters is this one

https://bugs.openjdk.java.net/browse/JDK-8212885


> Also, I'd like to know if TLS 1.3 with Java 11 is (or will be?)
> supported by HttpAsyncClient.
> 

HttpAsyncClient 4.1.4 should be fully TLSv1.3 / Java11 compatible but
one needs to upgrade its HttpCore dependency to version 4.4.11. 
 
Cheers

Oleg



-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

Re: NTLM Authentication fails

[Oleg Kalnichevski]

On Mon, 2019-07-08 at 09:27 +, Kirill Rajbhandary wrote:
> Dear HttpClient Support List,
> 
> I found out that when trying to make NTLM Authentication using
> httpclient-win-4.5.6.jar library it relies on
> CurrentWindowsCredentials instead of using credentials provided in
> WindowsCredentialsProvider which it seems to be incorrect for the
> case when web-container (Tomcat in my case) is running as a service
> under another "Local System" user on Windows machine. It retrieves
> incorrect username which is not authorized to pass NTLM
> authentication and gets 401 Unauthorized Error. Besides, if web
> container (Tomcat in my case) is running inside Docker Linux
> Container it does not work at all because the user specified inside
> Docker Container is completely different from the Windows one. I
> suppose that in WindowsNegotiateScheme.authenticate() method the
> below implementation should not rely on CurrentWindowsCredentials and
> throw Exception but have to use the Credentials specified in
> WindowsCredentialsProvider.
> 
> if (clientCred == null) {
> // ?? We don't use the credentials, should we allow anything?
> if (!(credentials instanceof CurrentWindowsCredentials)) {
> throw new InvalidCredentialsException(
> "Credentials cannot be used for " + getSchemeName() + "
> authentication: "
> + credentials.getClass().getName());
> }
> 
>
> Also WindowsCredentialsProvider should not use instance of
> CurrentWindowsCredentials in case of AuthSchemes.NTLM but use
> provider.getCredentials(authscope) one:
> 
> public Credentials getCredentials(final AuthScope authscope) {
> final String scheme = authscope.getScheme();
> if (AuthSchemes.NTLM.equalsIgnoreCase(scheme) ||
> AuthSchemes.SPNEGO.equalsIgnoreCase(scheme)) {
> return CurrentWindowsCredentials.INSTANCE;
> } else {
> return provider.getCredentials(authscope);
> }
> }
> 
> Besides, if user provides the credentials of another user which is
> different from the user logged in to Windows system, httpclient-win
> API should not try to get information about currently logged user via
> CurrentWindowsCredentials class but has to use those credentials
> provided in WindowsCredentialsProvider if there are provided. If the
> credentials are not provided, then probably makes sense to get user
> using CurrentWindowsCredentials.
> 
> Here is the code snippet how NTLM authentication was used in my case
> via httpclient-4.4.0.jar and httpclient-win-4.5.6.jar libraries:
> 
> HttpClientBuilder clientbuilder = HttpClients.custom();
> Registry authSchemeRegistry =
> RegistryBuilder.create()
> .register(AuthSchemes.NTLM, new WindowsNTLMSchemeFactory(null))
> .build();
> CredentialsProvider windowsCredentialsProvider = new
> WindowsCredentialsProvider(new SystemDefaultCredentialsProvider());
> windowsCredentialsProvider.setCredentials(AuthScope.ANY, new
> NTCredentials("username, "password", "workstation", "domain"));
> clientbuilder.setDefaultCredentialsProvider(windowsCredentialsProvide
> r);
> clientbuilder.setDefaultAuthSchemeRegistry(authSchemeRegistry);
> 
> RequestConfig.Builder requestBuilder = RequestConfig.custom();
> requestBuilder = requestBuilder.setConnectTimeout(connectionTimeout);
> requestBuilder =
> requestBuilder.setConnectionRequestTimeout(connectionTimeout);
> clientbuilder.setDefaultRequestConfig(requestBuilder.build());
> client = clientbuilder.build();
> 
> HttpGet get = new HttpGet("http://test.url/ntlm;);
> CloseableHttpResponse response = client.execute(get);
> 
> Could you please advise a workaround for the issue and make the
> corresponding fix if you consider my description as an issue?
> 

Hi Kirill

I am not sure I fully understand your logic here but also admittedly I
have little idea how things work in Windows these days. 

The HttpClient for Windows is an experimental module and is NOT
recommended for production use. On a number of occasions we have
considered dropping Windows specific code altogether.

However if you contribute a PR with your proposed improvements I will
happily review them.

Cheers

Oleg


-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

Re: Response timeout issue

[Oleg Kalnichevski]

On Wed, 2019-06-26 at 15:52 +0200, Joan grupoventus wrote:
> Hi Oleg,
> 
> Regarding this pending issue about response timeout, I could finally
> get traces.
> 
> Our response timeout is set to 2,000ms. This is a request that ended
> up with a response timeout:
> 62.xxx.xxx.xxx,23:59:47.606,A,1,A,5,A,4,B,,10.15.15.18:8081,,273,569,
> 0,776,http.responseTimeout,23:59:48.382
> 
> If we log this error that means the 'failed' method of our
> AsyncResponseConsumer received an exception that is an instance of a
> 'java.net.SocketTimeoutException'.
> 
> Here the trace in our error log:
> 2019-06-25 23:59:48 [62.xxx.xxx.xxx] ventusproxy : ToDi 2.1 :
> x : GetAccomodationAvail Response timeout
> java.net.SocketTimeoutException: 2,000 MILLISECONDS
> at
> org.apache.hc.core5.io.SocketTimeoutExceptionFactory.create(SocketTim
> eoutExceptionFactory.java:50)
> at
> org.apache.hc.core5.http.impl.nio.AbstractHttp1StreamDuplexer.onTimeo
> ut(AbstractHttp1StreamDuplexer.java:378)
> at
> org.apache.hc.core5.http.impl.nio.AbstractHttp1IOEventHandler.timeout
> (AbstractHttp1IOEventHandler.java:81)
> at
> org.apache.hc.core5.http.impl.nio.ClientHttp1IOEventHandler.timeout(C
> lientHttp1IOEventHandler.java:39)
> at
> org.apache.hc.core5.reactor.InternalDataChannel.onTimeout(InternalDat
> aChannel.java:188)
> at
> org.apache.hc.core5.reactor.InternalChannel.checkTimeout(InternalChan
> nel.java:67)
> at
> org.apache.hc.core5.reactor.SingleCoreIOReactor.checkTimeout(SingleCo
> reIOReactor.java:232)
> at
> org.apache.hc.core5.reactor.SingleCoreIOReactor.validateActiveChannel
> s(SingleCoreIOReactor.java:165)
> at
> org.apache.hc.core5.reactor.SingleCoreIOReactor.doExecute(SingleCoreI
> OReactor.java:127)
> at
> org.apache.hc.core5.reactor.AbstractSingleCoreIOReactor.execute(Abstr
> actSingleCoreIOReactor.java:81)
> at
> org.apache.hc.core5.reactor.IOReactorWorker.run(IOReactorWorker.java:
> 44)
> at java.lang.Thread.run(Thread.java:748)
> 
> The point is that the total time consumed by the request/response is
> just 776ms (end = 23:59:48.382, start = 23:59:47.606).
> 

Hi Juan

I found the culprit. Presently the internal data channel calculates the
deadline of read operation based on the last read access. Naturally the
last read operation timestamp on a persistent (kept-alive) HTTP
connection usually points back at the previous message exchange 

https://github.com/apache/httpcomponents-core/blob/master/httpcore5/src/main/java/org/apache/hc/core5/reactor/InternalChannel.java#L64

Please raise a JIRA for this defect.

Oleg


> Finally the http traces belonging to this request/response, where the
> start and end dates match perfectly:
> 23:59:47.606ex-017E91C3: preparing request execution
> 23:59:47.606ex-017E91C3: target auth state: UNCHALLENGED
> 23:59:47.606ex-017E91C3: proxy auth state: UNCHALLENGED
> 23:59:47.606ex-017E91C3: acquiring connection with route {}->
> http://10.15.15.18:8081
> 23:59:47.606ex-017E91C3: acquiring endpoint (1,000 MILLISECONDS)
> 23:59:47.606ex-017E91C3: endpoint lease request (1,000
> MILLISECONDS) [route: {}->http://10.15.15.18:8081][total kept alive:
> 131; route allocated: 2 of 1500; total allocated: 208 of 1500]
> 23:59:47.606ex-017E91C3: endpoint leased [route: {}->
> http://10.15.15.18:8081][total kept alive: 130; route allocated: 2 of
> 1500; total allocated: 208 of 1500]
> 23:59:47.606ex-017E91C3: acquired ep-017E91C2
> 23:59:47.606ex-017E91C3: acquired endpoint ep-017E91C2
> 23:59:47.606ex-017E91C3: executing POST /ApiServlet.Srv HTTP/1.1
> 23:59:47.606ep-017E91C2: start execution ex-017E91C3
> 23:59:47.606ep-017E91C2: executing exchange ex-017E91C3 over i/o-
> 00970788
> 23:59:47.607ex-017E91C3: send request POST /ApiServlet.Srv
> HTTP/1.1, entity len 810
> 23:59:47.607ex-017E91C3: produce request data
> 23:59:47.607ex-017E91C3: produce request data, len 810 bytes
> 23:59:47.607ex-017E91C3: end of request data
> 23:59:48.382ex-017E91C3: execution failed: 2,000 MILLISECONDS
> 23:59:48.382ex-017E91C3: request failed: 2,000 MILLISECONDS
> 23:59:48.382ep-017E91C2: close IMMEDIATE
> 23:59:48.382ep-017E91C2: endpoint closed
> 23:59:48.382ep-017E91C2: discarding endpoint
> 23:59:48.382ep-017E91C2: releasing endpoint
> 23:59:48.382ep-017E91C2: connection released [route: {}->
> http://10.15.15.18:8081][total kept alive: 106; route allocated: 1 of
> 1500; total allocated: 211 of 1500]
> 
> Important: if we are not wrong, it seems this is only happening when
> keep alive connections are used. If we disable keep alive from the
> pool, all response timeouts disappear.
> 
> Let me know if you need further information.
> 
> Thanks,
> Joan.
> 
> 
> 
> 
> -
> To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
> For 

Re: HTTP headers for Redirected HEAD requests

[Oleg Kalnichevski]

On Sun, 2019-06-23 at 13:32 +, Krishna Sankaran wrote:
>  Attached the curl output for the same URL.
> There is a 301, followed by 302 followed by 200.
>  Content-Type: text/html; charset=utf-8  is seen for the 301 and 302.
> and there is no Content-Length
> But finally, for the 200 i see the correct Content-Type and Length.
> Content-Type: video/mp4  <<<<
> Content-Length: 92894175   <<<<<
> I set the user agent to "some other string" to see if the server is
> doing something strange based on UserAgent.
> 

I am not sure what difference that makes. The server responds
differently to requests generated by curl. 

Oleg


> 
> curl -L -I  https://www.dropbox.com/s//test_out4.mp4
> HTTP/1.1 301 Moved Permanently
> Server: nginx
> Date: Sun, 23 Jun 2019 13:25:44 GMT
> Content-Type: text/html; charset=utf-8
> Connection: keep-alive
> Cache-Control: no-cache
> Content-Security-Policy: sandbox
> Location: /s/raw//test_out4.mp4
> Pragma: no-cache
> Referrer-Policy: origin-when-cross-origin
> Set-Cookie: locale=en; Domain=dropbox.com; expires=Fri, 21 Jun 2024
> 13:25:44 GMT; Path=/; secure
> Set-Cookie: gvc=MTA3NDcwOTg3MzE2NzIyNTE1MjMxNTg0OTM2OTA4NDcyMDExNzcx;
> expires=Fri, 21 Jun 2024 13:25:44 GMT; httponly; Path=/; secure
> Set-Cookie: flash=; Domain=dropbox.com; expires=Sun, 23 Jun 2019
> 13:25:44 GMT; Path=/; secure
> Set-Cookie: puc=; expires=Sun, 23 Jun 2019 13:25:44 GMT; httponly;
> Path=/; secure
> Set-Cookie: bang=; Domain=dropbox.com; expires=Sun, 23 Jun 2019
> 13:25:44 GMT; Path=/; secure
> Set-Cookie: t=dFl-48q8dhtsUpNl6OD627hR; Domain=dropbox.com;
> expires=Wed, 22 Jun 2022 13:25:44 GMT; httponly; Path=/; secure
> Set-Cookie: __Host-js_csrf=dFl-48q8dhtsUpNl6OD627hR; expires=Wed, 22
> Jun 2022 13:25:44 GMT; Path=/; secure
> X-Content-Type-Options: nosniff
> X-Dropbox-Request-Id: 95cdc3818032b5355abb335726e7093e
> X-Frame-Options: DENY
> X-Robots-Tag: noindex, nofollow, noimageindex
> X-Xss-Protection: 1; mode=block
> Strict-Transport-Security: max-age=15552000; includeSubDomains
> 
> HTTP/1.1 302 Found
> Server: nginx
> Date: Sun, 23 Jun 2019 13:25:44 GMT
> Content-Type: text/html; charset=utf-8
> Connection: keep-alive
> Cache-Control: no-cache
> Content-Security-Policy: sandbox
> Location: 
> https://.dl.dropboxusercontent.com/cd/0/inline/AjUbqHAmT0GH4EvTrTsHe6LtsGUcmKcxq__N0fAkvSybpqlIr_zo7f49YDzIcmr7O4cWwqH1Vp85Xd2Ro7S_wrWhux4LK2u_sNbF7-2qVwGLaw/file#
> Pragma: no-cache
> Referrer-Policy: origin-when-cross-origin
> Set-Cookie: locale=en; Domain=dropbox.com; expires=Fri, 21 Jun 2024
> 13:25:44 GMT; Path=/; secure
> Set-Cookie: gvc=Mjg5MTM5MTgzNzcwMDQ3MTE3ODE3ODEyMTIyODUwODg4MjI5Nzkw;
> expires=Fri, 21 Jun 2024 13:25:44 GMT; httponly; Path=/; secure
> Set-Cookie: flash=; Domain=dropbox.com; expires=Sun, 23 Jun 2019
> 13:25:44 GMT; Path=/; secure
> Set-Cookie: puc=; expires=Sun, 23 Jun 2019 13:25:44 GMT; httponly;
> Path=/; secure
> Set-Cookie: bang=; Domain=dropbox.com; expires=Sun, 23 Jun 2019
> 13:25:44 GMT; Path=/; secure
> Set-Cookie: t=GVbLM33tvYrFdQl3keuX6maj; Domain=dropbox.com;
> expires=Wed, 22 Jun 2022 13:25:44 GMT; httponly; Path=/; secure
> Set-Cookie: __Host-js_csrf=GVbLM33tvYrFdQl3keuX6maj; expires=Wed, 22
> Jun 2022 13:25:44 GMT; Path=/; secure
> X-Content-Type-Options: nosniff
> X-Dropbox-Request-Id: 8196ea9b627918cdc3057982fa5254fb
> X-Frame-Options: DENY
> X-Xss-Protection: 1; mode=block
> Strict-Transport-Security: max-age=15552000; includeSubDomains
> 
> HTTP/1.1 200 OK
> Server: nginx
> Date: Sun, 23 Jun 2019 13:25:44 GMT
> Content-Type: video/mp4  <<<<
> Content-Length: 92894175   <<<<<
> Connection: keep-alive
> content-disposition: inline; filename="test_out4.mp4"; filename*=UTF-
> 8''test_out4.mp4
> x-robots-tag: noindex, nofollow, noimageindex
> x-content-type-options: nosniff
> set-cookie:
> uc_session=pQdbyZ45ckq6BSRRKmfz247ugcFfboyCV55Od94quDvPkkGmSR5LfNhS8L
> f3nINL; Domain=dropboxusercontent.com; httponly; Path=/; secure
> accept-ranges: bytes
> content-security-policy: sandbox allow-forms allow-scripts allow-top-
> navigation allow-popups ; report-uri 
> https://www.dropbox.com/csp_log?policy_name=blockserver-usercontent
> content-security-policy: form-action 'none' ; report-uri 
> https://www.dropbox.com/csp_log?policy_name=blockserver-noscript ;
> script-src 'none'
> etag: 2d
> x-dropbox-request-id: f260f699a530d0694879d078aeffa804
> pragma: public
> cache-control: max-age=60
> referrer-policy: no-referrer
> Vary: Origin
> X-Server-Response-Time: 301
> Strict-Transport-Security: max-age=15552000; includeSubDomains
> On Sunday, June 23, 2019, 2:50:03 AM PDT, Oleg Kalnichevski <

Re: HTTP headers for Redirected HEAD requests

[Oleg Kalnichevski]

l 2019 17:13:32 GMT
> 2019/06/22 10:13:32:456 PDT [WARN] ResponseProcessCookies - Invalid
> cookie header: "Set-Cookie: locale=en; Domain=dropbox.com;
> expires=Thu, 20 Jun 2024 17:13:32 GMT; Path=/; secure". Invalid
> 'expires' attribute: Thu, 20 Jun 2024 17:13:32 GMT
> 2019/06/22 10:13:32:456 PDT [WARN] ResponseProcessCookies - Invalid
> cookie header: "Set-Cookie:
> gvc=Mzk5NDUxMzQzNjY3NzkzNDkwMTA4NTY4NzMxNzA3MDMyNzYzNDY%3D;
> expires=Thu, 20 Jun 2024 17:13:32 GMT; httponly; Path=/; secure".
> Invalid 'expires' attribute: Thu, 20 Jun 2024 17:13:32 GMT
> 2019/06/22 10:13:32:457 PDT [WARN] ResponseProcessCookies - Invalid
> cookie header: "Set-Cookie: flash=; Domain=dropbox.com; expires=Sat,
> 22 Jun 2019 17:13:32 GMT; Path=/; secure". Invalid 'expires'
> attribute: Sat, 22 Jun 2019 17:13:32 GMT
> 2019/06/22 10:13:32:457 PDT [WARN] ResponseProcessCookies - Invalid
> cookie header: "Set-Cookie: puc=; expires=Sat, 22 Jun 2019 17:13:32
> GMT; httponly; Path=/; secure". Invalid 'expires' attribute: Sat, 22
> Jun 2019 17:13:32 GMT
> 2019/06/22 10:13:32:457 PDT [WARN] ResponseProcessCookies - Invalid
> cookie header: "Set-Cookie: bang=; Domain=dropbox.com; expires=Sat,
> 22 Jun 2019 17:13:32 GMT; Path=/; secure". Invalid 'expires'
> attribute: Sat, 22 Jun 2019 17:13:32 GMT
> 2019/06/22 10:13:32:458 PDT [WARN] ResponseProcessCookies - Invalid
> cookie header: "Set-Cookie: t=NoGNAN4y5h84gjei3tzmqbGl;
> Domain=dropbox.com; expires=Tue, 21 Jun 2022 17:13:32 GMT; httponly;
> Path=/; secure". Invalid 'expires' attribute: Tue, 21 Jun 2022
> 17:13:32 GMT
> 2019/06/22 10:13:32:458 PDT [WARN] ResponseProcessCookies - Invalid
> cookie header: "Set-Cookie: __Host-js_csrf=NoGNAN4y5h84gjei3tzmqbGl;
> expires=Tue, 21 Jun 2022 17:13:32 GMT; Path=/; secure". Invalid
> 'expires' attribute: Tue, 21 Jun 2022 17:13:32 GMT
> 
> 
> 
> 
> On Friday, June 21, 2019, 9:03:41 AM PDT, Oleg Kalnichevski <
> ol...@apache.org> wrote:  
>  
>  On Fri, 2019-06-21 at 00:04 +, Krishna Sankaran wrote:
> > I am using the following java snippet for a HEAD request. The
> > request
> > gets redirected couple of times, a 301 followed by 302 and dinally
> > gets a 200.When i use the following code snippet, i get the status
> > as
> > 200 OK, however all the header fields are from the 1st response
> > (301). The Content-Length header is present in the last response,
> > but
> > i don't see that in the header's received by the code. Is there a
> > knob to get the last header or all headers?
> > 
> > private static final String SAMPLE_URL = "
> > https://www.dropbox.com/s//test_out4.mp4";
> > 
> > public static void main(String[] args) throws IOException,
> > URISyntaxException {
> > HttpClient client = HttpClientBuilder.create().build();
> > HttpHead request = new HttpHead(new URI(SAMPLE_URL));
> > HttpResponse response = client.execute(request);
> > System.out.println(response.getStatusLine());
> > for (Header header : response.getAllHeaders()) {
> > System.out.println(header.getName() + ": " +
> > header.getValue());
> > }
> > }
> > 
> > It returns the code as 200 OK, but the Content-Type is incorrect,
> > even 
> > the 
> > HTTP/1.1 200 OK <<< Status is 200 Server: nginx Date: Thu, 20 Jun
> > 2019 02:22:58 GMT Content-Type: text/html; charset=utf-8 << Content
> > type is char  <<<< Content-
> > Length 
> > is missing!
> > 
> > curl -I https://www.dropbox.com/s//test_out4.mp4 << Curl works
> > correctly
> > 
> > 
> > 
> >   HTTP/1.1 301 Moved Permanently <<< Status 301 Server: nginx 
> > Date: Thu, 20 Jun 2019 02:20:50 GMT Content-Type: text/html;
> > charset=utf-8 <<< Content type text Connection: keep-alive
> > 
> > HTTP/1.1 302 Found << second redirect Server: nginx 
> > Date: Thu, 20 Jun 2019 02:36:03 GMT 
> >   Content-Type: text/html; charset=utf-8
> > 
> > HTTP/1.1 200 OK <<< Status finally 200 Server: nginx 
> > Date: Thu, 20 Jun 2019 02:36:04 GMT 
> >   Content-Type: video/mp4 << content type correct 
> >   Content-Length: 92894175 << length correct Connection: keep-alive
> > 
> > 
> 
> Krishna
> 
> I _suppose_ this is due automatic content compression used by
> HttpClient per default. This can be easy to tell if you turn on
> HttpClient context / wire logging.
> 
> Try disabling automatic content compression.
> 
> ---
> CloseableHttpClient client = HttpClients.custom()
> .disableContentCompression()
> .build();
> ---
> 
> Oleg 
> 
> 
> -
> To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
> For additional commands, e-mail: httpclient-users-h...@hc.apache.org
> 
>   


-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

HttpClient 5.0 performance

[Oleg Kalnichevski]

Folks

I updated my (very unscientific and biased) HttpClient benchmark [1]
and got quite satisfactory results so far.

https://github.com/ok2c/httpclient-benchmark/wiki


Classic HttpClient 5.0 appears some 5 to 10% faster than HttpClient 4.5
and is presently the fastest.

HttpAsyncClient 5.0 appears to be 5 to 10% slower than HttpAsyncClient
4.1 but it does not look too bad as far as I am concerned and we have
not done _any_ performance optimization on the new 5.0 code yet. There
is still a change the performance gap can be reduced.

Feel free to take look and let me know if you get different results.

Oleg

[1] https://github.com/ok2c/httpclient-benchmark


-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

Re: HTTP headers for Redirected HEAD requests

[Oleg Kalnichevski]

On Fri, 2019-06-21 at 00:04 +, Krishna Sankaran wrote:
> I am using the following java snippet for a HEAD request. The request
> gets redirected couple of times, a 301 followed by 302 and dinally
> gets a 200.When i use the following code snippet, i get the status as
> 200 OK, however all the header fields are from the 1st response
> (301). The Content-Length header is present in the last response, but
> i don't see that in the header's received by the code. Is there a
> knob to get the last header or all headers?
> 
> private static final String SAMPLE_URL = "
> https://www.dropbox.com/s//test_out4.mp4";
> 
> public static void main(String[] args) throws IOException,
> URISyntaxException {
> HttpClient client = HttpClientBuilder.create().build();
> HttpHead request = new HttpHead(new URI(SAMPLE_URL));
> HttpResponse response = client.execute(request);
> System.out.println(response.getStatusLine());
> for (Header header : response.getAllHeaders()) {
> System.out.println(header.getName() + ": " +
> header.getValue());
> }
> }
> 
> It returns the code as 200 OK, but the Content-Type is incorrect,
> even 
> the 
> HTTP/1.1 200 OK <<< Status is 200 Server: nginx Date: Thu, 20 Jun
> 2019 02:22:58 GMT Content-Type: text/html; charset=utf-8 << Content
> type is char   Content-Length 
> is missing!
> 
> curl -I https://www.dropbox.com/s//test_out4.mp4 << Curl works
> correctly
> 
> 
> 
>  HTTP/1.1 301 Moved Permanently <<< Status 301 Server: nginx 
> Date: Thu, 20 Jun 2019 02:20:50 GMT Content-Type: text/html;
> charset=utf-8 <<< Content type text Connection: keep-alive
> 
> HTTP/1.1 302 Found << second redirect Server: nginx 
> Date: Thu, 20 Jun 2019 02:36:03 GMT 
>  Content-Type: text/html; charset=utf-8
> 
> HTTP/1.1 200 OK <<< Status finally 200 Server: nginx 
> Date: Thu, 20 Jun 2019 02:36:04 GMT 
>  Content-Type: video/mp4 << content type correct 
>  Content-Length: 92894175 << length correct Connection: keep-alive
> 
> 

Krishna

I _suppose_ this is due automatic content compression used by
HttpClient per default. This can be easy to tell if you turn on
HttpClient context / wire logging.

Try disabling automatic content compression.

---
CloseableHttpClient client = HttpClients.custom()
.disableContentCompression()
.build();
---

Oleg 


-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

[ANNOUNCEMENT] HttpComponents Client 4.5.9 GA Released

[Oleg Kalnichevski]

The Apache HttpComponents project is pleased to announce 4.5.9 GA
release of HttpComponents HttpClient.

This is a maintenance release that fixes a number defects discovered
since 4.5.8.

Download - 
Release notes - 

HttpComponents site - 

About HttpComponents HttpClient

The Hyper-Text Transfer Protocol (HTTP) is perhaps the most significant
protocol used on the Internet today. Web services, network-enabled
appliances and the growth of network computing continue to expand the
role of the HTTP protocol beyond user-driven web browsers, while
increasing the number of applications that require HTTP support.

Designed for extension while providing robust support for the base HTTP
protocol, HttpClient may be of interest to anyone building HTTP-aware
client applications such as web browsers, web service clients, or
systems that leverage or extend the HTTP protocol for distributed
communication.



-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

Re: Response timeout issue

[Oleg Kalnichevski]

On Fri, 2019-06-07 at 19:43 +0200, Joan grupoventus wrote:
> Hi,
> 
> > Now the current issue. Something similar is happening. After
> > several 
> > days of execution, requests start to throw an exception because
> > the 
> > container asynchronous timeout is reached. Tracing http, we saw
> > the 
> > pool exhausted ... so it's a behavior similar to issue nr. 1.
> > 08:14:03.054Connection request: [route: {}->
> > http://10.15.15.53:8083][total kept alive: 0; route allocated: 107
> > of 
> > 1500; total allocated: 1500 of 1500]
> > 
> 
> I could not find it anywhere in the logs that you have provided. 
> --> It's here, https://github.com/joanbalaguero/HttpClient , in the
> http.log I uploaded.
> 

But the pool is already been completely exhausted at the beginning of
the log

---
Connection request: [route: {}->http://10.15.15.22:8081][total kept
alive: 0; route allocated: 9 of 1500; total allocated: 1500 of 1500]
---

The log contains no information as to what has caused the pool
exhaustion in the first place. It is useless.


> > 
> I need to see a log of such 'hanging' message exchnage.
> --> Ok, to do this I need to enable the http traces for days waiting
> the "hang" to happen. But with the log I have enabled now there is a
> significant increase in cpu and io (because the responses are usually
> quite big). Is there any way to enable a kind of logging useful for
> you but avoiding trace the request and response messages?
> 

Try disabling I/O session and wire logging. That should massively
reduce the amount of chatter in the log.

---


















    
    




---

Oleg


> Joan.
> 
> 
> > Thanks,
> > 
> > Joan.
> > 
> > -Original Message-
> > From: Oleg Kalnichevski 
> > Sent: Friday, June 7, 2019 4:00 PM
> > To: HttpClient User Discussion 
> > Subject: Re: Response timeout issue
> > 
> > On Thu, 2019-06-06 at 18:56 +0200, Joan grupoventus wrote:
> > > Hi again,
> > > 
> > > Not sure if we have seen something strange in the current test
> > > with 
> > > the keep alive connections. It's just a GET method to get a gif 
> > > (this pool has no keep-alive, that means a 100ms keep-alive).
> > > Here 
> > > the complete http log: 
> > > https://github.com/joanbalaguero/HttpClient.git
> > > (it's the "http_get.log")
> > 
> > Joan
> > 
> > I am confused. I looked at http_get.log and I can see the exact
> > same 
> > number of connection lease requests and that of released
> > connections.
> > There are two 253 message exchanges and the same number of
> > connection 
> > lease / release operations.
> > 
> > I am not seeing any evidence of connections being leaked. I am
> > lost 
> > now. What is exactly the issue you are having?
> > 
> > Oleg
> > 
> > 
> > > Previously to this test, this pool received 4 requests today
> > > 06/06/2019 around 11:15:00 (so 6 hours before the test), these 4 
> > > requests ended up with a 4xx status code.
> > > 
> > > This is the first request we sent to this pool using this route.
> > > Should not keep-alive be 0 and total allocated 1 of 1000?
> > > ---
> > > 
> > > 
> > > 17:23:02.692ex-0105B9B8: preparing request execution
> > > 17:23:02.692Target auth state: UNCHALLENGED
> > > 17:23:02.692Proxy auth state: UNCHALLENGED
> > > 17:23:02.692ex-0105B9B8: acquiring connection with route {}->
> > > http://andalucia.viajesolympia.com:80
> > > 17:23:02.692Connection request: [route: {}->
> > > http://andalucia.viajesolympia.com:80][total kept alive: 1; route
> > > allocated: 0 of 1000; total allocated: 1 of 1000]
> > > 17:23:02.692Connection leased: [route: {}->
> > > http://andalucia.viajesolympia.com:80][total kept alive: 1; route
> > > allocated: 1 of 1000; total allocated: 2 of 1000]
> > > 17:23:02.692ep-010526BF: acquired null
> > > 17:23:02.692ex-0105B9B8: connection acquired
> > > 
> > > This is the second:
> > > 
> > > 17:23:02.954ex-0105B9B9: preparing request execution
> > > 17:23:02.955Target auth state: UNCHALLENGED
> > > 17:23:02.955Proxy auth state: UNCHALLENGED
> > &

Re: Response timeout issue

[Oleg Kalnichevski]

On Fri, 2019-06-07 at 19:06 +0200, Joan grupoventus wrote:
> Hi Oleg,
> 
> Ok, let me explain a bit more. This comes from a previous mistake in
> our AsyncResponseConsumer.
> 
> Some weeks ago I wrote you an email because we were having an issue
> where threads got stuck, streamEnd and failed methods were never
> called, and the thread was released when the container asynchronous
> timeout was reached. As you notice, the issue was in our
> updateCapacity method because we were doing nothing on it, instead of
> extending the capacity of the buffer. This was fixed.
> 
> Now the current issue. Something similar is happening. After several
> days of execution, requests start to throw an exception because the
> container asynchronous timeout is reached. Tracing http, we saw the
> pool exhausted ... so it's a behavior similar to issue nr. 1. 
> 08:14:03.054Connection request: [route: {}->
> http://10.15.15.53:8083][total kept alive: 0; route allocated: 107 of
> 1500; total allocated: 1500 of 1500]
> 

I could not find it anywhere in the logs that you have provided. 


> That's why I asked you if it's possible the 'updateCapacity' method
> to be called but without previously calling the 'consumeResponse'
> method with a non-null entity.

No, I do not think it is possible.


>  In this case the variable 'this.capacityToPerform' we use to extend
> the buffer will be 0 and we will be again in the first issue.
> 
> So I think the pool exhausted is a consequence the thread that is
> managing the 'AsyncResponseConsumer' is not ending for some reason
> (like in the first issue).
> 

Again, I need to see a _complete_ wire / context log of such message
exchange.

Is there something else (like updating the capacity) I have to take
> into account in the ' AsyncResponseConsumer ' to avoid hanging the
> execution?
> 

I need to see a log of such 'hanging' message exchnage.

Oleg


> Thanks,
> 
> Joan.
> 
> -Original Message-
> From: Oleg Kalnichevski  
> Sent: Friday, June 7, 2019 4:00 PM
> To: HttpClient User Discussion 
> Subject: Re: Response timeout issue
> 
> On Thu, 2019-06-06 at 18:56 +0200, Joan grupoventus wrote:
> > Hi again,
> > 
> > Not sure if we have seen something strange in the current test
> > with 
> > the keep alive connections. It's just a GET method to get a gif
> > (this 
> > pool has no keep-alive, that means a 100ms keep-alive). Here the 
> > complete http log: https://github.com/joanbalaguero/HttpClient.git
> > (it's the "http_get.log")
> 
> Joan
> 
> I am confused. I looked at http_get.log and I can see the exact same
> number of connection lease requests and that of released connections.
> There are two 253 message exchanges and the same number of connection
> lease / release operations. 
> 
> I am not seeing any evidence of connections being leaked. I am lost
> now. What is exactly the issue you are having?
> 
> Oleg 
> 
> 
> > Previously to this test, this pool received 4 requests today
> > 06/06/2019 around 11:15:00 (so 6 hours before the test), these 4 
> > requests ended up with a 4xx status code.
> > 
> > This is the first request we sent to this pool using this route.
> > Should not keep-alive be 0 and total allocated 1 of 1000?
> > ---
> > 
> > 17:23:02.692ex-0105B9B8: preparing request execution
> > 17:23:02.692Target auth state: UNCHALLENGED
> > 17:23:02.692Proxy auth state: UNCHALLENGED
> > 17:23:02.692ex-0105B9B8: acquiring connection with route {}->
> > http://andalucia.viajesolympia.com:80
> > 17:23:02.692Connection request: [route: {}->
> > http://andalucia.viajesolympia.com:80][total kept alive: 1; route
> > allocated: 0 of 1000; total allocated: 1 of 1000]
> > 17:23:02.692Connection leased: [route: {}->
> > http://andalucia.viajesolympia.com:80][total kept alive: 1; route
> > allocated: 1 of 1000; total allocated: 2 of 1000]
> > 17:23:02.692ep-010526BF: acquired null
> > 17:23:02.692ex-0105B9B8: connection acquired
> > 
> > This is the second:
> > 
> > 17:23:02.954ex-0105B9B9: preparing request execution
> > 17:23:02.955Target auth state: UNCHALLENGED
> > 17:23:02.955Proxy auth state: UNCHALLENGED
> > 17:23:02.955ex-0105B9B9: acquiring connection with route {}->
> > http://andalucia.viajesolympia.com:80
> > 17:23:02.955Connection request: [route: {}->
> > http://andalucia.viajesolympia.com:80][total kept alive: 2; route
> > allocated: 1 of 1000; total allocated: 2 of 1000] 
> &