Hi Oleg,
Real problem was in our customized AbstractClassicEntityProducer
class produce method, buffer flush was missing which was causing the
issue (*Connection
Closed* in case of GET requests and *Connection forcibly closed* in case of
POST requests). If we don't add a request method check for skipping adding
of entity to request, httpclient produces ContentLength header with 0 (In
*RequestContent.java* process method there is a check for entity and if
entity present it adds ContentLength header by default). Currently I see
the only way to avoid not adding ContentLength header is to not set an
entity to request in specific conditions (i.e., GET requests etc.,). Is
there any other way we can make httpclient skip adding ContentLength header
with 0?
Here is sample code with below modifications to actual code being used
- ContentType, ProtocolVersion and inputstream (i.e., Content) we get as
input from servlet
- AbstractClassicEntityProducer has been customized to generate content
length as well and is not using any executor instead produceData API is
predefined in that custom class
*Code*
HttpHost host = new HttpHost("host", "/printenv.pl", 80);
InputStream content = new BufferedInputStream(<Stream from
servlet>);
AsyncRequestProducer request =
AsyncRequestBuilder.get().setHttpHost(host).setEntity(
new AbstractClassicEntityProducer(1024,
ContentType.TEXT_XML, Executors.newSingleThreadExecutor()) {
@Override
protected void produceData(ContentType ct, OutputStream
os) throws IOException {
if (content != null) {
byte[] b = new byte[1024];
int bytesRead = 0;
while (content.read(b) != -1) {
os.write(b, bytesRead, b.length);
bytesRead += b.length;
}
}
}
}).build();
HttpClientContext clientContext = HttpClientContext.create();
CookieStore cookieStore = new BasicCookieStore();
clientContext.setCookieStore(cookieStore);
clientContext.setProtocolVersion(HttpVersion.HTTP_1_1);
try (CloseableHttpAsyncClient httpClient =
HttpAsyncClients.createDefault()) {
httpClient.execute(request, new BasicResponseConsumer<>(
new AbstractClassicEntityConsumer<byte[]>(1024,
Executors.newSingleThreadExecutor()) {
@Override
protected byte[] consumeData(ContentType ct,
InputStream is) throws IOException {
ByteArrayOutputStream baos = new
ByteArrayOutputStream();
byte[] b = new byte[1024];
int bytesRead = 0;
while (is.read(b) != -1) {
baos.write(b, bytesRead, b.length);
bytesRead += b.length;
}
return baos.toByteArray();
}
}), null, clientContext, null);
} finally {
if (content != null) {
content.close();
}
}
On Thu, Feb 23, 2023 at 12:12 AM Oleg Kalnichevski <ol...@apache.org> wrote:
> On Wed, 2023-02-22 at 16:18 +0530, sreenivas somavarapu wrote:
> > Hi Oleg,
> >
> > Thank you for the Content-Length query which helped in determining
> > the
> > actual cause of the issue. The problem was with sample code which is
> > adding
> > entity object to request builder for all methods.
>
>
> Could you please give the link to that sample code?
>
> > Once added a hardcoded
> > check to create and add entity (i.e., instance of
> > *AbstractClassicEntityProducer*) only for POST, PATCH and PUT methods
> > the
> > issue disappeared and we are able to get the response and now even
> > Content-Length header is not being sent in the request.
> >
>
> The problem was not so much the Content-Length header, as the custom
> message exchange handler terminating the request message body
> incorrectly.
>
>
> > In 5.x sync implementation it seems all methods extend the
> > *HttpUriRequestBase* class and there is no way to avoid adding an
> > entity to
> > a request object.
>
> Of course, there is. An entity can simply be null.
>
>
> > In 4.5.x we have all entity available methods extending
> > *HttpEntityEnclosingRequestBase* and based on this we could determine
> > whether to add an entity to a request object or not. Is there
> > anything
> > similar methodology available for Async implementation or 5.x
> > httpclient
> > implementation as well through which we can determine for which
> > request
> > method instances we need to add entity objects to requestbuilder
> > instead of
> > the current hardcoded check for few methods?
> >
>
> Unlike HC 4.x, HC 5.x conforms to RFC 7230 that makes it technically
> valid for all HTTP methods to have a message body (except TRACE. Oh
> well). This however does not force one to use request bodies with GET
> and DELETE requests where a message body makes no sense.
>
> Oleg
>
> > Regards,
> > Sreenivas
> >
> > On Wed, Feb 22, 2023 at 2:38 AM Oleg Kalnichevski <ol...@apache.org>
> > wrote:
> >
> > > On Wed, 2023-02-22 at 01:27 +0530, sreenivas somavarapu wrote:
> > > > Hi Team,
> > > >
> > > >
> > > > I have a perl script deployed on IIS server. When accessing this
> > > > perl
> > > > script we are getting *Connection closed* error (see below for
> > > > stack
> > > > trace)
> > > > from Async HTTPClient 5.2.x future callback. When further
> > > > debugged
> > > > found
> > > > that IIS is sending “Connection: Close” header after the response
> > > > is
> > > > sent
> > > > due to which http client is throwing the said error even after
> > > > reading /
> > > > processing the response. This is resulting in response lost what
> > > > IIS
> > > > has
> > > > sent. When I access the same resource using browser it is able to
> > > > display
> > > > the content. Is there any mechanism to handle these kind of
> > > > responses?
> > > >
> > > >
> > >
> > > HttpClient by itself handles 'Connection: Close' correctly but your
> > > custom message exchange handling logic appears wrong. What I see
> > > from
> > > the exception stack trace is the client is still busy sending the
> > > request body when the server sends back the response and drops the
> > > connection on it, and the entire message exchange fails with the
> > > ConnectionClosedException as a result.
> > >
> > > And why on earth are you sending 'Content-Length: 0' with a GET
> > > request? Basically your request producing logic appears wrong.
> > >
> > > Oleg
> > >
> > >
> > > >
> > > > *Async HTTPClient Code*
> > > >
> > > > Future<Message<HttpResponse, String>> responseHandler =
> > > > httpAsyncClient.execute(reqBuilder.build(),
> > > >
> > > > new BasicResponseConsumer<>(new
> > > > AbstractClassicEntityConsumer<String>(){…}, null, clientContext,
> > > > null);
> > > >
> > > > *Message<HttpResponse, String> response = responseHandler.get();*
> > > >
> > > > retCode = response.getHead().getCode();
> > > >
> > > >
> > > >
> > > > *Stack Trace / Exception stack*
> > > >
> > > > org.apache.hc.core5.http.ConnectionClosedException: Connection is
> > > > closed
> > > >
> > > > at
> > > > org.apache.hc.core5.http.impl.nio.ClientHttp1StreamDuplexer.disco
> > > > nnec
> > > > ted(ClientHttp1StreamDuplexer.java:212)
> > > >
> > > > at
> > > > org.apache.hc.core5.http.impl.nio.AbstractHttp1StreamDuplexer.onD
> > > > isco
> > > > nnect(AbstractHttp1StreamDuplexer.java:409)
> > > >
> > > > at
> > > > org.apache.hc.core5.http.impl.nio.AbstractHttp1IOEventHandler.dis
> > > > conn
> > > > ected(AbstractHttp1IOEventHandler.java:95)
> > > >
> > > > at
> > > > org.apache.hc.core5.http.impl.nio.ClientHttp1IOEventHandler.disco
> > > > nnec
> > > > ted(ClientHttp1IOEventHandler.java:41)
> > > >
> > > > at
> > > > org.apache.hc.client5.http.impl.async.LoggingIOSession$1.disconne
> > > > cted
> > > > (LoggingIOSession.java:258)
> > > >
> > > > at
> > > > org.apache.hc.core5.reactor.InternalDataChannel.disconnected(Inte
> > > > rnal
> > > > DataChannel.java:204)
> > > >
> > > > at
> > > > org.apache.hc.core5.reactor.SingleCoreIOReactor.processClosedSess
> > > > ions
> > > > (SingleCoreIOReactor.java:231)
> > > >
> > > > at
> > > > org.apache.hc.core5.reactor.SingleCoreIOReactor.doExecute(SingleC
> > > > oreI
> > > > OReactor.java:133)
> > > >
> > > > at
> > > > org.apache.hc.core5.reactor.AbstractSingleCoreIOReactor.execute(A
> > > > bstr
> > > > actSingleCoreIOReactor.java:86)
> > > >
> > > > at
> > > > org.apache.hc.core5.reactor.IOReactorWorker.run(IOReactorWorker.j
> > > > ava:
> > > > 44)
> > > >
> > > > at java.lang.Thread.run(Thread.java:748)
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > *HTTPClient log extract*
> > > >
> > > > 2023-02-21 15:37:59,600 DEBUG
> > > > [org.apache.hc.client5.http.impl.async.HttpAsyncMainClientExec]
> > > > ex-0000000003 send request GET /sample/ldapUD/16239/printenv.pl
> > > > HTTP/1.1,
> > > > entity len 0
> > > >
> > > > 2023-02-21 15:37:59,600 DEBUG
> > > > [org.apache.hc.client5.http.headers]
> > > > c-0000000002 >> GET /printenv.pl HTTP/1.1
> > > >
> > > > 2023-02-21 15:37:59,600 DEBUG
> > > > [org.apache.hc.client5.http.headers]
> > > > c-0000000002 >> Accept-Encoding: gzip, deflate
> > > >
> > > > 2023-02-21 15:37:59,600 DEBUG
> > > > [org.apache.hc.client5.http.headers]
> > > > c-0000000002 >> SMSDOMAIN: .abc.net
> > > >
> > > > 2023-02-21 15:37:59,600 DEBUG
> > > > [org.apache.hc.client5.http.headers]
> > > > c-0000000002 >> accept:
> > > > text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,
> > > > imag
> > > > e/webp,image/apng,*/*;q=0.8,application/signed-
> > > > exchange;v=b3;q=0.7
> > > >
> > > > 2023-02-21 15:37:59,600 DEBUG
> > > > [org.apache.hc.client5.http.headers]
> > > > c-0000000002 >> Cache-Control: max-age=0
> > > >
> > > > 2023-02-21 15:37:59,600 DEBUG
> > > > [org.apache.hc.client5.http.headers]
> > > > c-0000000002 >> Upgrade-Insecure-Requests: 1
> > > >
> > > > 2023-02-21 15:37:59,600 DEBUG
> > > > [org.apache.hc.client5.http.headers]
> > > > c-0000000002 >> *connection: keep-alive*
> > > >
> > > > 2023-02-21 15:37:59,600 DEBUG
> > > > [org.apache.hc.client5.http.headers]
> > > > c-0000000002 >> Accept-Language: en-US,en;q=0.9
> > > >
> > > > 2023-02-21 15:37:59,600 DEBUG
> > > > [org.apache.hc.client5.http.headers]
> > > > c-0000000002 >> user-agent: Mozilla/5.0 (Windows NT 10.0; Win64;
> > > > x64)
> > > > AppleWebKit/537.36 (KHTML, like Gecko) Chrome/110.0.0.0
> > > > Safari/537.36
> > > >
> > > > 2023-02-21 15:37:59,600 DEBUG
> > > > [org.apache.hc.client5.http.headers]
> > > > c-0000000002 >> Content-Length: 0
> > > >
> > > > 2023-02-21 15:37:59,600 DEBUG
> > > > [org.apache.hc.client5.http.headers]
> > > > c-0000000002 >> Host: iisserver
> > > >
> > > > 2023-02-21 15:37:59,600 DEBUG
> > > > [org.apache.hc.client5.http.impl.async.HttpAsyncMainClientExec]
> > > > ex-0000000003: produce request data
> > > >
> > > > ….
> > > >
> > > > ….
> > > >
> > > > ….
> > > >
> > > > 2023-02-21 15:37:59,647 DEBUG [org.apache.hc.client5.http.wire]
> > > > c-0000000002[ACTIVE][r:r] << 7N. .s..c... 37 4e fe 1f e3 73
> > > > 1b
> > > > a9 63
> > > > 10 00 00
> > > >
> > > > 2023-02-21 15:37:59,647 DEBUG
> > > > [org.apache.hc.client5.http.headers]
> > > > c-0000000002 << HTTP/1.1 200 OK
> > > >
> > > > 2023-02-21 15:37:59,647 DEBUG
> > > > [org.apache.hc.client5.http.headers]
> > > > c-0000000002 << Content-Type: text/plain; charset=iso-8859-1
> > > >
> > > > 2023-02-21 15:37:59,647 DEBUG
> > > > [org.apache.hc.client5.http.headers]
> > > > c-0000000002 << Content-Encoding: gzip
> > > >
> > > > 2023-02-21 15:37:59,647 DEBUG
> > > > [org.apache.hc.client5.http.headers]
> > > > c-0000000002 << Vary: Accept-Encoding
> > > >
> > > > 2023-02-21 15:37:59,647 DEBUG
> > > > [org.apache.hc.client5.http.headers]
> > > > c-0000000002 << Server: Microsoft-IIS/10.0
> > > >
> > > > 2023-02-21 15:37:59,647 DEBUG
> > > > [org.apache.hc.client5.http.headers]
> > > > c-0000000002 << set-cookie: APPSESSION=abcd;Domain=.test.com;
> > > > Path=/default/xyz
> > > >
> > > > 2023-02-21 15:37:59,647 DEBUG
> > > > [org.apache.hc.client5.http.headers]
> > > > c-0000000002 << X-Powered-By: ASP.NET
> > > >
> > > > 2023-02-21 15:37:59,647 DEBUG
> > > > [org.apache.hc.client5.http.headers]
> > > > c-0000000002 << Date: Tue, 28 Feb 2023 03:41:23 GMT
> > > >
> > > > 2023-02-21 15:37:59,647 DEBUG
> > > > [org.apache.hc.client5.http.headers]
> > > > c-0000000002 << *Connection: close*
> > > >
> > > > 2023-02-21 15:37:59,647 DEBUG
> > > > [org.apache.hc.client5.http.headers]
> > > > c-0000000002 << Content-Length: 2170
> > > >
> > > > 2023-02-21 15:37:59,647 DEBUG
> > > > [org.apache.hc.client5.http.impl.async.HttpAsyncMainClientExec]
> > > > ex-0000000003: consume response HTTP/1.1 200 OK, entity len 2170
> > > >
> > > > 2023-02-21 15:37:59,647 DEBUG
> > > > [org.apache.hc.client5.http.impl.async.HttpAsyncMainClientExec]
> > > > ex-0000000003: consume response data, len 2170 bytes
> > > >
> > > > 2023-02-21 15:37:59,647 DEBUG
> > > > [org.apache.hc.client5.http.impl.async.HttpAsyncMainClientExec]
> > > > ex-0000000003: end of response data
> > > >
> > > > 2023-02-21 15:37:59,647 DEBUG
> > > > [org.apache.hc.client5.http.impl.async.InternalHttpAsyncClient]
> > > > c-0000000002 Connection is not kept alive
> > > >
> > > > 2023-02-21 15:37:59,647 DEBUG
> > > > [org.apache.hc.client5.http.impl.async.HttpAsyncMainClientExec]
> > > > ex-0000000003: execution failed: Connection is closed
> > > >
> > > > 2023-02-21 15:37:59,647 DEBUG
> > > > [org.apache.hc.client5.http.impl.async.InternalAbstractHttpAsyncC
> > > > lien
> > > > t]
> > > > ex-0000000003 request failed: Connection is closed
> > > >
> > > > 2023-02-21 15:37:59,647 DEBUG
> > > > [org.apache.hc.client5.http.impl.nio.PoolingAsyncClientConnection
> > > > Mana
> > > > ger]
> > > > ep-0000000003 close IMMEDIATE
> > > >
> > > > 2023-02-21 15:37:59,647 DEBUG
> > > > [org.apache.hc.client5.http.impl.nio.DefaultManagedAsyncClientCon
> > > > nect
> > > > ion]
> > > > c-0000000002 Shutdown connection IMMEDIATE
> > > >
> > > > 2023-02-21 15:37:59,647 DEBUG
> > > > [org.apache.hc.client5.http.impl.async.InternalHttpAsyncClient]
> > > > ep-0000000003 endpoint closed
> > > >
> > > > 2023-02-21 15:37:59,647 DEBUG
> > > > [org.apache.hc.client5.http.impl.async.InternalHttpAsyncClient]
> > > > ep-0000000003 discarding endpoint
> > > >
> > > > 2023-02-21 15:37:59,647 DEBUG
> > > > [org.apache.hc.client5.http.impl.nio.PoolingAsyncClientConnection
> > > > Mana
> > > > ger]
> > > > ep-0000000003 releasing endpoint
> > > >
> > > >
> > >
> > >
> > > -------------------------------------------------------------------
> > > --
> > > To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
> > > For additional commands, e-mail:
> > > httpclient-users-h...@hc.apache.org
> > >
> > >
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
> For additional commands, e-mail: httpclient-users-h...@hc.apache.org
>
>
--
Cheers,
S. Sreenivas
