I am using the Maven httpclient 4.5.6 artifact and trying to make a PUT
request to a server. The server for some URLs requires authentication, it
is impossible to know which URLs do and don't ahead of time. I only want to
send the credentials if the server replies 401 Unauthorized.
I am trying to do a HTTP PUT to the server. If the server replies HTTP 401
Unauthorized, then I plan to retry with the authorization credentials.
I have found rather a strange issue, and I am not sure if the fault is with
the Apache HTTP Client or the server. However, I get different behaviour in
the Apache HTTP Client depending on how fast I can stream the data to the
server.
What I am seeing looks like this:
1. The Client opens a HTTP connection to the server
2. The Client starts sending data to the server
3. The Server starts receiving the data from the client, and determines
that the client is not authorized.
4. The Server responds HTTP 401 Unauthorized to the client
5. The server closes the connection.
At the client end when I only have a small amount of data to PUT, or use a
fast mechanism to write the data, all is fine and Apache HTTP Client gives
me the HTTP Response 401 from the server. All is good!
However, at the client end when I have a lot of data to PUT, or use a slow
mechanism to write the data, the Apache HTTP Client is still writting data
to the server when (5) occurs. The operating system then raises a "Broken
pipe (Write failed)" error, which Java relays to the Apache HTTP Client as
a SocketException. I then see errors like the following in my application:
Oct 10, 2018 10:10:58 PM org.apache.http.impl.execchain.RetryExec execute
INFO: I/O exception (java.net.SocketException) caught when processing
request to {}->http://localhost:8080: Broken pipe (Write failed)
Oct 10, 2018 10:10:58 PM org.apache.http.impl.execchain.RetryExec execute
INFO: Retrying request to {}->http://localhost:8080
Oct 10, 2018 10:10:58 PM org.apache.http.impl.execchain.RetryExec execute
INFO: I/O exception (java.net.SocketException) caught when processing
request to {}->http://localhost:8080: Broken pipe (Write failed)
Oct 10, 2018 10:10:58 PM org.apache.http.impl.execchain.RetryExec execute
INFO: Retrying request to {}->http://localhost:8080
Oct 10, 2018 10:10:58 PM org.apache.http.impl.execchain.RetryExec execute
INFO: I/O exception (java.net.SocketException) caught when processing
request to {}->http://localhost:8080: Broken pipe (Write failed)
Oct 10, 2018 10:10:58 PM org.apache.http.impl.execchain.RetryExec execute
INFO: Retrying request to {}->http://localhost:8080
Exception in thread "main" java.net.SocketException: Broken pipe (Write
failed)
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:111)
at java.net.SocketOutputStream.write(SocketOutputStream.java:155)
at
org.apache.http.impl.io.SessionOutputBufferImpl.streamWrite(SessionOutputBufferImpl.java:124)
at
org.apache.http.impl.io.SessionOutputBufferImpl.flushBuffer(SessionOutputBufferImpl.java:136)
at
org.apache.http.impl.io.SessionOutputBufferImpl.write(SessionOutputBufferImpl.java:167)
at
org.apache.http.impl.io.ChunkedOutputStream.flushCacheWithAppend(ChunkedOutputStream.java:122)
at
org.apache.http.impl.io.ChunkedOutputStream.write(ChunkedOutputStream.java:179)
at
org.apache.commons.io.output.ProxyOutputStream.write(ProxyOutputStream.java:89)
at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221)
at sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:282)
at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:125)
at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:135)
at java.io.OutputStreamWriter.write(OutputStreamWriter.java:220)
at java.io.Writer.write(Writer.java:157)
at Issue2186.copySlow(Issue2186.java:44)
at Issue2186.lambda$0(Issue2186.java:26)
at org.apache.http.entity.EntityTemplate.writeTo(EntityTemplate.java:73)
at
org.apache.http.impl.DefaultBHttpClientConnection.sendRequestEntity(DefaultBHttpClientConnection.java:156)
at
org.apache.http.impl.conn.CPoolProxy.sendRequestEntity(CPoolProxy.java:160)
at
org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:238)
at
org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123)
at
org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
at
org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
at
org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
at
org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
at
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
at
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108)
at Issue2186.main(Issue2186.java:55)
So what is going on here? Is there something I am doing wrong on the client
side, or should the server be waiting to receive all the data from the
client before replying with the 401 Unauthorized response?
My simple code to reproduce the issue looks like:
public class Issue2186 {
private static final Path file = Paths.get("/tmp/mediumfile.xml");
private static HttpEntity getRequestBody() {
final ContentProducer producer = os -> {
// copyFast(file, os);
// NOTE: shows the broken pipe error
copySlow(file, os);
};
final EntityTemplate entityTemplate = new EntityTemplate(producer);
entityTemplate.setContentType("application/xml");
entityTemplate.setChunked(true);
return entityTemplate;
}
private static void copyFast(final Path file, final OutputStream os)
throws IOException {
Files.copy(file, os);
}
private static void copySlow(final Path file, final OutputStream os)
throws IOException {
try(final LineNumberReader reader = new
LineNumberReader(Files.newBufferedReader(file));
final OutputStreamWriter writer = new
OutputStreamWriter(new CloseShieldOutputStream(os), UTF_8)) {
String line;
while((line = reader.readLine()) != null) {
writer.write(line + "\n");
}
}
}
public static void main(final String args[]) throws IOException {
final CloseableHttpClient httpClient = HttpClients.createDefault();
try {
final HttpPut httpPut = new HttpPut("
http://localhost:8080/exist/rest/db/mediumfile.xml";);
httpPut.setEntity(getRequestBody());
final CloseableHttpResponse httpResponse =
httpClient.execute(httpPut);
try {
System.out.println(httpResponse.getStatusLine());
} finally {
httpResponse.close();
}
} finally {
httpClient.close();
}
}
}
Thanks for your time.
--Adam Retter
eXist Core Developer
{ United Kingdom / United States }
a...@exist-db.org
