Re: Problem with half-closure related to connection-reset in Java 11

[Alan Bateman]

On 29/05/2018 17:28, Norman Maurer wrote:

:
Oh sorry I thought thats the right system to use. I just followed the wiki page 
(I think).
bugs.sun.com or bugreport.java.com is the right place. That routes the 
bugs to the JIRA, just not automatically to the right project as often 
issues need to be filtered or deleted. The bug tracking is JDK-8203937 [1].




:
Well it seems like it worked before on linux as well ? I mean with Java10 it 
seems to at least not fail on linux the same way as now.

The only platform that I'm aware of that will report the reset when 
reading and allow reading beyond the reset is Solaris and this is only 
because of the way that it reports networking errors to applications. I 
think the behavior change you observe is because the reset is reported 
when writing and this is impacting the reading from the java.net.Socket. 
This is really odd behavior that has been there since JDK 1.4, just not 
noticed because it would attempt to read until it got the reset. This is 
all unspecified behavior so this is why the testing with the changes in 
JDK 11 didn't show up anything. I agree we should just eliminate this so 
that it doesn't impact the read.


Would you have cycles to create a small, and standalone test, that 
demonstrates the issue? This could be something we turn into a 
regression test to test behavior.


-Alan

[1] https://bugs.openjdk.java.net/browse/JDK-8203937

Re: Problem with half-closure related to connection-reset in Java 11

[Norman Maurer]

> On 29. May 2018, at 18:26, Alan Bateman  wrote:
> 
> On 29/05/2018 15:26, Norman Maurer wrote:
>> :
>> 
>> 
>> Yes thats what I am saying… I think if a write fails due a connection-reset 
>> a read should still be possible until we are told by the OS that we also hit 
>> an error here. Honestly I think this scenario can happen quite often in 
>> reality where some software writes while draining data from the socket in 
>> chunks. With Java 11 this may lead to the situation where the user may never 
>> see the data even when its waiting on the socket to be read which I think is 
>> weird. What kind of problems this may cause in different programs is hard to 
>> know, but its definitely something that surprised me. Even more after I 
>> started to debug and could see the packets via tcpdump etc.
>> 
>> Also as a side-note when using SocketChannel this works perfectly fine, as 
>> before. I will open a bug with all the informations in this email as 
>> requested. I just wanted to make sure first that my observations are correct 
>> and wanted to provide as much details as possible ( + making a strong 
>> argument to why I think this is a regression).
>> 
> I see you've created an issue to the bugs.sun.com system - thanks! I'll move 
> that to the JDK project so that it's accessible via bugs.openjdk.java.net.

Oh sorry I thought thats the right system to use. I just followed the wiki page 
(I think). 
> 
> You are right that there is subtle behavior change, something that wasn't 
> really intended. The code to manage reading beyond connection resets has 
> always been problematic and only ever worked on Solaris. The SocketChannel 
> implementation has never attempt to do this, it just reflects the platform 
> behavior when reading after a reset.

Well it seems like it worked before on linux as well ? I mean with Java10 it 
seems to at least not fail on linux the same way as now.


> 
> -Alan.


Thanks
Norman

Re: Problem with half-closure related to connection-reset in Java 11

[Alan Bateman]

On 29/05/2018 15:26, Norman Maurer wrote:

:


Yes thats what I am saying… I think if a write fails due a connection-reset a 
read should still be possible until we are told by the OS that we also hit an 
error here. Honestly I think this scenario can happen quite often in reality 
where some software writes while draining data from the socket in chunks. With 
Java 11 this may lead to the situation where the user may never see the data 
even when its waiting on the socket to be read which I think is weird. What 
kind of problems this may cause in different programs is hard to know, but its 
definitely something that surprised me. Even more after I started to debug and 
could see the packets via tcpdump etc.

Also as a side-note when using SocketChannel this works perfectly fine, as 
before. I will open a bug with all the informations in this email as requested. 
I just wanted to make sure first that my observations are correct and wanted to 
provide as much details as possible ( + making a strong argument to why I think 
this is a regression).

I see you've created an issue to the bugs.sun.com system - thanks! I'll 
move that to the JDK project so that it's accessible via 
bugs.openjdk.java.net.


You are right that there is subtle behavior change, something that 
wasn't really intended. The code to manage reading beyond connection 
resets has always been problematic and only ever worked on Solaris. The 
SocketChannel implementation has never attempt to do this, it just 
reflects the platform behavior when reading after a reset.


-Alan.

Re: Problem with half-closure related to connection-reset in Java 11

[Norman Maurer]

> On 29. May 2018, at 16:19, Alan Bateman  wrote:
> 
> On 29/05/2018 14:52, Norman Maurer wrote:
>> Hi all,
>> 
>> After trying to run our testsuite in Netty [1] with Java11+ea15 I noticed we 
>> have one failing test that seems to be related to:
>> 
>> https://bugs.openjdk.java.net/browse/JDK-8199329
>> http://hg.openjdk.java.net/jdk/jdk/rev/92cca24c8807
>> 
>> I think the change here is not 100 % correct as it basically disallow 
>> draining any remaining bytes from the socket if a write causes a connection 
>> reset. This should be completely safe to do. At the moment if a write is 
>> causing a connection-reset you basically loose all the pending bytes that 
>> are sitting on the socket and are waiting to be read.
>> 
>> This happens because SocketOutputStream.write(…) may call 
>> AbstractPlainSocketImpl.setConnectionReset(…). Once this method is called 
>> any read(…) call will just throw a SocketException without even attempt to 
>> read any remaining data.
>> 
>> This was not the case with earlier Java versions, and I would argue its a 
>> bug.
>> 
>> Let me know what you think and please ask if you have any more questions,
> Reading beyond connection reset has always been problematic and never 
> guaranteed to work, esp. on the main stream platforms. I think you are 
> arguing that hitting connection reset when writing shouldn't impact another 
> thread reading. That probably make sense although it's hard to relate to 
> something that depends on such behavior. Can you submit a bug to track this 
> so that we at least track the behavior change?
> 
> -Alan


Yes thats what I am saying… I think if a write fails due a connection-reset a 
read should still be possible until we are told by the OS that we also hit an 
error here. Honestly I think this scenario can happen quite often in reality 
where some software writes while draining data from the socket in chunks. With 
Java 11 this may lead to the situation where the user may never see the data 
even when its waiting on the socket to be read which I think is weird. What 
kind of problems this may cause in different programs is hard to know, but its 
definitely something that surprised me. Even more after I started to debug and 
could see the packets via tcpdump etc. 

Also as a side-note when using SocketChannel this works perfectly fine, as 
before. I will open a bug with all the informations in this email as requested. 
I just wanted to make sure first that my observations are correct and wanted to 
provide as much details as possible ( + making a strong argument to why I think 
this is a regression).


Bye
Norman

Re: Problem with half-closure related to connection-reset in Java 11

[Alan Bateman]

On 29/05/2018 14:52, Norman Maurer wrote:

Hi all,

After trying to run our testsuite in Netty [1] with Java11+ea15 I 
noticed we have one failing test that seems to be related to:


https://bugs.openjdk.java.net/browse/JDK-8199329
http://hg.openjdk.java.net/jdk/jdk/rev/92cca24c8807

I think the change here is not 100 % correct as it basically disallow 
draining any remaining bytes from the socket if a write causes a 
connection reset. This should be completely safe to do. At the moment 
if a write is causing a connection-reset you basically loose all the 
pending bytes that are sitting on the socket and are waiting to be read.


This happens because SocketOutputStream.write(…) may call 
AbstractPlainSocketImpl.setConnectionReset(…). Once this method is 
called any read(…) call will just throw a SocketException without even 
attempt to read any remaining data.


This was not the case with earlier Java versions, and I would argue 
its a bug.


Let me know what you think and please ask if you have any more questions,
Reading beyond connection reset has always been problematic and never 
guaranteed to work, esp. on the main stream platforms. I think you are 
arguing that hitting connection reset when writing shouldn't impact 
another thread reading. That probably make sense although it's hard to 
relate to something that depends on such behavior. Can you submit a bug 
to track this so that we at least track the behavior change?


-Alan

Re: NullPointerException in jdk.incubator.http.internal.hpack.HeaderTable.Table

[Pavel Rappo]

A note for the readers of this mailing list. Apologies as this email has not
been visible for quite a while (it was sent on 2018-03-31 and appeared on the
list only on 2018-05-28). However, I have contacted the author and with his
great help the issue was resolved on the 5th of April. The patched version is
available with Early Access JDK 11 Builds here:

  http://jdk.java.net/11/

-Pavel

> On 31 Mar 2018, at 13:16, Андрей Турбанов  wrote:
> 
> Hello
> I'm trying to new use new shiny httpclient from JDK10. Code is pretty simple. 
> I use synchronous HTTP call form multiple threads: 
> 
> private final HttpClient httpClient = HttpClient.newBuilder()
> 
> 
> .executor(Utils.newFixedThreadPoolExecutor(1, "HttpClient"))
> 
> 
> .build();
> 
> 
> 
> private JsonObject useHttpClient(URL url, String params) throws Exception {
> 
> 
> HttpRequest req = HttpRequest.newBuilder()
> 
> 
> .uri(url.toURI())
> 
> 
> .setHeader("Connection", "keep-alive")
> 
> 
> .setHeader("Accept-Encoding", "gzip")
> 
> 
> .timeout(timeout)
> 
> 
> .POST(HttpRequest.BodyPublisher.fromString(params))
> 
> 
> .build();
> 
> 
> HttpResponse response = httpClient.send(req, 
> HttpResponse.BodyHandler.asInputStream());
> 
> 
> if (response.statusCode() != 200) {
> 
> 
> throw new IOException("Server returned " + response.statusCode());
> 
> 
> }
> 
> 
> String encoding = 
> response.headers().firstValue("content-encoding").orElse("");
> 
> 
> return parseResponseStream(encoding, response.body());
> }
> Sometimes I get NPEs:
> 
> java.lang.NullPointerException: null
> 
> at jdk
> .incubator.http.internal.hpack.HeaderTable$Table.remove(HeaderTable.java:455) 
> ~[jdk.incubator.httpclient:?]
> 
> at jdk
> .incubator.http.internal.hpack.HeaderTable.evictEntry(HeaderTable.java:264) 
> ~[jdk.incubator.httpclient:?]
> 
> at jdk
> .incubator.http.internal.hpack.HeaderTable.put(HeaderTable.java:233) 
> ~[jdk.incubator.httpclient:?]
> 
> at jdk
> .incubator.http.internal.hpack.HeaderTable.put(HeaderTable.java:215) 
> ~[jdk.incubator.httpclient:?]
> 
> at jdk
> .incubator.http.internal.hpack.Decoder.resumeLiteralWithIndexing(Decoder.java:464)
>  ~[jdk.incubator.httpclient:?]
> 
> at jdk
> .incubator.http.internal.hpack.Decoder.proceed(Decoder.java:268) 
> ~[jdk.incubator.httpclient:?]
> 
> at jdk
> .incubator.http.internal.hpack.Decoder.decode(Decoder.java:246) 
> ~[jdk.incubator.httpclient:?]
> 
> at jdk
> .incubator.http.Http2Connection.decodeHeaders(Http2Connection.java:471) 
> ~[jdk.incubator.httpclient:?]
> 
> at jdk
> .incubator.http.Http2Connection.processFrame(Http2Connection.java:635) 
> ~[jdk.incubator.httpclient:?]
> 
> at jdk
> .incubator.http.internal.frame.FramesDecoder.decode(FramesDecoder.java:156) 
> ~[jdk.incubator.httpclient:?]
> 
> at jdk
> .incubator.http.Http2Connection$FramesController.processReceivedData(Http2Connection.java:195)
>  ~[jdk.incubator.httpclient:?]
> 
> at jdk
> .incubator.http.Http2Connection.asyncReceive(Http2Connection.java:528) 
> ~[jdk.incubator.httpclient:?]
> 
> at jdk
> .incubator.http.Http2Connection$Http2TubeSubscriber.processQueue(Http2Connection.java:1054)
>  ~[jdk.incubator.httpclient:?]
> 
> at jdk
> .incubator.http.internal.common.SequentialScheduler$SynchronizedRestartableTask.run(SequentialScheduler.java:175)
>  ~[jdk.incubator.httpclient:?]
> 
> at jdk
> .incubator.http.internal.common.SequentialScheduler$CompleteRestartableTask.run(SequentialScheduler.java:147)
>  ~[jdk.incubator.httpclient:?]
> 
> at jdk
> .incubator.http.internal.common.SequentialScheduler$SchedulableTask.run(SequentialScheduler.java:198)
>  ~[jdk.incubator.httpclient:?]
> 
> at java
> .util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1135) 
> [?:?]
> Are there any problem in my code? What can be reason for this exception? 
> 
> C:\Program Files\Java\jdk-10\bin>java -
> version
> java version 
> "10" 2018-03-20
> Java(TM) SE Runtime Environment 18.3 (build 10+46)
> Java HotSpot(TM) 64-Bit Server VM 18.3 (build 10+46, mixed mode)
> 
> 
> Andrey Turbanov

Problem with half-closure related to connection-reset in Java 11

[Norman Maurer]

Hi all,

After trying to run our testsuite in Netty [1] with Java11+ea15 I noticed we 
have one failing test that seems to be related to:

https://bugs.openjdk.java.net/browse/JDK-8199329 

http://hg.openjdk.java.net/jdk/jdk/rev/92cca24c8807 


I think the change here is not 100 % correct as it basically disallow draining 
any remaining bytes from the socket if a write causes a connection reset. This 
should be completely safe to do. At the moment if a write is causing a 
connection-reset you basically loose all the pending bytes that are sitting on 
the socket and are waiting to be read.

This happens because SocketOutputStream.write(…) may call 
AbstractPlainSocketImpl.setConnectionReset(…). Once this method is called any 
read(…) call will just throw a SocketException without even attempt to read any 
remaining data.

This was not the case with earlier Java versions, and I would argue its a bug.

Let me know what you think and please ask if you have any more questions,
Norman



[1] 
https://github.com/netty/netty/blob/4.1/testsuite/src/main/java/io/netty/testsuite/transport/socket/SocketHalfClosedTest.java#L385