Re: possible NIO selector leak in 7u25
Glad to hear you got to the bottom of it. -Rob On 08/07/13 23:22, Bernd Eckenfels wrote: Hello, we found a cause for the leak, we did not use the latest xnio-nio release. Looking at the NioTcpChannel code I guess that for example this commit could fix a potential problem (we shutdown r+w before we close, which would not be passed on by the older xnio-nio): https://github.com/xnio/xnio/commit/71ebef70d11eedce9a0eb7e5de4d37ab22648b73 When we update the implementation to 3.0.8.GA we dont see the handles accumulate anymore. So sorry for blaming NIO (I am still not sure why this does not show up in the heapdumps. I have done some test programs where I havent closed() the channels or havent selected on the canceled keys. And in all those conditions I see Java heap objects representing the (open) FDs. But I dont see them in our test environment). Oh well... thanks for listening, it always helps to explain your problems to others, especially if they are home made :) Bernd
Re: possible NIO selector leak in 7u25
Hello, we found a cause for the leak, we did not use the latest xnio-nio release. Looking at the NioTcpChannel code I guess that for example this commit could fix a potential problem (we shutdown r+w before we close, which would not be passed on by the older xnio-nio): https://github.com/xnio/xnio/commit/71ebef70d11eedce9a0eb7e5de4d37ab22648b73 When we update the implementation to 3.0.8.GA we dont see the handles accumulate anymore. So sorry for blaming NIO (I am still not sure why this does not show up in the heapdumps. I have done some test programs where I havent closed() the channels or havent selected on the canceled keys. And in all those conditions I see Java heap objects representing the (open) FDs. But I dont see them in our test environment). Oh well... thanks for listening, it always helps to explain your problems to others, especially if they are home made :) Bernd
Re: possible NIO selector leak in 7u25
Sorry I should not write those mails on the mobile - full version below. Am 05.07.2013 um 16:05 schrieb Bernd Eckenfels : > Hello, > > I have two updates on the Issue: > > A) same Problem exists with June 7u40 EA JDK > B) there are no threads starting/terminating in the test > > In our test we run a workload with 1 transaction per second and it leaks one > socket each. They don't get cleaned up in a FullGC and when stopping the load > generator they don't clear up after longer time. What we found is, that we > don't pass shutdownRead/Write to NioTCPChannel (only close) the leak is gone.
Re: possible NIO selector leak in 7u25
On 7/4/13 1:53 PM, Alan Bateman wrote: On 04/07/2013 19:43, David M. Lloyd wrote: XNIO uses Selectors (usually PollSelectorImpls) which are cached per thread in order to mix blocking and non-blocking I/O. If you are starting many short-lived threads and doing blocking operations on XNIO channels then this might explain what is happening. The answer is basically "don't do that". When you say "usually PollSelectorImpls" then it normally run with it configured to use the poll based Selector? Yeah it will prefer PollSelectorImpl for temporary selectors, but will ultimately fall back to whatever the default provider gives. BTW: Sean Coffey mailed me off-list so say that he tried the original support/customer test case that lead to 7118373 and confirms that it doesn't duplicate with 7u25. Good to know, thanks. -- - DML
Re: possible NIO selector leak in 7u25
Hello, I will have a look at 7u40 and 8 as soon as I get some time. But I dont think it is thread(caching) related as it does not show up in the heapdump (and we have a controled thread creation rate). It looks more like some close/cancel interactions (and a real leak). I will let you know what I find. Thanks David, Alan and Sean! Greetings Bernd
Re: possible NIO selector leak in 7u25
On 04/07/2013 19:43, David M. Lloyd wrote: XNIO uses Selectors (usually PollSelectorImpls) which are cached per thread in order to mix blocking and non-blocking I/O. If you are starting many short-lived threads and doing blocking operations on XNIO channels then this might explain what is happening. The answer is basically "don't do that". When you say "usually PollSelectorImpls" then it normally run with it configured to use the poll based Selector? BTW: Sean Coffey mailed me off-list so say that he tried the original support/customer test case that lead to 7118373 and confirms that it doesn't duplicate with 7u25. Anyway, it would be good to hear more from Bernd to know if this duplicates with 7u40 or 8. Also interesting to know if this is a case of lots of short-lived threads doing timed reads on socket adapters (which was the motive for replace this part of the implementation in 8). -Alan.
Re: possible NIO selector leak in 7u25
On 7/4/13 4:42 AM, Alan Bateman wrote: On 04/07/2013 09:36, Bernd Eckenfels wrote: Hello, we see a possible handle/selector leak very similiar to this bug: http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=7118373 We see on linux unix domain sockets and on windows /dev/afd handles which are not backed up by any socket/selector/handle/channel in the heapdump. This is a applicxation using NIO (via JBoss XNIO). We would like to nail down the source of it and therefore it would be good if we have the actual code from the old bug above "e.java" to see if this helps us to reproduce the problem. Can any Oracle developer with access to it sent it? (and cvonfirm the bug is fixed) In our case we see >500 of those internal sockets, but the related tcp sockets (and java objects) are gone. Bernd Do you know if JBoss XNIO uses socket adapters to do timed reads? These were implemented as temporary Selectors and cached on a per thread basis. We've replaced this in jdk8 but for jdk7 and older then it is possible to have scenarios where there are a lot of threads or short-lived threads and the temporarily Selectors hang around until they are GC'ed. If you are using lsof or equivalent then it would be visible as an apparently build up of Unix domain sockets. You mention that the heap dump doesn't appear to have references and that make sense if the heap dump is generated from only the live objects (would be interesting to know if the Unix domain sockets are closed as as result of a heap dump that does a full GC first). There isn't an "e.java" attached to the bug. This bug is actually a "shadow bug" for something that came via another support/customer system so it has been filtered. When Rob fixed 7118373 then he wasn't able to come up with a reliable test case that demonstrated the issue in a reasonable amount of time. So are you able to duplicate the issue in your environment? I'm just wondering if you could try a preview build of 8 or event 7u40 to double check that the issue still duplicates. XNIO uses Selectors (usually PollSelectorImpls) which are cached per thread in order to mix blocking and non-blocking I/O. If you are starting many short-lived threads and doing blocking operations on XNIO channels then this might explain what is happening. The answer is basically "don't do that". -- - DML
Re: possible NIO selector leak in 7u25
On 04/07/2013 09:36, Bernd Eckenfels wrote: Hello, we see a possible handle/selector leak very similiar to this bug: http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=7118373 We see on linux unix domain sockets and on windows /dev/afd handles which are not backed up by any socket/selector/handle/channel in the heapdump. This is a applicxation using NIO (via JBoss XNIO). We would like to nail down the source of it and therefore it would be good if we have the actual code from the old bug above "e.java" to see if this helps us to reproduce the problem. Can any Oracle developer with access to it sent it? (and cvonfirm the bug is fixed) In our case we see >500 of those internal sockets, but the related tcp sockets (and java objects) are gone. Bernd Do you know if JBoss XNIO uses socket adapters to do timed reads? These were implemented as temporary Selectors and cached on a per thread basis. We've replaced this in jdk8 but for jdk7 and older then it is possible to have scenarios where there are a lot of threads or short-lived threads and the temporarily Selectors hang around until they are GC'ed. If you are using lsof or equivalent then it would be visible as an apparently build up of Unix domain sockets. You mention that the heap dump doesn't appear to have references and that make sense if the heap dump is generated from only the live objects (would be interesting to know if the Unix domain sockets are closed as as result of a heap dump that does a full GC first). There isn't an "e.java" attached to the bug. This bug is actually a "shadow bug" for something that came via another support/customer system so it has been filtered. When Rob fixed 7118373 then he wasn't able to come up with a reliable test case that demonstrated the issue in a reasonable amount of time. So are you able to duplicate the issue in your environment? I'm just wondering if you could try a preview build of 8 or event 7u40 to double check that the issue still duplicates. -Alan
possible NIO selector leak in 7u25
Hello, we see a possible handle/selector leak very similiar to this bug: http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=7118373 We see on linux unix domain sockets and on windows /dev/afd handles which are not backed up by any socket/selector/handle/channel in the heapdump. This is a applicxation using NIO (via JBoss XNIO). We would like to nail down the source of it and therefore it would be good if we have the actual code from the old bug above "e.java" to see if this helps us to reproduce the problem. Can any Oracle developer with access to it sent it? (and cvonfirm the bug is fixed) In our case we see >500 of those internal sockets, but the related tcp sockets (and java objects) are gone. Bernd