Re: possible NIO selector leak in 7u25
Glad to hear you got to the bottom of it. -Rob On 08/07/13 23:22, Bernd Eckenfels wrote: Hello, we found a cause for the leak, we did not use the latest xnio-nio release. Looking at the NioTcpChannel code I guess that for example this commit could fix a potential problem (we shutdown r+w before we close, which would not be passed on by the older xnio-nio): https://github.com/xnio/xnio/commit/71ebef70d11eedce9a0eb7e5de4d37ab22648b73 When we update the implementation to 3.0.8.GA we dont see the handles accumulate anymore. So sorry for blaming NIO (I am still not sure why this does not show up in the heapdumps. I have done some test programs where I havent closed() the channels or havent selected on the canceled keys. And in all those conditions I see Java heap objects representing the (open) FDs. But I dont see them in our test environment). Oh well... thanks for listening, it always helps to explain your problems to others, especially if they are home made :) Bernd
Re: possible NIO selector leak in 7u25
Hello, we found a cause for the leak, we did not use the latest xnio-nio release. Looking at the NioTcpChannel code I guess that for example this commit could fix a potential problem (we shutdown r+w before we close, which would not be passed on by the older xnio-nio): https://github.com/xnio/xnio/commit/71ebef70d11eedce9a0eb7e5de4d37ab22648b73 When we update the implementation to 3.0.8.GA we dont see the handles accumulate anymore. So sorry for blaming NIO (I am still not sure why this does not show up in the heapdumps. I have done some test programs where I havent closed() the channels or havent selected on the canceled keys. And in all those conditions I see Java heap objects representing the (open) FDs. But I dont see them in our test environment). Oh well... thanks for listening, it always helps to explain your problems to others, especially if they are home made :) Bernd
possible NIO selector leak in 7u25
Hello, we see a possible handle/selector leak very similiar to this bug: http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=7118373 We see on linux unix domain sockets and on windows /dev/afd handles which are not backed up by any socket/selector/handle/channel in the heapdump. This is a applicxation using NIO (via JBoss XNIO). We would like to nail down the source of it and therefore it would be good if we have the actual code from the old bug above e.java to see if this helps us to reproduce the problem. Can any Oracle developer with access to it sent it? (and cvonfirm the bug is fixed) In our case we see 500 of those internal sockets, but the related tcp sockets (and java objects) are gone. Bernd
Re: possible NIO selector leak in 7u25
On 04/07/2013 09:36, Bernd Eckenfels wrote: Hello, we see a possible handle/selector leak very similiar to this bug: http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=7118373 We see on linux unix domain sockets and on windows /dev/afd handles which are not backed up by any socket/selector/handle/channel in the heapdump. This is a applicxation using NIO (via JBoss XNIO). We would like to nail down the source of it and therefore it would be good if we have the actual code from the old bug above e.java to see if this helps us to reproduce the problem. Can any Oracle developer with access to it sent it? (and cvonfirm the bug is fixed) In our case we see 500 of those internal sockets, but the related tcp sockets (and java objects) are gone. Bernd Do you know if JBoss XNIO uses socket adapters to do timed reads? These were implemented as temporary Selectors and cached on a per thread basis. We've replaced this in jdk8 but for jdk7 and older then it is possible to have scenarios where there are a lot of threads or short-lived threads and the temporarily Selectors hang around until they are GC'ed. If you are using lsof or equivalent then it would be visible as an apparently build up of Unix domain sockets. You mention that the heap dump doesn't appear to have references and that make sense if the heap dump is generated from only the live objects (would be interesting to know if the Unix domain sockets are closed as as result of a heap dump that does a full GC first). There isn't an e.java attached to the bug. This bug is actually a shadow bug for something that came via another support/customer system so it has been filtered. When Rob fixed 7118373 then he wasn't able to come up with a reliable test case that demonstrated the issue in a reasonable amount of time. So are you able to duplicate the issue in your environment? I'm just wondering if you could try a preview build of 8 or event 7u40 to double check that the issue still duplicates. -Alan
Re: possible NIO selector leak in 7u25
On 7/4/13 4:42 AM, Alan Bateman wrote: On 04/07/2013 09:36, Bernd Eckenfels wrote: Hello, we see a possible handle/selector leak very similiar to this bug: http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=7118373 We see on linux unix domain sockets and on windows /dev/afd handles which are not backed up by any socket/selector/handle/channel in the heapdump. This is a applicxation using NIO (via JBoss XNIO). We would like to nail down the source of it and therefore it would be good if we have the actual code from the old bug above e.java to see if this helps us to reproduce the problem. Can any Oracle developer with access to it sent it? (and cvonfirm the bug is fixed) In our case we see 500 of those internal sockets, but the related tcp sockets (and java objects) are gone. Bernd Do you know if JBoss XNIO uses socket adapters to do timed reads? These were implemented as temporary Selectors and cached on a per thread basis. We've replaced this in jdk8 but for jdk7 and older then it is possible to have scenarios where there are a lot of threads or short-lived threads and the temporarily Selectors hang around until they are GC'ed. If you are using lsof or equivalent then it would be visible as an apparently build up of Unix domain sockets. You mention that the heap dump doesn't appear to have references and that make sense if the heap dump is generated from only the live objects (would be interesting to know if the Unix domain sockets are closed as as result of a heap dump that does a full GC first). There isn't an e.java attached to the bug. This bug is actually a shadow bug for something that came via another support/customer system so it has been filtered. When Rob fixed 7118373 then he wasn't able to come up with a reliable test case that demonstrated the issue in a reasonable amount of time. So are you able to duplicate the issue in your environment? I'm just wondering if you could try a preview build of 8 or event 7u40 to double check that the issue still duplicates. XNIO uses Selectors (usually PollSelectorImpls) which are cached per thread in order to mix blocking and non-blocking I/O. If you are starting many short-lived threads and doing blocking operations on XNIO channels then this might explain what is happening. The answer is basically don't do that. -- - DML
Re: possible NIO selector leak in 7u25
On 04/07/2013 19:43, David M. Lloyd wrote: XNIO uses Selectors (usually PollSelectorImpls) which are cached per thread in order to mix blocking and non-blocking I/O. If you are starting many short-lived threads and doing blocking operations on XNIO channels then this might explain what is happening. The answer is basically don't do that. When you say usually PollSelectorImpls then it normally run with it configured to use the poll based Selector? BTW: Sean Coffey mailed me off-list so say that he tried the original support/customer test case that lead to 7118373 and confirms that it doesn't duplicate with 7u25. Anyway, it would be good to hear more from Bernd to know if this duplicates with 7u40 or 8. Also interesting to know if this is a case of lots of short-lived threads doing timed reads on socket adapters (which was the motive for replace this part of the implementation in 8). -Alan.
Re: possible NIO selector leak in 7u25
Hello, I will have a look at 7u40 and 8 as soon as I get some time. But I dont think it is thread(caching) related as it does not show up in the heapdump (and we have a controled thread creation rate). It looks more like some close/cancel interactions (and a real leak). I will let you know what I find. Thanks David, Alan and Sean! Greetings Bernd
Re: possible NIO selector leak in 7u25
On 7/4/13 1:53 PM, Alan Bateman wrote: On 04/07/2013 19:43, David M. Lloyd wrote: XNIO uses Selectors (usually PollSelectorImpls) which are cached per thread in order to mix blocking and non-blocking I/O. If you are starting many short-lived threads and doing blocking operations on XNIO channels then this might explain what is happening. The answer is basically don't do that. When you say usually PollSelectorImpls then it normally run with it configured to use the poll based Selector? Yeah it will prefer PollSelectorImpl for temporary selectors, but will ultimately fall back to whatever the default provider gives. BTW: Sean Coffey mailed me off-list so say that he tried the original support/customer test case that lead to 7118373 and confirms that it doesn't duplicate with 7u25. Good to know, thanks. -- - DML
