To be more accurate, once you have HADOOP-4346,
fds for epoll and pipes = 3 * threads blocked on Hadoop I/O
Unless you have hundreds of threads at a time, you should not see
hundreds of these. These fds stay up to 10sec even after the
threads exit.
I am a bit confused about your exact situation. Please check number of
threads if you still facing the problem.
Raghu.
Raghu Angadi wrote:
since you have HADOOP-4346, you should not have excessive epoll/pipe fds
open. First of all do you still have the problem? If yes, how many
hadoop streams do you have at a time?
System.gc() won't help if you have HADOOP-4346.
Ragu.
Thanks for your opinion!
2009/6/22 Stas Oskin <stas.os...@gmail.com>
Ok, seems this issue is already patched in the Hadoop distro I'm using
(Cloudera).
Any idea if I still should call GC manually/periodically to clean out
all
the stale pipes / epolls?
2009/6/22 Steve Loughran <ste...@apache.org>
Stas Oskin wrote:
Hi.
So what would be the recommended approach to pre-0.20.x series?
To insure each file is used only by one thread, and then it safe to
close
the handle in that thread?
Regards.
good question -I'm not sure. For anythiong you get with
FileSystem.get(),
its now dangerous to close, so try just setting the reference to
null and
hoping that GC will do the finalize() when needed
