Hi. In my case it was actually ~ 12 fd's per stream, which included pipes and epolls.
Could it be that HDFS opens 3 x 3 (input - output - epoll) fd's per each thread, which make it close to the number I mentioned? Or it always 3 at maximum per thread / stream? Up to 10 sec looks quite the correct number, it seems it gets freed arround this time indeed. Regards. 2009/6/23 Raghu Angadi <rang...@yahoo-inc.com> > To be more accurate, once you have HADOOP-4346, > > fds for epoll and pipes = 3 * threads blocked on Hadoop I/O > > Unless you have hundreds of threads at a time, you should not see hundreds > of these. These fds stay up to 10sec even after the > threads exit. > > I am a bit confused about your exact situation. Please check number of > threads if you still facing the problem. > > Raghu. > > > Raghu Angadi wrote: > >> >> since you have HADOOP-4346, you should not have excessive epoll/pipe fds >> open. First of all do you still have the problem? If yes, how many hadoop >> streams do you have at a time? >> >> System.gc() won't help if you have HADOOP-4346. >> >> Ragu. >> >> Thanks for your opinion! >>> >>> 2009/6/22 Stas Oskin <stas.os...@gmail.com> >>> >>> Ok, seems this issue is already patched in the Hadoop distro I'm using >>>> (Cloudera). >>>> >>>> Any idea if I still should call GC manually/periodically to clean out >>>> all >>>> the stale pipes / epolls? >>>> >>>> 2009/6/22 Steve Loughran <ste...@apache.org> >>>> >>>> Stas Oskin wrote: >>>>> >>>>> Hi. >>>>> >>>>>> So what would be the recommended approach to pre-0.20.x series? >>>>>> >>>>>> To insure each file is used only by one thread, and then it safe to >>>>>> close >>>>>> the handle in that thread? >>>>>> >>>>>> Regards. >>>>>> >>>>>> good question -I'm not sure. For anythiong you get with >>>>> FileSystem.get(), >>>>> its now dangerous to close, so try just setting the reference to null >>>>> and >>>>> hoping that GC will do the finalize() when needed >>>>> >>>>> >>> >> >
