Hi. So if I open one stream, it should be 4?
2009/6/23 Raghu Angadi <rang...@yahoo-inc.com> > > how many threads do you have? Number of active threads is very important. > Normally, > > #fds = (3 * #threads_blocked_on_io) + #streams > > 12 per stream is certainly way off. > > Raghu. > > > Stas Oskin wrote: > >> Hi. >> >> In my case it was actually ~ 12 fd's per stream, which included pipes and >> epolls. >> >> Could it be that HDFS opens 3 x 3 (input - output - epoll) fd's per each >> thread, which make it close to the number I mentioned? Or it always 3 at >> maximum per thread / stream? >> >> Up to 10 sec looks quite the correct number, it seems it gets freed >> arround >> this time indeed. >> >> Regards. >> >> 2009/6/23 Raghu Angadi <rang...@yahoo-inc.com> >> >> To be more accurate, once you have HADOOP-4346, >>> >>> fds for epoll and pipes = 3 * threads blocked on Hadoop I/O >>> >>> Unless you have hundreds of threads at a time, you should not see >>> hundreds >>> of these. These fds stay up to 10sec even after the >>> threads exit. >>> >>> I am a bit confused about your exact situation. Please check number of >>> threads if you still facing the problem. >>> >>> Raghu. >>> >>> >>> Raghu Angadi wrote: >>> >>> since you have HADOOP-4346, you should not have excessive epoll/pipe fds >>>> open. First of all do you still have the problem? If yes, how many >>>> hadoop >>>> streams do you have at a time? >>>> >>>> System.gc() won't help if you have HADOOP-4346. >>>> >>>> Ragu. >>>> >>>> Thanks for your opinion! >>>> >>>>> 2009/6/22 Stas Oskin <stas.os...@gmail.com> >>>>> >>>>> Ok, seems this issue is already patched in the Hadoop distro I'm using >>>>> >>>>>> (Cloudera). >>>>>> >>>>>> Any idea if I still should call GC manually/periodically to clean out >>>>>> all >>>>>> the stale pipes / epolls? >>>>>> >>>>>> 2009/6/22 Steve Loughran <ste...@apache.org> >>>>>> >>>>>> Stas Oskin wrote: >>>>>> >>>>>>> Hi. >>>>>>> >>>>>>> So what would be the recommended approach to pre-0.20.x series? >>>>>>>> >>>>>>>> To insure each file is used only by one thread, and then it safe to >>>>>>>> close >>>>>>>> the handle in that thread? >>>>>>>> >>>>>>>> Regards. >>>>>>>> >>>>>>>> good question -I'm not sure. For anythiong you get with >>>>>>>> >>>>>>> FileSystem.get(), >>>>>>> its now dangerous to close, so try just setting the reference to null >>>>>>> and >>>>>>> hoping that GC will do the finalize() when needed >>>>>>> >>>>>>> >>>>>>> >> >
