On Wednesday 05 November 2008 15:27:34 Karl Anderson wrote: > I am running into a similar issue. It seems to be affected by the > number of simultaneous tasks.
For me, while I generally allow up to 4 mappers per node, in this particular instance I had only one mapper reading from a single gzipped text file. It did work when I ran the same code on a smaller file. > I upped the nofile limit in > /etc/security/limits.conf to 131072 for all users on all of my EC2 > images, but it didn't help. I'm never running more than one job at > once. After I upped the max number of fds to 16K, the job ran to completion. I was monitoring the number of open files/processes every 15s (by simply running ps and lsof | wc -l) and saw this: #processes open_files ... 13 646 13 648 12 2535 13 4860 12 4346 12 3842 12 3324 12 2823 12 2316 12 1852 12 1387 12 936 12 643 12 643 12 643 12 643 12 643 12 643 13 642 12 642 12 4775 12 2738 12 917 12 643 12 642 12 4992 12 4453 12 3943 12 3299 12 2855 12 2437 ... It looks like something (garbage collection?) cleans up fds periodically; the max I saw was 5007 (but again, there may have been more in between the 15s sampling interval). > The hadoop-ec2 tools launch clusters with one master which runs the > namenode and jobtracker, and slaves each running a datanode and > tasktracker. It seems that running more than 2 mappers per node isn't > feasable with this setup, which surprises me because the cluster setup > suggestions I've read advise using far more. Would changing the ratio > of datanodes to tasktrackers have an effect? Is this done in > practice? > > Are you running more than 2 mappers per node? Do you see any > differences in the number of failed tasks when you change the number > of tasks over the same input set? I'm running a hand-carved cluster on a bunch of heterogeneous systems, some crappy some good. I suspect your total open FDs = (#mappers) x (FDs/map) In my case the second factor was ~5K; so if I ran 8 mappers total might have been as high as 40K! This is totally insane. Perhaps playing with GC modes might help... > In general, I've had to do a lot of fine-tuning of my job paramaters > to balance memory, file handles, and task timeouts. I'm finding that > a setup that works with one input set breaks when I try it on an input > set which is twice the size. My productivity is not high while I'm > figuring this out, and I wonder why I don't hear about this more. > Perhaps this is a streaming issue, and streaming isn't being used very > much? I doubt in my case this is a specific to streaming, although streaming might exacerbate the problem by opening pipes, etc. In my case the vast majority of open files were to spills during sorting/shuffling which is not restricted to streaming. This is a scalability issue and I'd really like to hear from developers. -Yuri P.S. It looks like we need to file a jira on this one...
