Re: losing network interfaces during long running map-reduce jobs
David Howell wrote: But I haven't seen anything in the dmesg log. I'll have to try looking at the tcpdump output on Monday, once I can get console access again. My apologies that I'm so sketchy on details right now... so far, I haven't been any able to find any evidence of something going wrong except for the hadoop log entries when the IOExceptions start. Thanks, -David I just lost my networking again. This time, I had switched my cluster back to the build I was using before I switched to CDH2. It's Hadoop 0.20.1 with these patches applied (for Dumbo): HADOOP-1722-v0.20.1 HADOOP-5450 MAPREDUCE-764 HADOOP-5528 Now I'm wondering if something about my job is the culprit. I have 2 nodes, both 8 core machines. mapred.tasktracker.map|reduce.tasks.maximum are both set to 7. The job I'm running is combining lots of gzipped Apache log files into sequence files for later analysis... I'm going from one file per virtual host per server per day to file per virtual host per day. The last attempt had ~1400 maps/10 reduces. could be just file handles you are losing; have up upped the OS defaults?
Re: losing network interfaces during long running map-reduce jobs
could be just file handles you are losing; have up upped the OS defaults? I have not, and that does seem like a likely culprit. Although, it's a bit alarming that asking for one socket too many could take down the networking stack...
Re: losing network interfaces during long running map-reduce jobs
Hi David, On Fri, Apr 2, 2010 at 6:16 PM, David Howell dehow...@gmail.com wrote: I'm encountering a completely bizarre failure mode in my Hadoop cluster. A week ago, I switched from vanilla apache Hadoop 0.20.1 to CDH 2. Ever since then, my tasktracker/ datenode machines have been regularly losing their networking during long ( 1 hour) jobs. Restarting the network interface brings them back online immediately. Could you clarify wha you mean by losing their networking? Can you ping the node externally? If you access the node via the console (via ILOM, etc) and run tcpdump or tshark, can you see ethernet broadcast traffic at all? Do you see anything in dmesg on the machine in question? Thanks -Todd -- Todd Lipcon Software Engineer, Cloudera
Re: losing network interfaces during long running map-reduce jobs
Could you clarify wha you mean by losing their networking? Can you ping the node externally? If you access the node via the console (via ILOM, etc) and run tcpdump or tshark, can you see ethernet broadcast traffic at all? Do you see anything in dmesg on the machine in question? Thanks -Todd My cluster is small and the physical servers managed by my company's IT department... I just admin the Hadoop install and I don't have access except through ssh. When one of my nodes goes unresponsive, it doesn't respond to ping, ssh, or any traffic on any port. I've been limited so far to trying to investigate logs after my sysadmin restarts the networking interface. But I haven't seen anything in the dmesg log. I'll have to try looking at the tcpdump output on Monday, once I can get console access again. My apologies that I'm so sketchy on details right now... so far, I haven't been any able to find any evidence of something going wrong except for the hadoop log entries when the IOExceptions start. Thanks, -David
losing network interfaces during long running map-reduce jobs
I'm encountering a completely bizarre failure mode in my Hadoop cluster. A week ago, I switched from vanilla apache Hadoop 0.20.1 to CDH 2. Ever since then, my tasktracker/ datenode machines have been regularly losing their networking during long ( 1 hour) jobs. Restarting the network interface brings them back online immediately. I'm mystified as to how this can be happening: anyone care to venture a hypothesis? I'm running on Centos 5.2. Cheers, David
Re: losing network interfaces during long running map-reduce jobs
Hi David, Strange indeed. I assume nothing in your configs changed. Anything funny in the logs? You should also rule out the switch itself as being faulty. It's possible that CDH2 has a patch that's not in 0.20.1 that's causing this problem, but we haven't heard this exact problem from any of our other customers / users. - P On Fri, Apr 2, 2010 at 6:16 PM, David Howell dehow...@gmail.com wrote: I'm encountering a completely bizarre failure mode in my Hadoop cluster. A week ago, I switched from vanilla apache Hadoop 0.20.1 to CDH 2. Ever since then, my tasktracker/ datenode machines have been regularly losing their networking during long ( 1 hour) jobs. Restarting the network interface brings them back online immediately. I'm mystified as to how this can be happening: anyone care to venture a hypothesis? I'm running on Centos 5.2. Cheers, David
