Hadoop and IP on InfiniBand (IPoIB)
I'm a new user of Hadoop. I have a Linux cluster with both gigabit ethernet and InfiniBand communications interfaces. Could someone please tell me how to switch IP communication from ethernet (the default) to InfiniBand? Thanks. -- Russell A. Brown| Oracle russ.br...@oracle.com | UMPK14-260 (650) 786-3011 (office) | 14 Network Circle (650) 786-3453 (fax)| Menlo Park, CA 94025
Re: Hadoop and IP on InfiniBand (IPoIB)
On Jun 14, 2010, at 10:57 AM, Russell Brown wrote: > I'm a new user of Hadoop. I have a Linux cluster with both gigabit ethernet > and InfiniBand communications interfaces. Could someone please tell me how > to switch IP communication from ethernet (the default) to InfiniBand? Thanks. Hadoop will bind inbound connections via the interface settings in the various hadoop configuration files. Outbound connections are unbound and based solely on OS configuration. I filed a jira to fix this, but it is obviously low priority since few people run multi-nic boxes. Best bet is to down the ethernet and up the IB, changing routing, etc, as necessary.
Re: Hadoop and IP on InfiniBand (IPoIB)
Thanks, Allen, for responding. So, if I understand you correctly, the dfs.datanode.dns.interface and mapred.tasktracker.dns.interface options may be used to define inbound connections only? Concerning the OS configuration, my /etc/hosts files assign unique host names to the ethernet and IB interfaces. However, even if I specify the IB host names in the masters and slaves files, communication still occurs via ethernet, not via IB. Your recommendation would therefore be to define IB instead of ethernet as the default network interface connection, right? Thanks, Russ On 06/14/10 12:32 PM, Allen Wittenauer wrote: On Jun 14, 2010, at 10:57 AM, Russell Brown wrote: I'm a new user of Hadoop. I have a Linux cluster with both gigabit ethernet and InfiniBand communications interfaces. Could someone please tell me how to switch IP communication from ethernet (the default) to InfiniBand? Thanks. Hadoop will bind inbound connections via the interface settings in the various hadoop configuration files. Outbound connections are unbound and based solely on OS configuration. I filed a jira to fix this, but it is obviously low priority since few people run multi-nic boxes. Best bet is to down the ethernet and up the IB, changing routing, etc, as necessary. -- Russell A. Brown| Oracle russ.br...@oracle.com | UMPK14-260 (650) 786-3011 (office) | 14 Network Circle (650) 786-3453 (fax)| Menlo Park, CA 94025
Re: Hadoop and IP on InfiniBand (IPoIB)
On Jun 15, 2010, at 7:40 AM, Russell Brown wrote: > Thanks, Allen, for responding. > > So, if I understand you correctly, the dfs.datanode.dns.interface and > mapred.tasktracker.dns.interface options may be used to define inbound > connections only? Correct. The daemons will bind to those interfaces and use those names as their 'official' connection in. > Concerning the OS configuration, my /etc/hosts files assign unique host names > to the ethernet and IB interfaces. However, even if I specify the IB host > names in the masters and slaves files, communication still occurs via > ethernet, not via IB. BTW, are you doing this on Solaris or Linux? Solaris is notorious for not honoring inbound and outbound interfaces. [In other words, just because the packet came in on bge0, that is no guarantee that the reply will go out on bge0 if another route is available. Particularly frustrating with NFS and SunCluster.] > Your recommendation would therefore be to define IB instead of ethernet as > the default network interface connection, right? Yup. Or at least give it a lower cost in the routing table.
Re: Hadoop and IP on InfiniBand (IPoIB)
FYI, Allen Wittnauer, I'm using Linux not Solaris, but I'll pay attention to your comment about Solaris if I install Solaris on the cluster. Thanks again for your helpful comments. Russ On 06/15/10 11:10 AM, Allen Wittenauer wrote: On Jun 15, 2010, at 7:40 AM, Russell Brown wrote: Thanks, Allen, for responding. So, if I understand you correctly, the dfs.datanode.dns.interface and mapred.tasktracker.dns.interface options may be used to define inbound connections only? Correct. The daemons will bind to those interfaces and use those names as their 'official' connection in. Concerning the OS configuration, my /etc/hosts files assign unique host names to the ethernet and IB interfaces. However, even if I specify the IB host names in the masters and slaves files, communication still occurs via ethernet, not via IB. BTW, are you doing this on Solaris or Linux? Solaris is notorious for not honoring inbound and outbound interfaces. [In other words, just because the packet came in on bge0, that is no guarantee that the reply will go out on bge0 if another route is available. Particularly frustrating with NFS and SunCluster.] Your recommendation would therefore be to define IB instead of ethernet as the default network interface connection, right? Yup. Or at least give it a lower cost in the routing table. -- Russell A. Brown| Oracle russ.br...@oracle.com | UMPK14-260 (650) 786-3011 (office) | 14 Network Circle (650) 786-3453 (fax)| Menlo Park, CA 94025