Ah, I see that you are already setting it up with separate subdomains. My apologies, missed that section when I was quickly skimming :-)
Seems like really the only issue is the search path on the headnode seeing the 10 gig first before the 1 gig management network. Not sure if there's a way around that without changing the search path on the headnode and changing any jobs/applications you are running to look at the appropriate subdomain as opposed to relying on search paths to get the traffic on the fastest network. On 10/7/2013 5:50 PM, Russell Jones wrote: > > After setting that, everything looks fine, except for the lease > written into /var/lib/dhcpd/dhcpd.leases (still has the 10.233 subnet, > which is 10GbE). > > > It sounds like you are not using the headnode for DNS management of the > cluster address space, correct? > > It seems you are setting DNS up quite a bit differently than is usually > done (in my experience). The short name with no extra dashes is usually > used for the management network, then from there $HOSTNAME-eth1, > $HOSTNAME-eth2, etc resolves to their appropriate interfaces. > > In addition, most sites I have done cluster management at use a separate > sub/domain all together for the different networks. Example: > node1.cluster = management, node1.10g.cluster = storage, > node1.pub.cluster = public > > I'm not sure the best route to take for getting your IP addresses > resolved in the appropriate order. Relying on the search path to ensure > traffic gets where it needs to go seems like sort of a dangerous way of > doing it. > > > > > > On 10/7/2013 4:55 PM, Steve Crusan wrote: >> Russell, >> >> Thanks for the response. I think we've gotten closer. After setting >> that, everything looks fine, except for the lease written into >> /var/lib/dhcpd/dhcpd.leases (still has the 10.233 subnet, which is 10GbE). >> >> I'll pour through the dhcp.pm module to see if I find out what's >> happening. >> >> xCAT aficionado's out there, this problem occurs when I run: makedhcp >> $shortHostname. If you recall, I believe it's because xCAT does a hostname >> lookup on the nodeName, which in our case resolves to 10.233.1.68, due to >> how our DNS is setup (see first message). >> >> Any way to get around that? >> >> ~Steve >> >> On Oct 7, 2013, at 2:18 PM, Russell Jones wrote: >> >>> Set noderes.nfsserver to the correct IP address to use for your >>> kickstart repo. >>> >>> >>> >>> >>> On 10/7/2013 1:33 PM, Steve Crusan wrote: >>>> Hi all, >>>> >>>> Our network setup is like so: >>>> - 1Gb ethernet (deployment / xCAT / management network) >>>> - 10Gb ethernet >>>> - out-of-band (IMM/BMC) >>>> >>>> Right now, our DNS is handled by setting up search paths on our xcat >>>> server (via /etc/resolv.conf), with the *fastest* paths being resolved >>>> first. So, 10Gb is resolved before 1Gb, like such: >>>> >>>> $ cat /etc/resolv.conf >>>> nameserver 10.230.16.5 >>>> nameserver 10.230.16.6 >>>> search fst.domain.com mcp.domain.com lom.domain.com i2b.domain.com >>>> domain.com >>>> $ host testcoreb >>>> testcoreb.fst.domain.com has address 10.233.1.68 >>>> $ host testcoreb.mcp >>>> testcoreb.mcp.domain.com has address 10.230.1.68 >>>> >>>> 10.233 is 10Gb ethernet, and 10.230 is 1Gb ethernet. We DO NOT want to >>>> provision/manage (psh, etc) over 10Gb, but rather the 1G ethernet. We >>>> don't use slashes in our short hostnames (hostname-mgmt), which I think is >>>> causing a problem. Some hosts that do not have a 10GbE provision fine, >>>> because the short name lookup defaults to that node's 10.230 network, >>>> which is correct: >>>> $ host fsm07 >>>> fsm07.mcp.domain.com has address 10.230.1.87 >>>> >>>> We only have dhcpd listening on 1Gb ethernet as well: >>>> $ tabdump site | grep dhcp >>>> "dhcpinterfaces","eth0",, >>>> >>>> When I do a nodeset testcoreb osimage=imageName, the 10.233 interface is >>>> used, which is wrong: >>>> kcmdline=quiet repo=http://10.233.1.5:80/install/rhels6.4/x86_64 >>>> ks=http://10.233.1.5:80/install/autoinst/testcoreb.mcp >>>> ksdevice=40:f2:e9:0c:ef:04 cmdline console=tty0 console=ttyS19200,hard >>>> >>>>> From dhcpd.leases: >>>> host testcoreb { >>>> dynamic; >>>> hardware ethernet 40:f2:e9:0c:ef:04; >>>> fixed-address 10.233.1.68; >>>> supersede server.ddns-hostname = "testcoreb"; >>>> <SNIPPED> >>>> >>>> Is there a way to explicitly tell xCAT *not* to do a DNS lookup for the >>>> shortname, and determine which interface to set for the tftp server? I've >>>> tried the following in the noders table, without any success: >>>> >>>> $ tabdump noderes | grep testcoreb >>>> "testcoreb",,"pxe","10.230.1.5",,"10.230.1.5","10.230.1.5",,"mac","mac",,,,,,,,, >>>> >>>> I've tried manually setting all of this in the bootparam table, etc, >>>> without any success. >>>> >>>> So, why long-winded question is, am I just missing setting a key somewhere >>>> in one of the 40+ xCAT tables to override the short name lookups, and >>>> explicitly use the 1Gb ethernet network? I haven't even defined the 10Gb >>>> ethernet network in the xCAT networks table either… >>>> >>>> Thanks, >>>> >>>> ~Steve >>>> >>>> >>>> >>>> >>>> >>>> ------------------------------------------------------------------------------ >>>> October Webinars: Code for Performance >>>> Free Intel webinars can help you accelerate application performance. >>>> Explore tips for MPI, OpenMP, advanced profiling, and more. Get the most >>>> from >>>> the latest Intel processors and coprocessors. See abstracts and register > >>>> http://pubads.g.doubleclick.net/gampad/clk?id=60134071&iu=/4140/ostg.clktrk >>>> _______________________________________________ >>>> xCAT-user mailing list >>>> xCAT-user@lists.sourceforge.net >>>> https://lists.sourceforge.net/lists/listinfo/xcat-user >>>> >>> ------------------------------------------------------------------------------ >>> October Webinars: Code for Performance >>> Free Intel webinars can help you accelerate application performance. >>> Explore tips for MPI, OpenMP, advanced profiling, and more. Get the most >>> from >>> the latest Intel processors and coprocessors. See abstracts and register > >>> http://pubads.g.doubleclick.net/gampad/clk?id=60134071&iu=/4140/ostg.clktrk >>> _______________________________________________ >>> xCAT-user mailing list >>> xCAT-user@lists.sourceforge.net >>> https://lists.sourceforge.net/lists/listinfo/xcat-user >> -------------------------------------- >> Steve Crusan >> HPC System Administrator - Filesystems >> Leadership Computing Facility >> Argonne National Laboratory >> >> >> >> >> >> ------------------------------------------------------------------------------ >> October Webinars: Code for Performance >> Free Intel webinars can help you accelerate application performance. >> Explore tips for MPI, OpenMP, advanced profiling, and more. Get the most from >> the latest Intel processors and coprocessors. See abstracts and register > >> http://pubads.g.doubleclick.net/gampad/clk?id=60134071&iu=/4140/ostg.clktrk >> _______________________________________________ >> xCAT-user mailing list >> xCAT-user@lists.sourceforge.net >> https://lists.sourceforge.net/lists/listinfo/xcat-user >> > > ------------------------------------------------------------------------------ > October Webinars: Code for Performance > Free Intel webinars can help you accelerate application performance. > Explore tips for MPI, OpenMP, advanced profiling, and more. Get the most from > the latest Intel processors and coprocessors. See abstracts and register > > http://pubads.g.doubleclick.net/gampad/clk?id=60134071&iu=/4140/ostg.clktrk > _______________________________________________ > xCAT-user mailing list > xCAT-user@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/xcat-user > ------------------------------------------------------------------------------ October Webinars: Code for Performance Free Intel webinars can help you accelerate application performance. Explore tips for MPI, OpenMP, advanced profiling, and more. Get the most from the latest Intel processors and coprocessors. See abstracts and register > http://pubads.g.doubleclick.net/gampad/clk?id=60134071&iu=/4140/ostg.clktrk _______________________________________________ xCAT-user mailing list xCAT-user@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xcat-user
