Linda and Lissa, You all have been extremely helpful, thank you very much! I've just about got all of this worked out in my head. Just one final minor question. In a service node pool architecture, if I wanted a specific group of nodes to only be able to netboot from a specific group of service nodes, would the correct method of handling this be to set disjointdhcps to "1", and then define the service nodes I would like that group to boot from in noderes? Or is there a better configuration somewhere for this?
Also, when setting a node to install, the following error was emitted: c1n01: Error: Unable to find requested nfsserver from noderes, with node=c1n01 Error: Some nodes failed to set up install resources, aborting c1n01: Error: Unable to find requested nfsserver from noderes, with node=c1n01 Error: Some nodes failed to set up install resources, aborting c1n01: Error: Unable to find requested nfsserver from noderes, with node=c1n01 Error: Some nodes failed to set up install resources, aborting Are service node pools designed only for diskless nodes? On Tue, Jul 19, 2011 at 10:14 AM, Linda Mellor <[email protected]> wrote: > Dave, > Let me try to explain a little more for you. I understand all the questions > and that this can be confusing -- xCAT can behave differently based on > various settings, and it's difficult for us to document this well for all > cases. > > First, some attributes that are of interest: > > *site attributes:* (lsdef -t site -h) > > master - your xCAT management node (MN) > > dhcpsetup - use default of "y" (no value) to have xCAT manage DHCP for > you > dhcpinterfaces - the interfaces DHCP should listen on. This should have > separate entries for your MN and for your service nodes (SNs). It will be > used to configure the dhcpd.conf files on the respective servers to only > listen on those network interfaces. It is important to get this correct, > and > you should verify the network definitions created by 'makedhcp -n' in the > respective dhcpd.conf files on your MN and SNs. > disjointdhcps - use default of "n" (no value) to have xCAT put an entry > in EVERY dhcpd.leases file for EVERY node on EVERY MN, SN > > sharedtftp - use default of "y" (no value) to have the /tftpboot > directory mounted from the MN to every SN > tftpdir - the /tftpboot directory to use > > installloc - "/install" means that all SNs will mount /install from the > MN. No value means that the SNs will have a local /install directory which > must be kept in sync from the MN for many xCAT commands > installdir - the /install directory to use > > > *network attributes:* (lsdef -t netwwork -h) > > dhcpserver - the DHCP server that will serve dynamic IPs for this > network. (required when there are multiple DHCP servers on the network and > a > dynamicrange is specified) > tftpserver - the tftpserver for this network. Typically, you will want > to set the node tftpserver attribute and leave this one blank. > gateway - set to value "<xcatmaster>" to have each DHCP servers > dhcpd.conf "option routers" set to itself. > dynamicrange - only set if you need to serve dynamic IP addresses on > this network > > *node attributes:* (lsdef -t node -h) > > servicenode - the xCAT node name of the service node (as known by the > MN) for this node. If not set, will default to site.xcatmaster. Set to list > for service node pools. > xcatmaster - the service node interface this node will use to > communicate to the SN (will be different from servicenode if the MN-to-SN > and SN-to-compute node are on different networks). If not set, will default > to servicenode. Keep blank for service node pools. > tftpserver - the tftpserver for this compute node. In most cases, you > should explicitly set this to your xcatmaster value for service node > support. Keep blank for service node pools. > nfsserver - the server that will serve nfs, http, etc., for node > deployment. If not set, will default to xcatmaster. Keep blank for service > node pools. > > *service node attributes:* (lsdef -t node -h) > > These should all be set to "1" to have the service node handle these > services for their compute nodes: > setupdhcp > setuptftp > setupftp > setupnfs > > > And the network possibilities with all of this can start to be > mind-boggling, but we try to address the most common ones as best we can: > > - the entire cluster on one flat network. This means there will be > multiple DHCP servers (and tftp servers), and xCAT needs to configure any > DHCP server to respond correctly to a broadcast request on the network, so > all dhcpd.leases files will need to be identical, with the "next-server" > value set to the designated tftpserver for a given node. > - the MN to SNs on one network, and the SNs to CNs all on a separate > flat network. Two possibilities here: > - specific DHCP servers for different sets of compute nodes. DHCP > configuration handled same as previous case. xCAT will use the > servicenode > attribute as the service node to handle hierarchical xCAT runtime > management > commands (such as xdsh, nodestat, updatenode, etc.) > - pooled DHCP servers so any server in the list can serve any > compute node on the network -- the first DHCP server to respond will > also be > that node's tftpserver, nfsserver, xcatmaster, etc.. xCAT will use the > first > servicenode in this list for hierarchical xCAT commands (to manage > performance of these cmds, you may want to set different groups of > nodes to > a different order of servicenodes) > - The MN to SNs on one network, each SN to its CNs on separate networks > -- the DHCP server on that network needs to only manage its set of compute > nodes. > > So, because of all the different possibilities, the xCAT makedhcp and > nodeset commands for any node are run on the MN and EVERY SN for EVERY node. > We try to be a little smarter in some cases where we can and where it is > important to get the correct server values set, but will typically default > to just doing everything everywhere. And depending on whether you are using > sharedtftp or not, you may see different results in your /tftpboot/etc files > and other places. > > Also, if your /install directory is local on your SNs, after MN commands > such as genimage, packimage, liteimg, updating postscripts, or making > changes to other files in /install, you need to rsync /install out to all > service nodes to make sure nodeset and other commands have the correct data > to work with. > > For your question: > > "Can you define more than one statically assigned Service Node to a > compute node?" > the answer is "No". You can either have 1 static service node, or use > service node pools. We do support a manual "snmove" command to move a > compute node from one static service node to another. > > In all cases, xCAT is designed so that all xCAT commands are run from the > MN. xCAT will do all internal remote communications to the SNs as needed and > consolidate/process the results. > > > Hope all of this helped more than it added more confusion to the mix. If > you have a specific end-result you're trying to achieve, let us know, and > we'll try to help you figure out how the xcat database should be set to > accomplish that (if possible). > > Linda > > [image: Inactive hide details for Dave Barry ---07/18/2011 11:41:56 > AM---Thanks! I am just trying to understand this part of xCAT more]Dave > Barry ---07/18/2011 11:41:56 AM---Thanks! I am just trying to understand > this part of xCAT more clearly so that I can make the correct > > > From: > Dave Barry <[email protected]> > To: > xCAT Users Mailing list <[email protected]> > Date: > 07/18/2011 11:41 AM > Subject: > Re: [xcat-user] Confused regarding service node pools and defining specific > service nodes > ------------------------------ > > > > Thanks! I am just trying to understand this part of xCAT more clearly so > that I can make the correct decisions for my setup. So let me make sure I > have this straight, please feel free to correct as needed: > > - For any nodes who are specifically assigned a service node (as > opposed to assigned to a service node pool), their management commands > (such > as nodeset etc.) should be ran on the Service Node they are assigned > to, and > not the Management Node. - I did notice that when defining the > "xcatmaster" in noderes for this compute node as its service node, even > though the management node could still DHCP boot the node, the imgurl > for > where to pull the netboot image was statically assigned to the service > node. > - When using service node pools, it becomes a first-come, > first-serve basis, in that any service node will have its imgurl defined > back to itself in /tftpboot, so that if that service node ends up being > the > one that pxe boots the compute node, it essentially becomes that compute > node's master. Management commands should be ran on the management node > instead of the service node in a service node pool architecture. > Correct? > > > > Questions: > > - Can you define more than one statically assigned Service Node to a > compute node? How would the imgurl that is defined in tftpboot be > handled in > this situation? > - I cannot seem to figure out how the "servicenode" column actually > comes into play in the various services configuration. I even tried > putting > fake service node hostnames in there that do not exist, and was still > able > to makedhcp, nodeset, and boot both diskless and install diskful nodes > from > either service node or master node with, what appeared to be, no ill > effects. What does this column actually affect, and when does it come > into > play in fail over situations? > > > > Sorry for the multiple questions, just trying to gather as much information > as possible =) I've read the service node pools documentation and > unfortunately, unless I missed something, it doesn't seem to go into as much > depth as I am trying to gather. > > > Here are my lsdef's. c1n01 is a diskful install, and c1n02 is a diskless > node. I was able to successfully boot c1n02 from service01, which is not > defined as that compute node's service node, without an issue. I even made > sure it was booting from service01 by stopping dhcp on the master node. > That's what is confusing to me... it would seem to me that if I have > service10 and service11 defined as c1n02's service nodes (which by the way > are non-existant service nodes), service01 wouldn't care about c1n02 and as > a result wouldn't create the tftpboot/dhcp configuration needed for that > node, only service10/11 would (if they existed). What is the actual purpose > of the servicenode column in noderes in a service node pool setup? > > > > Object name: c1n01 > arch=x86_64 > chain=runcmd=standby > currchain=boot > currstate=install centos5.5-x86_64-compute > groups=compute,all > initrd=xcat/centos5.5/x86_64/initrd.img > installnic=eth0 > interface=eth0 > ip=192.168.1.2 > kcmdline=nofb utf8 > ks=*http://mn/install/autoinst/c1n01*<http://mn/install/autoinst/c1n01>ksdevice=eth0 > noipv6 > kernel=xcat/centos5.5/x86_64/vmlinuz > mac=00:50:56:11:11:11 > mgt=ipmi > netboot=pxe > nfsserver=mn > nodetype=osi > ondiscover=nodediscover > os=centos5.5 > postbootscripts=otherpkgs > postscripts=updaterepos.sh,syslog,remoteshell,syncfiles > power=ipmi > primarynic=eth0 > profile=compute > provmethod=install > servicenode=service10,service11 > status=booted > statustime=07-15-2011 18:32:09 > [root@mn ~]# lsdef c1n02 > > Object name: c1n02 > arch=x86_64 > chain=runcmd=standby > currchain=boot > currstate=netboot centos5.5-x86_64-compute > groups=compute,all > initrd=xcat/netboot/centos5.5/x86_64/compute/initrd-stateless.gz > installnic=eth0 > interface=eth0 > ip=192.168.1.3 > > kcmdline=imgurl=http://!myipfn!/install/netboot/centos5.5/x86_64/compute/rootimg.gz > XCAT=!myipfn!:3001 ifname=eth0:00:50:56:11:11:15 netdev=eth0 > kernel=xcat/netboot/centos5.5/x86_64/compute/kernel > mac=00:50:56:11:11:15 > mgt=ipmi > netboot=pxe > nodetype=osi > ondiscover=nodediscover > os=centos5.5 > postbootscripts=otherpkgs > postscripts=updaterepos.sh,syslog,remoteshell,syncfiles > power=ipmi > primarynic=eth0 > profile=compute > provmethod=netboot > servicenode=service10,service11 > status=booted > statustime=07-17-2011 20:29:51 > > > > > > Thanks! > > > > On Mon, Jul 18, 2011 at 7:31 AM, Lissa Valletta > <*[email protected]*<[email protected]>> > wrote: > > So first do you really want service node pools or as you indicate below > you just want to assign a compute node to a particular service. If you > check > this link the two sections have a description of setting up Service Nodes > and pools: > > * > ** > > https://sourceforge.net/apps/mediawiki/xcat/index.php?title=Setting_Up_a_Linux_Hierarchical_Cluster#Assigning_Nodes_to_their_Service_Nodes_.28_updating_the_noderes_table.29 > > *<https://sourceforge.net/apps/mediawiki/xcat/index.php?title=Setting_Up_a_Linux_Hierarchical_Cluster#Assigning_Nodes_to_their_Service_Nodes_.28_updating_the_noderes_table.29> > > If you want compute1 to just use the Management Node, you do not have > to put anything in the noderes table for it. That will be the default, or > just assign the xcatmaster attribute to the Management node as know by > compute1. > > If not a pool then in the noderes table:* > The servicenode attribute for a compute node should be set to the > hostname of the service node(s) that the management node knows it by. The > xcatmaster attribute in the noderes table should be set to the hostname of > the service node that the compute node knows it by. * > Make sure your service node is defined in the servicenode table. > > For pools, make sure you note this restriction.* > Note: the noderes table's xcatmaster, tftpserver,nfsserver attributes > should be blank for any node entry that has the noderes servicenode > attribute set to a pool of service nodes.* > > > The command(s) that is reference, is all xCAT commands that will > actually be run on the servicenode at this point instead of the Management > Node, because the Service Node is the master of the compute node. Some > examples are your nodeset , nodestat, xdsh. Some of these commands do some > work on the Management Node ( our preprocess setup) before the actual > executing the real work on the Service Node. > > Also run lsdef compute1 and lsdef compute2, we can check the entire > setup from that output. > > Lissa K. Valletta > 2-3/T12 > Poughkeepsie, NY 12601 > (tie 293) 433-3102 > > > > [image: Inactive hide details for Dave Barry ---07/17/2011 05:45:56 > PM---Hello! I am attempting to understand how to manually lay out s]Dave > Barry ---07/17/2011 05:45:56 PM---Hello! I am attempting to understand how > to manually lay out specific service nodes > > From: Dave Barry <*[email protected]* <[email protected]>> > To: *[email protected]* <[email protected]> > Date: 07/17/2011 05:45 PM > Subject: [xcat-user] Confused regarding service node pools and defining > specific service nodes > ------------------------------ > > > > Hello! > > I am attempting to understand how to manually lay out specific service > nodes that are responsible to specific compute nodes, but am having a hard > time doing so. I read the following paragraph: > > > To define a list of service nodes that support a set of compute > node(s), in the noderes table, in the service node attribute, put a > comma-delimited list of the > service nodes. The list will be processed left to right, picking the > first service node on the list to run the command. If that service node is > not available, then > the next service node on the list will be chosen until the command is > successful. Errors will be logged. If no service node on the list can > process the > command, then the error will be returned. You can provide some > load-balancing by assigning your service nodes as we do below. > > > > I have tried manually defining my node "compute1" to have its > servicenode (in the noderes table) to be my masternode, and then defining > compute2 to have its servicenode be sn1. However when I run "nodeset > compute1 netboot" > the command appears to be sent to both the master node and the service > node. The same happens if I do "nodeset compute2 netboot". The /tftpboot > files and /install/autoinst files are written out on both the masternode > and > service node > as if xcat is ignoring the fact that I have separated these two compute > nodes to different service nodes. I am successfully able to netboot > compute1 > from service01 without any problems. DHCP from service01 will happily > respond and boot this node even though it is not assigned as this node's > servicenode. > > > Am I misunderstanding how this is supposed to work? > > Also: "The list will be processed left to right, picking the first > service node on the list to run the command. If that service node is not > available, then > the next service node on the list will be chosen until the command is > successful." > > > What "command" is this documentation referring to? > > > > Thanks! > > ------------------------------------------------------------------------------ > AppSumo Presents a FREE Video for the SourceForge Community by Eric > Ries, the creator of the Lean Startup Methodology on "Lean Startup > Secrets Revealed." This video shows you how to validate your ideas, > optimize your ideas and identify your business strategy.* > > **http://p.sf.net/sfu/appsumosfdev2dev*<http://p.sf.net/sfu/appsumosfdev2dev> > _______________________________________________ > xCAT-user mailing list* > **[email protected]* <[email protected]>* > > **https://lists.sourceforge.net/lists/listinfo/xcat-user*<https://lists.sourceforge.net/lists/listinfo/xcat-user> > > > > > ------------------------------------------------------------------------------ > AppSumo Presents a FREE Video for the SourceForge Community by Eric > Ries, the creator of the Lean Startup Methodology on "Lean Startup > Secrets Revealed." This video shows you how to validate your ideas, > optimize your ideas and identify your business strategy.* > > **http://p.sf.net/sfu/appsumosfdev2dev*<http://p.sf.net/sfu/appsumosfdev2dev> > _______________________________________________ > xCAT-user mailing list* > **[email protected]* <[email protected]>* > > **https://lists.sourceforge.net/lists/listinfo/xcat-user*<https://lists.sourceforge.net/lists/listinfo/xcat-user> > > > ------------------------------------------------------------------------------ > AppSumo Presents a FREE Video for the SourceForge Community by Eric > Ries, the creator of the Lean Startup Methodology on "Lean Startup > Secrets Revealed." This video shows you how to validate your ideas, > optimize your ideas and identify your business strategy. > http://p.sf.net/sfu/appsumosfdev2dev > _______________________________________________ > xCAT-user mailing list > [email protected] > https://lists.sourceforge.net/lists/listinfo/xcat-user > > > > > ------------------------------------------------------------------------------ > Magic Quadrant for Content-Aware Data Loss Prevention > Research study explores the data loss prevention market. Includes in-depth > analysis on the changes within the DLP market, and the criteria used to > evaluate the strengths and weaknesses of these DLP solutions. > http://www.accelacomm.com/jaw/sfnl/114/51385063/ > _______________________________________________ > xCAT-user mailing list > [email protected] > https://lists.sourceforge.net/lists/listinfo/xcat-user > >
<<ecblank.gif>>
<<graycol.gif>>
------------------------------------------------------------------------------ Magic Quadrant for Content-Aware Data Loss Prevention Research study explores the data loss prevention market. Includes in-depth analysis on the changes within the DLP market, and the criteria used to evaluate the strengths and weaknesses of these DLP solutions. http://www.accelacomm.com/jaw/sfnl/114/51385063/
_______________________________________________ xCAT-user mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/xcat-user
