I'm working in a distributed computing project, in which we have many small, interconnected, clusters. So I maintain over a dozen installs of xCAT, but each xCAT install may only control 8 to 15 nodes. xCAT is necessary due to the churn in these nodes, which are reinstalled constantly.
So to answer your question, the two nodes I cited are in two completely different clusters. I don't believe name resolution is a problem, namely because in addition to DNS, I have static hosts files everywhere. Even on the nodes exhibiting the issue I described, I can resolve the xcatmaster from the node, and the node from the xcatmaster. If there is something specific to configuring xCAT for multiple domains...beyond the networks table...I'm unaware of it. Was there something specific you were thinking of? On 09/30/2013 03:31 PM, Linda Mellor wrote: > It looks like you have already done alot of very good debug on this. > One question: are both the bad node and the good node that you have > trace output for being managed by the same xCAT management node? The > reason I am asking is that you may have a name resolution problem. For > the bad node you listed: > > > '_xcat_clientfqdn' => [ 'sl-w8.local' ], > > > > but the good node is listed as > > > '_xcat_clientfqdn' => [ 'compute-0-0.cluster' ], > > > The nodes are in 2 different domains. Was that intentional? If so, and > if these are in the same cluster, do you have xCAT configured for > multiple domains? > > Linda > > Inactive hide details for Jonathan Mills ---09/25/2013 09:23:20 AM---No. > There is no firewall. On the client, iptables is disabJonathan Mills > ---09/25/2013 09:23:20 AM---No. There is no firewall. On the client, > iptables is disabled. On the server, the cluster-facing int > > From: Jonathan Mills <[email protected]> > To: xCAT Users Mailing list <[email protected]>, > Cc: xCAT Users Mailing list <[email protected]> > Date: 09/25/2013 09:23 AM > Subject: Re: [xcat-user] xCAT 2.8.2: Remoteshell postscript infinite > loop with getcredentials.awk failure > ------------------------------------------------------------------------ > > > > No. There is no firewall. On the client, iptables is disabled. On the > server, the cluster-facing interface is whitelisted to allow all traffic > > On Sep 25, 2013, at 8:34 AM, "Lissa Valletta" <[email protected]_ > <mailto:[email protected]>> wrote: > > > Is there any chance that a firewall is still enabled on the nodes > that are not working. > > Lissa K. Valletta > 8-3/B10 > Poughkeepsie, NY 12601 > (tie 293) 433-3102 > > > > <graycol.gif>Jonathan Mills ---09/24/2013 04:58:11 PM---I am having > serious trouble with this issue, and it is impeding all kinds of > work on our projects. > > From: Jonathan Mills <[email protected]_ <mailto:[email protected]>> > To: Ling Gao/Poughkeepsie/IBM@IBMUS, xCAT Users Mailing list > <[email protected]_ > <mailto:[email protected]>>, > Date: 09/24/2013 04:58 PM > Subject: Re: [xcat-user] xCAT 2.8.2: Remoteshell postscript infinite > loop with getcredentials.awk failure > > ------------------------------------------------------------------------ > > > > I am having serious trouble with this issue, and it is impeding all > kinds of work on our projects. I am no closer to solving this problem. > > All of the xCAT installations I'm using are 2.8.2 on CentOS 6.4, and to > my knowledge are all configured alike. Yet some are working and some > are not. > > The crux seems to come down to these awk scripts, wherein the script > pipes some XML-formatted directives through an openssl tunnel, and > waits > to hear a response from xcatd on port 300 (opened by allowcred.awk on > the client). > > To break down the issue, I've simulated what it does, but running it by > hand to watch it unfold. I've also enabled xcatdebug with > > xcatdebug -f enable -c /opt/xcat/share/xcat/samples/tracelevel0 -d > enable > > After that, I watch the debug log: > > tailf /var/log/xcat/subcallingtrace > > Then I mimic the interaction of getcredentials.awk attempting to fetch > the ssh_dsa_hostkey (as it does in 'remoteshell' postscript). You can > do this by running, on the client: > > allowcreds.awk & > openssl s_client -quiet -connect XCATMASTER:3001 2> /dev/null > > That opens an ssl connection. Then paste > > <xcatrequest> > <command>getcredentials</command> > <callback_port>300</callback_port> > <arg>"ssh_dsa_hostkey"</arg> > </xcatrequest> > > So here's what the whole scenario looks like, on a busted node: > > [root@sl-w8 xcatpost]# openssl s_client -quiet -connect 10.100.0.1:3001 > 2> /dev/null > <xcatrequest> > <command>getcredentials</command> > <callback_port>300</callback_port> > <arg>"ssh_dsa_hostkey"</arg> > </xcatrequest> > <xcatresponse> > <serverdone></serverdone> > </xcatresponse> > > >From the server side, here's what I see: > > /var/log/messages: > Sep 24 15:27:21 sl-hn xCAT: xCAT: Allowing getcredentials > "ssh_dsa_hostkey" from sl-w8 > > /var/log/xcat/subcallingtrace: > --------------------------------------------------------- > ***************Calling of subroutine > *main::service_connection*************** > Tue Sep 24 15:27:12 2013 > Arguments: > $VAR1 = bless( \*Symbol::GEN85, 'IO::Socket::SSL' ); > $VAR2 = undef; > $VAR3 = 'sl-w8'; > $VAR4 = 'sl-w8.local'; > $VAR5 = 'sl-w8.local'; > > Calling stack: > at /opt/xcat/sbin/xcatd line 1022 > > > ***************Calling of subroutine > *main::plugin_command*************** > Tue Sep 24 15:27:21 2013 > Arguments: > $VAR1 = { > '_xcat_clientfqdn' => [ > 'sl-w8.local' > ], > 'arg' => [ > '"ssh_dsa_hostkey"' > ], > '_xcat_clientport' => [ > 33436 > ], > '_xcat_authname' => [ > undef > ], > 'username' => [ > undef > ], > 'callback_port' => [ > '300' > ], > 'command' => [ > 'getcredentials' > ], > '_xcat_clienthost' => [ > 'sl-w8' > ] > }; > $VAR2 = bless( \*Symbol::GEN85, 'IO::Socket::SSL' ); > $VAR3 = sub { "DUMMY" }; > > Calling stack: > at /opt/xcat/sbin/xcatd line 1903 > eval {...} called at /opt/xcat/sbin/xcatd line 1849 > main::service_connection called at /opt/xcat/sbin/xcatd line > 2253 > main::__ANON__('IO::Socket::SSL=GLOB(0x3b5ff98)', undef, > 'sl-w8', 'sl-w8.local', 'sl-w8.local') called at /opt/xcat/sbin/xcatd > line 1022 > ----------------------------------------------------------------------- > > Above, '_xcat_clienthost' is resolved to 'sl-w8'. That node exists in > nodelist: > > # nodels sl-w8 > sl-w8 > > What's awful about this is that that output looks exactly the same as > what I see on the xCAT sites that are working correctly: > > [root@compute-0-0 xcatpost]# openssl s_client -quiet -connect > 10.4.0.10:3001 2> /dev/null > <xcatrequest> > <command>getcredentials</command> > <callback_port>300</callback_port> > <arg>"ssh_dsa_hostkey"</arg> > </xcatrequest> > <xcatresponse> > <data> > <content> > -----BEGIN DSA PRIVATE KEY----- > MIIBuwIBAAKBgQDAqbLr7+pGxXEvt054YNlxnCpZBq8EoQj9R+TKChudtWHPYvFw > EgGC62r5Cr9+cd3oJtaAnYE0s3GtrfMbtCNzL/IWdPsneHnimB5o0ziFCxXNPITb > AJpso+ab+Gs/Gb+N4+oWB1vZcxG3niY+ynMXuC/aph/zcw7WdqQ1sJEa1wIVAKHG > ....excerpted..... > 2Fk6qah4bPyStwwgBsGsc3TJdycPWrk662WkAJ9LxlboPyqJ9y47BOc0WotfbnOx > NP9CNrHxC0kOaVyKEaMMs2RsbL13PPgm4z85ZOEwzbQ3BbF+6X93R/8CFHgXYeSX > gXaXNEKdWfl1fWh05Amf > -----END DSA PRIVATE KEY----- > </content> > <desc>"ssh_dsa_hostkey"</desc> > </data> > <errorcode>0</errorcode> > </xcatresponse> > <xcatresponse> > <serverdone></serverdone> > </xcatresponse> > > And here is the server response from the server where this transaction > works: > > ***************Calling of subroutine > *main::service_connection*************** > Tue Sep 24 15:26:28 2013 > Arguments: > $VAR1 = bless( \*Symbol::GEN56, 'IO::Socket::SSL' ); > $VAR2 = undef; > $VAR3 = 'compute-0-0'; > $VAR4 = 'compute-0-0.cluster'; > $VAR5 = 'compute-0-0.cluster'; > > Calling stack: > at /opt/xcat/sbin/xcatd line 1022 > > Arguments: > $VAR1 = bless( \*Symbol::GEN56, 'IO::Socket::SSL' ); > $VAR2 = undef; > $VAR3 = 'compute-0-0'; > $VAR4 = 'compute-0-0.cluster'; > $VAR5 = 'compute-0-0.cluster'; > > Calling stack: > at /opt/xcat/sbin/xcatd line 1022 > > > ***************Calling of subroutine > *main::plugin_command*************** > Tue Sep 24 15:26:28 2013 > Arguments: > $VAR1 = { > '_xcat_clientfqdn' => [ > 'compute-0-0.cluster' > ], > 'arg' => [ > 'ssh_dsa_hostkey' > ], > '_xcat_clientport' => [ > 33782 > ], > '_xcat_authname' => [ > undef > ], > 'username' => [ > undef > ], > 'callback_port' => [ > '300' > ], > 'command' => [ > 'getcredentials' > ], > '_xcat_clienthost' => [ > 'compute-0-0' > ] > }; > $VAR2 = bless( \*Symbol::GEN56, 'IO::Socket::SSL' ); > $VAR3 = sub { "DUMMY" }; > > Calling stack: > at /opt/xcat/sbin/xcatd line 1903 > eval {...} called at /opt/xcat/sbin/xcatd line 1849 > main::service_connection called at /opt/xcat/sbin/xcatd line 2253 > main::__ANON__('IO::Socket::SSL=GLOB(0x34bfc70)', undef, 'compute-0-0', > 'compute-0-0.cluster', 'compute-0-0.cluster') called at > /opt/xcat/sbin/xcatd line 1022 > > > > On 09/12/2013 03:40 PM, Ling Gao wrote: > > One more thing to check: please make sure the name resolution is > > correct. Make sure node ip that's facing the management node > resolves to > > a node name in the nodelist table. > > > > Ling > > > > Ling Gao > > Poughkeepsie Unix Development Lab > > IBM Systems and Technology Group > > Internal: T/L 293-5692 > > External: [email protected]_ <mailto:[email protected]>, > 845-433-5692 > > > > "I never worry about the future. It comes soon enough." --- > Albert Einstein > > > > > > > > From: Jonathan Mills <[email protected]_ > <mailto:[email protected]>> > > To: xCAT Users Mailing list <[email protected]_ > <mailto:[email protected]>>, > > Date: 09/12/2013 02:57 PM > > Subject: Re: [xcat-user] xCAT 2.8.2: Remoteshell postscript infinite > > loop with getcredentials.awk failure > > > ------------------------------------------------------------------------ > > > > > > > > But it isn't missing, nor empty: > > > > # cat /etc/xcat/hostkeys/ssh_host_dsa_key > > -----BEGIN DSA PRIVATE KEY----- > > MIIBvAIBAAKBgQCPWuNyJmvv5cbM4fziswR5RT/boZooaFQdXz7jJw8zhx/Y+V3R > > K8C5ikBQuzJtm0nOy+C1J4sT9ncP4m04R5y3OCqCX60ka5x1N3DAtv2JMoBCu3uZ > > ...excerpted... > > 8084a+DNMO/TWcDU4m4IZeLgkVHIe/jg7tInjfTX6JFQlGwtuYGGs2NQP/LrYYZ8 > > QnnvHLm4615PKcgxOBltXUSzbEW4H180vS+ZmA8y2yOxfJtMDB+3+wkCFQDjSxJQ > > i7D2jHnS+6W0imBUvge1ng== > > -----END DSA PRIVATE KEY----- > > > > I will try a regenerate, but there doesn't appear to be anything > wrong > > with the exists contents of /etc/xcat/hostkeys. > > > > > > On 09/12/2013 02:49 PM, Lissa Valletta wrote: > > > One suggestion is that the hostkey we are trying to download, > is no > > > longer there or is there and the file is empty. Look in your > > > /etc/xcat/hostkeys directory on the Management Node. > > > You can regenerate them by running xcatconfig -s on the > management node. > > > > > > Thanks for pointing out the infinite loop though. I have opened a > > defect. > > > _https://sourceforge.net/p/xcat/bugs/3781/_ > > > > > > > > > > > > > > > Lissa K. Valletta > > > 8-3/B10 > > > Poughkeepsie, NY 12601 > > > (tie 293) 433-3102 > > > > > > > > > > > > Inactive hide details for Jonathan Mills ---09/12/2013 > 12:13:35 PM---I'm > > > looking at a real problem with my xCAT installation whJonathan > Mills > > > ---09/12/2013 12:13:35 PM---I'm looking at a real problem with > my xCAT > > > installation where the remoteshell postscript is failing > > > > > > From: Jonathan Mills <[email protected]_ > <mailto:[email protected]>> > > > To: xCAT Users Mailing list <[email protected]_ > <mailto:[email protected]>>, > > > Date: 09/12/2013 12:13 PM > > > Subject: [xcat-user] xCAT 2.8.2: Remoteshell postscript > infinite loop > > > with getcredentials.awk failure > > > > > > > ------------------------------------------------------------------------ > > > > > > > > > > > > I'm looking at a real problem with my xCAT installation where the > > > remoteshell postscript is failing to work. > > > > > > Digging into the script (line numbers on left), here it spawns a > > > listener on port 300: > > > > > > 66 allowcred.awk & > > > > > > You can see the result of that command like so: > > > # netstat -tulpn | grep awk > > > tcp 0 0 0.0.0.0:300 0.0.0.0:* > > > LISTEN 3119/awk > > > > > > Then in line 75 of remoteshell it runs getcredentials.awk with the > > > parameter 'ssh_dsa_hostkey', redirecting the output to > > /tmp/ssh_dsa_hostkey: > > > > > > 75 getcredentials.awk ssh_dsa_hostkey | grep -E -v > > > '</{0,1}xcatresponse>|</{0,1}serverdone>' | sed -e 's/</</' -e > > > 's/>/>/' -e 's/&/&/' -e 's/&qu ot/"/' -e "s/'/'/" > > > > /tmp/ssh_dsa_hostkey > > > > > > > > > What the getcredentials.awk script does is: > > > "openssl s_client -quiet -connect XCATSERVER:3001" and tells > xCATd to > > > respond on port 300 (opened above from allowcred.awk). > > > > > > A correct interaction of these parts has output that looks like: > > > > > > # ./getcredentials.awk ssh_dsa_hostkey > > > <xcatresponse> > > > <data> > > > <content> > > > -----BEGIN DSA PRIVATE KEY----- > > > MIIBuwIBAAKBgQCzrhm0rTC9FcCnTjuI5QGh7MV2oczIwqBWhBlid7748dcdF23M > > > KLwbuopN1y2/6l1XmyS2ama+vtbUnpFZ4yQcdX99QXZq3UgZPsVayKmPjgjNmV99 > > > xm+RMuoMBCkh12dL5JVc1auJKGupaz+yVEol4hCkRAYb/E4Ft/5GroP/jQIVAMHB > > > .....excerpt...... > > > dxEhemsBz9FwP7e3VB8yc8T/vXTswuYrDzslGL22Z3Il87MtHXO5pdO5HFLnmPan > > > w0NyyWKdxfSSwfxbTNRRTVDi7XTg3CfWnsIRI98EsIwmxUe0QcV6lwIVAJdtvPLA > > > Blck9nPWNaBqTbpBznj+ > > > -----END DSA PRIVATE KEY----- > > > </content> > > > <desc>ssh_dsa_hostkey</desc> > > > </data> > > > <errorcode>0</errorcode> > > > </xcatresponse> > > > <xcatresponse> > > > <serverdone></serverdone> > > > </xcatresponse> > > > > > > > > > However, on my faulty xCAT 2.8.2 install, here is what happens: > > > > > > # ./getcredentials.awk ssh_dsa_hostkey > > > <xcatresponse> > > > <serverdone></serverdone> > > > </xcatresponse> > > > > > > This leaves /tmp/ssh_dsa_hostkey empty. At line 81 of > remoteshell, it > > > tries to populate /etc/ssh/ssh_host_dsa_key with the value of > > > /tmp/ssh_dsa_hostkey: > > > > > > 81 cat /tmp/ssh_dsa_hostkey | grep -E -v > > > '</{0,1}data>|</{0,1}content>|</{0,1}desc>' > >/etc/ssh/ssh_host_dsa_key > > > > > > Then at lines 83 & 84 of remoteshell, it does this: > > > > > > 83 MYCONT=`cat /etc/ssh/ssh_host_dsa_key` > > > 84 while [ -z "$MYCONT" ]; do > > > > > > So $MYCONT is always an empty value, thus dropping that while > loop into > > > an infinite loop. > > > > > > > > > Moreover, please note that on my faulty xCAT install, the > policy table > > > is set to allow 'getcredentials': > > > # tabdump policy > > > > > > > #priority,name,host,commands,noderange,parameters,time,rule,comments,disable > > > "1","root",,,,,,"allow",, > > > ......excerpted > > > "4.4",,,"getpostscript",,,,"allow",, > > > "4.5",,,"getcredentials",,,,"allow",, > > > ....etc > > > > > > And in /var/log/messages on the XCATMASTER, I can even see > where xCATd > > > is granting these requests: > > > Sep 8 03:10:59 <xcatmaster_hostname> xCAT: xCAT: Allowing > > > getcredentials ssh_dsa_hostkey from <hostname> > > > > > > > > > It must be the case that there is a subtle misconfiguration in > this xCAT > > > 2.8.2 installation, as I have other installs of 2.8.2 where > this is > > > working. I would appreciate it if anyone can help me > understand why > > > xCATd is failing to return the ssh_dsa_hostkey. > > > > > > -- > > > Jonathan Mills > > > Systems Administrator > > > Renaissance Computing Institute > > > UNC-Chapel Hill > > > > > > > > > > ------------------------------------------------------------------------------ > > > How ServiceNow helps IT people transform IT departments: > > > 1. Consolidate legacy IT systems to a single system of record > for IT > > > 2. Standardize and globalize service processes across IT > > > 3. Implement zero-touch automation to replace manual, > redundant tasks > > > > > > > _http://pubads.g.doubleclick.net/gampad/clk?id=51271111&iu=/4140/ostg.clktrk_ > > > _______________________________________________ > > > xCAT-user mailing list > > > [email protected]_ > <mailto:[email protected]> > > > _https://lists.sourceforge.net/lists/listinfo/xcat-user_ > > > > > > > > > > -- > > Jonathan Mills > > Systems Administrator > > Renaissance Computing Institute > > UNC-Chapel Hill > > > > > > ------------------------------------------------------------------------------ > > How ServiceNow helps IT people transform IT departments: > > 1. Consolidate legacy IT systems to a single system of record for IT > > 2. Standardize and globalize service processes across IT > > 3. Implement zero-touch automation to replace manual, redundant tasks > > > > _http://pubads.g.doubleclick.net/gampad/clk?id=51271111&iu=/4140/ostg.clktrk_ > > _______________________________________________ > > xCAT-user mailing list > > [email protected]_ > <mailto:[email protected]> > > _https://lists.sourceforge.net/lists/listinfo/xcat-user_ > > > > > > -- > Jonathan Mills > Systems Administrator > Renaissance Computing Institute > UNC-Chapel Hill > > > ------------------------------------------------------------------------------ > October Webinars: Code for Performance > Free Intel webinars can help you accelerate application performance. > Explore tips for MPI, OpenMP, advanced profiling, and more. Get the > most from > the latest Intel processors and coprocessors. See abstracts and > register >_ > > __http://pubads.g.doubleclick.net/gampad/clk?id=60133471&iu=/4140/ostg.clktrk_ > _______________________________________________ > xCAT-user mailing list_ > [email protected]_ > <mailto:[email protected]>_ > __https://lists.sourceforge.net/lists/listinfo/xcat-user_ > > > > ------------------------------------------------------------------------------ > October Webinars: Code for Performance > Free Intel webinars can help you accelerate application performance. > Explore tips for MPI, OpenMP, advanced profiling, and more. Get the > most from > the latest Intel processors and coprocessors. See abstracts and > register >_ > > __http://pubads.g.doubleclick.net/gampad/clk?id=60133471&iu=/4140/ostg.clktrk_ > _______________________________________________ > xCAT-user mailing list_ > [email protected]_ > <mailto:[email protected]>_ > > __https://lists.sourceforge.net/lists/listinfo/xcat-user_------------------------------------------------------------------------------ > October Webinars: Code for Performance > Free Intel webinars can help you accelerate application performance. > Explore tips for MPI, OpenMP, advanced profiling, and more. Get the > most from > the latest Intel processors and coprocessors. See abstracts and > register > > > http://pubads.g.doubleclick.net/gampad/clk?id=60133471&iu=/4140/ostg.clktrk_______________________________________________ > xCAT-user mailing list > [email protected] > https://lists.sourceforge.net/lists/listinfo/xcat-user > -- Jonathan Mills Systems Administrator Renaissance Computing Institute UNC-Chapel Hill ------------------------------------------------------------------------------ October Webinars: Code for Performance Free Intel webinars can help you accelerate application performance. Explore tips for MPI, OpenMP, advanced profiling, and more. Get the most from the latest Intel processors and coprocessors. See abstracts and register > http://pubads.g.doubleclick.net/gampad/clk?id=60133471&iu=/4140/ostg.clktrk _______________________________________________ xCAT-user mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/xcat-user
