I am having serious trouble with this issue, and it is impeding all
kinds of work on our projects. I am no closer to solving this problem.
All of the xCAT installations I'm using are 2.8.2 on CentOS 6.4, and to
my knowledge are all configured alike. Yet some are working and some
are not.
The crux seems to come down to these awk scripts, wherein the script
pipes some XML-formatted directives through an openssl tunnel, and waits
to hear a response from xcatd on port 300 (opened by allowcred.awk on
the client).
To break down the issue, I've simulated what it does, but running it by
hand to watch it unfold. I've also enabled xcatdebug with
xcatdebug -f enable -c /opt/xcat/share/xcat/samples/tracelevel0 -d enable
After that, I watch the debug log:
tailf /var/log/xcat/subcallingtrace
Then I mimic the interaction of getcredentials.awk attempting to fetch
the ssh_dsa_hostkey (as it does in 'remoteshell' postscript). You can
do this by running, on the client:
allowcreds.awk &
openssl s_client -quiet -connect XCATMASTER:3001 2> /dev/null
That opens an ssl connection. Then paste
<xcatrequest>
<command>getcredentials</command>
<callback_port>300</callback_port>
<arg>"ssh_dsa_hostkey"</arg>
</xcatrequest>
So here's what the whole scenario looks like, on a busted node:
[root@sl-w8 xcatpost]# openssl s_client -quiet -connect 10.100.0.1:3001
2> /dev/null
<xcatrequest>
<command>getcredentials</command>
<callback_port>300</callback_port>
<arg>"ssh_dsa_hostkey"</arg>
</xcatrequest>
<xcatresponse>
<serverdone></serverdone>
</xcatresponse>
From the server side, here's what I see:
/var/log/messages:
Sep 24 15:27:21 sl-hn xCAT: xCAT: Allowing getcredentials
"ssh_dsa_hostkey" from sl-w8
/var/log/xcat/subcallingtrace:
---------------------------------------------------------
***************Calling of subroutine
*main::service_connection***************
Tue Sep 24 15:27:12 2013
Arguments:
$VAR1 = bless( \*Symbol::GEN85, 'IO::Socket::SSL' );
$VAR2 = undef;
$VAR3 = 'sl-w8';
$VAR4 = 'sl-w8.local';
$VAR5 = 'sl-w8.local';
Calling stack:
at /opt/xcat/sbin/xcatd line 1022
***************Calling of subroutine *main::plugin_command***************
Tue Sep 24 15:27:21 2013
Arguments:
$VAR1 = {
'_xcat_clientfqdn' => [
'sl-w8.local'
],
'arg' => [
'"ssh_dsa_hostkey"'
],
'_xcat_clientport' => [
33436
],
'_xcat_authname' => [
undef
],
'username' => [
undef
],
'callback_port' => [
'300'
],
'command' => [
'getcredentials'
],
'_xcat_clienthost' => [
'sl-w8'
]
};
$VAR2 = bless( \*Symbol::GEN85, 'IO::Socket::SSL' );
$VAR3 = sub { "DUMMY" };
Calling stack:
at /opt/xcat/sbin/xcatd line 1903
eval {...} called at /opt/xcat/sbin/xcatd line 1849
main::service_connection called at /opt/xcat/sbin/xcatd line 2253
main::__ANON__('IO::Socket::SSL=GLOB(0x3b5ff98)', undef,
'sl-w8', 'sl-w8.local', 'sl-w8.local') called at /opt/xcat/sbin/xcatd
line 1022
-----------------------------------------------------------------------
Above, '_xcat_clienthost' is resolved to 'sl-w8'. That node exists in
nodelist:
# nodels sl-w8
sl-w8
What's awful about this is that that output looks exactly the same as
what I see on the xCAT sites that are working correctly:
[root@compute-0-0 xcatpost]# openssl s_client -quiet -connect
10.4.0.10:3001 2> /dev/null
<xcatrequest>
<command>getcredentials</command>
<callback_port>300</callback_port>
<arg>"ssh_dsa_hostkey"</arg>
</xcatrequest>
<xcatresponse>
<data>
<content>
-----BEGIN DSA PRIVATE KEY-----
MIIBuwIBAAKBgQDAqbLr7+pGxXEvt054YNlxnCpZBq8EoQj9R+TKChudtWHPYvFw
EgGC62r5Cr9+cd3oJtaAnYE0s3GtrfMbtCNzL/IWdPsneHnimB5o0ziFCxXNPITb
AJpso+ab+Gs/Gb+N4+oWB1vZcxG3niY+ynMXuC/aph/zcw7WdqQ1sJEa1wIVAKHG
....excerpted.....
2Fk6qah4bPyStwwgBsGsc3TJdycPWrk662WkAJ9LxlboPyqJ9y47BOc0WotfbnOx
NP9CNrHxC0kOaVyKEaMMs2RsbL13PPgm4z85ZOEwzbQ3BbF+6X93R/8CFHgXYeSX
gXaXNEKdWfl1fWh05Amf
-----END DSA PRIVATE KEY-----
</content>
<desc>"ssh_dsa_hostkey"</desc>
</data>
<errorcode>0</errorcode>
</xcatresponse>
<xcatresponse>
<serverdone></serverdone>
</xcatresponse>
And here is the server response from the server where this transaction
works:
***************Calling of subroutine
*main::service_connection***************
Tue Sep 24 15:26:28 2013
Arguments:
$VAR1 = bless( \*Symbol::GEN56, 'IO::Socket::SSL' );
$VAR2 = undef;
$VAR3 = 'compute-0-0';
$VAR4 = 'compute-0-0.cluster';
$VAR5 = 'compute-0-0.cluster';
Calling stack:
at /opt/xcat/sbin/xcatd line 1022
Arguments:
$VAR1 = bless( \*Symbol::GEN56, 'IO::Socket::SSL' );
$VAR2 = undef;
$VAR3 = 'compute-0-0';
$VAR4 = 'compute-0-0.cluster';
$VAR5 = 'compute-0-0.cluster';
Calling stack:
at /opt/xcat/sbin/xcatd line 1022
***************Calling of subroutine *main::plugin_command***************
Tue Sep 24 15:26:28 2013
Arguments:
$VAR1 = {
'_xcat_clientfqdn' => [
'compute-0-0.cluster'
],
'arg' => [
'ssh_dsa_hostkey'
],
'_xcat_clientport' => [
33782
],
'_xcat_authname' => [
undef
],
'username' => [
undef
],
'callback_port' => [
'300'
],
'command' => [
'getcredentials'
],
'_xcat_clienthost' => [
'compute-0-0'
]
};
$VAR2 = bless( \*Symbol::GEN56, 'IO::Socket::SSL' );
$VAR3 = sub { "DUMMY" };
Calling stack:
at /opt/xcat/sbin/xcatd line 1903
eval {...} called at /opt/xcat/sbin/xcatd line 1849
main::service_connection called at /opt/xcat/sbin/xcatd line 2253
main::__ANON__('IO::Socket::SSL=GLOB(0x34bfc70)', undef, 'compute-0-0',
'compute-0-0.cluster', 'compute-0-0.cluster') called at
/opt/xcat/sbin/xcatd line 1022
On 09/12/2013 03:40 PM, Ling Gao wrote:
> One more thing to check: please make sure the name resolution is
> correct. Make sure node ip that's facing the management node resolves to
> a node name in the nodelist table.
>
> Ling
>
> Ling Gao
> Poughkeepsie Unix Development Lab
> IBM Systems and Technology Group
> Internal: T/L 293-5692
> External: [email protected], 845-433-5692
>
> "I never worry about the future. It comes soon enough." --- Albert Einstein
>
>
>
> From: Jonathan Mills <[email protected]>
> To: xCAT Users Mailing list <[email protected]>,
> Date: 09/12/2013 02:57 PM
> Subject: Re: [xcat-user] xCAT 2.8.2: Remoteshell postscript infinite
> loop with getcredentials.awk failure
> ------------------------------------------------------------------------
>
>
>
> But it isn't missing, nor empty:
>
> # cat /etc/xcat/hostkeys/ssh_host_dsa_key
> -----BEGIN DSA PRIVATE KEY-----
> MIIBvAIBAAKBgQCPWuNyJmvv5cbM4fziswR5RT/boZooaFQdXz7jJw8zhx/Y+V3R
> K8C5ikBQuzJtm0nOy+C1J4sT9ncP4m04R5y3OCqCX60ka5x1N3DAtv2JMoBCu3uZ
> ...excerpted...
> 8084a+DNMO/TWcDU4m4IZeLgkVHIe/jg7tInjfTX6JFQlGwtuYGGs2NQP/LrYYZ8
> QnnvHLm4615PKcgxOBltXUSzbEW4H180vS+ZmA8y2yOxfJtMDB+3+wkCFQDjSxJQ
> i7D2jHnS+6W0imBUvge1ng==
> -----END DSA PRIVATE KEY-----
>
> I will try a regenerate, but there doesn't appear to be anything wrong
> with the exists contents of /etc/xcat/hostkeys.
>
>
> On 09/12/2013 02:49 PM, Lissa Valletta wrote:
> > One suggestion is that the hostkey we are trying to download, is no
> > longer there or is there and the file is empty. Look in your
> > /etc/xcat/hostkeys directory on the Management Node.
> > You can regenerate them by running xcatconfig -s on the management node.
> >
> > Thanks for pointing out the infinite loop though. I have opened a
> defect.
> > https://sourceforge.net/p/xcat/bugs/3781/
> >
> >
> >
> >
> > Lissa K. Valletta
> > 8-3/B10
> > Poughkeepsie, NY 12601
> > (tie 293) 433-3102
> >
> >
> >
> > Inactive hide details for Jonathan Mills ---09/12/2013 12:13:35 PM---I'm
> > looking at a real problem with my xCAT installation whJonathan Mills
> > ---09/12/2013 12:13:35 PM---I'm looking at a real problem with my xCAT
> > installation where the remoteshell postscript is failing
> >
> > From: Jonathan Mills <[email protected]>
> > To: xCAT Users Mailing list <[email protected]>,
> > Date: 09/12/2013 12:13 PM
> > Subject: [xcat-user] xCAT 2.8.2: Remoteshell postscript infinite loop
> > with getcredentials.awk failure
> >
> > ------------------------------------------------------------------------
> >
> >
> >
> > I'm looking at a real problem with my xCAT installation where the
> > remoteshell postscript is failing to work.
> >
> > Digging into the script (line numbers on left), here it spawns a
> > listener on port 300:
> >
> > 66 allowcred.awk &
> >
> > You can see the result of that command like so:
> > # netstat -tulpn | grep awk
> > tcp 0 0 0.0.0.0:300 0.0.0.0:*
> > LISTEN 3119/awk
> >
> > Then in line 75 of remoteshell it runs getcredentials.awk with the
> > parameter 'ssh_dsa_hostkey', redirecting the output to
> /tmp/ssh_dsa_hostkey:
> >
> > 75 getcredentials.awk ssh_dsa_hostkey | grep -E -v
> > '</{0,1}xcatresponse>|</{0,1}serverdone>' | sed -e 's/</</' -e
> > 's/>/>/' -e 's/&/&/' -e 's/&qu ot/"/' -e "s/'/'/" >
> > /tmp/ssh_dsa_hostkey
> >
> >
> > What the getcredentials.awk script does is:
> > "openssl s_client -quiet -connect XCATSERVER:3001" and tells xCATd to
> > respond on port 300 (opened above from allowcred.awk).
> >
> > A correct interaction of these parts has output that looks like:
> >
> > # ./getcredentials.awk ssh_dsa_hostkey
> > <xcatresponse>
> > <data>
> > <content>
> > -----BEGIN DSA PRIVATE KEY-----
> > MIIBuwIBAAKBgQCzrhm0rTC9FcCnTjuI5QGh7MV2oczIwqBWhBlid7748dcdF23M
> > KLwbuopN1y2/6l1XmyS2ama+vtbUnpFZ4yQcdX99QXZq3UgZPsVayKmPjgjNmV99
> > xm+RMuoMBCkh12dL5JVc1auJKGupaz+yVEol4hCkRAYb/E4Ft/5GroP/jQIVAMHB
> > .....excerpt......
> > dxEhemsBz9FwP7e3VB8yc8T/vXTswuYrDzslGL22Z3Il87MtHXO5pdO5HFLnmPan
> > w0NyyWKdxfSSwfxbTNRRTVDi7XTg3CfWnsIRI98EsIwmxUe0QcV6lwIVAJdtvPLA
> > Blck9nPWNaBqTbpBznj+
> > -----END DSA PRIVATE KEY-----
> > </content>
> > <desc>ssh_dsa_hostkey</desc>
> > </data>
> > <errorcode>0</errorcode>
> > </xcatresponse>
> > <xcatresponse>
> > <serverdone></serverdone>
> > </xcatresponse>
> >
> >
> > However, on my faulty xCAT 2.8.2 install, here is what happens:
> >
> > # ./getcredentials.awk ssh_dsa_hostkey
> > <xcatresponse>
> > <serverdone></serverdone>
> > </xcatresponse>
> >
> > This leaves /tmp/ssh_dsa_hostkey empty. At line 81 of remoteshell, it
> > tries to populate /etc/ssh/ssh_host_dsa_key with the value of
> > /tmp/ssh_dsa_hostkey:
> >
> > 81 cat /tmp/ssh_dsa_hostkey | grep -E -v
> > '</{0,1}data>|</{0,1}content>|</{0,1}desc>' >/etc/ssh/ssh_host_dsa_key
> >
> > Then at lines 83 & 84 of remoteshell, it does this:
> >
> > 83 MYCONT=`cat /etc/ssh/ssh_host_dsa_key`
> > 84 while [ -z "$MYCONT" ]; do
> >
> > So $MYCONT is always an empty value, thus dropping that while loop into
> > an infinite loop.
> >
> >
> > Moreover, please note that on my faulty xCAT install, the policy table
> > is set to allow 'getcredentials':
> > # tabdump policy
> >
> #priority,name,host,commands,noderange,parameters,time,rule,comments,disable
> > "1","root",,,,,,"allow",,
> > ......excerpted
> > "4.4",,,"getpostscript",,,,"allow",,
> > "4.5",,,"getcredentials",,,,"allow",,
> > ....etc
> >
> > And in /var/log/messages on the XCATMASTER, I can even see where xCATd
> > is granting these requests:
> > Sep 8 03:10:59 <xcatmaster_hostname> xCAT: xCAT: Allowing
> > getcredentials ssh_dsa_hostkey from <hostname>
> >
> >
> > It must be the case that there is a subtle misconfiguration in this xCAT
> > 2.8.2 installation, as I have other installs of 2.8.2 where this is
> > working. I would appreciate it if anyone can help me understand why
> > xCATd is failing to return the ssh_dsa_hostkey.
> >
> > --
> > Jonathan Mills
> > Systems Administrator
> > Renaissance Computing Institute
> > UNC-Chapel Hill
> >
> >
> ------------------------------------------------------------------------------
> > How ServiceNow helps IT people transform IT departments:
> > 1. Consolidate legacy IT systems to a single system of record for IT
> > 2. Standardize and globalize service processes across IT
> > 3. Implement zero-touch automation to replace manual, redundant tasks
> >
> http://pubads.g.doubleclick.net/gampad/clk?id=51271111&iu=/4140/ostg.clktrk
> > _______________________________________________
> > xCAT-user mailing list
> > [email protected]
> > https://lists.sourceforge.net/lists/listinfo/xcat-user
> >
> >
>
> --
> Jonathan Mills
> Systems Administrator
> Renaissance Computing Institute
> UNC-Chapel Hill
>
> ------------------------------------------------------------------------------
> How ServiceNow helps IT people transform IT departments:
> 1. Consolidate legacy IT systems to a single system of record for IT
> 2. Standardize and globalize service processes across IT
> 3. Implement zero-touch automation to replace manual, redundant tasks
> http://pubads.g.doubleclick.net/gampad/clk?id=51271111&iu=/4140/ostg.clktrk
> _______________________________________________
> xCAT-user mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/xcat-user
>
>
--
Jonathan Mills
Systems Administrator
Renaissance Computing Institute
UNC-Chapel Hill
------------------------------------------------------------------------------
October Webinars: Code for Performance
Free Intel webinars can help you accelerate application performance.
Explore tips for MPI, OpenMP, advanced profiling, and more. Get the most from
the latest Intel processors and coprocessors. See abstracts and register >
http://pubads.g.doubleclick.net/gampad/clk?id=60133471&iu=/4140/ostg.clktrk
_______________________________________________
xCAT-user mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/xcat-user
