Russell,
A couple things to try:
1. xcatmaster=evxcat, you do not need to set up xcatmaster for a node if
there is no service node involved. It default to site.master. I usually
put an ip address that is facing the node for the site.master.
2. Regarding the pping error, (not sure why pping is called). By default
pping calls nmap. nmap sometimes times out and gives error for slow
networks. You can set a site table attribute to let it use fping instead.
site.usefping=1. Make sure fping is installed on the management node.
Ling
Ling Gao
Poughkeepsie Unix Development Lab
IBM Systems and Technology Group
Internal: T/L 293-5692
External: [email protected], 845-433-5692
"I never worry about the future. It comes soon enough." --- Albert
Einstein
From: Russell Jones <[email protected]>
To: xCAT Users Mailing list <[email protected]>
Date: 02/05/2015 02:03 PM
Subject: Re: [xcat-user] xCAT 2.9 Syncfiles error
Here you go. There is no service node, just the management node (evxcat).
[root@evxcat ~]# lsdef n1
Object name: n1
arch=x86_64
currstate=netboot centos6.6-x86_64-v6.6.0-dl-compute-apu
groups=all,compute,v6.6.0-dl-compute-apu
initrd=xcat/osimage/v6.6.0-dl-compute-apu/initrd-stateless.gz
ip=172.21.1.2
kcmdline=imgurl=http://evxcat:80//install/netboot/centos6.6/x86_64/v6.6.0-dl-compute-apu/rootimg.gz
XCAT=evxcat:3001 NODE=n1 FC=0 ifname=eth0:74:D4:35:9B:CF:FE netdev=eth0
kernel=xcat/osimage/v6.6.0-dl-compute-apu/kernel
mac=74:D4:35:9B:CF:FE
mgt=ipmi
netboot=xnba
nichostnamesuffixes.eth0=-eth0
nichostnamesuffixes.ib0=-ib0
nicips.ib0=172.40.130.1
nicips.eth0=172.21.1.2
nicnetworks.ib0=172_40_0_0_Infiniband
nicnetworks.eth0=172_21_1_0_APU
nictypes.ib0=Infiniband
nictypes.eth0=Ethernet
os=centos6.6
postscripts=remoteshell
profile=v6.6.0-dl-compute-apu
provmethod=v6.6.0-dl-compute-apu
status=booted
statustime=02-04-2015 13:43:56
updatestatus=synced
updatestatustime=02-04-2015 13:32:41
xcatmaster=evxcat
[root@evxcat ~]# lsdef -t osimage v6.6.0-dl-compute-apu
Object name: v6.6.0-dl-compute-apu
groups=all
imagetype=linux
nodebootif=eth0
osarch=x86_64
osdistroname=centos6.6-x86_64
osname=Linux
osvers=centos6.6
otherpkgdir=/install/custom/v6.6.0-dl-compute-apu/otherpkgs
otherpkglist=/install/custom/v6.6.0-dl-compute-apu/v6.6.0-dl-compute-apu.otherpkgs.pkglist
permission=755
pkgdir=/install/centos6.6/x86_64
pkglist=/install/custom/v6.6.0-dl-compute-apu/v6.6.0-dl-compute-apu.pkglist
postbootscripts=otherpkgs
postinstall=/install/custom/v6.6.0-dl-compute-apu/v6.6.0-dl-compute-apu.postinstall
postscripts=confignics,syslog,syncfiles,setupntp,addsiteyum,mount_lustre2.sh,mount_home.sh,enable_slurm.sh
profile=v6.6.0-dl-compute-apu
provmethod=netboot
rootimgdir=/install/netboot/centos6.6/x86_64/v6.6.0-dl-compute-apu
synclists=/install/custom/v6.6.0-dl-compute-apu/v6.6.0-dl-compute-apu.synclist
On 05.02.2015 12:52, Ling Gao wrote:
Russell,
Could you send me the output of lsdef n1? Is there a service node
involve in the cluster?
Thanks,
Ling
Ling Gao
Poughkeepsie Unix Development Lab
IBM Systems and Technology Group
Internal: T/L 293-5692
External: [email protected], 845-433-5692
"I never worry about the future. It comes soon enough." --- Albert
Einstein
From: Russell Jones <[email protected]>
To: xCAT Users Mailing list <[email protected]>
Date: 02/05/2015 10:44 AM
Subject: Re: [xcat-user] xCAT 2.9 Syncfiles error
Hi Ling,
I'm confused too! Here are answers to your questions, thanks for your
help!
> Is this diskless or diskful deployment? When you say "reboot", did you
mean redeploy?
This is a diskless node netbooting a CentOS 6.6 x64 image.
> Could you grep /var/log/messages for "Allowing xdcp" and show me the
output.
That message seems to only show up when doing a packimage. When the node
is booting it shows "Allowing syncfiles". Here's the output when that is
happening. Note this is recorded on the management node as I have the
syslog postscript being ran on the node before syncfiles. Also note that
it seems to have tried to run Syncfiles several times before showing the
error message. When this error does not occur, syncfiles is only ran once,
so there must be a retry in the code somewhere. Time is also off on n1
until the setupntp script ran. I've since reordered to have setupntp run
first, however rebooting the node several times before reordering the
script had Syncfiles run properly while time is still off, so I don't
believe this is a time-related issue:
Feb 4 19:27:26 n1 xCAT: Install: syslog setup
Feb 4 19:27:26 n1 xCAT: Performing syncfiles postscript
Feb 4 19:27:26 n1 xCAT: ./syncfiles: the OS name = Linux
Feb 4 13:27:25 evxcat xCAT[20079]: xCAT: Allowing syncfiles from n1
Feb 4 19:27:27 n1 sshd[4392]: Accepted publickey for root from 172.21.0.1
port 41221 ssh2
Feb 4 19:27:27 n1 sshd[4392]: pam_unix(sshd:session): session opened for
user root by (uid=0)
Feb 4 19:27:27 n1 sshd[4392]: Received disconnect from 172.21.0.1: 11:
disconnected by user
<snip>
Feb 4 19:27:29 n1 kernel: ADDRCONF(NETDEV_CHANGE): ib0: link becomes
ready
Feb 4 19:27:40 n1 logger[4528]: openibd: Set node_desc for mlx5_0: n1
HCA-1
Feb 4 19:27:40 n1 kernel: ib0: no IPv6 routers present
Feb 4 13:27:43 evxcat xCAT[20181]: xCAT: Allowing syncfiles from n1
Feb 4 19:27:45 n1 sshd[4532]: Accepted publickey for root from 172.21.0.1
port 41233 ssh2
Feb 4 19:27:45 n1 sshd[4532]: pam_unix(sshd:session): session opened for
user root by (uid=0)
Feb 4 19:27:45 n1 sshd[4532]: Received disconnect from 172.21.0.1: 11:
disconnected by user
Feb 4 19:27:45 n1 sshd[4532]: pam_unix(sshd:session): session closed for
user root
<snip>
Feb 4 13:28:04 evxcat xCAT[20262]: xCAT: Allowing syncfiles from n1
Feb 4 19:28:06 n1 sshd[4640]: Accepted publickey for root from 172.21.0.1
port 41241 ssh2
Feb 4 19:28:06 n1 sshd[4640]: pam_unix(sshd:session): session opened for
user root by (uid=0)
Feb 4 19:28:06 n1 sshd[4640]: Received disconnect from 172.21.0.1: 11:
disconnected by user
Feb 4 19:28:06 n1 sshd[4640]: pam_unix(sshd:session): session closed for
user root
<snip>
Feb 4 13:28:21 evxcat xCAT[20341]: xCAT: Allowing syncfiles from n1
Feb 4 19:28:22 n1 sshd[4748]: Accepted publickey for root from 172.21.0.1
port 41249 ssh2
Feb 4 19:28:22 n1 sshd[4748]: pam_unix(sshd:session): session opened for
user root by (uid=0)
Feb 4 19:28:22 n1 sshd[4748]: Received disconnect from 172.21.0.1: 11:
disconnected by user
Feb 4 19:28:22 n1 sshd[4748]: pam_unix(sshd:session): session closed for
user root
<snip>
Feb 4 13:28:41 evxcat xCAT[20420]: xCAT: Allowing syncfiles from n1
Feb 4 19:28:43 n1 sshd[4863]: Accepted publickey for root from 172.21.0.1
port 41257 ssh2
Feb 4 19:28:43 n1 sshd[4863]: pam_unix(sshd:session): session opened for
user root by (uid=0)
Feb 4 19:28:43 n1 sshd[4863]: Received disconnect from 172.21.0.1: 11:
disconnected by user
Feb 4 19:28:43 n1 sshd[4863]: pam_unix(sshd:session): session closed for
user root
<snip>
Feb 4 13:29:02 evxcat xCAT[20507]: xCAT: Allowing syncfiles from n1
Feb 4 19:29:04 n1 sshd[4972]: Accepted publickey for root from 172.21.0.1
port 41271 ssh2
Feb 4 19:29:04 n1 sshd[4972]: pam_unix(sshd:session): session opened for
user root by (uid=0)
Feb 4 19:29:04 n1 sshd[4972]: Received disconnect from 172.21.0.1: 11:
disconnected by user
Feb 4 19:29:04 n1 sshd[4972]: pam_unix(sshd:session): session closed for
user root
<snip>
Feb 4 19:29:05 n1 xCAT: ./syncfiles: Perform Syncing File action
encountered error
Feb 4 19:29:05 n1 xcat: Install: Setup NTP
Feb 4 19:29:06 n1 xcat: ntpdate -t5 172.21.0.1
Feb 4 13:29:06 n1 ntpd[5141]: ntpd [email protected] Sat Nov 23 18:21:48
UTC 2013 (1)
> “Error from pping” can only happen when calling xdcp with -v option. But
I checked the code for updatenode -F and did not see xdcp was called with
a -v option, so I am confused.
Strange, but that's exactly what happened! Here's a direct copy/paste from
my terminal:
[root@evxcat ~]# updatenode n1 -F
Error from pping
File synchronization has completed for nodes.
[root@evxcat ~]# updatenode n1 -F
File synchronization has completed for nodes.
[root@evxcat postscripts]# pping n1
n1: ping
> BTW, the server side error is usually logged in /var/log/messages. Could
you check the time stamps of the messages and see what errors it had
during the deployment time.
There's no other errors shown during deployment, just this one from
syncfiles. Everything also runs fine, including syncfiles when it shows
this error. All of my files are synced including those that are being
appended. The node comes up and is 100% healthy.
Thanks for the help!
On 05.02.2015 08:42, Ling Gao wrote:
BTW, the server side error is usually logged in /var/log/messages. Could
you check the time stamps of the messages and see what errors it had
during the deployment time.
Thanks,
Ling
Ling Gao
Poughkeepsie Unix Development Lab
IBM Systems and Technology Group
Internal: T/L 293-5692
External: [email protected], 845-433-5692
"I never worry about the future. It comes soon enough." --- Albert
Einstein
From: Ling Gao/Poughkeepsie/IBM
To: xCAT Users Mailing list <[email protected]>
Date: 02/05/2015 09:34 AM
Subject: Re: [xcat-user] xCAT 2.9 Syncfiles error
Hi Russell,
Thanks for the debugging.
Is this diskless or diskful deployment? When you say "reboot", did you
mean redeploy?
Could you grep /var/log/messages for "Allowing xdcp" and show me the
output. “Error from pping” can only happen when calling xdcp with -v
option. But I checked the code for updatenode -F and did not see xdcp was
called with a -v option, so I am confused.
Thanks,
Ling
Ling Gao
Poughkeepsie Unix Development Lab
IBM Systems and Technology Group
Internal: T/L 293-5692
External: [email protected], 845-433-5692
"I never worry about the future. It comes soon enough." --- Albert
Einstein
From: Russell Jones <[email protected]>
To: [email protected]
Date: 02/04/2015 02:55 PM
Subject: Re: [xcat-user] xCAT 2.9 Syncfiles error
Some further debugging, I ran "updatenode n1 -F" to try a manual syncfiles
on the node that showed this error during boot and it completed, but also
stated "Error from pping". I ran pping and updatenode -F again, and I am
also unable to reproduce that error from pping. Further reboots of n1 does
not result in any syncfiles errors.
Really difficult to track this down. Any insight would be appreciated!
On 2/4/2015 1:30 PM, Russell Jones wrote:
Hi Ling,
Unfortunately I am running into this issue again, seemingly at random. Can
we revisit this? Are there areas within the syncfiles scripts where I can
get it to tell me *what* the error is that it is encountering as opposed
to just saying "Perform Syncing File action encountered error" ?
Thanks!
On 1/28/2015 3:34 PM, Ling Gao wrote:
Hi Russell,
Sure, that is a good news. I tried on a RH6.4 node and it worked
fine.
Thanks,
Ling
Ling Gao
Poughkeepsie Unix Development Lab
IBM Systems and Technology Group
Internal: T/L 293-5692
External: [email protected], 845-433-5692
"I never worry about the future. It comes soon enough." --- Albert
Einstein
From: Russell Jones <[email protected]>
To: [email protected]
Date: 01/28/2015 04:25 PM
Subject: Re: [xcat-user] xCAT 2.9 Syncfiles error
Hi Ling,
Thanks for looking at this! I was troubleshooting this issue some more
this morning and something, unbeknownst to me, fixed the issue. I have no
idea what changed but it seems to be resolved now....
Sure wish I knew what was breaking it! :-)
On 1/28/2015 2:49 PM, Ling Gao wrote:
Russell,
Can you comment out line 106 "return 1" from
/opt/xcat/lib/perl/xCAT_plugin/syncfiles.pm and try again? Please restart
xcatd.
Thanks,
Ling
Ling Gao
Poughkeepsie Unix Development Lab
IBM Systems and Technology Group
Internal: T/L 293-5692
External: [email protected], 845-433-5692
"I never worry about the future. It comes soon enough." --- Albert
Einstein
From: Russell Jones <[email protected]>
To: <[email protected]>
Date: 01/27/2015 05:31 PM
Subject: [xcat-user] xCAT 2.9 Syncfiles error
xCAT 2.9
Hi all,
I have a centos 6.6 image with the following contents in the synclist file
for the osimage:
/install/custom/v6.6.0-dl-compute-apu/sync/etc/modprobe.d/lustre.conf ->
/etc/modprobe.d/lustre.conf
APPEND:
/install/custom/v6.6.0-dl-compute-apu/sync/etc/fstab -> /etc/fstab
When my compute nodes boot up and execute the script it takes a couple of
minutes, which seems way too long for syncing 2 text files. I get a
"syncfiles exited with code 0" in the xcat.log file on the node, while in
/var/log/messages on the management node I get:
Jan 27 16:07:57 evxcat xCAT[27685]: xCAT: Allowing syncfiles from n0
Jan 27 16:08:09 evxcat xCAT[27753]: xCAT: Allowing syncfiles from n0
Jan 27 16:08:24 evxcat xCAT[27805]: xCAT: Allowing syncfiles from n0
Jan 27 16:08:45 evxcat xCAT[27858]: xCAT: Allowing syncfiles from n0
Jan 27 16:09:00 evxcat xCAT[27909]: xCAT: Allowing syncfiles from n0
Jan 27 16:09:16 evxcat xCAT[27960]: xCAT: Allowing syncfiles from n0
Jan 27 22:09:18 n0 xCAT: ./syncfiles: Perform Syncing File action
encountered error
Looking at the end result the files are synced fine, and my /etc/fstab is
appended properly.
So it seems like the Syncfiles script can't seem to make up its mind if it
exited with an error or not (code 0 usually = good!).
Any ideas what the issue could be?
------------------------------------------------------------------------------
Dive into the World of Parallel Programming. The Go Parallel Website,
sponsored by Intel and developed in partnership with Slashdot Media, is
your
hub for all things parallel software development, from weekly thought
leadership blogs to news, videos, case studies, tutorials and more. Take a
look and join the conversation now. http://goparallel.sourceforge.net/
_______________________________________________
xCAT-user mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/xcat-user
------------------------------------------------------------------------------
Dive into the World of Parallel Programming. The Go Parallel Website,
sponsored by Intel and developed in partnership with Slashdot Media, is
your
hub for all things parallel software development, from weekly thought
leadership blogs to news, videos, case studies, tutorials and more. Take a
look and join the conversation now. http://goparallel.sourceforge.net/
_______________________________________________
xCAT-user mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/xcat-user
------------------------------------------------------------------------------
Dive into the World of Parallel Programming. The Go Parallel Website,
sponsored by Intel and developed in partnership with Slashdot Media, is
your
hub for all things parallel software development, from weekly thought
leadership blogs to news, videos, case studies, tutorials and more. Take a
look and join the conversation now. http://goparallel.sourceforge.net/
_______________________________________________
xCAT-user mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/xcat-user
------------------------------------------------------------------------------
Dive into the World of Parallel Programming. The Go Parallel Website,
sponsored by Intel and developed in partnership with Slashdot Media, is
your
hub for all things parallel software development, from weekly thought
leadership blogs to news, videos, case studies, tutorials and more. Take a
look and join the conversation now. http://goparallel.sourceforge.net/
_______________________________________________
xCAT-user mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/xcat-user
------------------------------------------------------------------------------
Dive into the World of Parallel Programming. The Go Parallel Website,
sponsored by Intel and developed in partnership with Slashdot Media, is
your
hub for all things parallel software development, from weekly thought
leadership blogs to news, videos, case studies, tutorials and more. Take a
look and join the conversation now. http://goparallel.sourceforge.net/
_______________________________________________
xCAT-user mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/xcat-user
------------------------------------------------------------------------------
Dive into the World of Parallel Programming. The Go Parallel Website,
sponsored by Intel and developed in partnership with Slashdot Media, is
your
hub for all things parallel software development, from weekly thought
leadership blogs to news, videos, case studies, tutorials and more. Take a
look and join the conversation now. http://goparallel.sourceforge.net/
_______________________________________________
xCAT-user mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/xcat-user
------------------------------------------------------------------------------
Dive into the World of Parallel Programming. The Go Parallel Website,
sponsored by Intel and developed in partnership with Slashdot Media, is
your
hub for all things parallel software development, from weekly thought
leadership blogs to news, videos, case studies, tutorials and more. Take a
look and join the conversation now. http://goparallel.sourceforge.net/
_______________________________________________
xCAT-user mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/xcat-user
------------------------------------------------------------------------------
Dive into the World of Parallel Programming. The Go Parallel Website,
sponsored by Intel and developed in partnership with Slashdot Media, is
your
hub for all things parallel software development, from weekly thought
leadership blogs to news, videos, case studies, tutorials and more. Take a
look and join the conversation now. http://goparallel.sourceforge.net/
_______________________________________________
xCAT-user mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/xcat-user
------------------------------------------------------------------------------
Dive into the World of Parallel Programming. The Go Parallel Website,
sponsored by Intel and developed in partnership with Slashdot Media, is
your
hub for all things parallel software development, from weekly thought
leadership blogs to news, videos, case studies, tutorials and more. Take a
look and join the conversation now. http://goparallel.sourceforge.net/
_______________________________________________
xCAT-user mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/xcat-user
------------------------------------------------------------------------------
Dive into the World of Parallel Programming. The Go Parallel Website,
sponsored by Intel and developed in partnership with Slashdot Media, is
your
hub for all things parallel software development, from weekly thought
leadership blogs to news, videos, case studies, tutorials and more. Take a
look and join the conversation now. http://goparallel.sourceforge.net/
_______________________________________________
xCAT-user mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/xcat-user
------------------------------------------------------------------------------
Dive into the World of Parallel Programming. The Go Parallel Website,
sponsored by Intel and developed in partnership with Slashdot Media, is your
hub for all things parallel software development, from weekly thought
leadership blogs to news, videos, case studies, tutorials and more. Take a
look and join the conversation now. http://goparallel.sourceforge.net/
_______________________________________________
xCAT-user mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/xcat-user
