We’ve used similar scripts in the past without any checks to prevent unintended
disasters. It would be pretty easy to use an if or case statement to ensure
anything destructive only happens on the right systems.
Regards,
Christian Caruthers
Lenovo Professional Services
Mobile: 757-289-9872
From: Vinícius Ferrão [mailto:[email protected]]
Sent: Monday, November 27, 2017 2:17 PM
To: xCAT Users Mailing list <[email protected]>
Subject: Re: [xcat-user] Local scratch for stateless compute nodes
Very good info indeed. I will be looking on the script, Kevin.
It would be sufficient for a while, but about the feature, it would be nice to
be fixed/documented/explained how to use it, because it’s a pretty common use
case of stateless nodes.
Perhaps someone on the dev team can look at this? Should we open a ticket on
the issue tracker?
V.
On 27 Nov 2017, at 17:02, Gilad Berman
<[email protected]<mailto:[email protected]>> wrote:
THX Kevin for the tip!!
We actually used similar method, but the first post on the thread reminded me
of the localdisk feature and I thought it can be very nice to use it, if
working.
<image001.jpg>
Gilad Berman
HPC Architect
Lenovo EMEA
<image002.png>+972-52-2554262
<image003.png>[email protected]<mailto:[email protected]>
Lenovo.com <http://www.lenovo.com/>
Twitter<http://twitter.com/lenovo> | Facebook<http://www.facebook.com/lenovo> |
Instagram<https://instagram.com/lenovo> | Blogs<http://blog.lenovo.com/> |
Forums<http://forums.lenovo.com/>
<image004.jpg>
From: Kevin Keane [mailto:[email protected]]
Sent: Monday, November 27, 2017 7:22 PM
To: xCAT Users Mailing list
<[email protected]<mailto:[email protected]>>
Subject: Re: [xcat-user] Local scratch for stateless compute nodes
To address this, we are using the syncfiles mechanism to copy an rc.local file
into the compute node (we could probably also put it directly into the image)
This rc.local contains statements to mount the /tmp volume. We originally also
used it to partition and format the physical hard disk, but that proved too
dangerous when somebody accidentally ran the script on the management node and
wiped out the partition table... You could probably do something similar with a
swap partition.
Here is the script we are using.
A few notes:
- a prerequisite is an entry that mounts /dev/sda1 as /localscratch in fstab.
- moving the content from /tmp to /localscratch/tmp actually isn't working
flawlessly; it is just good enough for our purposes.
#!/bin/sh
#
# This script will be executed *after* all the other init scripts.
# You can put your own initialization stuff in here if you don't
# want to do the full Sys V style init stuff.
touch /var/lock/subsys/local
# Let's see if the local disk is already formatted and set up - if so,
# we won't redo it.
if [ -d /localscratch/tmp ]
then
echo "Localscratch disk is already formatted"
else
umount /dev/sda1
##################################################################
# DANGER DANGER DANGER! This code can, and will, blindly destroy
# partitions on whatever computer it is run. Partition recovery
# is not easy.
##################################################################
# Format the disk in the compute node. Single partition, mounted
# as localscratch.
#dd if=/dev/zero of=/dev/sda bs=1M count=100
#parted -s /dev/sda mklabel gpt
#parted -s -a optimal /dev/sda mkpart primary ext3 0% 100%
#mkfs -t ext3 /dev/sda1
mount -a
fi
# Create the /tmp and /var/tmp directories.
for i in /tmp /var/tmp /var/log
do
# Make sure mv includes .dotfiles
shopt -s dotglob
mkdir -p /localscratch$i
chmod 755 /localscratch
case "$i" in
/tmp)
# The first digit in the mode is the sticky bit.
chmod 1777 /localscratch$i
;;
/var/tmp)
# The first digit in the mode is the sticky bit.
chmod 1777 /localscratch$i
;;
*)
chmod 755 /localscratch$i
esac
if [ ! -h $i ]
then
if [ -n "$(ls -A $i)" ]
then
mv $i/* /localscratch$i
fi
# In theory, the directory should be empty because we moved everything
# out of the way. But that may have failed if the localscratch directory
# was already used.
rm -rf $i
ln -sf /localscratch$i $i
fi
shopt -u dotglob
done
mkdir -p /localscratch/ansys
chmod 777 /localscratch/ansys
On Mon, Nov 27, 2017 at 6:57 AM, Gilad Berman
<[email protected]<mailto:[email protected]>> wrote:
1. I use local disk for scratch and swap. Somethings logs as well (in this
case you can think of it as sort of statelite, but from xCAT perspective, it is
still stateless).
2. I took only the part that not relate to statelite from the instructions
– not working.
<image009.jpg>
Gilad Berman
HPC Architect
Lenovo EMEA
<image010.png>+972-52-2554262<tel:+972%2052-255-4262>
<image011.png>[email protected]<mailto:[email protected]>
Lenovo.com <http://www.lenovo.com/>
Twitter<http://twitter.com/lenovo> | Facebook<http://www.facebook.com/lenovo> |
Instagram<https://instagram.com/lenovo> | Blogs<http://blog.lenovo.com/> |
Forums<http://forums.lenovo.com/>
<image012.jpg>
From: Russ Auld [mailto:[email protected]<mailto:[email protected]>]
Sent: Monday, November 27, 2017 4:50 PM
To: xCAT Users Mailing list
<[email protected]<mailto:[email protected]>>
Subject: Re: [xcat-user] Local scratch for stateless compute nodes
If you're using netboot and local disk, then isn't that "statelite"?
Do the satellite instructions not work?
On Nov 27, 2017 9:26 AM, Gilad Berman
<[email protected]<mailto:[email protected]>> wrote:
All,
I would like to join this question –
Does even localdisk works with stateless? From the docs it seems that should be
supported (because it is under stateless), however –
- the instructions are taken from statelite and refer to statelite code
(litefile)
- The rc.localdisk code is under statelite
- In the linuximage man – “Partitionfile - Only available for diskful
osimages and statelite osimages(localdisk enabled)“
A very quick trial on my statless nodes results in nothing ☺, it seems there is
simply no reference to localdisk with stateless.
So, can someone please help clarify it?
** as always, there is a chance I missed something very basic and it should be
working ☺
THX in advance!
Gilad Berman
HPC Architect
Lenovo EMEA
+972-52-2554262<tel:+972%2052-255-4262>
[email protected]<mailto:[email protected]>
<mailto:[email protected]>
Lenovo.com
Twitter | Facebook | Instagram | Blogs | Forums<mailto:[email protected]>
<mailto:[email protected]>
<mailto:[email protected]>
From: Vinícius Ferrão [mailto:[email protected]]
Sent: Wednesday, November 22, 2017 4:09 AM
To: [email protected]
Subject: [xcat-user] Local scratch for stateless compute
nodes<mailto:[email protected]>
<mailto:[email protected]>
Hello,<mailto:[email protected]>
<mailto:[email protected]>
I would like to enable swap and local /tmp on my stateless nodes, but after
following the documentation on the following link nothing appears to
work:<mailto:[email protected]>
http://xcat-docs.readthedocs.io/en/stable/advanced/hierarchy/provision/diskless_sn.html<mailto:[email protected]>
<mailto:[email protected]>
I’m aware that the documentation is for service nodes and not for compute
nodes, but I was thinking the procedure would be
similar.<mailto:[email protected]>
<mailto:[email protected]>
At this point I’m with this settings on osimage:<mailto:[email protected]>
[root@headnode xcat]# lsdef -t osimage centos7.4-x86_64-netboot-compute
exlist=/opt/xcat/share/xcat/netboot/centos/compute.centos7.exlist
imagetype=linux
osarch=x86_64
osdistroname=centos7.4-x86_64
osname=Linux
osvers=centos7.4
otherpkgdir=/install/post/otherpkgs/centos7.4/x86_64<mailto:[email protected]>
****
partitionfile=/install/custom/netboot/centos7.4-x86_64-netboot-compute/partitionfile<mailto:[email protected]>
permission=755
pkgdir=/install/centos7.4/x86_64
pkglist=/opt/xcat/share/xcat/netboot/centos/compute.centos7.pkglist
postinstall=/opt/xcat/share/xcat/netboot/centos/compute.centos7.postinstall
profile=compute
provmethod=netboot
rootimgdir=/install/netboot/centos7.4/x86_64/compute
synclists=/install/custom/netboot/compute.synclist<mailto:[email protected]>
<mailto:[email protected]>
And the content of the partition file is the
following:<mailto:[email protected]>
cat /install/custom/netboot/centos7.4-x86_64-netboot-compute/partitionfile
enable=yes
enablepart=yes
[disk]
dev=/dev/sda
clear=yes
parts=10,90
[swapspace]
dev=/dev/sda1
[localspace]
dev=/dev/sda2
fstype=xfs<mailto:[email protected]>
<mailto:[email protected]>
Finally the following commands were executed:<mailto:[email protected]>
chtab priority=7.1 policy.commands=getpartition
policy.rule=allow<mailto:[email protected]>
chtab litefile.image=centos7.4-x86_64-netboot-compute litefile.file=/var/log/
litefile.options=localdisk<mailto:[email protected]>
chtab litefile.image=centos7.4-x86_64-netboot-compute litefile.file=/tmp/
litefile.options=localdisk<mailto:[email protected]>
<mailto:[email protected]>
After a new genimage/packimage the local scratch does not appears to be made
nor working.<mailto:[email protected]>
<mailto:[email protected]>
Someone knows what to do next?<mailto:[email protected]>
<mailto:[email protected]>
Thanks,<mailto:[email protected]>
V.<mailto:[email protected]>
<mailto:[email protected]>
<mailto:[email protected]>
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
xCAT-user mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/xcat-user<mailto:[email protected]>
-- <mailto:[email protected]>
_______________________________________________________________________
Kevin Keane | Systems Architect | University of San Diego ITS |
[email protected]
Maher Hall, 192 |5998 Alcalá Park | San Diego, CA 92110-2492 |
619.260.6859<mailto:[email protected]>
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org<http://slashdot.org/>!
http://sdm.link/slashdot_______________________________________________
xCAT-user mailing list
[email protected]<mailto:[email protected]>
https://lists.sourceforge.net/lists/listinfo/xcat-user
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
xCAT-user mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/xcat-user
