Hi all I successfully installed two similar boxes running FC4 w/linux-vserver, implementing a hot-standby of each vserver on the other box. All VServers are simply installed on both boxes in the same directory structure; half of the servers are active on box A and standby on box B, the other half is active on box B and standby on box A. This is what I call a "poor man's warm standby solution", i.e. without the need for a SAN infrastructure. This setup does "load balancing" by having the active vserver on the CPU of my choice statically. On both boxes a cronjob runs once every hour (of course not both at the same time) that takes a LVM snapshot of the vserver partition and then uses rsync to mirror the active server(s) to the standby box. This setup works very well now for several weeks. Depending on the data changing and overall load, the hourly script needs about 1-5 minutes to sync 8 vservers in my case, together occupying about 23 GB of disk space (running on a x86_64 2.8GHz Dual Xeon box w/software RAID 1 SATA disks).
For anyone interested, find my scripts below that do the work of syncing the servers and migrate a vserver from one mirror to the other (switchover).. As a prerequisite, each of the two boxes should be able to connect to the other one via SSH without using passwords, i.e. by public key authentication. The same scripts are installed on both servers, they automatically find out where they are running and what to do. I also have a .lan domain on my DNS to avoid having IP addresses in the script, but of course this can be adapted easily. In my own setup, I have a few additional site-specific scripts that take care of networking issues (firewalling, VLAN setup, static routes etc), this is necessary in my case since my vservers have RFC1918 IPs and are accessible from the outside ony via NAT. Syncing the /etc/vserver directory has to be done manually, my script just rsyncs the /vservers partition. Of course, the scripts come without any warranty whatsoever, all bugs have been added by me ;) Regards --Marcel <file rsync_vservers> #!/bin/bash # synchronizes all running vservers to mirror and checks the status. # Does not automatically launch vservers; this has to be done manually. # stdout can typically be piped into logger from within a cronjob, e.g. # /etc/rsync_vservers | /usr/bin/logger -t rsync_vservers # 3-MAR-2006 milprog/mg # 10-MAR-2006 milprog/mg : adapted to retry snapshot creation and wait 1 second before mounting if [ "$HOSTNAME" = "milprogmail.ch" ] then THISHOST=milprogmail.milnet.lan else THISHOST=$HOSTNAME fi MIRROR1=lepus.milnet.lan MIRROR2=milprogmail.milnet.lan # maximum 1 MB/s bandwidth please during rsync BWLIMIT=1024 SNAPSHOTSIZE=4096M VSERVERVG=/dev/vg00 VSERVERLV=lv00_vservers VSERVERSNAPSHOT=vservers_snapshot SNAPSHOTMOUNT=/mnt/vservers-snapshot VSERVERBASE=/vservers # no need to change anything below (hopefully) VSERVERCMD=/usr/sbin/vserver RSYNCCMD=/usr/bin/rsync LVCREATECMD=/usr/sbin/lvcreate LVREMOVECMD=/usr/sbin/lvremove MOUNTCMD=/bin/mount UMOUNTCMD=/bin/umount SSHCMD=/usr/bin/ssh LVSCMD=/usr/sbin/lvs AWKCMD=/bin/awk BASENAMECMD=/bin/basename DATECMD=/bin/date SYNCCMD=/bin/sync SLEEPCMD=/bin/sleep if [ "$THISHOST" = "$MIRROR1" ] then MIRROR=$MIRROR2 fi if [ "$THISHOST" = "$MIRROR2" ] then MIRROR=$MIRROR1 fi if [ "$MIRROR" = "" ] then echo "ERROR: VServer resync must be installed either on $MIRROR1 or $MIRROR2, not here ($THISHOST)" exit 1 fi TIMESTART=`$DATECMD +%s` RETRIES=0 MAXRETRIES=5 $SYNCCMD if $LVSCMD --noheadings ${VSERVERVG}/${VSERVERSNAPSHOT} >/dev/null 2>&1 then echo "ERROR: Snapshot overlap; cannot rsync from $THISHOST to $MIRROR" else # echo "creating LV..." # $LVSCMD if $LVCREATECMD -L${SNAPSHOTSIZE} -s --chunksize 4k -n ${VSERVERVG}/${VSERVERSNAPSHOT} ${VSERVERVG}/${VSERVERLV} >/dev/null 2>&1 && $MOUNTCMD -t ext2 -ni -o noatime,ro ${VSERVERVG}/${VSERVERSNAPSHOT} ${SNAPSHOTMOUNT} >/dev/null 2>&1 then : else # echo "retrying to create snapshot..." while [ $RETRIES -lt "$MAXRETRIES" ] do RETRIES=`expr $RETRIES + 1` $SYNCCMD $SLEEPCMD 1 $UMOUNTCMD -ni $SNAPSHOTMOUNT $SLEEPCMD 1 $LVREMOVECMD -f ${VSERVERVG}/${VSERVERSNAPSHOT} >/dev/null 2>&1 $SLEEPCMD 1 $SYNCCMD $LVREMOVECMD -f ${VSERVERVG}/${VSERVERSNAPSHOT} >/dev/null 2>&1 $SLEEPCMD 1 if $LVCREATECMD -L${SNAPSHOTSIZE} -s --chunksize 4k -n ${VSERVERVG}/${VSERVERSNAPSHOT} ${VSERVERVG}/${VSERVERLV} >/dev/null 2>&1 && $MOUNTCMD -t ext2 -ni -o noatime,ro ${VSERVERVG}/${VSERVERSNAPSHOT} ${SNAPSHOTMOUNT} >/dev/null 2>&1 then break fi done if [ $RETRIES -eq "$MAXRETRIES"] then echo "ERROR: cannot create snapshot after `expr $MAXRETRIES-1` retries" $LVREMOVECMD -f ${VSERVERVG}/${VSERVERSNAPSHOT} >/dev/null 2>&1 exit 2 fi fi NACTIVE=0 NSTANDBY=0 NINACTIVE=0 NERROR=0 for i in /etc/vservers/*; do VSERVER=`$BASENAMECMD $i` if $VSERVERCMD $VSERVER running then # echo "vserver $VSERVER running on $THISHOST" if $SSHCMD $MIRROR $VSERVERCMD $VSERVER running then NERROR=`expr $NERROR + 1` echo "ERROR: vserver $VSERVER running both on $THISHOST and $MIRROR" else NACTIVE=`expr $NACTIVE + 1` # echo "rsyncing vserver $VSERVER from $THISHOST to $MIRROR..." if $RSYNCCMD -aqzpotSx4 --numeric-ids --bwlimit $BWLIMIT --recursive --safe-links --delete --force -e ssh ${SNAPSHOTMOUNT}/${VSERVER}/ $MIRROR:${VSERVERBASE}/${VSERVER}/ then # echo "rsync vserver $VSERVER from $THISHOST to $MIRROR done" : else echo "ERROR: cannot rsync $VSERVER from $THISHOST to $MIRROR" fi fi else if $SSHCMD $MIRROR $VSERVERCMD $VSERVER running then # echo "vserver $VSERVER running on $MIRROR" NSTANDBY=`expr $NSTANDBY + 1` else NINACTIVE=`expr $NSTANDBY + 1` echo "WARNING: vserver $VSERVER inactive both on $THISHOST and $MIRROR" fi fi; done $SYNCCMD if $UMOUNTCMD -ni $SNAPSHOTMOUNT then # echo "unmount of $SNAPSHOTMOUNT done" : else echo "ERROR: cannot unmount LV snapshot" fi ## else ## echo -n `$DATECMD +"%x %X" ` ## echo " ERROR: cannot mount LV snapshot" ## fi SNAPPERCENT=`$LVSCMD --noheadings ${VSERVERVG}/${VSERVERSNAPSHOT} | $AWKCMD '{print $6}'` $SYNCCMD if $LVREMOVECMD -f ${VSERVERVG}/${VSERVERSNAPSHOT} >/dev/null 2>&1 then TIMEEND=`$DATECMD +%s` echo "NOTICE: rsync_vservers complete; elapsedsecs=`expr $TIMEEND - $TIMESTART`; snapshotsize=$SNAPSHOTSIZE; snap%=$SNAPPERCENT; active=$NACTIVE; standby=$NSTANDBY; inactive=$NINACTIVE; error=$NERROR; snapshotretries=$RETRIES" # $LVSCMD else echo "ERROR: cannot remove LV snapshot" fi fi </file> <file migrate_vserver> #!/bin/bash # gets 2 parameters: vserver_name source_server # ensures that vserver_name is running on source_server and standby on the corresponding mirror # stops the vserver on source_server, does rsync of the remaining data and restarts on mirror # if only the vserver_name is given, the script assumes that the server it's running on should become the new active server # # 19-MAR-2006 milprog/mg if [ "$HOSTNAME" = "milprogmail.ch" ] then THISHOST=milprogmail.milnet.lan else THISHOST=$HOSTNAME fi MIRROR1=lepus.milnet.lan MIRROR2=milprogmail.milnet.lan SNAPSHOTSIZE=4096M VSERVERVG=/dev/vg00 VSERVERLV=lv00_vservers VSERVERSNAPSHOT=vservers_snapshot SNAPSHOTMOUNT=/mnt/vservers-snapshot # parameters for taking the snapshot: number of retries, interval in seconds MAXSNAPSHOTRETRIES=3 RETRYINTERVAL=250 # no need to change anything below (hopefully) LVCREATECMD=/usr/sbin/lvcreate LVREMOVECMD=/usr/sbin/lvremove LVSCMD=/usr/sbin/lvs VSERVERCMD=/usr/sbin/vserver VSERVERROOT=/vservers RSYNCCMD=/usr/bin/rsync SSHCMD=/usr/bin/ssh AWKCMD=/bin/awk BASENAMECMD=/bin/basename DATECMD=/bin/date SYNCCMD=/bin/sync SLEEPCMD=/bin/sleep if [ "$THISHOST" = "$MIRROR1" ] then MIRROR=$MIRROR2 fi if [ "$THISHOST" = "$MIRROR2" ] then MIRROR=$MIRROR1 fi if [ "$MIRROR" = "" ] then echo "ERROR: $0 must be installed either on $MIRROR1 or $MIRROR2, not here ($THISHOST)" exit 1 fi VSERVER=$1 if [ "$VSERVER" == "" ] then echo "ERROR: VServer name missing" exit 2 fi MIGRATEFROM=$2 if [ "$MIGRATEFROM" == "" ] then MIGRATEFROM=$MIRROR MIGRATETO=$THISHOST else if [ $MIGRATEFROM == "$THISHOST" ] then MIGRATETO=$MIRROR else if [ $MIGRATEFROM == "$MIRROR" ] then MIGRATETO=$THISHOST else echo "ERROR: $MIGRATEFROM is not a valid host for this script (use $THISHOST or $MIRROR)" exit 3 fi fi fi if [ $MIGRATEFROM == "$THISHOST" ] then RUNFROM="" RUNTO="$SSHCMD $MIGRATETO " else RUNFROM="$SSHCMD $MIGRATEFROM " RUNTO="" fi if $RUNFROM $VSERVERCMD $VSERVER running >/dev/null 2>&1 then if $RUNTO $VSERVERCMD $VSERVER running >/dev/null 2>&1 then echo "ERROR: vserver $VSERVER active on both $MIGRATEFROM and $MIGRATETO" exit 5 else if [ $? -gt 1 ] then echo "ERROR: vserver $VSERVER unknown on $MIGRATETO as a standby server" exit 6 else if $RUNFROM $LVCREATECMD -L${SNAPSHOTSIZE} -s --chunksize 4k -n ${VSERVERVG}/${VSERVERSNAPSHOT} ${VSERVERVG}/${VSERVERLV} >/dev/null 2>&1 then TIMESTART=`$DATECMD +%s` if $RUNFROM $VSERVERCMD $VSERVER stop >/dev/null 2>&1 then if $RUNFROM $RSYNCCMD -aqzpotSx4 --numeric-ids --recursive --safe-links --delete --force -e ssh ${VSERVERROOT}/${VSERVER}/ $MIGRATETO:${VSERVERROOT}/${VSERVER}/ then if $RUNTO $VSERVERCMD $VSERVER start >/dev/null 2>1 then TIMEEND=`$DATECMD +%s` echo "NOTICE: vserver $VSERVER migrated from $MIGRATEFROM to $MIGRATETO; elapsedsecs=`expr $TIMEEND - $TIMESTART`" else if $RUNFROM $VSERVERCMD $VSERVER start >/dev/null 2>&1 then echo "ERROR: cannot start $VSERVER after migration from $MIGRATEFROM to $MIGRATETO; server was restarted again on $MIGRATEFROM" exit 8 else echo "ERROR: cannot start $VSERVER after migration from $MIGRATEFROM to $MIGRATETO; server could not be restarted again on $MIGRATEFROM and is now dead on both mirrors" exit 9 fi fi else if $RUNFROM $VSERVERCMD $VSERVER start >/dev/null 2>&1 then echo "ERROR: cannot rsync $VSERVER for migration from $MIGRATEFROM to $MIGRATETO; server was restarted again on $MIGRATEFROM" exit 10 else echo "ERROR: cannot rsync $VSERVER for migration from $MIGRATEFROM to $MIGRATETO; server could not be restarted again on $MIGRATEFROM and is now dead on both mirrors" exit 11 fi fi else echo "ERROR: vserver $VSERVER cannot be stopped on $MIGRATEFROM" exit 7 fi if $RUNFROM $LVREMOVECMD -f ${VSERVERVG}/${VSERVERSNAPSHOT} >/dev/null 2>&1 then : else echo "ERROR: snapshot ${VSERVERVG}/${VSERVERSNAPSHOT} cannot be removed on $MIGRATEFROM after successful migration" exit 13 fi else echo "ERROR: cannot migrate $VSERVER from $MIGRATEFROM to $MIGRATETO because snapshot ${VSERVERVG}/${VSERVERSNAPSHOT} cannot be established (possible overlap with backup/cronjob)" exit 12 fi fi fi else echo "ERROR: vserver $VSERVER not running on $MIGRATEFROM" exit 4 fi </file> _______________________________________________ Vserver mailing list Vserver@list.linux-vserver.org http://list.linux-vserver.org/mailman/listinfo/vserver