The branch, master has been updated via ecbe6c6 ctdb-scripts: Add GlusterFS support to nfs-ganesha-callout via 83226d9 ctdb-scripts: Section off GPFS-specific functionality in nfs-ganesha-callout via 8b59705 ctdb-scripts: Add config options for use by clustered NFS via c7dbf72 ctdb-scripts: Parametize symlink checking in nfs-ganesha-callout via a540a0d ctdb-scripts: Cleanup service_check() in nfs-ganesha-callout via e9b507c ctdb-scripts: Use D-Bus messages to trigger grace in nfs-ganesha-callout via d375fd8 ctdb-scripts: Add register action to nfs-ganesha-callout via 755f060 ctdb-scripts: Organize global variables in nfs_ganesha_callout via a857e33 ctdb-scripts: Various small fixes to example nfs-ganesha-callout from f9d4cb4 ctdb-recoverd: Unify takeover run triggering code in main loop
https://git.samba.org/?p=samba.git;a=shortlog;h=master - Log ----------------------------------------------------------------- commit ecbe6c669f00ca3db6be76b865f06b3ea9f2c014 Author: Jose A. Rivera <jar...@samba.org> Date: Thu Apr 28 21:24:16 2016 -0500 ctdb-scripts: Add GlusterFS support to nfs-ganesha-callout Signed-off-by: Jose A. Rivera <jar...@samba.org> Reviewed-by: Martin Schwenke <mar...@meltin.net> Autobuild-User(master): Martin Schwenke <mart...@samba.org> Autobuild-Date(master): Sat May 14 03:06:05 CEST 2016 on sn-devel-144 commit 83226d98e59b17b99533f3298420675618bb663e Author: Jose A. Rivera <jar...@samba.org> Date: Tue May 3 01:53:07 2016 -0500 ctdb-scripts: Section off GPFS-specific functionality in nfs-ganesha-callout Signed-off-by: Jose A. Rivera <jar...@samba.org> Reviewed-by: Martin Schwenke <mar...@meltin.net> commit 8b597052521ed67234d12e899d336df5f1eb354b Author: Jose A. Rivera <jar...@samba.org> Date: Thu Apr 28 21:18:05 2016 -0500 ctdb-scripts: Add config options for use by clustered NFS Add CTDB_NFS_STATE_FS_TYPE and CTDB_NFS_STATE_MNT config options, show use in nfs-ganesha-callout. Since the callout script is only an example, we officially don't have default values for these. Signed-off-by: Jose A. Rivera <jar...@samba.org> Reviewed-by: Martin Schwenke <mar...@meltin.net> commit c7dbf720d830bb0a107adb3369946daab08a4429 Author: Jose A. Rivera <jar...@samba.org> Date: Wed May 4 14:16:27 2016 -0500 ctdb-scripts: Parametize symlink checking in nfs-ganesha-callout Signed-off-by: Jose A. Rivera <jar...@samba.org> Reviewed-by: Martin Schwenke <mar...@meltin.net> commit a540a0d213b41010fd084caa05cd57e3199d6a20 Author: Jose A. Rivera <jar...@samba.org> Date: Tue May 10 16:27:43 2016 +0200 ctdb-scripts: Cleanup service_check() in nfs-ganesha-callout Signed-off-by: Jose A. Rivera <jar...@samba.org> Reviewed-by: Martin Schwenke <mar...@meltin.net> commit e9b507c11c2538b18de3b0d3fd8685bd8d195a37 Author: Jose A. Rivera <jar...@samba.org> Date: Wed May 4 13:03:29 2016 -0500 ctdb-scripts: Use D-Bus messages to trigger grace in nfs-ganesha-callout Signed-off-by: Jose A. Rivera <jar...@samba.org> Reviewed-by: Martin Schwenke <mar...@meltin.net> commit d375fd80f02fcdc5af35ea83639a9acf75577710 Author: Jose A. Rivera <jar...@samba.org> Date: Thu Apr 28 21:12:33 2016 -0500 ctdb-scripts: Add register action to nfs-ganesha-callout Signed-off-by: Jose A. Rivera <jar...@samba.org> Reviewed-by: Martin Schwenke <mar...@meltin.net> commit 755f060d52125f771e64d472744aec79eaf9fb11 Author: Jose A. Rivera <jar...@samba.org> Date: Tue May 10 16:50:10 2016 +0200 ctdb-scripts: Organize global variables in nfs_ganesha_callout Signed-off-by: Jose A. Rivera <jar...@samba.org> Reviewed-by: Martin Schwenke <mar...@meltin.net> commit a857e33deb973735813aa12a93eb1f4bf0133695 Author: Jose A. Rivera <jar...@samba.org> Date: Thu Apr 28 20:58:30 2016 -0500 ctdb-scripts: Various small fixes to example nfs-ganesha-callout Comment typos and clarifications, erroneous variable names, corrected pathnames, reorganizing variables, and squashing a few non-fatal scripting errors. Signed-off-by: Jose A. Rivera <jar...@samba.org> Reviewed-by: Martin Schwenke <mar...@meltin.net> ----------------------------------------------------------------------- Summary of changes: ctdb/config/events.d/60.nfs | 8 ++ ctdb/doc/ctdbd.conf.5.xml | 33 +++-- ctdb/doc/examples/nfs-ganesha-callout | 233 ++++++++++++++++++++-------------- 3 files changed, 168 insertions(+), 106 deletions(-) Changeset truncated at 500 lines: diff --git a/ctdb/config/events.d/60.nfs b/ctdb/config/events.d/60.nfs index c25b73e..57cfbff 100755 --- a/ctdb/config/events.d/60.nfs +++ b/ctdb/config/events.d/60.nfs @@ -21,6 +21,14 @@ export CTDB_NFS_CALLOUT # If the callout wants to use this then it must create it export CTDB_NFS_CALLOUT_STATE_DIR="${service_state_dir}/callout-state" +# Export, if set, for use by clustered NFS callouts +if [ -n "$CTDB_NFS_STATE_FS_TYPE" ] ; then + export CTDB_NFS_STATE_FS_TYPE +fi +if [ -n "$CTDB_NFS_STATE_MNT" ] ; then + export CTDB_NFS_STATE_MNT +fi + nfs_callout_cache="${service_state_dir}/nfs_callout_cache" nfs_callout_cache_callout="${nfs_callout_cache}/CTDB_NFS_CALLOUT" nfs_callout_cache_ops="${nfs_callout_cache}/ops" diff --git a/ctdb/doc/ctdbd.conf.5.xml b/ctdb/doc/ctdbd.conf.5.xml index a364c9f..f69f705 100644 --- a/ctdb/doc/ctdbd.conf.5.xml +++ b/ctdb/doc/ctdbd.conf.5.xml @@ -1062,19 +1062,6 @@ CTDB_PER_IP_ROUTING_TABLE_ID_HIGH=9000 <variablelist> <varlistentry> - <term>CTDB_CLUSTER_FILESYSTEM_TYPE=gpfs</term> - <listitem> - <para> - The type of cluster filesystem to use with NFS-ganesha. - Currently only "gpfs" is supported. - </para> - <para> - Default is "gpfs". - </para> - </listitem> - </varlistentry> - - <varlistentry> <term>CTDB_MANAGES_NFS=yes|no</term> <listitem> <para> @@ -1148,6 +1135,26 @@ CTDB_PER_IP_ROUTING_TABLE_ID_HIGH=9000 </listitem> </varlistentry> + <varlistentry> + <term>CTDB_NFS_STATE_FS_TYPE=<parameter>TYPE</parameter></term> + <listitem> + <para> + The type of filesystem used for a clustered NFS' shared + state. No default. + </para> + </listitem> + </varlistentry> + + <varlistentry> + <term>CTDB_NFS_STATE_MNT=<parameter>DIR</parameter></term> + <listitem> + <para> + The directory where a clustered NFS' shared state will be + located. No default. + </para> + </listitem> + </varlistentry> + </variablelist> </refsect2> diff --git a/ctdb/doc/examples/nfs-ganesha-callout b/ctdb/doc/examples/nfs-ganesha-callout index a3d0471..34794c1 100755 --- a/ctdb/doc/examples/nfs-ganesha-callout +++ b/ctdb/doc/examples/nfs-ganesha-callout @@ -14,10 +14,19 @@ # * Set CTDB_NFS_CALLOUT in your CTDB configuration to point to this # script # -# * Rename nfs-checks.d/{20.nfs.check,30.nlockmgr.check,50.mountd.check} -# so that they no longer have the ".check" suffix +# * Rename the following files in nfs-checks.d so that they no longer +# have the ".check" suffix: +# * 10.status.check +# * 20.nfs.check +# * 30.nlockmgr.check +# * 40.mountd.check +# * 50.rquotad.check # # * Install 20.nfs-ganesha.check to nfs-checks.d/20.nfs.check +# +# * It is recommended, but not required, to install the grace_period +# script (usually shipped in a utils package for NFS-Ganesha) to +# /usr/bin/grace_period # I (Martin Schwenke) hereby relicense all of my contributions to this # callout (and, previously, to 60.ganesha) to a license compatible @@ -30,14 +39,34 @@ # Exit on 1st error set -e -if [ -z "$CTDB_CLUSTER_FILESYSTEM_TYPE" ] ; then - CTDB_CLUSTER_FILESYSTEM_TYPE="gpfs" -fi +# Filesystem type and mount point for the (typically clustered) +# volume that will contain the NFS-Ganesha state. +state_fs="${CTDB_NFS_STATE_FS_TYPE:-gpfs}" +state_dir="${CTDB_NFS_STATE_MNT}" # No sane default. + +# To change the following, edit the default values below. Do not set +# these - they aren't configuration variables, just hooks for testing. +nfs_exports_file="${CTDB_NFS_EXPORTS_FILE:-/etc/ganesha/ganesha.conf}" +nfs_service="${CTDB_NFS_SERVICE:-nfs-ganesha}" +ganesha_rec_subdir=${CTDB_GANESHA_REC_SUBDIR:-.ganesha} +procfs=${PROCFS_PATH:-/proc} + +case $state_fs in + gpfs) + GANRECDIR="/var/lib/nfs/ganesha" + ;; + glusterfs) + if [ -z "${state_dir}" ]; then + echo "CTDB_NFS_STATE_MNT not defined for GlusterFS" + exit 1 + fi + host=`hostname` + NODESTATEDIR="$state_dir/nfs-ganesha/$host" + GANSTATEDIR="$state_dir/nfs-ganesha/.noderefs" + NODESTATELN="$GANSTATEDIR/$host" + ;; +esac -# Override for unit testing -if [ -z "$PROCFS_PATH" ] ; then - PROCFS_PATH="/proc" -fi ################################################## @@ -57,8 +86,6 @@ EOF ################################################## # Basic service stop and start -nfs_service="nfs-ganesha-$CTDB_CLUSTER_FILESYSTEM_TYPE" - basic_stop () { case "$1" in @@ -117,50 +144,82 @@ service_start () ################################################## # Nitty gritty - monitoring and IP handling -GANRECDIR="/var/lib/nfs/ganesha" -GANRECDIR2="/var/lib/nfs/ganesha/recevents" -GANRECDIR3="/var/lib/nfs/ganesha_local" +# Check that a symlink exists, create it otherwise. +# Usage: check_ln <TARGET> <LINK> +check_ln () +{ + if [ ! -L "${2}" ] ; then + rm -vrf "${2}" + else + _t=$(readlink "${2}") + if [ "$_t" != "${1}" ] ; then + rm -v "${2}" + fi + fi + # This is not an "else". It also re-creates the link if it was + # removed above! + if [ ! -e "${2}" ]; then + ln -sfv "${1}" "${2}" + fi +} +# Return 'active' if the shared filesystem is accessible. get_cluster_fs_state () { - case $CTDB_CLUSTER_FILESYSTEM_TYPE in + case $state_fs in gpfs) /usr/lpp/mmfs/bin/mmgetstate | awk 'NR == 4 { print $3 }' ;; + glusterfs) + # Since we're past create_ganesha_recdirs(), we're active. + echo "active" + ;; *) - die "File system $CTDB_CLUSTER_FILESYSTEM_TYPE not supported" + echo "File system $state_fs not supported" + exit 1 ;; esac } create_ganesha_recdirs () { - [ -n "$CTDB_GANESHA_REC_SUBDIR" ] || CTDB_GANESHA_REC_SUBDIR=".ganesha" - - _mounts=$(mount -t $CTDB_CLUSTER_FILESYSTEM_TYPE) + if ! _mounts=$(mount | grep $state_fs); then + echo "Failed to find mounts of type $state_fs" + exit 1 + fi if [ -z "$_mounts" ]; then - echo "startup $CTDB_CLUSTER_FILESYSTEM_TYPE not ready" + echo "startup $state_fs not ready" exit 0 fi - _mntpt=$(echo "$_mounts" | sort | awk 'NR == 1 {print $3}') - _link_dst="${_mntpt}/${CTDB_GANESHA_REC_SUBDIR}" - mkdir -vp "$_link_dst" - if [ ! -L "$GANRECDIR" ] ; then - rm -vrf "$GANRECDIR" - else - _t=$(readlink "$GANRECDIR") - if [ "$_t" != "$_link_dst" ] ; then - rm -v "$GANRECDIR" - fi - fi - # This is not an "else". It also re-creates the link if it was - # removed above! - if [ ! -e "$GANRECDIR" ]; then - ln -sv "$_link_dst" "$GANRECDIR" - fi - mkdir -p "$GANRECDIR2" - mkdir -p "$GANRECDIR3" + case $state_fs in + gpfs) + _mntpt=$(echo "$_mounts" | sort | awk 'NR == 1 {print $3}') + _link_dst="${_mntpt}/${ganesha_rec_subdir}" + mkdir -vp "$_link_dst" + check_ln "$_link_dst" "$GANRECDIR" + ;; + glusterfs) + [ -d /var/lib/nfs.backup ] || mv /var/lib/nfs /var/lib/nfs.backup + check_ln ${NODESTATEDIR} /var/lib/nfs + + mkdir -p ${NODESTATEDIR}/ganesha/v4recov + mkdir -p ${NODESTATEDIR}/ganesha/v4old + mkdir -p ${NODESTATEDIR}/statd/sm + mkdir -p ${NODESTATEDIR}/statd/sm.bak + touch ${NODESTATEDIR}/state + touch ${NODESTATEDIR}/statd/state + + mkdir -p ${GANSTATEDIR} + check_ln ${NODESTATEDIR} ${NODESTATELN} + for node in `ls ${GANSTATEDIR}`; do + if [ "${node}" != "${host}" ]; then + check_ln ${GANSTATEDIR}/${node}/ganesha ${NODESTATEDIR}/ganesha/${node} + check_ln ${GANSTATEDIR}/${node}/statd ${NODESTATEDIR}/statd/${node} + fi + done + ;; + esac } service_check () @@ -170,78 +229,48 @@ service_check () # Always succeed if cluster filesystem is not active _cluster_fs_state=$(get_cluster_fs_state) if [ $_cluster_fs_state != "active" ] ; then - exit 0 + return 0 fi # Check that NFS Ganesha is running, according to PID file _pidfile="/var/run/ganesha.pid" - _ganesha="/usr/bin/$CTDB_CLUSTER_FILESYSTEM_TYPE.ganesha.nfsd" + _ganesha="/usr/bin/ganesha.nfsd" if ! { read _pid < "$_pidfile" && \ - grep "$_ganesha" "${PROCFS_PATH}/${_pid}/cmdline" ; } >/dev/null 2>&1 ; then + grep "$_ganesha" "${procfs}/${_pid}/cmdline" ; } >/dev/null 2>&1 ; then echo "ERROR: NFS Ganesha not running according to PID file" return 1 fi - # Check red conditions against limit - _reds_max=2 - _reds=$(ls $GANRECDIR3 | grep -c "red") - - if [ $_reds -ge $_reds_max ] ; then - echo "Too many red conditions (${_reds}/${_reds_max})" - return 1 - fi - - # Check for stall - _stall_max=120 - _now=$(date +"%s") - _last=$(ls -t $GANRECDIR3 | sed -n -e '1s@_.*@@p') - [ -n $_last ] || _last=$_now # Handle startup - _stall=$(($_now - $_last)) - if [ $_stall -ge $_stall_max ] ; then - echo "ERROR: Stalled for ${_stall} second(s)" - return 1 - fi - return 0 } #------------------------------------------------- -get_nodenum () -{ - _nodenum_file="${GANRECDIR}/gpfs_nodenum" - - if [ ! -f "$_nodenum_file" ]; then - /usr/lpp/mmfs/bin/mmlsconfig myNodeConfigNumber | - awk '{print $2}' >"$_nodenum_file" - fi - - cat "$_nodenum_file" -} - nfs_releaseip () { - case $CLUSTER_FILESYSTEM_TYPE in - gpfs) - _nnum=$(get_nodenum) - _tdate=$(date +"%s") - _touchtgt="releaseip_${_tdate}_${_nnum}_${2}_${3}_${1}" - touch "${GANRECDIR2}/${_touchtgt}" - touch "$GANRECDIR2/my${_touchtgt}" - ;; - esac + if [ -x "/usr/bin/grace_period" ]; then + /usr/bin/grace_period "2:${2}" + else + dbus-send --print-reply --system --dest=org.ganesha.nfsd \ + /org/ganesha/nfsd/admin org.ganesha.nfsd.admin.grace \ + string:"2:${2}" + fi } nfs_takeip () { - case $CLUSTER_FILESYSTEM_TYPE in - gpfs) - _nnum=$(get_nodenum) - _tdate=$(date +"%s") - _touchtgt="takeip_${_tdate}_${_nnum}_${2}_${3}_${1}" - touch "${GANRECDIR2}/${_touchtgt}" - ;; + case $state_fs in + glusterfs) + check_ln ${NODESTATEDIR} ${GANSTATEDIR}/${2} + ;; esac + if [ -x "/usr/bin/grace_period" ]; then + /usr/bin/grace_period "5:${2}" + else + dbus-send --print-reply --system --dest=org.ganesha.nfsd \ + /org/ganesha/nfsd/admin org.ganesha.nfsd.admin.grace \ + string:"5:${2}" + fi } ################################################## @@ -254,11 +283,12 @@ nfs_shutdown () nfs_startup () { + basic_stop "nfs" || true + create_ganesha_recdirs - basic_stop "nfs" || true basic_start "nfs" - _f="${PROCFS_PATH}/sys/net/ipv4/tcp_tw_recycle" + _f="${procfs}/sys/net/ipv4/tcp_tw_recycle" if [ "$_f" ] ; then echo 1 >"$_f" fi @@ -269,13 +299,29 @@ nfs_startup () nfs_monitor_list_shares () { - grep Path /etc/ganesha/$CTDB_CLUSTER_FILESYSTEM_TYPE.ganesha.exports.conf | + grep Path $nfs_exports_file | cut -f2 -d\" | sort -u } ################################################## +nfs_register () +{ + cat <<EOF +shutdown +startup +stop +start +check +releaseip +takeip +monitor-list-shares +EOF +} + +################################################## + action="$1" shift @@ -288,7 +334,8 @@ case "$action" in releaseip) nfs_releaseip "$@" ;; takeip) nfs_takeip "$@" ;; monitor-list-shares) nfs_monitor_list_shares ;; - register|monitor-pre|monitor-post) + register) nfs_register ;; + monitor-pre|monitor-post) # Not required/implemented : ;; -- Samba Shared Repository