If I might chip in here, this can cause an issue when rebooting nodes unless you make sure to stop the Gluster processes first. If you don't stop the processes, then the Gluster volumes can pause for the default 42 seconds(?) until the other nodes time out the rebooting node. This is of course long enough to cause any VMs running their volumes on gluster to show I/O errors and re-mount them as read-only potentially causing all sorts of mischief.
I *think* there's a systemd solution to this somewhere, perhaps the RedHat packages?, that does stop the gluster processes prior to a reboot or halt. Certainly the Debian packages *don't* have this solution in place. I wish they did but I've never mastered enough systemd foo to sort if out myself. :-). Ronny Aravinda wrote on 05/02/2024 19:09: > Hi Anant, > > It was intentional design decision to not stop any gluster processes if > Glusterd need to be upgraded or if Glusterd crashes. Because of this Volume > availability will not be affected if any issues with Glusterd or Glusterd is > upgraded. All the mounts will reconnect once the Glusterd comes back up. CLI > operations from that node may not be available but IO will not be affected if > the Glusterd is down (New mounts can't be created, but existing mounts should > work without Glusterd). > > stop-all-gluster-processes.sh is available as part of installation, it can be > used to stop all the processes (Check in /usr/share/glusterfs/scripts) > > -- > Thanks and Regards > Aravinda > Kadalu Technologies > > > > ---- On Mon, 05 Feb 2024 22:40:30 +0530 *Anant Saraswat > <anant.saras...@techblue.co.uk>* wrote --- > > Hello Everyone, > > > I am using GlusterFS 9.4, and whenever we use the systemctl command to > stop the Gluster server, it leaves many Gluster processes running. So, I just > want to check how to shut down the Gluster server in a graceful manner. > > > Is there any specific sequence or trick I need to follow? Currently, I am > using the following command: > > > [root@master2 ~]# systemctl stop glusterd.service > > [root@master2 ~]# ps aux | grep gluster > root 2710138 14.1 0.0 2968372 216852 ? Ssl Jan27 170:27 > /usr/sbin/glusterfsd -s master2 --volfile-id > tier1data.master2.opt-tier1data2019-brick -p > /var/run/gluster/vols/tier1data/master2-opt-tier1data2019-brick.pid -S > /var/run/gluster/97da28e3d5c23317.socket --brick-name > /opt/tier1data2019/brick -l > /var/log/glusterfs/bricks/opt-tier1data2019-brick.log --xlator-option > *-posix.glusterd-uuid=c1591bde-df1c-41b4-8cc3-5eaa02c5b89d --process-name > brick --brick-port 49152 --xlator-option tier1data-server.listen-port=49152 > root 2710196 0.0 0.0 1298116 11544 ? Ssl Jan27 0:01 > /usr/sbin/glusterfs -s localhost --volfile-id shd/tier1data -p > /var/run/gluster/shd/tier1data/tier1data-shd.pid -l > /var/log/glusterfs/glustershd.log -S /var/run/gluster/1ac2284f75671ffa.socket > --xlator-option *replicate*.node-uuid=c1591bde-df1c-41b4-8cc3-5eaa02c5b89d > --process-name glustershd --client-pid=-6 > root 3730742 0.0 0.0 288264 14388 ? Ssl 18:44 0:00 > /usr/bin/python3 /usr/libexec/glusterfs/python/syncdaemon/gsyncd.py > --path=/opt/tier1data2019/brick --monitor -c > /var/lib/glusterd/geo-replication/tier1data_drtier1data_drtier1data/gsyncd.conf > --iprefix=/var :tier1data > --glusterd-uuid=c1591bde-df1c-41b4-8cc3-5eaa02c5b89d drtier1data::drtier1data > root 3730763 2.4 0.0 2097216 35904 ? Sl 18:44 0:09 > python3 /usr/libexec/glusterfs/python/syncdaemon/gsyncd.py worker tier1data > drtier1data::drtier1data --feedback-fd 9 --local-path > /opt/tier1data2019/brick --local-node master2 --local-node-id > c1591bde-df1c-41b4-8cc3-5eaa02c5b89d --slave-id > eca32e08-c3f8-4883-bef5-84bfb89f4d56 --subvol-num 1 --resource-remote > drtier1data --resource-remote-id 28f3e75b-56aa-43a1-a0ea-a0e5d44d59ea > root 3730768 0.7 0.0 50796 9668 ? S 18:44 0:02 ssh > -oPasswordAuthentication=no -oStrictHostKeyChecking=no -i > /var/lib/glusterd/geo-replication/secret.pem -p 22 -oControlMaster=auto -S > /tmp/gsyncd-aux-ssh-ep7a14up/75785990b3233f5dbbab9f43cc3ed895.sock > drtier1data /nonexistent/gsyncd slave tier1data drtier1data::drtier1data > --master-node master2 --master-node-id c1591bde-df1c-41b4-8cc3-5eaa02c5b89d > --master-brick /opt/tier1data2019/brick --local-node drtier1data > --local-node-id 28f3e75b-56aa-43a1-a0ea-a0e5d44d59ea --slave-timeout 120 > --slave-log-level INFO --slave-gluster-log-level INFO > --slave-gluster-command-dir /usr/sbin --master-dist-count 1 > root 3730795 1.1 0.0 1108268 55596 ? Ssl 18:44 0:04 > /usr/sbin/glusterfs --aux-gfid-mount --acl --log-level=INFO > --log-file=/var/log/glusterfs/geo-replication/tier1data_drtier1data_drtier1data/mnt-opt-tier1data2019-brick.log > --volfile-server=localhost --volfile-id=tier1data --client-pid=-1 > /tmp/gsyncd-aux-mount-9210kh43 > root 3772665 0.0 0.0 12208 2400 ? S 18:51 0:00 rsync > -aR0 --inplace --files-from=- --super --stats --numeric-ids --no-implied-dirs > --existing --xattrs --acls --ignore-missing-args . -e ssh > -oPasswordAuthentication=no -oStrictHostKeyChecking=no -i > /var/lib/glusterd/geo-replication/secret.pem -p 22 -oControlMaster=auto -S > /tmp/gsyncd-aux-ssh-ep7a14up/75785990b3233f5dbbab9f43cc3ed895.sock > drtier1data:/proc/897118/cwd > root 3772667 0.0 0.0 44156 5640 ? S 18:51 0:00 ssh > -oPasswordAuthentication=no -oStrictHostKeyChecking=no -i > /var/lib/glusterd/geo-replication/secret.pem -p 22 -oControlMaster=auto -S > /tmp/gsyncd-aux-ssh-ep7a14up/75785990b3233f5dbbab9f43cc3ed895.sock > drtier1data rsync --server -logDtpAXRe.LsfxC --super --stats --numeric-ids > --existing --inplace --no-implied-dirs . /proc/897118/cwd > > For now, We are using > https://github.com/gluster/glusterfs/blob/master/extras/stop-all-gluster-processes.sh > to kill all the remaining processes. > > Thanks, > Anant > > > DISCLAIMER: This email and any files transmitted with it are confidential > and intended solely for the use of the individual or entity to whom they are > addressed. If you have received this email in error, please notify the > sender. This message contains confidential information and is intended only > for the individual named. If you are not the named addressee, you should not > disseminate, distribute or copy this email. Please notify the sender > immediately by email if you have received this email by mistake and delete > this email from your system. > > If you are not the intended recipient, you are notified that disclosing, > copying, distributing or taking any action in reliance on the contents of > this information is strictly prohibited. Thanks for your cooperation. > > ________ > > > > Community Meeting Calendar: > > Schedule - > Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC > Bridge: https://meet.google.com/cpu-eiue-hvk > Gluster-users mailing list > Gluster-users@gluster.org > https://lists.gluster.org/mailman/listinfo/gluster-users > > > > ________ > > > > Community Meeting Calendar: > > Schedule - > Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC > Bridge: https://meet.google.com/cpu-eiue-hvk > Gluster-users mailing list > Gluster-users@gluster.org > https://lists.gluster.org/mailman/listinfo/gluster-users -- Ronny Adsetts Technical Director Amazing Internet Ltd, London t: +44 20 8977 8943 w: www.amazinginternet.com Registered office: 85 Waldegrave Park, Twickenham, TW1 4TJ Registered in England. Company No. 4042957
________ Community Meeting Calendar: Schedule - Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC Bridge: https://meet.google.com/cpu-eiue-hvk Gluster-users mailing list Gluster-users@gluster.org https://lists.gluster.org/mailman/listinfo/gluster-users
