On March 9, 2020 1:05:13 PM GMT+02:00, Christoforos Christoforou <[email protected]> wrote: >> Start: RAID1, LVM, fs, nsfsserver, exportfs, IP address (stop is the >other way 'round) >If your IP resource stops before the nfsserver, I would think it's >possible that connections and file handles are left hanging. >If you don't have an nfsnotify service, then this could be your >problem. >Does your nfsserver resource also have nfs_no_notify=true? > >I'd say test the following: >Test1: Set nfs_no_notify=true on your nfsserver resource and create an >nfsnotify resource that starts after the IP and stops first. >Something like this: pcs resource create r_nfsnotify nfsnotify >source_host=IPADDRESS >Test2: Set the IP to stop after the nfsserver resource and have >nfs_no_notify=false (default) > >Christoforos Christoforou >Senior Systems Administrator >Global Reach Internet Productions >Twitter | Facebook | LinkedIn >p (515) 996-0996 | globalreach.com > >-----Original Message----- >From: Ulrich Windl <[email protected]> >Sent: Monday, March 9, 2020 12:37 PM >To: [email protected]; [email protected] >Subject: RE: Antw: [EXT] [ClusterLabs] Multiple nfsserver resource >groups > >>>> "Christoforos Christoforou" <[email protected]> schrieb >>>> am >09.03.2020 um 10:38 in Nachricht ><[email protected]>: >> Thanks for the advice. >> We haven’t had any issues with the time it takes to prepare the >> exportfs resources so far, and we've been running this setup for 2 >> years now, but I will keep it in mind as we increase the number of >> exportfs resources. I have > >> already implemented the solution discussed and merged all filesystems > >> and exports into one resource group and everything looks good. >> >> For your problem, what is the order in which your resources >> startup/shutdown? > >Start: RAID1, LVM, fs, nsfsserver, exportfs, IP address (stop is the >other way >'round) > >> Is your nfs info dir a filesystem resource or an LVM resource? > >We have everything as LVs (see above) > >> Do you have an nfsnotify resource in place? > >Not explicitly. We are using NFSv3 only. > >> We have found that the order in which resources startup/shutdown >> without any > >> problems is to have the nfsnotify resource stop first, then stop the >exportfs >> resources, nfsserver after that, filesystems (one of which is the nfs > >> shared > >> info dir) and finally the virtual IP resource. Startup is the reverse > >> of that. > >Regards, >Ulrich > > >> >> -----Original Message----- >> From: Ulrich Windl <[email protected]> >> Sent: Monday, March 9, 2020 9:26 AM >> To: [email protected]; [email protected] >> Subject: Antw: [EXT] [ClusterLabs] Multiple nfsserver resource groups >> >>>>> "Christoforos Christoforou" <[email protected]> schrieb > >>>>> am >> 06.03.2020 um 18:56 in Nachricht >> ><25205_1583517421_5E628EE5_25205_75_1_01e001d5f3e0$804a6240$80df26c0$@globalr >> eac >> .com>: >>> Hello, >>> >>> >>> >>> We have a PCS cluster running on 2 CentOS 7 nodes, exposing 2 NFSv3 >>> volumes which are then mounted to multiple servers (around 8). >>> >>> We want to have 2 more sets of additional shared NFS volumes, for a >>> total >> of >>> 6. >>> >>> >>> >>> I have successfully configured 3 resource groups, with each group >>> having >> the >>> following resources: >>> >>> * 1x ocf_heartbeat_IPaddr2 resource for the Virtual IP that exposes >>> the NFS share assigned to its own NIC. >>> * 3x ocf_heartbeat_Filesystem resources (1 is for the >>> nfs_shared_infodir and the other 2 are the ones exposed via the NFS >server) >>> * 1x ocf_heartbeat_nfsserver resource that uses the aforementioned >>> nfs_shared_infodir. >>> * 2x ocf_heartbeat_exportfs resources that expose the other 2 >>> filesystems as NFS shares. >>> * 1x ocf_heartbeat_nfsnotify resource that has the Virtual IP set as >>> its own source_host. >>> >>> >>> >>> All 9 filesystem volumes are mounted via iSCSI to the PCS nodes in >>> /dev/mapper/mpathX >>> >>> So the structure is like so: >>> >>> Resource group 1: >>> >>> * /dev/mapper/mpatha ‑ shared volume 1 >>> * /dev/mapper/mpathb ‑ shared volume 2 >>> * /dev/mapper/mpathc ‑ nfs_shared_infodir for resource group 1 >>> >>> Resource group 2: >>> >>> * /dev/mapper/mpathd ‑ shared volume 3 >>> * /dev/mapper/mpathe ‑ shared volume 4 >>> * /dev/mapper/mpathf ‑ nfs_shared_infodir for resource group 2 >>> >>> Resource group 3: >>> >>> * /dev/mapper/mpathg ‑ shared volume 5 >>> * /dev/mapper/mpathh ‑ shared volume 6 >>> * /dev/mapper/mpathi ‑ nfs_shared_infodir for resource group 3 >>> >>> >>> >>> My concern is that when I run a df command on the active node, the >>> last ocf_heartbeat_nfsserver volume (/dev/mapper/mpathi) mounted to >> /var/lib/nfs. >>> I understand that I cannot change this, but I can change the >location >>> of >> the >>> rpc_pipefs folder. >>> >>> >>> >>> I have had this setup running with 2 resource groups in our >>> development environment, and have not noticed any issues, but since >>> we're planning to move to production and add a 3rd resource group, I > >>> want to make sure that this setup will not cause any issues. I am by > >>> no means an expert on NFS, so some insight is appreciated. >>> >>> >>> >>> If this kind of setup is not supported or recommended, I have 2 >>> alternate plans in mind: >>> >>> 1. Have all resources in the same resource group, in a setup that >will >>> look like this: >>> >>> a. 1x ocf_heartbeat_IPaddr2 resource for the Virtual IP that exposes >>> the NFS share. >>> b. 7x ocf_heartbeat_Filesystem resources (1 is for the >>> nfs_shared_infodir and 6 exposed via the NFS server) >>> c. 1x ocf_heartbeat_nfsserver resource that uses the aforementioned >>> nfs_shared_infodir. >>> d. 6x ocf_heartbeat_exportfs resources that expose the other 6 >>> filesystems as NFS shares. Use the clientspec option to restrict to >>> IPs and prevent unwanted mounts. >>> e. 1x ocf_heartbeat_nfsnotify resource that has the Virtual IP set >as >>> its own source_host. >>> >>> 2. Setup 2 more clusters to accommodate our needs >>> >>> >>> >>> I really want to avoid #2, due to the fact that it will be overkill >>> for our case. >> >> Things you might consider is to get reid of the groups and use >> explicit colocation and orderings. The advantages will be that you >can >> execute >several >> agents in parallel (e.g. prepare all fileysstems in parallel). In the > >> past >we >> had made the experience that exportfs resources can take quite some >> time and > >> if you have like 20 or more of them, it delays the shutdown/startup >> significatly. >> So we moved to using netgroups provided by LDAP instead, and we could > >> reduce > >> the number of exportfs statements drastically. >> However we have one odd problem (SLES12 SP4): The NFS resource using >> systemd > >> does not shut down clearly due some unmount issue related the shared >> info dir. >> >>> >>> Thanks >>> >>> >>> >>> Christoforos Christoforou >>> >>> Senior Systems Administrator >>> >>> Global Reach Internet Productions >>> >>> <http://www.twitter.com/globalreach> Twitter | >>> <http://www.facebook.com/globalreach> Facebook | >>> <https://www.linkedin.com/company/global‑reach‑internet‑productions> >>> LinkedIn >>> >>> p (515) 996‑0996 | <http://www.globalreach.com/> globalreach.com >>> >>> > > > > > > > >_______________________________________________ >Manage your subscription: >https://lists.clusterlabs.org/mailman/listinfo/users > >ClusterLabs home: https://www.clusterlabs.org/
Actually Red Hat's documentation uses the nfsnotify variant . Best Regards, Strahil Nikolov _______________________________________________ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
