>>> "Christoforos Christoforou" <[email protected]> schrieb am 09.03.2020 um 10:38 in Nachricht <[email protected]>: > Thanks for the advice. > We haven’t had any issues with the time it takes to prepare the exportfs > resources so far, and we've been running this setup for 2 years now, but I > will keep it in mind as we increase the number of exportfs resources. I have
> already implemented the solution discussed and merged all filesystems and > exports into one resource group and everything looks good. > > For your problem, what is the order in which your resources > startup/shutdown? Start: RAID1, LVM, fs, nsfsserver, exportfs, IP address (stop is the other way 'round) > Is your nfs info dir a filesystem resource or an LVM resource? We have everything as LVs (see above) > Do you have an nfsnotify resource in place? Not explicitly. We are using NFSv3 only. > We have found that the order in which resources startup/shutdown without any > problems is to have the nfsnotify resource stop first, then stop the exportfs > resources, nfsserver after that, filesystems (one of which is the nfs shared > info dir) and finally the virtual IP resource. Startup is the reverse of > that. Regards, Ulrich > > -----Original Message----- > From: Ulrich Windl <[email protected]> > Sent: Monday, March 9, 2020 9:26 AM > To: [email protected]; [email protected] > Subject: Antw: [EXT] [ClusterLabs] Multiple nfsserver resource groups > >>>> "Christoforos Christoforou" <[email protected]> schrieb >>>> am > 06.03.2020 um 18:56 in Nachricht > <25205_1583517421_5E628EE5_25205_75_1_01e001d5f3e0$804a6240$80df26c0$@globalr > eac > .com>: >> Hello, >> >> >> >> We have a PCS cluster running on 2 CentOS 7 nodes, exposing 2 NFSv3 >> volumes which are then mounted to multiple servers (around 8). >> >> We want to have 2 more sets of additional shared NFS volumes, for a >> total > of >> 6. >> >> >> >> I have successfully configured 3 resource groups, with each group >> having > the >> following resources: >> >> * 1x ocf_heartbeat_IPaddr2 resource for the Virtual IP that exposes >> the NFS share assigned to its own NIC. >> * 3x ocf_heartbeat_Filesystem resources (1 is for the >> nfs_shared_infodir and the other 2 are the ones exposed via the NFS server) >> * 1x ocf_heartbeat_nfsserver resource that uses the aforementioned >> nfs_shared_infodir. >> * 2x ocf_heartbeat_exportfs resources that expose the other 2 >> filesystems as NFS shares. >> * 1x ocf_heartbeat_nfsnotify resource that has the Virtual IP set as >> its own source_host. >> >> >> >> All 9 filesystem volumes are mounted via iSCSI to the PCS nodes in >> /dev/mapper/mpathX >> >> So the structure is like so: >> >> Resource group 1: >> >> * /dev/mapper/mpatha ‑ shared volume 1 >> * /dev/mapper/mpathb ‑ shared volume 2 >> * /dev/mapper/mpathc ‑ nfs_shared_infodir for resource group 1 >> >> Resource group 2: >> >> * /dev/mapper/mpathd ‑ shared volume 3 >> * /dev/mapper/mpathe ‑ shared volume 4 >> * /dev/mapper/mpathf ‑ nfs_shared_infodir for resource group 2 >> >> Resource group 3: >> >> * /dev/mapper/mpathg ‑ shared volume 5 >> * /dev/mapper/mpathh ‑ shared volume 6 >> * /dev/mapper/mpathi ‑ nfs_shared_infodir for resource group 3 >> >> >> >> My concern is that when I run a df command on the active node, the >> last ocf_heartbeat_nfsserver volume (/dev/mapper/mpathi) mounted to > /var/lib/nfs. >> I understand that I cannot change this, but I can change the location >> of > the >> rpc_pipefs folder. >> >> >> >> I have had this setup running with 2 resource groups in our >> development environment, and have not noticed any issues, but since >> we're planning to move to production and add a 3rd resource group, I >> want to make sure that this setup will not cause any issues. I am by >> no means an expert on NFS, so some insight is appreciated. >> >> >> >> If this kind of setup is not supported or recommended, I have 2 >> alternate plans in mind: >> >> 1. Have all resources in the same resource group, in a setup that will >> look like this: >> >> a. 1x ocf_heartbeat_IPaddr2 resource for the Virtual IP that exposes >> the NFS share. >> b. 7x ocf_heartbeat_Filesystem resources (1 is for the >> nfs_shared_infodir and 6 exposed via the NFS server) >> c. 1x ocf_heartbeat_nfsserver resource that uses the aforementioned >> nfs_shared_infodir. >> d. 6x ocf_heartbeat_exportfs resources that expose the other 6 >> filesystems as NFS shares. Use the clientspec option to restrict to >> IPs and prevent unwanted mounts. >> e. 1x ocf_heartbeat_nfsnotify resource that has the Virtual IP set as >> its own source_host. >> >> 2. Setup 2 more clusters to accommodate our needs >> >> >> >> I really want to avoid #2, due to the fact that it will be overkill >> for our case. > > Things you might consider is to get reid of the groups and use explicit > colocation and orderings. The advantages will be that you can execute several > agents in parallel (e.g. prepare all fileysstems in parallel). In the past we > had made the experience that exportfs resources can take quite some time and > if you have like 20 or more of them, it delays the shutdown/startup > significatly. > So we moved to using netgroups provided by LDAP instead, and we could reduce > the number of exportfs statements drastically. > However we have one odd problem (SLES12 SP4): The NFS resource using systemd > does not shut down clearly due some unmount issue related the shared info > dir. > >> >> Thanks >> >> >> >> Christoforos Christoforou >> >> Senior Systems Administrator >> >> Global Reach Internet Productions >> >> <http://www.twitter.com/globalreach> Twitter | >> <http://www.facebook.com/globalreach> Facebook | >> <https://www.linkedin.com/company/global‑reach‑internet‑productions> >> LinkedIn >> >> p (515) 996‑0996 | <http://www.globalreach.com/> globalreach.com >> >> _______________________________________________ Manage your subscription: https://lists.clusterlabs.org/mailman/listinfo/users ClusterLabs home: https://www.clusterlabs.org/
