Hi Muhammad! Could you be a little bit more elaborate on your fencing-setup! I read about you using SBD but I don't see any sbd-fencing-resource. For the case you wanted to use watchdog-fencing with SBD this would require stonith-watchdog-timeout property to be set. But watchdog-fencing relies on quorum (without 2-node trickery) and thus wouldn't work on a 2-node-cluster anyway.
Didn't read through the whole thread - so I might be missing something ... Regards, Klaus On 03/12/2018 12:51 PM, Muhammad Sharfuddin wrote: > Hello Gang, > > as informed, previously cluster was fixed to start the ocfs2 resources by > > a) crm resource start dlm > > b) mount/umount the ocfs2 file system manually. (this step was the fix) > > and then starting the clone group(which include dlm, ocfs2 file > systems) worked fine: > > c) crm resource start base-clone. > > Now I crash the nodes intentionally and then keep only one node > online, again cluster stopped starting the ocfs2 resources. I again > tried to follow your instructions i.e > > i) crm resource start dlm > > then try to mount the ocfs2 file system manually which got hanged this > time(previously manually mounting helped me): > > # cat /proc/3966/stack > [<ffffffffa039f18e>] do_uevent+0x7e/0x200 [dlm] > [<ffffffffa039fe0a>] new_lockspace+0x80a/0xa70 [dlm] > [<ffffffffa03a02d9>] dlm_new_lockspace+0x69/0x160 [dlm] > [<ffffffffa038e758>] user_cluster_connect+0xc8/0x350 [ocfs2_stack_user] > [<ffffffffa03c2872>] ocfs2_cluster_connect+0x192/0x240 [ocfs2_stackglue] > [<ffffffffa045eefc>] ocfs2_dlm_init+0x31c/0x570 [ocfs2] > [<ffffffffa04a9983>] ocfs2_fill_super+0xb33/0x1200 [ocfs2] > [<ffffffff8120e130>] mount_bdev+0x1a0/0x1e0 > [<ffffffff8120ea1a>] mount_fs+0x3a/0x170 > [<ffffffff81228bf2>] vfs_kern_mount+0x62/0x110 > [<ffffffff8122b123>] do_mount+0x213/0xcd0 > [<ffffffff8122bed5>] SyS_mount+0x85/0xd0 > [<ffffffff81614b0a>] entry_SYSCALL_64_fastpath+0x1e/0xb6 > [<ffffffffffffffff>] 0xffffffffffffffff > > I killed the mount.ocfs2 process stop(crm resource stop dlm) the dlm > process, and then try to start(crm resource start dlm) the dlm(which > previously always get started successfully), this time dlm didn't > start and I checked the dlm_controld process > > cat /proc/3754/stack > [<ffffffff8121dc55>] poll_schedule_timeout+0x45/0x60 > [<ffffffff8121f0bc>] do_sys_poll+0x38c/0x4f0 > [<ffffffff8121f2dd>] SyS_poll+0x5d/0xe0 > [<ffffffff81614b0a>] entry_SYSCALL_64_fastpath+0x1e/0xb6 > [<ffffffffffffffff>] 0xffffffffffffffff > > Nutshell: > > 1 - this cluster is configured to run when single node is online > > 2 - this cluster does not start the ocfs2 resources after a crash when > only one node is online. > > -- > Regards, > Muhammad Sharfuddin | +923332144823 | nds.com.pk > > On 3/12/2018 12:41 PM, Gang He wrote: >> >> >>> Hello Gang, >>> >>> to follow your instructions, I started the dlm resource via: >>> >>> crm resource start dlm >>> >>> then mount/unmount the ocfs2 file system manually..(which seems to be >>> the fix of the situation). >>> >>> Now resources are getting started properly on a single node.. I am >>> happy >>> as the issue is fixed, but at the same time I am lost because I have >>> no idea >>> >>> how things get fixed here(merely by mounting/unmounting the ocfs2 file >>> systems) >> >From your description. >> I just wonder the DLM resource does not work normally under that >> situation. >> Yan/Bin, do you have any comments about two-node cluster? which >> configuration settings will affect corosync quorum/DLM ? >> >> >> Thanks >> Gang >> >> >>> >>> -- >>> Regards, >>> Muhammad Sharfuddin >>> >>> On 3/12/2018 10:59 AM, Gang He wrote: >>>> Hello Muhammad, >>>> >>>> Usually, ocfs2 resource startup failure is caused by mount command >>>> timeout >>> (or hanged). >>>> The sample debugging method is, >>>> remove ocfs2 resource from crm first, >>>> then mount this file system manually, see if the mount command will be >>> timeout or hanged. >>>> If this command is hanged, please watch where is mount.ocfs2 >>>> process hanged >>> via "cat /proc/xxx/stack" command. >>>> If the back trace is stopped at DLM kernel module, usually the root >>>> cause is >>> cluster configuration problem. >>>> >>>> Thanks >>>> Gang >>>> >>>> >>>>> On 3/12/2018 7:32 AM, Gang He wrote: >>>>>> Hello Muhammad, >>>>>> >>>>>> I think this problem is not in ocfs2, the cause looks like the >>>>>> cluster >>>>> quorum is missed. >>>>>> For two-node cluster (does not three-node cluster), if one node >>>>>> is offline, >>>>> the quorum will be missed by default. >>>>>> So, you should configure two-node related quorum setting >>>>>> according to the >>>>> pacemaker manual. >>>>>> Then, DLM can work normal, and ocfs2 resource can start up. >>>>> Yes its configured accordingly, no-quorum is set to "ignore". >>>>> >>>>> property cib-bootstrap-options: \ >>>>> have-watchdog=true \ >>>>> stonith-enabled=true \ >>>>> stonith-timeout=80 \ >>>>> startup-fencing=true \ >>>>> no-quorum-policy=ignore >>>>> >>>>>> Thanks >>>>>> Gang >>>>>> >>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> This two node cluster starts resources when both nodes are >>>>>>> online but >>>>>>> does not start the ocfs2 resources >>>>>>> >>>>>>> when one node is offline. e.g if I gracefully stop the cluster >>>>>>> resources >>>>>>> then stop the pacemaker service on >>>>>>> >>>>>>> either node, and try to start the ocfs2 resource on the online >>>>>>> node, it >>>>>>> fails. >>>>>>> >>>>>>> logs: >>>>>>> >>>>>>> pipci001 pengine[17732]: notice: Start dlm:0#011(pipci001) >>>>>>> pengine[17732]: notice: Start p-fssapmnt:0#011(pipci001) >>>>>>> pengine[17732]: notice: Start p-fsusrsap:0#011(pipci001) >>>>>>> pipci001 pengine[17732]: notice: Calculated transition 2, saving >>>>>>> inputs in /var/lib/pacemaker/pengine/pe-input-339.bz2 >>>>>>> pipci001 crmd[17733]: notice: Processing graph 2 >>>>>>> (ref=pe_calc-dc-1520613202-31) derived from >>>>>>> /var/lib/pacemaker/pengine/pe-input-339.bz2 >>>>>>> crmd[17733]: notice: Initiating start operation dlm_start_0 >>>>>>> locally on >>>>>>> pipci001 >>>>>>> lrmd[17730]: notice: executing - rsc:dlm action:start call_id:69 >>>>>>> dlm_controld[19019]: 4575 dlm_controld 4.0.7 started >>>>>>> lrmd[17730]: notice: finished - rsc:dlm action:start call_id:69 >>>>>>> pid:18999 exit-code:0 exec-time:1082ms queue-time:1ms >>>>>>> crmd[17733]: notice: Result of start operation for dlm on >>>>>>> pipci001: 0 (ok) >>>>>>> crmd[17733]: notice: Initiating monitor operation >>>>>>> dlm_monitor_60000 >>>>>>> locally on pipci001 >>>>>>> crmd[17733]: notice: Initiating start operation >>>>>>> p-fssapmnt_start_0 >>>>>>> locally on pipci001 >>>>>>> lrmd[17730]: notice: executing - rsc:p-fssapmnt action:start >>>>>>> call_id:71 >>>>>>> Filesystem(p-fssapmnt)[19052]: INFO: Running start for >>>>>>> /dev/mapper/sapmnt on /sapmnt >>>>>>> kernel: [ 4576.529938] dlm: Using TCP for communications >>>>>>> kernel: [ 4576.530233] dlm: BFA9FF042AA045F4822C2A6A06020EE9: >>>>>>> joining >>>>>>> the lockspace group. >>>>>>> dlm_controld[19019]: 4629 fence work wait for quorum >>>>>>> dlm_controld[19019]: 4634 BFA9FF042AA045F4822C2A6A06020EE9 wait >>>>>>> for quorum >>>>>>> lrmd[17730]: warning: p-fssapmnt_start_0 process (PID 19052) >>>>>>> timed out >>>>>>> kernel: [ 4636.418223] dlm: BFA9FF042AA045F4822C2A6A06020EE9: group >>>>>>> event done -512 0 >>>>>>> kernel: [ 4636.418227] dlm: BFA9FF042AA045F4822C2A6A06020EE9: >>>>>>> group join >>>>>>> failed -512 0 >>>>>>> lrmd[17730]: warning: p-fssapmnt_start_0:19052 - timed out >>>>>>> after 60000ms >>>>>>> lrmd[17730]: notice: finished - rsc:p-fssapmnt action:start >>>>>>> call_id:71 >>>>>>> pid:19052 exit-code:1 exec-time:60002ms queue-time:0ms >>>>>>> kernel: [ 4636.420628] ocfs2: Unmounting device (254,1) on (node 0) >>>>>>> crmd[17733]: error: Result of start operation for p-fssapmnt on >>>>>>> pipci001: Timed Out >>>>>>> crmd[17733]: warning: Action 11 (p-fssapmnt_start_0) on >>>>>>> pipci001 failed >>>>>>> (target: 0 vs. rc: 1): Error >>>>>>> crmd[17733]: notice: Transition aborted by operation >>>>>>> p-fssapmnt_start_0 'modify' on pipci001: Event failed >>>>>>> crmd[17733]: warning: Action 11 (p-fssapmnt_start_0) on >>>>>>> pipci001 failed >>>>>>> (target: 0 vs. rc: 1): Error >>>>>>> crmd[17733]: notice: Transition 2 (Complete=5, Pending=0, >>>>>>> Fired=0, >>>>>>> Skipped=0, Incomplete=6, >>>>>>> Source=/var/lib/pacemaker/pengine/pe-input-339.bz2): Complete >>>>>>> pengine[17732]: notice: Watchdog will be used via SBD if >>>>>>> fencing is >>>>>>> required >>>>>>> pengine[17732]: notice: On loss of CCM Quorum: Ignore >>>>>>> pengine[17732]: warning: Processing failed op start for >>>>>>> p-fssapmnt:0 on >>>>>>> pipci001: unknown error (1) >>>>>>> pengine[17732]: warning: Processing failed op start for >>>>>>> p-fssapmnt:0 on >>>>>>> pipci001: unknown error (1) >>>>>>> pengine[17732]: warning: Forcing base-clone away from pipci001 >>>>>>> after >>>>>>> 1000000 failures (max=2) >>>>>>> pengine[17732]: warning: Forcing base-clone away from pipci001 >>>>>>> after >>>>>>> 1000000 failures (max=2) >>>>>>> pengine[17732]: notice: Stop dlm:0#011(pipci001) >>>>>>> pengine[17732]: notice: Stop p-fssapmnt:0#011(pipci001) >>>>>>> pengine[17732]: notice: Calculated transition 3, saving inputs in >>>>>>> /var/lib/pacemaker/pengine/pe-input-340.bz2 >>>>>>> pengine[17732]: notice: Watchdog will be used via SBD if >>>>>>> fencing is >>>>>>> required >>>>>>> pengine[17732]: notice: On loss of CCM Quorum: Ignore >>>>>>> pengine[17732]: warning: Processing failed op start for >>>>>>> p-fssapmnt:0 on >>>>>>> pipci001: unknown error (1) >>>>>>> pengine[17732]: warning: Processing failed op start for >>>>>>> p-fssapmnt:0 on >>>>>>> pipci001: unknown error (1) >>>>>>> pengine[17732]: warning: Forcing base-clone away from pipci001 >>>>>>> after >>>>>>> 1000000 failures (max=2) >>>>>>> pipci001 pengine[17732]: warning: Forcing base-clone away from >>>>>>> pipci001 >>>>>>> after 1000000 failures (max=2) >>>>>>> pengine[17732]: notice: Stop dlm:0#011(pipci001) >>>>>>> pengine[17732]: notice: Stop p-fssapmnt:0#011(pipci001) >>>>>>> pengine[17732]: notice: Calculated transition 4, saving inputs in >>>>>>> /var/lib/pacemaker/pengine/pe-input-341.bz2 >>>>>>> crmd[17733]: notice: Processing graph 4 >>>>>>> (ref=pe_calc-dc-1520613263-36) >>>>>>> derived from /var/lib/pacemaker/pengine/pe-input-341.bz2 >>>>>>> crmd[17733]: notice: Initiating stop operation p-fssapmnt_stop_0 >>>>>>> locally on pipci001 >>>>>>> lrmd[17730]: notice: executing - rsc:p-fssapmnt action:stop >>>>>>> call_id:72 >>>>>>> Filesystem(p-fssapmnt)[19189]: INFO: Running stop for >>>>>>> /dev/mapper/sapmnt >>>>>>> on /sapmnt >>>>>>> pipci001 lrmd[17730]: notice: finished - rsc:p-fssapmnt >>>>>>> action:stop >>>>>>> call_id:72 pid:19189 exit-code:0 exec-time:83ms queue-time:0ms >>>>>>> pipci001 crmd[17733]: notice: Result of stop operation for >>>>>>> p-fssapmnt >>>>>>> on pipci001: 0 (ok) >>>>>>> crmd[17733]: notice: Initiating stop operation dlm_stop_0 >>>>>>> locally on >>>>>>> pipci001 >>>>>>> pipci001 lrmd[17730]: notice: executing - rsc:dlm action:stop >>>>>>> call_id:74 >>>>>>> pipci001 dlm_controld[19019]: 4636 shutdown ignored, active >>>>>>> lockspaces >>>>>>> >>>>>>> >>>>>>> resource configuration: >>>>>>> >>>>>>> primitive p-fssapmnt Filesystem \ >>>>>>> params device="/dev/mapper/sapmnt" directory="/sapmnt" >>>>>>> fstype=ocfs2 \ >>>>>>> op monitor interval=20 timeout=40 \ >>>>>>> op start timeout=60 interval=0 \ >>>>>>> op stop timeout=60 interval=0 >>>>>>> primitive dlm ocf:pacemaker:controld \ >>>>>>> op monitor interval=60 timeout=60 \ >>>>>>> op start interval=0 timeout=90 \ >>>>>>> op stop interval=0 timeout=100 >>>>>>> clone base-clone base-group \ >>>>>>> meta interleave=true target-role=Started >>>>>>> >>>>>>> cluster properties: >>>>>>> property cib-bootstrap-options: \ >>>>>>> have-watchdog=true \ >>>>>>> stonith-enabled=true \ >>>>>>> stonith-timeout=80 \ >>>>>>> startup-fencing=true \ >>>>>>> >>>>>>> >>>>>>> Software versions: >>>>>>> >>>>>>> kernel version: 4.4.114-94.11-default >>>>>>> pacemaker-1.1.16-4.8.x86_64 >>>>>>> corosync-2.3.6-9.5.1.x86_64 >>>>>>> ocfs2-kmp-default-4.4.114-94.11.3.x86_64 >>>>>>> ocfs2-tools-1.8.5-1.35.x86_64 >>>>>>> dlm-kmp-default-4.4.114-94.11.3.x86_64 >>>>>>> libdlm3-4.0.7-1.28.x86_64 >>>>>>> libdlm-4.0.7-1.28.x86_64 >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Regards, >>>>>>> Muhammad Sharfuddin >>>>>>> >>>>>>> >>>>>>> --- >>>>>>> This email has been checked for viruses by Avast antivirus >>>>>>> software. >>>>>>> https://www.avast.com/antivirus >>>>>>> >>>>>>> _______________________________________________ >>>>>>> Users mailing list: Users@clusterlabs.org >>>>>>> https://lists.clusterlabs.org/mailman/listinfo/users >>>>>>> >>>>>>> Project Home: http://www.clusterlabs.org >>>>>>> Getting started: >>>>>>> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf >>>>>>> Bugs: http://bugs.clusterlabs.org >>>>>> _______________________________________________ >>>>>> Users mailing list: Users@clusterlabs.org >>>>>> https://lists.clusterlabs.org/mailman/listinfo/users >>>>>> >>>>>> Project Home: http://www.clusterlabs.org >>>>>> Getting started: >>>>>> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf >>>>>> Bugs: http://bugs.clusterlabs.org >>>>>> >>>>> -- >>>>> Regards, >>>>> Muhammad Sharfuddin >>>>> >>>>> _______________________________________________ >>>>> Users mailing list: Users@clusterlabs.org >>>>> https://lists.clusterlabs.org/mailman/listinfo/users >>>>> >>>>> Project Home: http://www.clusterlabs.org >>>>> Getting started: >>>>> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf >>>>> Bugs: http://bugs.clusterlabs.org >>>> _______________________________________________ >>>> Users mailing list: Users@clusterlabs.org >>>> https://lists.clusterlabs.org/mailman/listinfo/users >>>> >>>> Project Home: http://www.clusterlabs.org >>>> Getting started: >>>> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf >>>> Bugs: http://bugs.clusterlabs.org >>>> >>> _______________________________________________ >>> Users mailing list: Users@clusterlabs.org >>> https://lists.clusterlabs.org/mailman/listinfo/users >>> >>> Project Home: http://www.clusterlabs.org >>> Getting started: >>> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf >>> Bugs: http://bugs.clusterlabs.org >> _______________________________________________ >> Users mailing list: Users@clusterlabs.org >> https://lists.clusterlabs.org/mailman/listinfo/users >> >> Project Home: http://www.clusterlabs.org >> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf >> Bugs: http://bugs.clusterlabs.org >> > > _______________________________________________ > Users mailing list: Users@clusterlabs.org > https://lists.clusterlabs.org/mailman/listinfo/users > > Project Home: http://www.clusterlabs.org > Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf > Bugs: http://bugs.clusterlabs.org _______________________________________________ Users mailing list: Users@clusterlabs.org https://lists.clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org