Hello Gang He,
which type of corosync rrp_mode you use? Passive or Active? Did you try test
both? Also, what kernel version you use? I see some SCTP fixes in latest
kernels.
> On 8 Mar 2018, at 08:52, Gang He <g...@suse.com> wrote:
>
> Hello list and David Teigland,
>
> I got a problem under a two rings cluster, the problem can be reproduced with
> the below steps.
> 1) setup a two rings cluster with two nodes.
> e.g.
> clvm1(nodeid 172204569) addr_list eth0 10.67.162.25 eth1 192.168.152.240
> clvm2(nodeid 172204570) addr_list eth0 10.67.162.26 eth1 192.168.152.103
>
> 2) the whole cluster works well, then I put eth0 down on node clvm2, and
> restart pacemaker service on that node.
> ifconfig eth0 down
> rcpacemaker restart
>
> 3) the whole cluster still work well (that means corosync is very smooth to
> switch to the other ring).
> Then, I can mount ocfs2 file system on node clvm2 quickly with the command
> mount /dev/sda /mnt/ocfs2
>
> 4) Next, I do the same mount on node clvm1, the mount command will be hanged
> for about 5 mins, and finally the mount command is done.
> But, if we setup a ocfs2 file system resource in pacemaker,
> the pacemaker resource agent will consider ocfs2 file system resource startup
> failure before this command returns,
> the pacemaker will fence node clvm1.
> This problem is impacting our customer's estimate, since they think the two
> rings can be switched smoothly.
>
> According to this problem, I can see the mount command is hanged with the
> below back trace,
> clvm1:/ # cat /proc/6688/stack
> [<ffffffffa04b8f2d>] new_lockspace+0x92d/0xa70 [dlm]
> [<ffffffffa04b92d9>] dlm_new_lockspace+0x69/0x160 [dlm]
> [<ffffffffa04db758>] user_cluster_connect+0xc8/0x350 [ocfs2_stack_user]
> [<ffffffffa0483872>] ocfs2_cluster_connect+0x192/0x240 [ocfs2_stackglue]
> [<ffffffffa0577efc>] ocfs2_dlm_init+0x31c/0x570 [ocfs2]
> [<ffffffffa05c2983>] ocfs2_fill_super+0xb33/0x1200 [ocfs2]
> [<ffffffff8120e130>] mount_bdev+0x1a0/0x1e0
> [<ffffffff8120ea1a>] mount_fs+0x3a/0x170
> [<ffffffff81228bf2>] vfs_kern_mount+0x62/0x110
> [<ffffffff8122b123>] do_mount+0x213/0xcd0
> [<ffffffff8122bed5>] SyS_mount+0x85/0xd0
> [<ffffffff81614b0a>] entry_SYSCALL_64_fastpath+0x1e/0xb6
> [<ffffffffffffffff>] 0xffffffffffffffff
>
> The root cause is in sctp_connect_to_sock() function in lowcomms.c,
> 1075
> 1076 log_print("connecting to %d", con->nodeid);
> 1077
> 1078 /* Turn off Nagle's algorithm */
> 1079 kernel_setsockopt(sock, SOL_TCP, TCP_NODELAY, (char *)&one,
> 1080 sizeof(one));
> 1081
> 1082 result = sock->ops->connect(sock, (struct sockaddr *)&daddr,
> addr_len,
> 1083 O_NONBLOCK); <<= here, this invoking
> will cost > 5 mins before return ETIMEDOUT(-110).
> 1084 printk(KERN_ERR "sctp_connect_to_sock connect: %d\n", result);
> 1085
> 1086 if (result == -EINPROGRESS)
> 1087 result = 0;
> 1088 if (result == 0)
> 1089 goto out;
>
> Then, I want to know if this problem was found/fixed before?
> it looks DLM can not switch the second ring very quickly, this will impact
> the above application (e.g. CLVM, ocfs2) to create a new lock space before
> it's startup.
>
> Thanks
> Gang
>
>
> _______________________________________________
> Users mailing list: Users@clusterlabs.org
> https://lists.clusterlabs.org/mailman/listinfo/users
>
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org
_______________________________________________
Users mailing list: Users@clusterlabs.org
https://lists.clusterlabs.org/mailman/listinfo/users
Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org
