Re: [ClusterLabs] corosync - CS_ERR_BAD_HANDLE when multiple nodes are starting up

[Jan Friesse]

ping


echo reply ;)



On 10/14/2015 02:10 PM, Thomas Lamprecht wrote:

Hi,

On 10/08/2015 10:57 AM, Jan Friesse wrote:

Hi,

Thomas Lamprecht napsal(a):

[snip]

Hello,

we are using corosync version needle (2.3.5) for our cluster
filesystem
(pmxcfs).
The situation is the following. First we start up the pmxcfs,
which is
an fuse fs. And if there is an cluster configuration, we start also
corosync.
This allows the filesystem to exist on one node 'cluster's or
forcing it
in an local mode. We use CPG to send our messages to all members,
the filesystem is in the RAM and all fs operations are sent
'over the
wire'.

The problem is now the following:
When we're restarting all (in my test case 3) nodes at the same
time, I
get in 1 from 10 cases only CS_ERR_BAD_HANDLE back when calling


I'm really unsure how to understand what are you doing. You are
restarting all nodes and get CS_ERR_BAD_HANDLE? I mean, if you are
restarting all nodes, which node returns CS_ERR_BAD_HANDLE? Or
are you
restarting just pmxcfs? Or just coorsync?

Clarification, sorry was a bit unspecific. I can see the error
behaviour
in two cases:
1) I restart three physical hosts (= nodes) at the same time, one of
them - normally the last one coming up again - joins successfully the
corosync cluster the filesystem (pmxcfs) notices that, but then
cpg_mcast_joined receives only CS_ERR_BAD_HANDLE errors.


Ok, that is weird. Are you able to reproduce same behavior restarting
pmxcfs? Or really membership change (= restart of node) is needed?
Also are you sure network interface is up when corosync starts?

No, tried quite a few times to restart pmxcfs but that didn't trigger
the problem, yet. But I could trigger it once with only restarting one
node, so restarting all only makes the problem worse but isn't
needed in
the first place.


Ok. So let's expect change of membership come into play.

Do you think you can try to test (in cycle):
- start corosync
- start pmxcfs
- stop pmxcfs
- stop corosync

On one node? Because if problem appears, we will have at least
reproducer.


Hmm, yeah can do that but you have to know that we start pmxcfs
_before_ corosync, as we want to access the data when quorum is lost
or when it's only a one node cluster, and thus corosync is not running.

So the cycle to replicate this problems would be:
- start pmxcfs
- start corosync
- stop corosync
- stop pmxcfs

if I'm not mistaken.



Ok, I know nothing about pmxcfs, but what we want to hunt down is easier 
reproducer (easier than restarting node).







corosync.log of failing node may be interesting.


My nodes hostnames are [one, two, three], this time the came up in the
order they're named.
This time it was on two nodes the first and second node coming up
again.
corosync log seems normal, although I haven't had debug mode enabled,
don't know what difference that makes when no errors shows up in the
normal log.

Oct 07 09:06:36 [1335] two corosync notice [MAIN  ] Corosync Cluster
Engine ('2.3.5'): started and ready to provide service.
Oct 07 09:06:36 [1335] two corosync info[MAIN  ] Corosync built-in
features: augeas systemd pie relro bindnow
Oct 07 09:06:36 [1335] two corosync notice  [TOTEM ] Initializing
transport (UDP/IP Multicast).
Oct 07 09:06:36 [1335] two corosync notice  [TOTEM ] Initializing
transmit/receive security (NSS) crypto: aes256 hash: sha1
Oct 07 09:06:36 [1335] two corosync notice  [TOTEM ] The network
interface [10.10.1.152] is now up.
Oct 07 09:06:36 [1335] two corosync notice  [SERV  ] Service engine
loaded: corosync configuration map access [0]
Oct 07 09:06:36 [1335] two corosync info[QB] server name: cmap
Oct 07 09:06:36 [1335] two corosync notice  [SERV  ] Service engine
loaded: corosync configuration service [1]
Oct 07 09:06:36 [1335] two corosync info[QB] server name: cfg
Oct 07 09:06:36 [1335] two corosync notice  [SERV  ] Service engine
loaded: corosync cluster closed process group service v1.01 [2]
Oct 07 09:06:36 [1335] two corosync info[QB] server name: cpg
Oct 07 09:06:36 [1335] two corosync notice  [SERV  ] Service engine
loaded: corosync profile loading service [4]
Oct 07 09:06:36 [1335] two corosync notice  [QUORUM] Using quorum
provider corosync_votequorum
Oct 07 09:06:36 [1335] two corosync notice  [SERV  ] Service engine
loaded: corosync vote quorum service v1.0 [5]
Oct 07 09:06:36 [1335] two corosync info[QB] server name:
votequorum
Oct 07 09:06:36 [1335] two corosync notice  [SERV  ] Service engine
loaded: corosync cluster quorum service v0.1 [3]
Oct 07 09:06:36 [1335] two corosync info[QB] server name:
quorum
Oct 07 09:06:36 [1335] two corosync notice  [TOTEM ] A new membership
(10.10.1.152:92) was formed. Members joined: 2
Oct 07 09:06:36 [1335] two corosync notice  [QUORUM] Members[1]: 2
Oct 07 09:06:36 [1335] two corosync notice  [MAIN  ] Completed service
synchronization, ready to provide service.


Looks good


Then pmxcfs results in:

Oct 07 09:06:38 two pmxcfs[952]: [status

Re: [ClusterLabs] two node cluster not behaving right

[Jan Friesse]

user.clusterlabs@siimnet.dk napsal(a):

Been new to pacemaker, I’m trying to create my first cluster of two nodes, but 
it seems to behave a little strange.
Following this guide: http://clusterlabs.org/quickstart-redhat-6.html 


but am unable to do f.ex.:

[root@afnA ~]# pcs property set stonith-enabled=false
Error: Unable to update cib
Call cib_replace failed (-62): Timer expired


only thing I find in logs are continued corosync events:

Nov 06 01:30:54 corosync [TOTEM ] Retransmit List: 96 97
Nov 06 01:30:56 corosync [TOTEM ] Retransmit List: 96 97
Nov 06 01:30:57 corosync [TOTEM ] Retransmit List: 96 97
Nov 06 01:30:59 corosync [TOTEM ] Retransmit List: 96 97
Nov 06 01:31:01 corosync [TOTEM ] Retransmit List: 96 97


This means something is blocking successful delivery of packets. Make 
sure to:

- Properly configure firewall (for testing you can disable it completely)
- Make sure you have properly configured multicast. As alternative, you 
can try udpu. Udpu is usually better compatible with switches and for 
two node use case performance is same.


Regards,
  Honza


...

Let me known if more info would help!

TIA

pcs cluster report: 
https://dl.dropboxusercontent.com/u/13225502/pacemaker.report.tar.bz2 



CentOS 6.7 w/:
pacemaker-1.1.12-8.el6.x86_64
pcs-0.9.139-9.el6_7.1.x86_64
ccs-0.16.2-81.el6.x86_64
resource-agents-3.9.5-24.el6.x86_64
cman-3.0.12.1-73.el6.1.x86_64
corosync-1.4.7-2.el6.x86_64

[root@afnB ~]# pacemakerd --features
Pacemaker 1.1.11 (Build: 97629de)
  Supporting v3.0.9:  generated-manpages agent-manpages ascii-docs ncurses 
libqb-logging libqb-ipc nagios  corosync-plugin cman acls

[root@afnB ~]# corosync-quorumtool -l
Nodeid Name
1   afnA.mxi.tdcfoo
2   afnB.mxi.tdcfoo
[root@afnB ~]# corosync-quorumtool -s
Version:  1.4.7
Nodes:2
Ring ID:  8
Quorum type:  quorum_cman
Quorate:  Yes
[root@afnB ~]# pcs status
Cluster name: afn-cluster
WARNING: no stonith devices and stonith-enabled is not false
Last updated: Fri Nov  6 01:35:30 2015
Last change: Fri Nov  6 01:29:37 2015
Stack: cman
Current DC: afna - partition with quorum
Version: 1.1.11-97629de
2 Nodes configured
0 Resources configured


Online: [ afna afnb ]


Full list of resources:



[root@afnB ~]# cat /etc/cluster/cluster.conf

   
   
 
   
 
   
 
   
 
 
   
 
   
 
   
 
   
   
   
 
   
   
 
 
   



[root@afnB ~]# grep -v '#' /etc/corosync/corosync.conf
compatibility: whitetank

totem {
 version: 2

 secauth: off
 threads: 0

 window_size: 150

 interface {
 ringnumber: 0
 bindnetaddr: 10.45.69.0
 mcastaddr: 239.255.15.1
 mcastport: 5405
 ttl: 1
 }
}

logging {
 fileline: off
 to_stderr: no
 to_logfile: yes
 logfile: /var/log/cluster/corosync.log
 to_syslog: yes
 debug: off
 timestamp: on
 logger_subsys {
 subsys: AMF
 debug: off
 }
}




___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org




___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] two node cluster not behaving right

[Jan Friesse]

Steffen,

user.clusterlabs@siimnet.dk napsal(a):



On 6. nov. 2015, at 08.42, Jan Friesse <jfrie...@redhat.com> wrote:

This means something is blocking successful delivery of packets. Make sure to:
- Properly configure firewall (for testing you can disable it completely)
- Make sure you have properly configured multicast. As alternative, you can try 
udpu. Udpu is usually better compatible with switches and for two node use case 
performance is same.

Seem unicast fixed the issue ;)


Good.

Multicast is always source of problems. Also if you decide one day to 
scale more and try multicast, you can try 
https://alteeve.ca/w/Fencing_KVM_Virtual_Servers#Fedora_18_Host.3B_Bridge_Configuration_Issue


Regards,
  Honza




This was done by changing cman configuration to udpu transport in cluster.conf 
like this:

http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org




___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] continous QUORUM messages in a 3-node cluster

[Jan Friesse]

Illia,



Hi,

We are using a 3-node pacemaker/corosync cluster on CentOs 7.
We have several identical setups in our QA/DEV orgs, and a couple of them 
continuously spew the following messages on all 3 nodes:

Oct  8 17:18:20 42-hw-rig4-L3-2 corosync[15105]: [TOTEM ] A new membership 
(10.1.13.134:1553572) was formed. Members
Oct  8 17:18:20 42-hw-rig4-L3-2 corosync[15105]: [QUORUM] Members[3]: 2 1 3
Oct  8 17:18:20 42-hw-rig4-L3-2 corosync[15105]: [MAIN  ] Completed service 
synchronization, ready to provide service.
Oct  8 17:18:22 42-hw-rig4-L3-2 corosync[15105]: [TOTEM ] A new membership 
(10.1.13.134:1553576) was formed. Members
Oct  8 17:18:22 42-hw-rig4-L3-2 corosync[15105]: [QUORUM] Members[3]: 2 1 3
Oct  8 17:18:22 42-hw-rig4-L3-2 corosync[15105]: [MAIN  ] Completed service 
synchronization, ready to provide service.
Oct  8 17:18:24 42-hw-rig4-L3-2 corosync[15105]: [TOTEM ] A new membership 
(10.1.13.134:1553580) was formed. Members
Oct  8 17:18:24 42-hw-rig4-L3-2 corosync[15105]: [QUORUM] Members[3]: 2 1 3
Oct  8 17:18:24 42-hw-rig4-L3-2 corosync[15105]: [MAIN  ] Completed service 
synchronization, ready to provide service.
Oct  8 17:18:26 42-hw-rig4-L3-2 corosync[15105]: [TOTEM ] A new membership 
(10.1.13.134:1553584) was formed. Members
Oct  8 17:18:26 42-hw-rig4-L3-2 corosync[15105]: [QUORUM] Members[3]: 2 1 3
Oct  8 17:18:26 42-hw-rig4-L3-2 corosync[15105]: [MAIN  ] Completed service 
synchronization, ready to provide service.

The cluster seems to be generally happy:

root@42-hw-rig4-L3-2 ~]# pcs cluster status
Cluster Status:
  Last updated: Thu Oct  8 17:24:02 2015
  Last change: Thu Oct  8 16:46:57 2015
  Stack: corosync
  Current DC: dq-ceph9.clearsky-data.net (3) - partition with quorum
  Version: 1.1.12-a14efad
  3 Nodes configured
  17 Resources configured

PCSD Status:
   42-hw-back-1.clearsky-data.net: Online
   41-hw-back-1.clearsky-data.net: Online
   dq-ceph9.clearsky-data.net: Online

The corosync config is:

totem {
version: 2
secauth: off
cluster_name: L3_cluster
transport: udpu
}

nodelist {
   node {
 ring0_addr: 42-hw-back-1.clearsky-data.net
 nodeid: 1
}
   node {
 ring0_addr: 41-hw-back-1.clearsky-data.net
 nodeid: 2
}
   node {
 ring0_addr: dq-ceph9.clearsky-data.net
 nodeid: 3
}
}

quorum {
provider: corosync_votequorum

}

logging {
to_syslog: yes
debug : off
}

What do these messages mean, and how can we stop them?

Any help would be very appreciated.

Ilia Sokolinski

PS

I have tried to enable corosync debug and got the following logs:

Oct  8 16:18:47 42-hw-rig4-L3-2 corosync[15105]: [CFG   ] Config reload 
requested by node 1
Oct  8 16:18:47 42-hw-rig4-L3-2 corosync[15105]: [QB] HUP conn 
(15105-46913-25)
Oct  8 16:18:47 42-hw-rig4-L3-2 corosync[15105]: [QB] 
qb_ipcs_disconnect(15105-46913-25) state:2
Oct  8 16:18:47 42-hw-rig4-L3-2 corosync[15105]: [QB] epoll_ctl(del): Bad 
file descriptor (9)
Oct  8 16:18:47 42-hw-rig4-L3-2 corosync[15105]: [MAIN  ] 
cs_ipcs_connection_closed()
Oct  8 16:18:47 42-hw-rig4-L3-2 corosync[15105]: [MAIN  ] 
cs_ipcs_connection_destroyed()
Oct  8 16:18:47 42-hw-rig4-L3-2 corosync[15105]: [QB] Free'ing ringbuffer: 
/dev/shm/qb-cfg-response-15105-46913-25-header
Oct  8 16:18:47 42-hw-rig4-L3-2 corosync[15105]: [QB] Free'ing ringbuffer: 
/dev/shm/qb-cfg-event-15105-46913-25-header
Oct  8 16:18:47 42-hw-rig4-L3-2 corosync[15105]: [QB] Free'ing ringbuffer: 
/dev/shm/qb-cfg-request-15105-46913-25-header
Oct  8 16:18:47 42-hw-rig4-L3-2 corosync[15105]: [VOTEQ ] got nodeinfo message 
from cluster node 3
Oct  8 16:18:47 42-hw-rig4-L3-2 corosync[15105]: [VOTEQ ] nodeinfo message[3]: 
votes: 1, expected: 3 flags: 1
Oct  8 16:18:47 42-hw-rig4-L3-2 corosync[15105]: [VOTEQ ] flags: quorate: Yes 
Leaving: No WFA Status: No First: No Qdevice: No QdeviceAlive: No 
QdeviceCastVote: No QdeviceMasterWins: No
Oct  8 16:18:47 42-hw-rig4-L3-2 corosync[15105]: [VOTEQ ] got nodeinfo message 
from cluster node 3
Oct  8 16:18:47 42-hw-rig4-L3-2 corosync[15105]: [VOTEQ ] nodeinfo message[0]: 
votes: 0, expected: 0 flags: 0
Oct  8 16:18:47 42-hw-rig4-L3-2 corosync[15105]: [VOTEQ ] got nodeinfo message 
from cluster node 2
Oct  8 16:18:47 42-hw-rig4-L3-2 corosync[15105]: [VOTEQ ] nodeinfo message[2]: 
votes: 1, expected: 3 flags: 1
Oct  8 16:18:47 42-hw-rig4-L3-2 corosync[15105]: [VOTEQ ] flags: quorate: Yes 
Leaving: No WFA Status: No First: No Qdevice: No QdeviceAlive: No 
QdeviceCastVote: No QdeviceMasterWins: No
Oct  8 16:18:47 42-hw-rig4-L3-2 corosync[15105]: [VOTEQ ] got nodeinfo message 
from cluster node 2
Oct  8 16:18:47 42-hw-rig4-L3-2 corosync[15105]: [VOTEQ ] nodeinfo message[0]: 
votes: 0, expected: 0 flags: 0
Oct  8 16:18:47 42-hw-rig4-L3-2 corosync[15105]: [VOTEQ ] got nodeinfo message 
from cluster node 1
Oct  8 16:18:47 42-hw-rig4-L3-2 corosync[15105]: [VOTEQ ] nodeinfo message[1]: 
votes: 1, expected: 3 flags: 1
Oct  8 16:18:47 42-hw-rig4-L3-2 

Re: [ClusterLabs] corosync - CS_ERR_BAD_HANDLE when multiple nodes are starting up

[Jan Friesse]

Thomas,

Thomas Lamprecht napsal(a):

Hi,

thanks for the response!
I added some information and clarification below.

On 10/01/2015 09:23 AM, Jan Friesse wrote:

Hi,

Thomas Lamprecht napsal(a):

Hello,

we are using corosync version needle (2.3.5) for our cluster filesystem
(pmxcfs).
The situation is the following. First we start up the pmxcfs, which is
an fuse fs. And if there is an cluster configuration, we start also
corosync.
This allows the filesystem to exist on one node 'cluster's or forcing it
in an local mode. We use CPG to send our messages to all members,
the filesystem is in the RAM and all fs operations are sent 'over the
wire'.

The problem is now the following:
When we're restarting all (in my test case 3) nodes at the same time, I
get in 1 from 10 cases only CS_ERR_BAD_HANDLE back when calling


I'm really unsure how to understand what are you doing. You are
restarting all nodes and get CS_ERR_BAD_HANDLE? I mean, if you are
restarting all nodes, which node returns CS_ERR_BAD_HANDLE? Or are you
restarting just pmxcfs? Or just coorsync?

Clarification, sorry was a bit unspecific. I can see the error behaviour
in two cases:
1) I restart three physical hosts (= nodes) at the same time, one of
them - normally the last one coming up again - joins successfully the
corosync cluster the filesystem (pmxcfs) notices that, but then
cpg_mcast_joined receives only CS_ERR_BAD_HANDLE errors.


Ok, that is weird. Are you able to reproduce same behavior restarting 
pmxcfs? Or really membership change (= restart of node) is needed? Also 
are you sure network interface is up when corosync starts?


corosync.log of failing node may be interesting.




2) I disconnect the network interface on which corosync runs, and
reconnect it a bit later. This triggers the same as above, but also not
every time.


Just to make sure. Don't do ifdown. Corosync reacts to ifdown pretty 
badly. Also NetworkManager does ifdown on cable unplug if not configured 
in server mode. If you want to test network split, ether use iptables 
(make sure to block all traffic needed by corosync, so if you are using 
multicast make sure to block both unicast and multicast packets on input 
and output - 
https://github.com/jfriesse/csts/blob/master/tests/inc/fw.sh), or use 
blocking on switch.




Currently I'm trying to get an somewhat reproduce able test and try it
also on bigger setups and other possible causes, need to do a bit more
home work here and report back later.


Actually, smaller clusters are better for debugging, but yes, larger 
setup may show problem faster.





cpg_mcast_joined to send out the data, but only one node.
corosyn-quorumtool shows that we have quorum, and the logs are also
showing a healthy connect to the corosync cluster. The failing handle is
initialized once at the initialization of our filesystem. Should it be
reinitialized on every reconnect?


Again, I'm unsure what you mean by reconnect. On Corosync shudown you
have to reconnect (I believe this is not the case because you are
getting error only with 10% probability).

Yes, we reconnect to Corosync, and it's not a corosync shutdown, the
whole host reboots or the network interfaces goes down and then a bit
later up again. The probability is just an estimation but the main
problem is that I can not reproduce it all the time.



Restarting the filesystem solves this problem, the strange thing is that
isn't clearly reproduce-able and often works just fine.

Are there some known problems or steps we should look for?


Hard to tell but generally:
- Make sure cpg_init really returns CS_OK. If not, returned handle is
invalid
- Make sure there is no memory corruption and handle is really valid
(valgrind may be helpful).

cpg_init checks are in place and should be OK.
Yes, will use Valgrind, but one questions ahead:

Can the handle get lost somehow? Is there a need to reinitialize the cpg
with cpg_initialize/cpg_model_initialize after we left and later
rejoined the cluster?


I'm still unsure what you mean after we left and later rejoined. As long 
as corosync is running client application "don't need to care about" 
membership changes. It's corosync problem. So if network split happens, 
you don't have to call cpg_initialize. Only place where cpg_initalize is 
needed is initial connection and reconnection after corosync main 
process exit.



Regards,
  Honza



Regards,
  Honza




___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org



___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clust

Re: [ClusterLabs] "0 Nodes configured" in crm_mon

[Jan Friesse]

Stanislav,

> Hi Ken,


thanks for the info, I will try 2.3.4 or maybe even 2.3.3 like in
original compilation guide.


also maybe you are hitting same problem as was discussed on list in 
thread (Corosync: 100% cpu (corosync 2.3.5, libqb 0.17.1, pacemaker 1.1.13)


Solution is ether apply 7f56f58 on libqb 0.17.1 (see 
https://github.com/ClusterLabs/libqb/issues/139 and 
https://github.com/ClusterLabs/libqb/pull/141), or upgrade to 0.17.2.


Regards,
  Honza



Best,
Stan

2015-08-28 19:04 GMT+02:00 Ken Gaillot :

On 08/28/2015 10:59 AM, Stanislav Kopp wrote:

Hi Andrew,

yeah, sorry about that, I need good glasses, it's working now. However
(and It maybe slight off-topic of my initial mail) the cluster is
reeeaally slow, the nodes appear online after 1-2 min after corosync
and pacemaker start and CPU is often at 100% for corosync process,
resource migration takes many seconds too (no such problem with same
IPaddr2 resource on Debian's Wheezy or Ubuntu's 14.04 pacemakers)
Once again, I dont really see errors in corosync.log
http://pastebin.com/zLwQJaqu
besides maybe

crmd:  warning: do_log: FSA: Input I_DC_TIMEOUT from
crm_timer_popped() received in state S_PENDING

and many CPU warnings.

Best,
Stan


I see you're using corosync 2.3.5. I played a little bit with a test
cluster on Fedora 22 (which has 2.3.5) and found it to be much slower
than clusters running on top of 2.3.4. I haven't had time to investigate
it yet, so I can't say whether that's actually to blame, but you might
try 2.3.4 and see if that changes anything.




2015-08-28 6:09 GMT+02:00 Andrew Beekhof :



On 25 Aug 2015, at 1:45 am, Stanislav Kopp  wrote:

Hi all,

I'm trying to run corosync2 + pacemaker setup on Debian Jessie (only
for testing purpose), I've successfully compiled all components using
this guide: http://clusterlabs.org/wiki/Compiling_on_Debian

Unfortunately, if I run "crm_mon" I don't see any nodes.

###
Last updated: Mon Aug 24 17:36:00 2015
Last change: Mon Aug 24 17:17:42 2015
Current DC: NONE
0 Nodes configured
0 Resources configured


I don't see any errors in corosync log either: http://pastebin.com/bJX66B9e


really?

Aug 24 17:16:10 [1723] pm1   crmd:error: cluster_connect_quorum:
Corosync quorum is not configured

Looks like you forgot to uncomment:

#provider: corosync_votequorum



This is my corosync.conf

###

# Please read the corosync.conf.5 manual page
totem {
version: 2

crypto_cipher: none
crypto_hash: none

interface {
ringnumber: 0
bindnetaddr: 192.168.122.0
mcastport: 5405
ttl: 1
}
transport: udpu
}

logging {
fileline: off
to_logfile: yes
to_syslog: no
logfile: /var/log/cluster/corosync.log
debug: off
timestamp: on
logger_subsys {
subsys: QUORUM
debug: off
}
}

nodelist {
node {
ring0_addr: 192.168.122.172
#nodeid: 1
}

node {
ring0_addr: 192.168.122.113
#nodeid: 2
}
}

quorum {
# Enable and configure quorum subsystem (default: off)
# see also corosync.conf.5 and votequorum.5
#provider: corosync_votequorum
}



used components:

pacemaker: 1.1.12
corosync: 2.3.5
libqb: 0.17.1


Did I miss something?

Thanks!
Stan



___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org


___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org




___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] EL6, cman, rrp, unicast and iptables

[Jan Friesse]

Digimer napsal(a):

Hi all,

   Starting a new thread from the "Clustered LVM with iptables issue"
thread...

   I've decided to review how I do networking entirely in my cluster. I
make zero claims to being great at networks, so I would love some feedback.

   I've got three active/passive bonded interfaces; Back-Channel, Storage
and Internet-Facing networks. The IFN is "off limits" to the cluster as
it is dedicated to hosted server traffic only.

   So before, I uses only the BCN for cluster traffic for cman/corosync
multicast traffic, no rrp. A couple months ago, I had a cluster
partition when VM live migration (also on the BCN) congested the
network. So I decided to enable RRP using the SN as backup, which has
been marginally successful.

   Now, I want to switch to unicast (

Don't do ifdown. Corosync reacts on ifdown very badly (long time known 
issue, also it's one of the reason for knet in future version).


Also rrp active is not so well tested as passive, so give a try to passive.

Honza



Thanks!




___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] corosync - CS_ERR_BAD_HANDLE when multiple nodes are starting up

[Jan Friesse]

Hi,

Thomas Lamprecht napsal(a):

Hello,

we are using corosync version needle (2.3.5) for our cluster filesystem
(pmxcfs).
The situation is the following. First we start up the pmxcfs, which is
an fuse fs. And if there is an cluster configuration, we start also
corosync.
This allows the filesystem to exist on one node 'cluster's or forcing it
in an local mode. We use CPG to send our messages to all members,
the filesystem is in the RAM and all fs operations are sent 'over the
wire'.

The problem is now the following:
When we're restarting all (in my test case 3) nodes at the same time, I
get in 1 from 10 cases only CS_ERR_BAD_HANDLE back when calling


I'm really unsure how to understand what are you doing. You are 
restarting all nodes and get CS_ERR_BAD_HANDLE? I mean, if you are 
restarting all nodes, which node returns CS_ERR_BAD_HANDLE? Or are you 
restarting just pmxcfs? Or just coorsync?



cpg_mcast_joined to send out the data, but only one node.
corosyn-quorumtool shows that we have quorum, and the logs are also
showing a healthy connect to the corosync cluster. The failing handle is
initialized once at the initialization of our filesystem. Should it be
reinitialized on every reconnect?


Again, I'm unsure what you mean by reconnect. On Corosync shudown you 
have to reconnect (I believe this is not the case because you are 
getting error only with 10% probability).



Restarting the filesystem solves this problem, the strange thing is that
isn't clearly reproduce-able and often works just fine.

Are there some known problems or steps we should look for?


Hard to tell but generally:
- Make sure cpg_init really returns CS_OK. If not, returned handle is 
invalid
- Make sure there is no memory corruption and handle is really valid 
(valgrind may be helpful).


Regards,
  Honza




___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org



___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Anyone successfully install Pacemaker/Corosync on Freebsd?

[Jan Friesse]

Christine Caulfield napsal(a):

On 21/12/15 16:12, Ken Gaillot wrote:

On 12/19/2015 04:56 PM, mike wrote:

Hi All,

just curious if anyone has had any luck at one point installing
Pacemaker and Corosync on FreeBSD. I have to install from source of
course and I've run into an issue when running ./configure while trying
to install Corosync. The process craps out at nss with this error:


FYI, Ruben Kerkhof has done some recent work to get the FreeBSD build
working. It will go into the next 1.1.14 release candidate. In the
meantime, make sure you have the very latest code from upstream's 1.1
branch.



I also strongly recommend using the latest (from git) version of libqb
has it has some FreeBSD bugs fixed in it. We plan to do a proper release
of this in the new year.


Same applies also for corosync. Use git and it should work (even with 
clang).


Honza



Chrissie


checking for nss... configure: error: in `/root/heartbeat/corosync-2.3.3':
configure: error: The pkg-config script could not be found or is too
old. Make sure it
is in your PATH or set the PKG_CONFIG environment variable to the full
path to pkg-config.​
Alternatively, you may set the environment variables nss_CFLAGS
and nss_LIBS to avoid the need to call pkg-config.
See the pkg-config man page for more details.

I've looked unsuccessfully for a package called pkg-config and nss
appears to be installed as you can see from this output:

root@wellesley:~/heartbeat/corosync-2.3.3 # pkg install nss
Updating FreeBSD repository catalogue...
FreeBSD repository is up-to-date.
All repositories are up-to-date.
Checking integrity... done (0 conflicting)
The most recent version of packages are already installed

Anyway - just looking for any suggestions. Hoping that perhaps someone
has successfully done this.

thanks in advance
-mgb



___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org




___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org




___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Few questions regarding corosync authkey

[Jan Friesse]

Hi,

Would like to understand how secure is the corosync authkey.
As the authkey is a binary file, how is the private key saved inside the
authkey?


Corosync uses symmetric encryption, so there is no public certificate. 
authkey = private key



What safeguard mechanisms are in place if the private key is compromised?


No safeguard mechanisms. Compromised authkey = problem.


For e.g I don't think it uses any temporary session key which refreshes
periodically.


Exactly


Is it possible to dynamically update the key without causing any outage?


Nope

Regards,
  Honza



-Thanks
Nikhil



___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org




___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] [corosync] Virtual Synchrony Property guarantees in case of network partition

[Jan Friesse]

Hi,


Hello,

Virtual Synchrony Property - messages are delivered in agreed order and
configuration changes are delivered in agreed order relative to message.

What happen to this property when network is partitioned the cluster into
two. Consider following scenario (which I took from one of the
previous query by Andrei Elkin):

* N1, N2 and N3 are in state sync with m(k-1) messages are delivered.


What exactly you mean by "state sync"?


* N1 sends m(k) and just now network partition N1 node from N2 and N3.

Does CPG_TYPE_AGREED guarantee that virtual synchrony is held?


Yes it does (actually higher level of VS called EVS)



When property is held, configuration change message C1 is guaranteed to
delivered before m(k) to N1.
N1 will see: m(k-1) C1 m(k)
N2 and N3 will see: m(k-1) C1

But if this property is violated:
N1 will see: m(k-1) m(k) C1
N2 and N3 will see: m(k-1) C1

Violation will screw any user application running on the cluster.

Could someone please explain what is the behavior of Corosync in this
scenario with CPG_TYPE_AGREED ordering.


For description how exactly totem synchronization works take a look to 
http://corosync.github.com/corosync/doc/DAAgarwal.thesis.ps.gz


After totem is synchronized, there is another level of synchronization 
of services (not described in above doc). All services synchronize in 
very similar way, so you can take a look to CPG as example. Basically 
only state held by CPG is connected clients. So every node sends it's 
connected clients list to every other node. If sync is aborted (change 
of membership), it's restarted. These sync messages has priority over 
user messages (actually it's not possible to send messages during sync). 
User app can be sure that message was delivered only after it gets it's 
own message. Also app gets configuration change message so it knows, who 
got the message.


Regards,
  Honza



Regards,
Satish



___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org




___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] [corosync] Virtual Synchrony Property guarantees in case of network partition

[Jan Friesse]

But C1 is *guaranteed *to deliver *before *m(k)? No case where C1 is


Yes


delivered after m(k)?


Nope.




Regards,
Satish

On Mon, Jun 6, 2016 at 8:10 PM, Jan Friesse <jfrie...@redhat.com> wrote:


satish kumar napsal(a):

Hello honza, thanks for the response !


With state sync, I simply mean that 'k-1' messages were delivered to N1,
N2
and N3 and they have applied these messages to change their program state.
N1.state = apply(m(k-1);
N2.state = apply(m(k-1);
N3.state = apply(m(k-1);

The document you shared cleared many doubts. However I still need one
clarification.

According to the document:
"The configuration change messages warn the application that a membership
change has occurred, so that the application program can take appropriate
action based on the membership change. Extended virtual synchrony
guarantees a consistent order of messages delivery across a partition,
which is essential if the application program are to be able to reconcile
their states following repair of a failed processor or reemerging of the
partitioned network."

I just want to know that this property is not something related to
CPG_TYPE_SAFE, which is still not implemented.
Please consider this scenario:
0. N1, N2 and N3 has received the message m(k-1).
1. N1 mcast(CPG_TYPE_AGREED) m(k) message.
2. As it is not CPG_TYPE_SAFE, m(k) delievered to N1 but was not yet
delivered to N2 and N3.
3. Network partition separate N1 from N2 and N3. N2 and N3 can never see
m(k).
4. Configuration change message is now delivered to N1, N2 and N3.

Here, N1 will change its state to N1.state = apply(m(k), thinking all in
the current configuration has received the message.

According to your reply it looks like N1 will not receive m(k). So this is
what each node will see:
N1 will see: m(k-1) -> C1 (config change)
N2 will see: m(k-1) -> C1 (config change)
N3 will see: m(k-1) -> C1 (config change)



For N2 and N3, it's not same C1. So let's call it C2. Because C1 for N1 is
(N2 and N3 left) and C2 for N2 and N3 is (N1 left).




Message m(k) will be discarded, and will not be delivered to N1 even if it
was sent by N1 before the network partition.



No. m(k) will be delivered to app running on N1. So N1 will see m(k-1),
C1, m(k). So application exactly knows which node got message m(k).

Regards,
   Honza




This is the expected behavior with CPG_TYPE_AGREED?

Regards,
Satish


On Mon, Jun 6, 2016 at 4:15 PM, Jan Friesse <jfrie...@redhat.com> wrote:

Hi,


Hello,



Virtual Synchrony Property - messages are delivered in agreed order and
configuration changes are delivered in agreed order relative to message.

What happen to this property when network is partitioned the cluster
into
two. Consider following scenario (which I took from one of the
previous query by Andrei Elkin):

* N1, N2 and N3 are in state sync with m(k-1) messages are delivered.



What exactly you mean by "state sync"?

* N1 sends m(k) and just now network partition N1 node from N2 and N3.



Does CPG_TYPE_AGREED guarantee that virtual synchrony is held?



Yes it does (actually higher level of VS called EVS)


When property is held, configuration change message C1 is guaranteed to

delivered before m(k) to N1.
N1 will see: m(k-1) C1 m(k)
N2 and N3 will see: m(k-1) C1

But if this property is violated:
N1 will see: m(k-1) m(k) C1
N2 and N3 will see: m(k-1) C1

Violation will screw any user application running on the cluster.

Could someone please explain what is the behavior of Corosync in this
scenario with CPG_TYPE_AGREED ordering.



For description how exactly totem synchronization works take a look to
http://corosync.github.com/corosync/doc/DAAgarwal.thesis.ps.gz

After totem is synchronized, there is another level of synchronization of
services (not described in above doc). All services synchronize in very
similar way, so you can take a look to CPG as example. Basically only
state
held by CPG is connected clients. So every node sends it's connected
clients list to every other node. If sync is aborted (change of
membership), it's restarted. These sync messages has priority over user
messages (actually it's not possible to send messages during sync). User
app can be sure that message was delivered only after it gets it's own
message. Also app gets configuration change message so it knows, who got
the message.

Regards,
Honza


Regards,

Satish



___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started:
http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org




___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scra

Re: [ClusterLabs] [corosync][Problem] Very long "pause detect ... " was detected.

[Jan Friesse]

Hideo,


Hi All,

Our user constituted a cluster in corosync and Pacemaker in the next 
environment.
The cluster constituted it among guests.

* Host/Guest : RHEL6.6 - kernel : 2.6.32-504.el6.x86_64
* libqb 0.17.1
* corosync 2.3.4
* Pacemaker 1.1.12

The cluster worked well.
When a user stopped an active guest, the next log was output in standby guests 
repeatedly.


What exactly you mean by "active guest" and "standby guests"?



May xx xx:25:53 standby-guest corosync[6311]:  [TOTEM ] Process pause detected 
for 5515870 ms, flushing membership messages.
May xx xx:25:53 standby-guest corosync[6311]:  [TOTEM ] Process pause detected 
for 5515920 ms, flushing membership messages.
May xx xx:25:53 standby-guest corosync[6311]:  [TOTEM ] Process pause detected 
for 5515971 ms, flushing membership messages.
May xx xx:25:53 standby-guest corosync[6311]:  [TOTEM ] Process pause detected 
for 5516021 ms, flushing membership messages.
May xx xx:25:53 standby-guest corosync[6311]:  [TOTEM ] Process pause detected 
for 5516071 ms, flushing membership messages.
May xx xx:25:53 standby-guest corosync[6311]:  [TOTEM ] Process pause detected 
for 5516121 ms, flushing membership messages.
May xx xx:25:53 standby-guest corosync[6311]:  [TOTEM ] Process pause detected 
for 5516171 ms, flushing membership messages.
May xx xx:25:53 standby-guest corosync[6311]:  [TOTEM ] Process pause detected 
for 5516221 ms, flushing membership messages.
May xx xx:25:53 standby-guest corosync[6311]:  [TOTEM ] Process pause detected 
for 5516271 ms, flushing membership messages.
May xx xx:25:53 standby-guest corosync[6311]:  [TOTEM ] Process pause detected 
for 5516322 ms, flushing membership messages.
May xx xx:25:53 standby-guest corosync[6311]:  [TOTEM ] Process pause detected 
for 5516372 ms, flushing membership messages.
(snip)
May xx xx:26:03 standby-guest corosync[6311]:  [TOTEM ] Process pause detected 
for 5526172 ms, flushing membership messages.
May xx xx:26:03 standby-guest corosync[6311]:  [MAIN  ] Totem is unable to form 
a cluster because of an operating system or network fault. The most common 
cause of this message is that the local firewall is configured improperly.
May xx xx:26:03 standby-guest corosync[6311]:  [TOTEM ] Process pause detected 
for 5526222 ms, flushing membership messages.
(snip)



This is weird. Not because of enormous pause length but because corosync 
has a "scheduler pause" detector which warns before "Process pause 
detected ..." error is logged.



As a result, the standby guest failed in the construction of the independent 
cluster.

It is recorded in log as if a timer stopped for 91 minutes.
It is abnormal length for 91 minutes.

Did you see a similar problem?


Never



Possibly I think whether it is libqb or Kernel or some kind of problems.


What virtualization technology are you using? KVM?


* I suspect that the set of the timer failed in reset_pause_timeout().


You can try to put asserts into this function, but there is really not 
too much reasons why it should fail (ether malloc returns NULL or some 
nasty memory corruption).


Regards,
  Honza



Best Regards,
Hideo Yamauchi.


___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org




___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Corosync with passive rrp, udpu - Unable to reset after "Marking ringid 1 interface 127.0.0.1 FAULTY"

[Jan Friesse]

Martin,



Hi Jan

Thanks for your super quick response !

We do not use a Network Manager - it's all static on these Ubuntu 14.04 nodes
(/etc/network/interfaces).


Good



I do not think we did an ifdown on the network interface manually. However, the
IP-Addresses are assigned to bond0 and bond1 - we use 4x physical network
interfaces with 2x bond'ed into a public (bond1) and 2x bond'ed into a private
network (bond0).

Could this have anything to do with it ?


I don't think so. Problem really happens only when corosync is 
configured to ip address which disappears so it has to rebind to 
127.0.0.1. You would then see "The network interface is down" in the 
logs. Try to find that msg, if it's really problem I was referring about.


Regards,
  Honza


Regards,
Martin Schlegel

___

 From /etc/network/interfaces, i.e.

auto bond0
iface bond0 inet static
#pre-up /sbin/ethtool -s bond0 speed 1000 duplex full autoneg on
post-up ifenslave bond0 eth0 eth2
pre-down ifenslave -d bond0 eth0 eth2
bond-slaves none
bond-mode 4
bond-lacp-rate fast
bond-miimon 100
bond-downdelay 0
bond-updelay 0
bond-xmit_hash_policy 1
address  [...]


Jan Friesse <jfrie...@redhat.com> hat am 16. Juni 2016 um 17:55 geschrieben:

Martin Schlegel napsal(a):


Hello everyone,

we run a 3 node Pacemaker (1.1.14) / Corosync (2.3.5) cluster for a couple
of
months successfully and we have started seeing a faulty ring with unexpected
  127.0.0.1 binding that we cannot reset via "corosync-cfgtool -r".


This is problem. Bind to 127.0.0.1 = ifdown happened = problem and with
RRP it means BIG problem.


We have had this once before and only restarting Corosync (and everything
else)
on the node showing the unexpected 127.0.0.1 binding made the problem go
away.
However, in production we obviously would like to avoid this if possible.


Just don't do ifdown. Never. If you are using NetworkManager (which does
ifdown by default if cable is disconnected), use something like
NetworkManager-config-server package (it's just change of configuration
so you can adopt it to whatever distribution you are using).

Regards,
  Honza


So from the following description - how can I troubleshoot this issue and/or
does anybody have a good idea what might be happening here ?

We run 2x passive rrp rings across different IP-subnets via udpu and we get
the
following output (all IPs obfuscated) - please notice the unexpected
interface
binding 127.0.0.1 for host pg2.

If we reset via "corosync-cfgtool -r" on each node heartbeat ring id 1
briefly
shows "no faults" but goes back to "FAULTY" seconds later.

Regards,
Martin Schlegel
_

root@pg1:~# corosync-cfgtool -s
Printing ring status.
Local node ID 1
RING ID 0
  id = A.B.C1.5
  status = ring 0 active with no faults
RING ID 1
  id = D.E.F1.170
  status = Marking ringid 1 interface D.E.F1.170 FAULTY

root@pg2:~# corosync-cfgtool -s
Printing ring status.
Local node ID 2
RING ID 0
  id = A.B.C2.88
  status = ring 0 active with no faults
RING ID 1
  id = 127.0.0.1
  status = Marking ringid 1 interface 127.0.0.1 FAULTY

root@pg3:~# corosync-cfgtool -s
Printing ring status.
Local node ID 3
RING ID 0
  id = A.B.C3.236
  status = ring 0 active with no faults
RING ID 1
  id = D.E.F3.112
  status = Marking ringid 1 interface D.E.F3.112 FAULTY

_

/etc/corosync/corosync.conf from pg1 0 other nodes use different subnets and
IPs, but are otherwise identical:
===
quorum {
  provider: corosync_votequorum
  expected_votes: 3
}

totem {
  version: 2

  crypto_cipher: none
  crypto_hash: none

  rrp_mode: passive
  interface {
  ringnumber: 0
  bindnetaddr: A.B.C1.0
  mcastport: 5405
  ttl: 1
  }
  interface {
  ringnumber: 1
  bindnetaddr: D.E.F1.64
  mcastport: 5405
  ttl: 1
  }
  transport: udpu
}

nodelist {
  node {
  ring0_addr: pg1
  ring1_addr: pg1p
  nodeid: 1
  }
  node {
  ring0_addr: pg2
  ring1_addr: pg2p
  nodeid: 2
  }
  node {
  ring0_addr: pg3
  ring1_addr: pg3p
  nodeid: 3
  }
}

logging {
  to_syslog: yes
}

===

___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org







___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Corosync with passive rrp, udpu - Unable to reset after "Marking ringid 1 interface 127.0.0.1 FAULTY"

[Jan Friesse]

Martin Schlegel napsal(a):

Hello everyone,

we run a 3 node Pacemaker (1.1.14) / Corosync (2.3.5) cluster for a couple of
months successfully and we have started seeing a faulty ring with unexpected
  127.0.0.1 binding that we cannot reset via "corosync-cfgtool -r".


This is problem. Bind to 127.0.0.1 = ifdown happened = problem and with 
RRP it means BIG problem.




We have had this once before and only restarting Corosync (and everything else)
on the node showing the unexpected 127.0.0.1 binding made the problem go away.
However, in production we obviously would like to avoid this if possible.


Just don't do ifdown. Never. If you are using NetworkManager (which does 
ifdown by default if cable is disconnected), use something like 
NetworkManager-config-server package (it's just change of configuration 
so you can adopt it to whatever distribution you are using).


Regards,
  Honza



So from the following description - how can I troubleshoot this issue and/or
does anybody have a good idea what might be happening here ?

We run 2x passive rrp rings across different IP-subnets via udpu and we get the
following output (all IPs obfuscated) - please notice the unexpected interface
binding 127.0.0.1 for host pg2.

If we reset via "corosync-cfgtool -r" on each node heartbeat ring id 1 briefly
shows "no faults" but goes back to "FAULTY" seconds later.

Regards,
Martin Schlegel
_

root@pg1:~# corosync-cfgtool -s
Printing ring status.
Local node ID 1
RING ID 0
 id  = A.B.C1.5
 status  = ring 0 active with no faults
RING ID 1
 id  = D.E.F1.170
 status  = Marking ringid 1 interface D.E.F1.170 FAULTY

root@pg2:~# corosync-cfgtool -s
Printing ring status.
Local node ID 2
RING ID 0
 id  = A.B.C2.88
 status  = ring 0 active with no faults
RING ID 1
 id  = 127.0.0.1
 status  = Marking ringid 1 interface 127.0.0.1 FAULTY

root@pg3:~# corosync-cfgtool -s
Printing ring status.
Local node ID 3
RING ID 0
 id  = A.B.C3.236
 status  = ring 0 active with no faults
RING ID 1
 id  = D.E.F3.112
 status  = Marking ringid 1 interface D.E.F3.112 FAULTY


_


/etc/corosync/corosync.conf from pg1 0 other nodes use different subnets and
IPs, but are otherwise identical:
===
quorum {
 provider: corosync_votequorum
 expected_votes: 3
}

totem {
 version: 2

 crypto_cipher: none
 crypto_hash: none

 rrp_mode: passive
 interface {
 ringnumber: 0
 bindnetaddr: A.B.C1.0
 mcastport: 5405
 ttl: 1
 }
 interface {
 ringnumber: 1
 bindnetaddr: D.E.F1.64
 mcastport: 5405
 ttl: 1
 }
 transport: udpu
}

nodelist {
 node {
 ring0_addr: pg1
 ring1_addr: pg1p
 nodeid: 1
 }
 node {
 ring0_addr: pg2
 ring1_addr: pg2p
 nodeid: 2
 }
 node {
 ring0_addr: pg3
 ring1_addr: pg3p
 nodeid: 3
 }
}

logging {
 to_syslog: yes
}

===

___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org




___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

[ClusterLabs] Temporarily corosync.org unavailability

[Jan Friesse]

I would like to inform all corosync.org user about temporarily 
corosync.org unavailability. We are working on fixing this issue.


For now just use http://corosync.github.io/corosync/ (and 
http://build.clusterlabs.org/corosync/releases/) for downloading releases.


We are planing to restore corosync.org discuss mailing list, but only in 
Read only mode. For new questions, suggestions, ... just use 
users@clusterlabs.org (what most of the users are already doing anyway).


If you like to send patches, use GitHub pull request as documented on 
Corosync wiki https://github.com/corosync/corosync/wiki/Submitting-patches.


We are very sorry for the inconvenience.

Regards,
  Honza


___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Too quick node reboot leads to failed corosync assert on other node(s)

[Jan Friesse]

Michal,



Hello.

The subject is just a hypothesis that I'd like to confirm/discuss here.

TL;DR Token timeout shouldn't be greater than reboot cycle, is that correct?



actually when I was reading your mail it was like "this sounds so 
familiar, we solved this problem looong time ago". Chrissie, are you 
able to remember/find patch solving this problem.




The situation with the assert is as follows:

node01:
18:08:02 scheduling node01 for STONITH
[ gap ]
18:08:26 start of log after reboot
18:08:36 corosync started

node02:
18:08:02 it starts queuing outgoing messages (it has no token)
18:08:32 timeout, it realizes token is lost in OPERATIONAL state
18:08:32 it enters gather->commit->recovery process
18:08:37 it received commit token for the 2nd time and wants to
  enter recovery state
18:08:37 it thinks that node01 is transitioning from the old ring
  configuration to the new and wants to retransmit old messages
  with seq number greater that node01's aru -- problem is that
  the difference is too large and
  'assert(range < QUEUE_RTR_ITEMS_SIZE_MAX)' in
  memb_state_recovery_enter fails

IOW, node01 shouldn't be considered transitioning from old ring to new
ring only because it is present in both configurations because it was
restarted(!) in between. In the end, its new aru is zero, however, the
seq number before the token loss might have been arbitrarily high,
especially greater than QUEUE_RTR_ITEMS_SIZE_MAX and thus the assertion
fails.


Yep, sounds very reasonable.



I'm referring to the Shift_to_recovery routine as described in paper [1]
on page 21. There are two types of nodes:
- my_new_memb (they are fresh new in the ring),
- my_trans_memb (they transition from the old ring).

I've looked more into the paper, rather than into the code, so my
reasoning might not fit the reality. This happened with corosync-1.4.7,
however, the relevant code of memb_state_recovery_enter in upstream is
still the same.

Below is last output [2] from corosync-fplay on the node which failed
the assertion.
Does this make sense to anyone?
FTR, totem:token value is set to 3ms, perhaps decreasing this below
reboot cycle would prevent the issue.


Highly probably. Also did you find any difference between case where 
node was fenced (so corosync wasn't killed correctly) and case where 
corosync was properly terminated?


Do you have any reproducer for this problem? It's for sure something 
good to have fixed.


Regards,
  Honza




Thanks,
Michal

[1] http://corosync.github.com/corosync/doc/tocssrp95.ps.gz
[2] corosync-fplay output

rec=[5444857] time=[2015-10-24 18:08:02] Tracing(1) Messsage=Delivering
MCAST message with seq 12a9b4 to pending delivery queue
rec=[5444858] time=[2015-10-24 18:08:02] Tracing(1) Messsage=releasing
messages up to and including 12a9b3
rec=[5444859] time=[2015-10-24 18:08:02] Tracing(1) Messsage=releasing
messages up to and including 12a9b4

rec=[5444860] time=[2015-10-24 18:08:04] Tracing(1) Messsage=mcasted
message added to pending queue
rec=[5444861] time=[2015-10-24 18:08:06] Tracing(1) Messsage=mcasted
message added to pending queue
rec=[5444862] time=[2015-10-24 18:08:07] Tracing(1) Messsage=mcasted
message added to pending queue
rec=[5444863] time=[2015-10-24 18:08:09] Tracing(1) Messsage=mcasted
message added to pending queue
rec=[5444864] time=[2015-10-24 18:08:10] Tracing(1) Messsage=mcasted
message added to pending queue
rec=[5444865] time=[2015-10-24 18:08:12] Tracing(1) Messsage=mcasted
message added to pending queue
rec=[5444866] time=[2015-10-24 18:08:13] Tracing(1) Messsage=mcasted
message added to pending queue
rec=[5444867] time=[2015-10-24 18:08:15] Tracing(1) Messsage=mcasted
message added to pending queue
rec=[5444868] time=[2015-10-24 18:08:16] Tracing(1) Messsage=mcasted
message added to pending queue
rec=[5444869] time=[2015-10-24 18:08:18] Tracing(1) Messsage=mcasted
message added to pending queue
rec=[5444870] time=[2015-10-24 18:08:20] Tracing(1) Messsage=mcasted
message added to pending queue
rec=[5444871] time=[2015-10-24 18:08:21] Tracing(1) Messsage=mcasted
message added to pending queue
rec=[5444872] time=[2015-10-24 18:08:23] Tracing(1) Messsage=mcasted
message added to pending queue
rec=[5444873] time=[2015-10-24 18:08:24] Tracing(1) Messsage=mcasted
message added to pending queue
rec=[5444874] time=[2015-10-24 18:08:26] Tracing(1) Messsage=mcasted
message added to pending queue
rec=[5444875] time=[2015-10-24 18:08:27] Tracing(1) Messsage=mcasted
message added to pending queue
rec=[5444876] time=[2015-10-24 18:08:30] Tracing(1) Messsage=mcasted
message added to pending queue
rec=[5444877] time=[2015-10-24 18:08:31] Tracing(1) Messsage=mcasted
message added to pending queue

rec=[5444878] time=[2015-10-24 18:08:32] Log Message=The token was lost
in the OPERATIONAL state.
rec=[5444879] time=[2015-10-24 18:08:32] Log Message=A processor failed,
forming new configuration.
rec=[5444880] time=[2015-10-24 18:08:32] 

Re: [ClusterLabs] Connection to the CPG API failed: Library error (2)

[Jan Friesse]

Karthikeyan Ramasamy napsal(a):

Hi Honza,
   A team is already working on moving to the latest linux.  However, we 
delivered a feature with RHEL6.5, which has run into this issue.  Any tip would 
be very helpful for us!


It's highly probably caused by corosync, but it's really hard to say, 
what patch can help solve problem you hit (we did rebase in RHEL 6.6). 
So no real hint. Just upgrade to something supported.


Regards,
  Honza



Thanks,
Karthik.

-Original Message-
From: Jan Friesse [mailto:jfrie...@redhat.com]
Sent: 01 பிப்ரவரி 2016 13:43
To: Cluster Labs - All topics related to open-source clustering welcomed
Subject: Re: [ClusterLabs] Connection to the CPG API failed: Library error (2)

Karthikeyan Ramasamy napsal(a):

Sorry missed important information.

We are using RHEL6.5 and Pacemaker 1.1.10-14.


6.5 EUS is over (ended November 30, 2015), so it's good idea to update.

Regards,
Honza




Thanks,
Karthik.

-Original Message-
From: Karthikeyan Ramasamy [mailto:karthikeyan.ramas...@ericsson.com]
Sent: 01 பிப்ரவரி 2016 13:21
To: Cluster Labs - All topics related to open-source clustering
welcomed
Subject: [ClusterLabs] Connection to the CPG API failed: Library error
(2)

Hello Pacemaker Users,
We are having a problem with cman failing, when the host server is
under load and processing traffic.  Following information is available
in the logs,


Jan 30 06:53:13 [2041] vmc0135  attrd:error: pcmk_cpg_dispatch: 
Connection to the CPG API failed: Library error (2)
Jan 30 06:53:13 [2043] vmc0135   crmd:error: pcmk_cpg_dispatch: 
Connection to the CPG API failed: Library error (2)
Jan 30 06:53:13 [2043] vmc0135   crmd:error: crmd_cs_destroy:   
connection terminated
Jan 30 06:53:13 [2041] vmc0135  attrd: crit: attrd_cs_destroy:  Lost 
connection to Corosync service!
Jan 30 06:53:13 [2043] vmc0135   crmd: info: qb_ipcs_us_withdraw:   
withdrawing server sockets
Jan 30 06:53:13 [2038] vmc0135cib: info: crm_cs_flush:  Sent 0 
CPG messages  (1 remaining, last=129): Library error (2)
Jan 30 06:53:13 [2038] vmc0135cib:error: pcmk_cpg_dispatch: 
Connection to the CPG API failed: Library error (2)
Jan 30 06:53:13 [2038] vmc0135cib:error: cib_cs_destroy:
Corosync connection lost!  Exiting.
Jan 30 06:53:13 [2038] vmc0135cib: info: terminate_cib: 
cib_cs_destroy: Exiting fast...
Jan 30 06:53:13 [2038] vmc0135cib: info: crm_client_destroy:
Destroying 0 events
Jan 30 06:53:13 [2041] vmc0135  attrd:   notice: main:  Exiting...
Jan 30 06:53:13 [2041] vmc0135  attrd:   notice: main:  Disconnecting 
client 0x611c30, pid=2043...
Jan 30 06:53:13 [2038] vmc0135cib: info: qb_ipcs_us_withdraw:   
withdrawing server sockets
Jan 30 06:53:13 [2038] vmc0135cib: info: crm_client_destroy:
Destroying 0 events
Jan 30 06:53:13 [2039] vmc0135 stonith-ng:error: pcmk_cpg_dispatch: 
Connection to the CPG API failed: Library error (2)
Jan 30 06:53:13 [2043] vmc0135   crmd: info: 
tengine_stonith_connection_destroy:Fencing daemon disconnected
Jan 30 06:53:13 [2043] vmc0135   crmd:   notice: crmd_exit: Forcing 
immediate exit: Link has been severed (67)
Jan 30 06:53:13 [2043] vmc0135   crmd: info: crm_xml_cleanup:   
Cleaning up memory from libxml2
Jan 30 06:53:13 [2039] vmc0135 stonith-ng:error: stonith_peer_cs_destroy:   
Corosync connection terminated
Jan 30 06:53:13 [2041] vmc0135  attrd:error: 
attrd_cib_connection_destroy:  Connection to the CIB terminated...
Jan 30 06:53:13 [2021] vmc0135 pacemakerd:error: cfg_connection_destroy:
Connection destroyed
Jan 30 06:53:13 [2039] vmc0135 stonith-ng: info: stonith_shutdown:  
Terminating with  1 clients
Jan 30 06:53:13 [2021] vmc0135 pacemakerd:   notice: pcmk_shutdown_worker:  
Shuting down Pacemaker
Jan 30 06:53:13 [2021] vmc0135 pacemakerd:   notice: stop_child:
Stopping crmd: Sent -15 to process 2043
Jan 30 06:53:13 [2021] vmc0135 pacemakerd:error: pcmk_cpg_dispatch: 
Connection to the CPG API failed: Library error (2)
Jan 30 06:53:13 [2021] vmc0135 pacemakerd:error: mcp_cpg_destroy:   
Connection destroyed
Jan 30 06:53:13 [2021] vmc0135 pacemakerd: info: crm_xml_cleanup:   
Cleaning up memory from libxml2
Jan 30 06:53:13 [2039] vmc0135 stonith-ng: info: cib_connection_destroy:
Connection to the CIB closed.
Jan 30 06:53:13 [2039] vmc0135 stonith-ng: info: crm_client_destroy:
Destroying 0 events
Jan 30 06:53:13 [2038] vmc0135cib: info: crm_client_destroy:
Destroying 0 events
Jan 30 06:53:13 [2039] vmc0135 stonith-ng: info: qb_ipcs_us_withdraw:   
withdrawing server sockets
Jan 30 06:53:13 [2039] vmc0135 stonith-ng: info: main:  Done
Jan 30 06:53:13 [2039] vmc0135 stonith-ng: info: crm_xml_cleanup

Re: [ClusterLabs] Corosync main process was not scheduled for 115935.2266 ms (threshold is 800.0000 ms). Consider token timeout increase.

[Jan Friesse]

Adam Spiers napsal(a):

Hi all,

Jan Friesse <jfrie...@redhat.com> wrote:

There is really no help. It's best to make sure corosync is scheduled

regularly.
I may sound silly, but how can I do it?


It's actually very hard to say. Pauses like 30 sec is really unusual
and shouldn't happen (specially with RT scheduling). It is usually
happening on VM where host is overcommitted.


It's funny you are discussing this during the same period where my
team is seeing this happen fairly regularly within VMs on a host which
is overcommitted.  In other words, I can confirm Jan's statement above
is true.


Yep, sadly VM affects scheduling a lot. For cluster nodes it actually 
really make sense to map every virtual cpu core to physical cpu core.




Like Konstiantyn, we have also sometimes seen no fencing occur as a
result of these pauses, e.g.

Feb 24 02:53:04 d52-54-77-77-77-02 corosync[18939]:   [MAIN  ] Corosync main 
process was not scheduled for 7343.1909 ms (threshold is 4000. ms). 
Consider token timeout increase.
Feb 24 02:53:04 d52-54-77-77-77-02 corosync[18939]:   [TOTEM ] A processor 
failed, forming new configuration.
Feb 24 02:53:04 d52-54-77-77-77-02 corosync[18939]:   [CLM   ] CLM 
CONFIGURATION CHANGE
Feb 24 02:53:04 d52-54-77-77-77-02 corosync[18939]:   [CLM   ] New 
Configuration:
Feb 24 02:53:04 d52-54-77-77-77-02 corosync[18939]:   [CLM   ] #011r(0) 
ip(192.168.2.82)
Feb 24 02:53:04 d52-54-77-77-77-02 corosync[18939]:   [CLM   ] #011r(0) 
ip(192.168.2.84)
Feb 24 02:53:04 d52-54-77-77-77-02 corosync[18939]:   [CLM   ] Members Left:
Feb 24 02:53:04 d52-54-77-77-77-02 corosync[18939]:   [CLM   ] Members Joined:
Feb 24 02:53:04 d52-54-77-77-77-02 corosync[18939]:   [pcmk  ] notice: 
pcmk_peer_update: Transitional membership event on ring 32: memb=2, new=0, 
lost=0
Feb 24 02:53:04 d52-54-77-77-77-02 corosync[18939]:   [pcmk  ] info: 
pcmk_peer_update: memb: d52-54-77-77-77-01 1084752466
Feb 24 02:53:04 d52-54-77-77-77-02 corosync[18939]:   [pcmk  ] info: 
pcmk_peer_update: memb: d52-54-77-77-77-02 1084752468
Feb 24 02:53:04 d52-54-77-77-77-02 corosync[18939]:   [CLM   ] CLM 
CONFIGURATION CHANGE
Feb 24 02:53:04 d52-54-77-77-77-02 corosync[18939]:   [CLM   ] New 
Configuration:
Feb 24 02:53:04 d52-54-77-77-77-02 corosync[18939]:   [CLM   ] #011r(0) 
ip(192.168.2.82)
Feb 24 02:53:04 d52-54-77-77-77-02 corosync[18939]:   [CLM   ] #011r(0) 
ip(192.168.2.84)
Feb 24 02:53:04 d52-54-77-77-77-02 corosync[18939]:   [CLM   ] Members Left:
Feb 24 02:53:04 d52-54-77-77-77-02 corosync[18939]:   [CLM   ] Members Joined:
Feb 24 02:53:04 d52-54-77-77-77-02 corosync[18939]:   [pcmk  ] notice: 
pcmk_peer_update: Stable membership event on ring 32: memb=2, new=0, lost=0
Feb 24 02:53:04 d52-54-77-77-77-02 corosync[18939]:   [pcmk  ] info: 
pcmk_peer_update: MEMB: d52-54-77-77-77-01 1084752466
Feb 24 02:53:04 d52-54-77-77-77-02 corosync[18939]:   [pcmk  ] info: 
pcmk_peer_update: MEMB: d52-54-77-77-77-02 1084752468
Feb 24 02:53:04 d52-54-77-77-77-02 corosync[18939]:   [TOTEM ] A processor 
joined or left the membership and a new membership was formed.
Feb 24 02:53:04 d52-54-77-77-77-02 corosync[18939]:   [CPG   ] chosen downlist: 
sender r(0) ip(192.168.2.82) ; members(old:2 left:0)
Feb 24 02:53:04 d52-54-77-77-77-02 corosync[18939]:   [MAIN  ] Completed 
service synchronization, ready to provide service.

I don't understand why it claims a processor failed, forming a new
configuration, when the configuration appears no different from
before: no members joined or left.  Can anyone explain this?


Corosync uses token (very similar to old token-ring, in corosync used 
for concession control (only node with token can send messages) and 
ordering of messages) timeout (= maximum time to wait for token. If 
token doesn't arrive it's considered to be lost) to detect problems in 
network or more generally failed nodes. If corosync is not scheduled for 
a long time, it's same situation (from the affected node point of view) 
as lost token. Corosync can find out if it was not scheduled for token 
timeout but there is really no change in following steps. Node really 
cannot be sure if other nodes didn't create different membership 
(without affected node) so it has to go thru gather state (contact all 
nodes, get their world view, decide) even if (for given node) nothing 
really changed (other nodes may see it differently).


Hope it helps a bit.

Regards,
  Honza



___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org




___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs

Re: [ClusterLabs] Security with Corosync

[Jan Friesse]

Nikhil,

Nikhil Utane napsal(a):

Hi,

I changed some configuration and captured packets. I can see that the data
is already garbled and not in the clear.
So does corosync already have this built-in?
Can somebody provide more details as to what all security features are
incorporated?


See man page corosync.conf(5) options crypto_hash, crypto_cipher (for 
corosync 2.x) and potentially secauth (for coorsync 1.x and 2.x).


Basically corosync by default uses aes256 for encryption and sha1 for 
hmac authentication.


Pacemaker uses corosync cpg API so as long as encryption is enabled in 
the corosync.conf, messages interchanged between nodes are encrypted.


Regards,
  Honza



-Thanks
Nikhil

On Fri, Mar 11, 2016 at 11:38 AM, Nikhil Utane 
wrote:


Hi,

Does corosync provide mechanism to secure the communication path between
nodes of a cluster?
I would like all the data that gets exchanged between all nodes to be
encrypted.

A quick google threw up this link:
https://github.com/corosync/corosync/blob/master/SECURITY

Can I make use of it with pacemaker?

-Thanks
Nikhil






___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org




___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Security with Corosync

[Jan Friesse]

Nikhil Utane napsal(a):

Follow-up question.
I noticed that secauth was turned off in my corosync.conf file. I enabled
it on all 3 nodes and restarted the cluster. Everything was working fine.
However I just noticed that I had forgotten to copy the authkey to one of
the node. It is present on 2 nodes but not the third. And I did a failover
and the third node took over without any issue.
How is the 3rd node participating in the cluster if it doesn't have the
authkey?


It's just not possible. If you would enabled secauth correctly and you 
didn't have /etc/corosync/authkey, message like "Could not open 
/etc/corosync/authkey: No such file or directory" would show up. There 
are few exceptions:

- you have changed totem.keyfile with file existing on all nodes
- you are using totem.key then everything works as expected (it has 
priority over default authkey file but not over totem.keyfile)
- you are using COROSYNC_TOTEM_AUTHKEY_FILE env with file existing on 
all nodes


Regards,
  Honza



On Fri, Mar 11, 2016 at 4:15 PM, Nikhil Utane <nikhil.subscri...@gmail.com>
wrote:


Perfect. Thanks for the quick response Honza.

Cheers
Nikhil

On Fri, Mar 11, 2016 at 4:10 PM, Jan Friesse <jfrie...@redhat.com> wrote:


Nikhil,

Nikhil Utane napsal(a):


Hi,

I changed some configuration and captured packets. I can see that the
data
is already garbled and not in the clear.
So does corosync already have this built-in?
Can somebody provide more details as to what all security features are
incorporated?



See man page corosync.conf(5) options crypto_hash, crypto_cipher (for
corosync 2.x) and potentially secauth (for coorsync 1.x and 2.x).

Basically corosync by default uses aes256 for encryption and sha1 for
hmac authentication.

Pacemaker uses corosync cpg API so as long as encryption is enabled in
the corosync.conf, messages interchanged between nodes are encrypted.

Regards,
   Honza



-Thanks
Nikhil

On Fri, Mar 11, 2016 at 11:38 AM, Nikhil Utane <
nikhil.subscri...@gmail.com>
wrote:

Hi,


Does corosync provide mechanism to secure the communication path between
nodes of a cluster?
I would like all the data that gets exchanged between all nodes to be
encrypted.

A quick google threw up this link:
https://github.com/corosync/corosync/blob/master/SECURITY

Can I make use of it with pacemaker?

-Thanks
Nikhil






___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org




___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org








___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org




___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Totem is unable to form a cluster because of an operating system or network fault

[Jan Friesse]

Hi ,
Am trying to configure my sql on ubuntu according to this article :
https://azure.microsoft.com/en-in/documentation/articles/virtual-machines-linux-classic-mysql-cluster/

two node cluster


looking on corosync log :


Apr 12 11:01:09 corosync [TOTEM ] Totem is unable to form a cluster because
of an operating system or network fault. The most common cause of this
message is that the local firewall is configured improperly.
Apr 12 11:01:11 corosync [TOTEM ] Totem is unable to form a cluster because
of an operating system or network fault. The most common cause of this
message is that the local firewall is configured improperly.
Apr 12 11:01:13 corosync [TOTEM ] Totem is unable to form a cluster because
of an operating system or network fault. The most common cause of this
message is that the local firewall is configured improperly.
Apr 12 11:01:16 corosync [TOTEM ] Totem is unable to form a cluster because
of an operating system or network fault. The most common cause of this
message is that the local firewall is configured improperly.
Apr 12 11:01:18 corosync [TOTEM ] Totem is unable to form a cluster because
of an operating system or network fault. The most common cause of this
message is that the local firewall is configured improperly.
Apr 12 11:01:20 corosync [TOTEM ] Totem is unable to form a cluster because
of an operating system or network fault. The most common cause of this
message is that the local firewall is configured improperly.
Apr 12 11:01:22 corosync [TOTEM ] Totem is unable to form a cluster because
of an operating system or network fault. The most common cause of this
message is that the local firewall is configured improperly.
Apr 12 11:01:24 corosync [TOTEM ] Totem is unable to form a cluster because
of an operating system or network fault. The most common cause of this
message is that the local firewall is configured improperly.
Apr 12 11:01:27 corosync [TOTEM ] Totem is unable to form a cluster because
of an operating system or network fault. The most common cause of this
message is that the local firewall is configured improperly.
Apr 12 11:01:29 corosync [TOTEM ] Totem is unable to form a cluster because
of an operating system or network fault. The most common cause of this
message is that the local firewall is configured improperly.
Apr 12 11:01:31 corosync [TOTEM ] Totem is unable to form a cluster because
of an operating system or network fault. The most common cause of this
message is that the local firewall is configured improperly.



totem {
   version: 2
   crypto_cipher: none
   crypto_hash: none
   interface {
 ringnumber: 0
 bindnetaddr: 10.1.0.0
 mcastport: 5405
 ttl: 1
   }
   transport: udpu
}
logging {
   fileline: off
   to_logfile: yes
   to_syslog: yes
   logfile: /var/log/corosync/corosync.log
   debug: off
   timestamp: on
   logger_subsys {
 subsys: QUORUM
 debug: off
 }
   }
nodelist {
   node {
 ring0_addr: 10.1.0.6
 nodeid: 1
   }
   node {
 ring0_addr: 10.1.0.7
 nodeid: 2
   }
}
quorum {
   provider: corosync_votequorum
}


If I initiate a tcpdump on node 2 and start either a netcat or nmap I see
packet arrives to destination host for port 5405 UDP traffic



I do see Corosync listening on the IP/PORT





root@node-2:/home/dinor# netstat -an | grep -i 5405

udp0  0 10.1.0.7:5405   0.0.0.0:*






root@node-1:/home/dinor# netstat -an | grep -i 5405

udp0  0 10.1.0.6:5405   0.0.0.0:*





On node 1 I start a netcat to port 5405 via udp



netcat -D -4 -u 10.1.0.7 5405



In here you type some text and hit enter



On node 1 tcpdump we see data sent to IP 10.1.0.7



root@node-1:/var/log/corosync# tcpdump -n udp port 5405

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode

listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes

10:08:24.484533 IP 10.1.0.6.44299 > 10.1.0.7.5405: UDP, length 26







On node 2 tcpdump I see the data arrive



root@node-2:/var/log/corosync# tcpdump -n udp port 5405

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode

listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes

10:08:24.484892 IP 10.1.0.6.44299 > 10.1.0.7.5405: UDP, length 26





Tested also sending UDP packets from node 2 – all ok.



So connectivity seems to be ok.



Port scanner also shows the port as Open





root@node-1:/home/dinor# nmap -sUV 10.1.0.7 -p 5402-5405



Starting Nmap 5.21 ( http://nmap.org ) at 2016-04-12 10:31 UTC

Nmap scan report for node-2 (10.1.0.7)

Host is up (0.00060s latency).

PORT STATE SERVICE VERSION

5402/udp closedunknown

5403/udp closedunknown

5404/udp closedunknown

*5405/udp open|filtered unknown*

MAC Address: 12:34:56:78:9A:BC (Unknown)



Service detection performed. Please report any incorrect results at
http://nmap.org/submit/ .

Nmap done: 1 IP address (1 host up) scanned in 79.07 seconds





There is no FW and 

Re: [ClusterLabs] getting "Totem is unable to form a cluster" error

[Jan Friesse]

pacemaker 1.1.12-11.12
openais 1.1.4-5.24.5
corosync 1.4.7-0.23.5

Its a two node active/passive cluster and we just upgraded the SLES 11
SP 3 to SLES 11 SP 4(nothing  else) but when we try to start the cluster
service we get the following error:

"Totem is unable to form a cluster because of an operating system or
network fault."

Firewall is stopped and disabled on both the nodes. Both nodes can
ping/ssh/vnc each other.


Hard to help. First of all, I would recommend to ask SUSE support 
because I don't really have access to source code of corosync 
1.4.7-0.23.5 package, so really don't know what patches are added.





corosync.conf:
aisexec {
 group:root
 user:root
}
service {
 use_mgmtd:yes
 use_logd:yes
 ver:0
 name:pacemaker
}
totem {
 rrp_mode:none
 join:60
 max_messages:20
 vsftype:none
 token:5000
 consensus:6000

 interface {
 bindnetaddr:192.168.150.0

 member {
 memberaddr: 192.168.150.12
 }
 member {
 memberaddr:  192.168.150.13
 }
 mcastport:5405

 ringnumber:0

 }
 secauth:off
 version:2
 transport:udpu
 token_retransmits_before_loss_const:10
 clear_node_high_bit:new
}
logging {
 to_logfile:no
 to_syslog:yes
 debug:off
 timestamp:off
 to_stderr:no
 fileline:off
 syslog_facility:daemon
}
amf {
 mode:disable
}

/var/log/messages:
Apr  6 17:51:49 prd1 corosync[8672]:  [MAIN  ] Corosync Cluster Engine
('1.4.7'): started and ready to provide service.
Apr  6 17:51:49 prd1 corosync[8672]:  [MAIN  ] Corosync built-in
features: nss
Apr  6 17:51:49 prd1 corosync[8672]:  [MAIN  ] Successfully configured
openais services to load
Apr  6 17:51:49 prd1 corosync[8672]:  [MAIN  ] Successfully read main
configuration file '/etc/corosync/corosync.conf'.
Apr  6 17:51:49 prd1 corosync[8672]:  [TOTEM ] Initializing transport
(UDP/IP Unicast).
Apr  6 17:51:49 prd1 corosync[8672]:  [TOTEM ] Initializing
transmit/receive security: libtomcrypt SOBER128/SHA1HMAC (mode 0).
Apr  6 17:51:49 prd1 corosync[8672]:  [TOTEM ] The network interface is
down.


^^^ This is important line. It means corosync was unable to find 
interface for bindnetaddr 192.168.150.0. Make sure interface with this 
network address exists.


Regards,
  Honza


___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] getting "Totem is unable to form a cluster" error

[Jan Friesse]

08.04.2016 17:51, Jan Friesse пишет:

On 04/08/16 13:01, Jan Friesse wrote:
  >> pacemaker 1.1.12-11.12
  >> openais 1.1.4-5.24.5
  >> corosync 1.4.7-0.23.5
  >>
  >> Its a two node active/passive cluster and we just upgraded the
SLES 11
  >> SP 3 to SLES 11 SP 4(nothing  else) but when we try to start the
cluster
  >> service we get the following error:
  >>
  >> "Totem is unable to form a cluster because of an operating system or
  >> network fault."
  >>
  >> Firewall is stopped and disabled on both the nodes. Both nodes can
  >> ping/ssh/vnc each other.
  >
  > Hard to help. First of all, I would recommend to ask SUSE support
because I don't really have access to source code of corosync
1.4.7-0.23.5 package, so really don't know what patches are added.
  >
  >
Yup, ticket opened with SUSE Support.

  >>
  >>
  >>
  >> /var/log/messages:
  >> Apr  6 17:51:49 prd1 corosync[8672]:  [MAIN  ] Corosync Cluster
Engine
  >> ('1.4.7'): started and ready to provide service.
  >> Apr  6 17:51:49 prd1 corosync[8672]:  [MAIN  ] Corosync built-in
  >> features: nss
  >> Apr  6 17:51:49 prd1 corosync[8672]:  [MAIN  ] Successfully
configured
  >> openais services to load
  >> Apr  6 17:51:49 prd1 corosync[8672]:  [MAIN  ] Successfully read main
  >> configuration file '/etc/corosync/corosync.conf'.
  >> Apr  6 17:51:49 prd1 corosync[8672]:  [TOTEM ] Initializing transport
  >> (UDP/IP Unicast).
  >> Apr  6 17:51:49 prd1 corosync[8672]:  [TOTEM ] Initializing
  >> transmit/receive security: libtomcrypt SOBER128/SHA1HMAC (mode 0).
  >> Apr  6 17:51:49 prd1 corosync[8672]:  [TOTEM ] The network
interface is
  >> down.
  >
  > ^^^ This is important line. It means corosync was unable to find
interface for bindnetaddr 192.168.150.0. Make sure interface with this
network address exists.
  >
  >
this machine has two IP address assigned on interface bond0

bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP
  link/ether 74:e6:e2:73:e5:61 brd ff:ff:ff:ff:ff:ff
  inet 10.150.20.91/24 brd 10.150.20.55 scope global bond0
  inet 192.168.150.12/22 brd 192.168.151.255 scope global
bond0:cluster
  inet6 fe80::76e6:e2ff:fe73:e561/64 scope link
 valid_lft forever preferred_lft forever


This is ifconfig output? I'm just wondering how you were able to set two
ipv4 addresses (in this format, I would expect another interface like
bond0:1 or nothing at all)?



That is how Linux stack works for the last 10 or 15 years. The bond0:1
is legacy emulation for ifconfig addicts.

ip addr add 10.150.20.91/24 dev bond0


Hmm.

RHEL 6:

# tunctl -p
Set 'tap0' persistent and owned by uid 0

# ip addr add 192.168.7.1/24 dev tap0
# ip addr add 192.168.8.1/24 dev tap0
# ifconfig tap0
tap0  Link encap:Ethernet  HWaddr 22:95:B1:85:67:3F
  inet addr:192.168.7.1  Bcast:0.0.0.0  Mask:255.255.255.0
  BROADCAST MULTICAST  MTU:1500  Metric:1
  RX packets:0 errors:0 dropped:0 overruns:0 frame:0
  TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
  collisions:0 txqueuelen:500
  RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)

RHEL 7:
# ip tuntap add dev tap0 mode tap
#  ip addr add 192.168.7.1/24 dev tap0
# ip addr add 192.168.8.1/24 dev tap0
# ifconfig tap0
tap0: flags=4098<BROADCAST,MULTICAST>  mtu 1500
inet 192.168.7.1  netmask 255.255.255.0  broadcast 0.0.0.0
ether 36:02:5c:ff:29:ea  txqueuelen 500  (Ethernet)
RX packets 0  bytes 0 (0.0 B)
RX errors 0  dropped 0  overruns 0  frame 0
TX packets 0  bytes 0 (0.0 B)
TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

So where do you see 192.168.8.1 in ifconfig output?




Anyway, I was trying to create bonding interface and set second ipv4
(via ip addr) and corosync (flatiron what is 1.4.8 + 4 for your problem
completely unrelated patches) was able to detect it without any problem.

I can recommend you to try:
- Set bindnetaddr to IP address of given node (so you have to change
bindnetaddr on both nodes)
- Try upstream corosync 1.4.8/flatiron

Regards,
   Honza



And I can ping 192.168.150.12 from this machine and from other machines
on network



--
Regards,

Muhammad Sharfuddin

___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org



___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.

Re: [ClusterLabs] Antw: Re: reproducible split brain

[Jan Friesse]

Christopher,



If I ignore pacemaker's existence, and just run corosync, corosync
disagrees about node membership in the situation presented in the first
email. While it's true that stonith just happens to quickly correct the
situation after it occurs it still smells like a bug in the case where
corosync in used in isolation. Corosync is after all a membership and
total ordering protocol, and the nodes in the cluster are unable to
agree on membership.

The Totem protocol specifies a ring_id in the token passed in a ring.
Since all of the 3 nodes but one have formed a new ring with a new id
how is it that the single node can survive in a ring with no other
members passing a token with the old ring_id?

Are there network failure situations that can fool the Totem membership
protocol or is this an implementation problem? I don't see how it could


The main problem (as you noted in original mail) is really about 
blocking only one direction (input one). This is called byzantine 
failure and it's something what corosync is unable to handle. Totem was 
simply never designed to solve byzantine failures.


Regards,
  Honza



not be one or the other, and it's bad either way.

On Thu, Mar 17, 2016, at 02:08 PM, Digimer wrote:

On 17/03/16 01:57 PM, vija ar wrote:

root file system is fine ...

but fencing is not a necessity a cluster shld function without it .. i
see the issue with corosync which has all been .. a inherent way of not
working neatly or smoothly ..


Absolutely wrong.

If you have a service that can run on both/all nodes at the same time
without coordination, you don't need a cluster, just run your services
everywhere.

If that's not the case, then you need fencing so that the (surviving)
node(s) can be sure that they know where services are and are not
running.


for e.g. take an issue where the live node is hung in db cluster .. now
db perspective transactions r not happening and tht is fine as the node
is having some issue .. now there is no need to fence this hung node but
just to switch over to passive one .. but tht doesnt happens and fencing
takes place either by reboot or shut .. which further makes the DB dirty
or far more than tht in non-recoverable state which wouldnt have happen
if a normal switch to other node as in cluster would have happened ...

i see fencing is not a solution its only required to forcefully take
control which is not the case always

On Thu, Mar 17, 2016 at 12:49 PM, Ulrich Windl
> wrote:

 >>> Christopher Harvey  schrieb am 16.03.2016 um 21:04
 in Nachricht
 <1458158684.122207.551267810.11f73...@webmail.messagingengine.com
 >:
 [...]
 >> > Would stonith solve this problem, or does this look like a bug?
 >>
 >> It should, that is its job.
 >
 > is there some log I can enable that would say
 > "ERROR: hey, I would use stonith here, but you have it disabled! your
 > warranty is void past this point! do not pass go, do not file a bug"?

 What should the kernel say during boot if the user has not defined a
 root file system?

 Maybe the "stonith-enabled=false" setting should be called either
 "data-corruption-mode=true" or "hang-forever-on-error=true" ;-)

 Regards,
 Ulrich



 ___
 Users mailing list: Users@clusterlabs.org 
 http://clusterlabs.org/mailman/listinfo/users

 Project Home: http://www.clusterlabs.org
 Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
 Bugs: http://bugs.clusterlabs.org




___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org




--
Digimer
Papers and Projects: https://alteeve.ca/w/
What if the cure for cancer is trapped in the mind of a person without
access to education?

___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org


___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org




___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: 

Re: [ClusterLabs] Security with Corosync

[Jan Friesse]

Nikhil Utane napsal(a):

Honza,

In my CIB I see the infrastructure being set to cman. pcs status is
reporting the same.



[root@node3 corosync]# pcs status
Cluster name: mycluster
Last updated: Wed Mar 16 16:57:46 2016
Last change: Wed Mar 16 16:56:23 2016
Stack: *cman*

But corosync also is running fine.

[root@node2 nikhil]# pcs status nodes corosync
Corosync Nodes:
  Online: node2 node3
  Offline: node1

I did a cibadmin query and replace from cman to corosync but it doesn't
change (even though replace operation succeeds)
I read that CMAN internally uses corosync but in corosync 2 CMAN support is
removed.
Totally confused. Please help.


Best start is to find out what versions you are using? If you have 
corosync 1.x and really using cman (what is highly probable), 
corosync.conf is completely ignored and instead cluster.conf 
(/etc/cluster/cluster.conf) is used. cluster.conf uses cman keyfile and 
if this is not provided, encryption key is simply cluster name. This is 
probably reason why everything worked when you haven't had authkey on 
one of nodes.


Honza



-Thanks
Nikhil

On Mon, Mar 14, 2016 at 1:19 PM, Jan Friesse <jfrie...@redhat.com> wrote:


Nikhil Utane napsal(a):


Follow-up question.
I noticed that secauth was turned off in my corosync.conf file. I enabled
it on all 3 nodes and restarted the cluster. Everything was working fine.
However I just noticed that I had forgotten to copy the authkey to one of
the node. It is present on 2 nodes but not the third. And I did a failover
and the third node took over without any issue.
How is the 3rd node participating in the cluster if it doesn't have the
authkey?



It's just not possible. If you would enabled secauth correctly and you
didn't have /etc/corosync/authkey, message like "Could not open
/etc/corosync/authkey: No such file or directory" would show up. There are
few exceptions:
- you have changed totem.keyfile with file existing on all nodes
- you are using totem.key then everything works as expected (it has
priority over default authkey file but not over totem.keyfile)
- you are using COROSYNC_TOTEM_AUTHKEY_FILE env with file existing on all
nodes

Regards,
   Honza




On Fri, Mar 11, 2016 at 4:15 PM, Nikhil Utane <
nikhil.subscri...@gmail.com>
wrote:

Perfect. Thanks for the quick response Honza.


Cheers
Nikhil

On Fri, Mar 11, 2016 at 4:10 PM, Jan Friesse <jfrie...@redhat.com>
wrote:

Nikhil,


Nikhil Utane napsal(a):

Hi,


I changed some configuration and captured packets. I can see that the
data
is already garbled and not in the clear.
So does corosync already have this built-in?
Can somebody provide more details as to what all security features are
incorporated?



See man page corosync.conf(5) options crypto_hash, crypto_cipher (for
corosync 2.x) and potentially secauth (for coorsync 1.x and 2.x).

Basically corosync by default uses aes256 for encryption and sha1 for
hmac authentication.

Pacemaker uses corosync cpg API so as long as encryption is enabled in
the corosync.conf, messages interchanged between nodes are encrypted.

Regards,
Honza


-Thanks

Nikhil

On Fri, Mar 11, 2016 at 11:38 AM, Nikhil Utane <
nikhil.subscri...@gmail.com>
wrote:

Hi,



Does corosync provide mechanism to secure the communication path
between
nodes of a cluster?
I would like all the data that gets exchanged between all nodes to be
encrypted.

A quick google threw up this link:
https://github.com/corosync/corosync/blob/master/SECURITY

Can I make use of it with pacemaker?

-Thanks
Nikhil






___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started:
http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org




___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started:
http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org








___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org




___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org





___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf

Re: [ClusterLabs] service network restart and corosync

[Jan Friesse]

Hi,

In our deployment, due to some requirement, we need to do a :
service network restart


What is exact reason for doing network restart?



Due to this corosync crashes and the associated pacemaker processes crash
as well.

As per the last comment on this issue,
---
Corosync reacts oddly to that. It's better to use an iptables rule to
block traffic (or crash the node with something like 'echo c >
/proc/sysrq-trigge




But other network services, like Postgres, do not crash due to this
network service restart :
I can login to psql , issue queries, without any problem.

In view of this, I would like to understand if it is possible to prevent a
corosync (and a corresponding Pacemaker) crash ?
Since postgres is somehow surviving this restart.

Any pointer to socket-level details for this behaviour will help me
understand (and explain the stakeholders) the problems better.


https://github.com/corosync/corosync/pull/32 should help.

Regards,
  Honza



Regards,
Debabrata Pani





___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org




___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Corosync do not send traffic

[Jan Friesse]

Roberto Munoz Gomez napsal(a):

2016-03-30 13:42 GMT+02:00 Jan Friesse <jfrie...@redhat.com>:


Roberto Munoz Gomez napsal(a):


2016-03-30 11:27 GMT+02:00 Jan Friesse <jfrie...@redhat.com>:

Hello,




Due to a change in the switch in one of the datacenters now I have an
odd
behaviour in the cluster.

I am using cman with corosync and pacemaker. The versions are:

pacemaker-1.1.10-14.el6_5.3.x86_64
corosync-1.4.1-15.el6_4.1.x86_64
cman-3.0.12.1-49.el6.x86_64

The problem is, when I launch /etc/init.d/pacemaker start and the
cman_tool
launch corosync, I don't see any UDP traffic, so the cluster is "broken"

But if I launch manually the same command "corosync -f" I do see udp
traffic and the totem is correctly sent between nodes.

It all began with the change in the switch, but I set the tcpdump in the
hosts and I do not see traffic.

I have tried the multicast and unicast configuration, different network,
but all with the same behaviour.

What am I missing?



I'm pretty sure main problem is that:
- corosync -f uses /etc/corosync/corosync.conf as a config file
- cman_tool join uses /etc/cluster/cluster.conf as a config file


Make sure to "replicate" (specially udpu/udp, ...) configuration from

corosync.conf to cluster.conf and I'm pretty sure it will work.





It worked!

I didn't knew cman_tool do not use corosync.conf. Setting udpu in
cluster.conf made it work again.

I am trying to migrate the bindnetaddr option from corosync.conf to
cluster.conf but couldn't make it work. I want to use this interface for
corosync traffic.


cluster.conf configuration file is more similar to corosync.conf with 
nodelist defined than corosync.conf without it. For cluster.conf you 
don't need to define bindnetaddr. Just put node names into /etc/hosts 
and add clusternode tag into cluster.conf. That's it.




inet addr:10.76.125.236  Bcast:10.76.125.239  Mask:255.255.255.240

In man cluster.conf says the last octect must be 0. But despite ccs_config
validate says it is ok, the cman_tool fails.


Do not use network address. Just put nodes into cluster.conf.

Regards,
  Honza




Regards.



___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org




___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] service network restart and corosync

[Jan Friesse]

Hi Jan Friesse,

Thank you for the update.

I have responded inline.
A few queries, as well.

Regards,
Debabrata Pani

On 29/03/16 14:24, "Jan Friesse" <jfrie...@redhat.com> wrote:




Hi (Jan Friesse)

I studied the issue mentioned in the github url.
It looks the crash that I am talking about is slightly different from
the
one mentioned in the original issue. May be they are related, but I
would
like to
Highlight my setup for ease.

Three node cluster , one is in maintenance mode to prevent any
scheduling
of resources.
=
Stack: classic openais (with plugin)


^^ I'm pretty sure you don't want to use plugin based pcmk.


Corosync version:
Corosync Cluster Engine, version '1.4.7'
Copyright (c) 2006-2009 Red Hat, Inc.


IS there any (non-plugin) version available for cents 6.5 ? What  is the


AFAIK (I'm not pcmk expert) CentOS contains both plugin and cman non 
plugin version. Also 6.5 is is not supported any longer so it's good 
idea to upgrade.



setup that is recommended ?
I am slightly puzzled by CMAN/plugin thing so some pointers will be really
helpful.


https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html-single/Configuring_the_Red_Hat_High_Availability_Add-On_with_Pacemaker/index.html






Current DC: vm2cen66.mobileum.com - partition with quorum
Version: 1.1.11-97629de
3 Nodes configured, 3 expected votes
6 Resources configured


Node vm3cent66.mobileum.com: maintenance
Online: [ vm1cen66.mobileum.com vm2cen66.mobileum.com ]


I login to vm1cen66 and do `ifdown eth0`
In vm1cen66, I don¹t see any change in the crm_mon -Afr output.
It remains the same, as shown below

Stack: classic openais (with plugin)
Current DC: vm2cen66.mobileum.com - partition with quorum
Version: 1.1.11-97629de
3 Nodes configured, 3 expected votes
6 Resources configured


Node vm3cent66.mobileum.com: maintenance
Online: [ vm1cen66.mobileum.com vm2cen66.mobileum.com ]
===


But if we login to the other nodes like vm2cen66, vem3cent66, we can
correctly see that the node vm1cen66 is offline.


That is expected




But if we look into the corosync.log of vm1cen66 we see the following

===
Mar 28 14:55:09 corosync [MAIN  ] Totem is unable to form a cluster
because of an operating system or network fault. The most common cause
of
this message is that the local firewall is configured improperly.
pgsql(TestPostgresql)[28203]:   2016/03/28_14:55:10 INFO: Master does
not
exist.
pgsql(TestPostgresql)[28203]:   2016/03/28_14:55:10 WARNING: My data is
out-of-date. status=DISCONNECT
Mar 28 14:55:11 corosync [MAIN  ] Totem is unable to form a cluster
because of an operating system or network fault. The most common cause
of
this message is that the local firewall is configured improperly.
Mar 28 14:55:12 corosync [MAIN  ] Totem is unable to form a cluster
because of an operating system or network fault. The most common cause
of
this message is that the local firewall is configured improperly.
==



This is result of ifdown. Just don't do that.

What exact version of corosync are you using?



Corosync version:
Corosync Cluster Engine, version '1.4.7'
Copyright (c) 2006-2009 Red Hat, Inc.






Pgsql resource (the postgresql resource agent) is running on this
particular node . I did a pgrep of the process and found it running. Not
attaching the logs for now.

The ³crash² happens when the ethernet interface is brought up. vm1cen66
is
unable to reconnect to the cluster because corosync has crashed, taking
some processes of pacemaker along with it.
crm_mon too stops working (it was working previously, before putting the
interface up)


I have to restart the corosync and pacemaker services to make it work
again.


That's why I keep saying don't do ifdown.




The main observation is that the node where the ethernet interface is
down, does not really ³get² it. It assumes that the other nodes are
still
online, although the logs do say that the interface is down.

Queries/Observations:
1- node vm1cen66 should realise that the other nodes are offline


That would be correct behavior, yes.


Does this mean that it should also realise that it does not have the
quorum and act according to the no-quorum-policy ?
Since we don’t have stonith hardware agents, this would be really useful


Are you sure that your server doesn't have IPMI?


for us. Do we have some other way of handling this ?


Not right now.






2- From the discussion in the github issue it seems that in case of
ethernet failure we want it to run as a single node setup. Is that so ?


Not exactly. It should behave like all other nodes gone down.


2a. If that is the case will it honour no-quorum-policy=ignore and stop
processes ?
2b. Or will it assume that it is a single node cluster and decided
accordingly ?
3- After doing an interface down, if we grep for the corosync port in
the
netstat command , we see that the corosync process has now bound the
loopback interface. Previously it was bound t

Re: [ClusterLabs] corosync-2.3.5 with ipv6 and multicast

[Jan Friesse]

Hi,
Thanks for your confirmation, and I 've two questions:

On 03/30/2016 05:11 PM, Jan Friesse wrote:



Hi, Honza
   I want to use corosync-2.3.5 with ipv6 and multicast, and I see
that I must specify nodeid in nodelist,
but I also see  nodeid in totem from totemconfig.c. Are they the same?


For given node it's same. Generally it's recommended to use nodelist
nodeid. It's better simply because you can then copy config file to
other nodes without changes.

Also if you specify both, warning is logged and nodelist nodeid is used.


Can I specify the nodeid in totem
instead of the on in nodelist.


You can but as long as you don't have really good reason (really
dynamic cluster without knowledge of nodes beforehand) it's better to
use node list.


If I use node list with multicast, should I add all nodes to the node
list? And if so, what's the difference between unicast and multicast in


It's better to add all nodes to node list (even it's probably going to 
work (somehow) without them).



this case except the sock addr?


Multicast should be more effective. I'm unsure what you mean by sock 
addr. If you mean mcastaddr, it's not needed if you set cluster_name and 
completely remove interface section (AFAIK pcs generated config use this 
style).


Regards,
  Honza




Regards,
  Honza



Thanks
Bin Liu

___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org



___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org




___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org



___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] corosync-2.3.5 with ipv6 and multicast

[Jan Friesse]

Hi, Honza
   I want to use corosync-2.3.5 with ipv6 and multicast, and I see
that I must specify nodeid in nodelist,
but I also see  nodeid in totem from totemconfig.c. Are they the same?


For given node it's same. Generally it's recommended to use nodelist 
nodeid. It's better simply because you can then copy config file to 
other nodes without changes.


Also if you specify both, warning is logged and nodelist nodeid is used.


Can I specify the nodeid in totem
instead of the on in nodelist.


You can but as long as you don't have really good reason (really dynamic 
cluster without knowledge of nodes beforehand) it's better to use node list.


Regards,
  Honza



Thanks
Bin Liu

___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org



___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] service network restart and corosync

[Jan Friesse]

Hi (Jan Friesse)

I studied the issue mentioned in the github url.
It looks the crash that I am talking about is slightly different from the
one mentioned in the original issue. May be they are related, but I would
like to
Highlight my setup for ease.

Three node cluster , one is in maintenance mode to prevent any scheduling
of resources.
=
Stack: classic openais (with plugin)


^^ I'm pretty sure you don't want to use plugin based pcmk.


Current DC: vm2cen66.mobileum.com - partition with quorum
Version: 1.1.11-97629de
3 Nodes configured, 3 expected votes
6 Resources configured


Node vm3cent66.mobileum.com: maintenance
Online: [ vm1cen66.mobileum.com vm2cen66.mobileum.com ]


I login to vm1cen66 and do `ifdown eth0`
In vm1cen66, I don¹t see any change in the crm_mon -Afr output.
It remains the same, as shown below

Stack: classic openais (with plugin)
Current DC: vm2cen66.mobileum.com - partition with quorum
Version: 1.1.11-97629de
3 Nodes configured, 3 expected votes
6 Resources configured


Node vm3cent66.mobileum.com: maintenance
Online: [ vm1cen66.mobileum.com vm2cen66.mobileum.com ]
===


But if we login to the other nodes like vm2cen66, vem3cent66, we can
correctly see that the node vm1cen66 is offline.


That is expected




But if we look into the corosync.log of vm1cen66 we see the following

===
Mar 28 14:55:09 corosync [MAIN  ] Totem is unable to form a cluster
because of an operating system or network fault. The most common cause of
this message is that the local firewall is configured improperly.
pgsql(TestPostgresql)[28203]:   2016/03/28_14:55:10 INFO: Master does not
exist.
pgsql(TestPostgresql)[28203]:   2016/03/28_14:55:10 WARNING: My data is
out-of-date. status=DISCONNECT
Mar 28 14:55:11 corosync [MAIN  ] Totem is unable to form a cluster
because of an operating system or network fault. The most common cause of
this message is that the local firewall is configured improperly.
Mar 28 14:55:12 corosync [MAIN  ] Totem is unable to form a cluster
because of an operating system or network fault. The most common cause of
this message is that the local firewall is configured improperly.
==



This is result of ifdown. Just don't do that.

What exact version of corosync are you using?



Pgsql resource (the postgresql resource agent) is running on this
particular node . I did a pgrep of the process and found it running. Not
attaching the logs for now.

The ³crash² happens when the ethernet interface is brought up. vm1cen66 is
unable to reconnect to the cluster because corosync has crashed, taking
some processes of pacemaker along with it.
crm_mon too stops working (it was working previously, before putting the
interface up)


I have to restart the corosync and pacemaker services to make it work
again.


That's why I keep saying don't do ifdown.




The main observation is that the node where the ethernet interface is
down, does not really ³get² it. It assumes that the other nodes are still
online, although the logs do say that the interface is down.

Queries/Observations:
1- node vm1cen66 should realise that the other nodes are offline


That would be correct behavior, yes.


2- From the discussion in the github issue it seems that in case of
ethernet failure we want it to run as a single node setup. Is that so ?


Not exactly. It should behave like all other nodes gone down.


2a. If that is the case will it honour no-quorum-policy=ignore and stop
processes ?
2b. Or will it assume that it is a single node cluster and decided
accordingly ?
3- After doing an interface down, if we grep for the corosync port in the
netstat command , we see that the corosync process has now bound the
loopback interface. Previously it was bound to the ip on eth0.
Is this expected ? As per the discussion it should be so. But the crash
did not happen immediately. It crashes when we bring the ethernet
interface up.


This is expected.


If the corosync did crash, why were we observing the logs in 
corosync.log
4- Is it possible to prevent the corosync crash that we witnessed when the
ethernet interface is brought up.


Nope. Just don't do ifdown.


5- Will preventing the corosync crash really matter ? Because the node
vm1cen66 has now converted into a single node cluster ? Or will it
automatically re-bind to eth0 when interface is brought up
(Could not verify because of the crash)


It's rebound to eth0, send wrong information to other nodes and totally 
destroy membership. Again, just don't do ifdown.



6- What about the split brain situation due to pacemaker not shutting down
the services on that single node ?
In a master-slave configuration this causes some confusion as to which
instance should be made a master after the node joins back.
As per the suggestion from the group , we need to configure stonith for
it. Configuring stonith seems to be the topmost priority in pacemaker
clusters.


It's not exactly topmost priority, but it's easy way

Re: [ClusterLabs] Pacemaker with Zookeeper??

[Jan Friesse]

Hi,

I have an idea: use Pacemaker with Zookeeper (instead of Corosync). Is
it possible?
Is there any examination about that?


From my point of view (and yes, I'm biased), biggest problem of 
Zookeper is need to have quorum 
(https://zookeeper.apache.org/doc/trunk/zookeeperAdmin.html#sc_designing). 
Direct consequence is inability to tolerate one node failure in 2 node 
cluster -> no 2 node clusters (and such deployment is extremely 
popular). Also Corosync can operate completely without quorum.


Regards,
  Honza



Thanks for your help!
Hai Nguyen





___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Pacemaker with Zookeeper??

[Jan Friesse]

Ken Gaillot napsal(a):

On 05/17/2016 09:54 AM, Digimer wrote:

On 16/05/16 04:35 AM, Bogdan Dobrelya wrote:

On 05/16/2016 09:23 AM, Jan Friesse wrote:

Hi,

I have an idea: use Pacemaker with Zookeeper (instead of Corosync). Is
it possible?
Is there any examination about that?


Indeed, would be *great* to have a Pacemaker based control plane on top
of other "pluggable" distributed KVS & messaging systems, for example
etcd as well :)
I'm looking forward to joining any dev efforts around that, although I'm
not a Java or Go developer.


Part of open source is the freedom to do whatever you want, of course.

Let me ask though; What problems would zookeeper, etcd or other systems
solve that can't be solved in corosync?

I ask because the HA community just finished a multi-year effort to
merge different projects into one common HA stack. This has a lot of
benefits to the user base, not least of which is lack of confusion.

Strikes me that the significant time investment in supporting a new
comms layer would be much more beneficially spent on improving the
existing stack.

Again, anyone is free to do whatever they want... I just don't see the
motivator personally.

digimer


I see one big difference that is both a strength and a weakness: these
other packages have a much wider user base beyond the HA cluster use
case. The strength is that there will be many more developers working to
fix bugs, add features, etc. The weakness is that most of those


This is exactly what I was thinking about during 2.x developement. If 
replacement of Corosync wouldn't make more sense than continue 
developing of Corosync. I was able to accept implementing some features. 
Sadly, there was exactly ONE project which would be able to replace 
corosync (Spread toolkit) which is even less widespread than Corosync.


From my point of view, replacement of corosync must be (at least) able to:
- Work without quorum
- Support 2 node clusters
- Allow multiple links (something like RRP)
- Don't include SPOF (so nothing like configuration stored on one node 
only and/or different machine on network)

- Provide EVS/VS
- Provide something like qdevice

Both zookeeper and etcd builds on top of quite simple to understand 
membership mechanism (zookeeper = elected master, something like amoeba, 
etcd = raft), what's nice, because it means more contributors. Sadly 
bare metal HA must work even in situations where "simple" quorum is not 
enough.




developers are ignorant of HA clustering and could easily cause more
problems for the HA use case than they fix.

Another potential benefit is the familiarity factor -- people are more
comfortable with things they recognize from somewhere else. So it might
help Pacemaker adoption, especially in the communities that already use
these packages.

I'm not aware of any technical advantages, and I wouldn't expect any,
given corosync's long HA focus.


 From my point of view (and yes, I'm biased), biggest problem of Zookeper
is need to have quorum
(https://zookeeper.apache.org/doc/trunk/zookeeperAdmin.html#sc_designing).
Direct consequence is inability to tolerate one node failure in 2 node
cluster -> no 2 node clusters (and such deployment is extremely
popular). Also Corosync can operate completely without quorum.

Regards,
   Honza



Thanks for your help!
Hai Nguyen


___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org




___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] [ClusterLab] : Corosync not initializing successfully

[Jan Friesse]

As your hardware is probably capable of running ppcle and if you have an
environment
at hand without too much effort it might pay off to try that.
There are of course distributions out there support corosync on
big-endian architectures
but I don't know if there is an automatized regression for corosync on
big-endian that
would catch big-endian-issues right away with something as current as
your 2.3.5.


No we are not testing big-endian.

So totally agree with Klaus. Give a try to ppcle. Also make sure all 
nodes are little-endian. Corosync should work in mixed BE/LE environment 
but because it's not tested, it may not work (and it's a bug, so if 
ppcle works I will try to fix BE).


Regards,
  Honza



Regards,
Klaus

On 05/02/2016 06:44 AM, Nikhil Utane wrote:

Re-sending as I don't see my post on the thread.

On Sun, May 1, 2016 at 4:21 PM, Nikhil Utane
> wrote:

 Hi,

 Looking for some guidance here as we are completely blocked
 otherwise :(.

 -Regards
 Nikhil

 On Fri, Apr 29, 2016 at 6:11 PM, Sriram > wrote:

 Corrected the subject.

 We went ahead and captured corosync debug logs for our ppc board.
 After log analysis and comparison with the sucessful logs(
 from x86 machine) ,
 we didnt find *"[ MAIN  ] Completed service synchronization,
 ready to provide service.*" in ppc logs.
 So, looks like corosync is not in a position to accept
 connection from Pacemaker.
 Even I tried with the new corosync.conf with no success.

 Any hints on this issue would be really helpful.

 Attaching ppc_notworking.log, x86_working.log, corosync.conf.

 Regards,
 Sriram



 On Fri, Apr 29, 2016 at 2:44 PM, Sriram > wrote:

 Hi,

 I went ahead and made some changes in file system(Like I
 brought in /etc/init.d/corosync and /etc/init.d/pacemaker,
 /etc/sysconfig ), After that I was able to run  "pcs
 cluster start".
 But it failed with the following error
  # pcs cluster start
 Starting Cluster...
 Starting Pacemaker Cluster Manager[FAILED]
 Error: unable to start pacemaker

 And in the /var/log/pacemaker.log, I saw these errors
 pacemakerd: info: mcp_read_config:  cmap connection
 setup failed: CS_ERR_TRY_AGAIN.  Retrying in 4s
 Apr 29 08:53:47 [15863] node_cu pacemakerd: info:
 mcp_read_config:  cmap connection setup failed:
 CS_ERR_TRY_AGAIN.  Retrying in 5s
 Apr 29 08:53:52 [15863] node_cu pacemakerd:  warning:
 mcp_read_config:  Could not connect to Cluster
 Configuration Database API, error 6
 Apr 29 08:53:52 [15863] node_cu pacemakerd:   notice:
 main: Could not obtain corosync config data, exiting
 Apr 29 08:53:52 [15863] node_cu pacemakerd: info:
 crm_xml_cleanup:  Cleaning up memory from libxml2


 And in the /var/log/Debuglog, I saw these errors coming
 from corosync
 20160429 085347.487050  airv_cu
 daemon.warn corosync[12857]:   [QB] Denied connection,
 is not ready (12857-15863-14)
 20160429 085347.487067  airv_cu
 daemon.info  corosync[12857]:   [QB
 ] Denied connection, is not ready (12857-15863-14)


 I browsed the code of libqb to find that it is failing in

 https://github.com/ClusterLabs/libqb/blob/master/lib/ipc_setup.c

 Line 600 :
 handle_new_connection function

 Line 637:
 if (auth_result == 0 &&
 c->service->serv_fns.connection_accept) {
 res = c->service->serv_fns.connection_accept(c,
  c->euid, c->egid);
 }
 if (res != 0) {
 goto send_response;
 }

 Any hints on this issue would be really helpful for me to
 go ahead.
 Please let me know if any logs are required,

 Regards,
 Sriram

 On Thu, Apr 28, 2016 at 2:42 PM, Sriram
 > wrote:

 Thanks Ken and Emmanuel.
 Its a big endian machine. I will try with running "pcs
 cluster setup" and "pcs cluster start"
 Inside cluster.py, "service pacemaker start" and
 "service corosync start" are executed to bring up
 pacemaker and corosync.
 Those service scripts and the infrastructure needed 

Re: [ClusterLabs] [ClusterLab] : Corosync not initializing successfully

[Jan Friesse]

Nikhil


Found the root-cause.
In file schedwrk.c, the function handle2void() uses a union which was not
initialized.
Because of that the handle value was computed incorrectly (lower half was
garbage).

  56 static hdb_handle_t
  57 void2handle (const void *v) { union u u={}; u.v = v; return u.h; }
  58 static const void *
  59 handle2void (hdb_handle_t h) { union u u={}; u.h = h; return u.v; }

After initializing (as highlighted), the corosync initialization seems to
be going through fine. Will check other things.


Your patch is incorrect and actually doesn't work. As I said (when 
pointing you to schedwrk.c), I will send you proper patch, but fix that 
issue correctly is not easy.


Regards,
  Honza



-Regards
Nikhil

On Tue, May 3, 2016 at 7:04 PM, Nikhil Utane <nikhil.subscri...@gmail.com>
wrote:


Thanks for your response Dejan.

I do not know yet whether this has anything to do with endianness.
FWIW, there could be something quirky with the system so keeping all
options open. :)

I added some debug prints to understand what's happening under the hood.

*Success case: (on x86 machine): *
[TOTEM ] entering OPERATIONAL state.
[TOTEM ] A new membership (10.206.1.7:137220) was formed. Members joined:
181272839
[TOTEM ] Nikhil: Inside messages_deliver_to_app. end_point=0,
my_high_delivered=0
[TOTEM ] Nikhil: Inside messages_deliver_to_app. end_point=1,
my_high_delivered=0
[TOTEM ] Delivering 0 to 1
[TOTEM ] Delivering MCAST message with seq 1 to pending delivery queue
[SYNC  ] Nikhil: Inside sync_deliver_fn. header->id=1
[TOTEM ] Nikhil: Inside messages_deliver_to_app. end_point=2,
my_high_delivered=1
[TOTEM ] Delivering 1 to 2
[TOTEM ] Delivering MCAST message with seq 2 to pending delivery queue
[SYNC  ] Nikhil: Inside sync_deliver_fn. header->id=0
[SYNC  ] Nikhil: Entering sync_barrier_handler
[SYNC  ] Committing synchronization for corosync configuration map access
.
[TOTEM ] Delivering 2 to 4
[TOTEM ] Delivering MCAST message with seq 3 to pending delivery queue
[TOTEM ] Delivering MCAST message with seq 4 to pending delivery queue
[CPG   ] comparing: sender r(0) ip(10.206.1.7) ; members(old:0 left:0)
[CPG   ] chosen downlist: sender r(0) ip(10.206.1.7) ; members(old:0
left:0)
[SYNC  ] Committing synchronization for corosync cluster closed process
group service v1.01
*[MAIN  ] Completed service synchronization, ready to provide service.*


*Failure case: (on ppc)*:
[TOTEM ] entering OPERATIONAL state.
[TOTEM ] A new membership (10.207.24.101:16) was formed. Members joined:
181344357
[TOTEM ] Nikhil: Inside messages_deliver_to_app. end_point=0,
my_high_delivered=0
[TOTEM ] Nikhil: Inside messages_deliver_to_app. end_point=1,
my_high_delivered=0
[TOTEM ] Delivering 0 to 1
[TOTEM ] Delivering MCAST message with seq 1 to pending delivery queue
[SYNC  ] Nikhil: Inside sync_deliver_fn header->id=1
[TOTEM ] Nikhil: Inside messages_deliver_to_app. end_point=1,
my_high_delivered=1
[TOTEM ] Nikhil: Inside messages_deliver_to_app. end_point=1,
my_high_delivered=1
Above message repeats continuously.

So it appears that in failure case I do not receive messages with sequence
number 2-4.
If somebody can throw some ideas that'll help a lot.

-Thanks
Nikhil

On Tue, May 3, 2016 at 5:26 PM, Dejan Muhamedagic <deja...@fastmail.fm>
wrote:


Hi,

On Mon, May 02, 2016 at 08:54:09AM +0200, Jan Friesse wrote:

As your hardware is probably capable of running ppcle and if you have

an

environment
at hand without too much effort it might pay off to try that.
There are of course distributions out there support corosync on
big-endian architectures
but I don't know if there is an automatized regression for corosync on
big-endian that
would catch big-endian-issues right away with something as current as
your 2.3.5.


No we are not testing big-endian.

So totally agree with Klaus. Give a try to ppcle. Also make sure all
nodes are little-endian. Corosync should work in mixed BE/LE
environment but because it's not tested, it may not work (and it's a
bug, so if ppcle works I will try to fix BE).


I tested a cluster consisting of big endian/little endian nodes
(s390 and x86-64), but that was a while ago. IIRC, all relevant
bugs in corosync got fixed at that time. Don't know what is the
situation with the latest version.

Thanks,

Dejan


Regards,
   Honza



Regards,
Klaus

On 05/02/2016 06:44 AM, Nikhil Utane wrote:

Re-sending as I don't see my post on the thread.

On Sun, May 1, 2016 at 4:21 PM, Nikhil Utane
<nikhil.subscri...@gmail.com <mailto:nikhil.subscri...@gmail.com>>

wrote:


 Hi,

 Looking for some guidance here as we are completely blocked
 otherwise :(.

 -Regards
 Nikhil

 On Fri, Apr 29, 2016 at 6:11 PM, Sriram <sriram...@gmail.com
 <mailto:sriram...@gmail.com>> wrote:

 Corrected the subject.

 We went ahead and captured corosync debug logs for our ppc

board.

 After log analysis and comparison with the sucessful l

[ClusterLabs] Corosync 2.4.1 is available at corosync.org!

[Jan Friesse]

I am pleased to announce the latest maintenance release of Corosync
2.4.1 available immediately from our website at
http://build.clusterlabs.org/corosync/releases/.

This release contains fix for one regression and few more smaller fixes.

During 2.3.6 development the bug which is causing pacemaker to not work 
after corosync configuration file is reloaded happened. Solution is 
ether to use this fixed version (recommended) or as a quick workaround 
(for users who wants to stay on 2.3.6 or 2.4.0) is to create file 
pacemaker (file name can be arbitrary) in /etc/corosync/uidgid.d 
directory with following content (you can also put same stanza into 
/etc/corosync/corosync.conf):


uidgid {
gid: haclient
}
 CUT HERE ---

If pacemaker is using different group, replace haclient with the name of 
that group.


Complete changelog for 2.4.1:

Bin Liu (1):
  cts: Make it run with pacemaker-1.13+

Christine Caulfield (2):
  qdevice: Fix 'tie_breaker' in man page
  qdevice: some more small man page fixes

HideoYamauchi (1):
  Low: totemsrp: Addition of the log.

Jan Friesse (2):
  Config: Flag config uidgid entries
  Spec: Qdevice require same version of corosync

Upgrade is (as usually) highly recommended.

Thanks/congratulations to all people that contributed to achieve this
great milestone.


___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] changing pacemaker.log location

[Jan Friesse]

Ken Gaillot napsal(a):

On 08/12/2016 10:19 AM, Christopher Harvey wrote:

I'm surprised I'm having such a hard time figuring this out on my own.
I'm running pacemaker 1.1.13 and corosync-2.3.4 and want to change the
location of pacemaker.log.

By default it is located in /var/log.

I looked in corosync.c and found the following lines:
 get_config_opt(config, local_handle, KEY_PREFIX "to_logfile",
 _enabled, "on");
 get_config_opt(config, local_handle, KEY_PREFIX "logfile",
 , "/var/log/pacemaker.log");
in mcp_read_config

I can't find any other documentation.

Here is my corosync.conf file.

totem {
   version: 2
   # Need a cluster name for now:
   #   https://github.com/corosync/corosync/issues/137
   cluster_name: temp
   crypto_cipher: aes256
   crypto_hash: sha512

   interface {
 ringnumber: 0
 bindnetaddr: 192.168.132.10
 mcastport: 5405
   }
   transport: udpu
   heartbeat_failures_allowed: 3
}

nodelist {
   node {
 ring0_addr: 192.168.132.25
 nodeid: 1
 name: a
   }

   node {
 ring0_addr: 192.168.132.21
 nodeid: 2
 name: b
   }

   node {
 ring0_addr: 192.168.132.10
 nodeid: 3
 name: c
   }
}

logging {
   # Log the source file and line where messages are being
   # generated. When in doubt, leave off. Potentially useful for
   # debugging.
   fileline: on
   # Log to standard error. When in doubt, set to no. Useful when
   # running in the foreground (when invoking 'corosync -f')
   to_stderr: no
   # Log to a log file. When set to 'no', the 'logfile' option
   # must not be set.
   to_logfile: yes
   logfile: /my/new/location/corosync.log


By default, pacemaker will use the same log file as corosync, so this
should be sufficient.


btw. this is something we have to fix. Basically what happens now (if I 
didn't overlooked something) is that pcmk is opening same file so it can 
be overwriting whatever corosync logs because corosync is using 
non-synced threaded logging.


Honza



Alternatively, you can explicitly tell Pacemaker what detail log file to
use with the environment variable PCMK_logfile (typically set in a
distro-specific location such as /etc/sysconfig/pacemaker or
/etc/default/pacemaker).


   # Log to the system log daemon. When in doubt, set to yes.
   to_syslog: yes
   # Log debug messages (very verbose). When in doubt, leave off.
   debug: off
   # Log messages with time stamps. When in doubt, set to on
   # (unless you are only logging to syslog, where double
   # timestamps can be annoying).
   timestamp: on
   logger_subsys {
 subsys: QUORUM
 debug: off
   }
}
quorum {
   provider: corosync_votequorum
   expected_votes: 3
}

Thanks,
Chris


___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org




___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] [corosync] [build] the configure shows UNKNOWN version and runtime exits 17

[Jan Friesse]

Hello.
I use this guide [0] to build libqb, corosync, pacemaker
and test them as pid-space linked docker containers [1].

A Pacemaker builds OK and shows the v1.1.15 runtime, a build-time it
complains about an unknown libqb version. I workarounded it by running a
Pacemaker build with these env vars set:

export PREFIX=/usr
export PKG_CONFIG_PATH=${PREFIX}/lib/pkgconfig
export libqb_CFLAGS="-I${PREFIX}/include"
export libqb_LIBS="-L${PREFIX}/lib"


But corosync ./configure shows an unknown version:

corosync configuration:
Version = UNKNOWN


https://github.com/corosync/corosync/issues/116

Use official signed tarbal ( 
http://build.clusterlabs.org/corosync/releases/corosync-2.3.6.tar.gz) or 
full git clone. Do not use github "releases".


Regards,
  Honza



Although it builds, runtime it throws the exit code 17 on me, which I
failed to google / understand from sources. Any help with that, folks?
Thank you!

[0] http://clusterlabs.org/wiki/SourceInstall
[1]
https://github.com/docker/docker/blob/master/docs/reference/run.md#pid-settings---pid




___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

[ClusterLabs] Corosync 2.4.0 is available at corosync.org!

[Jan Friesse]

I am pleased to announce the latest release of Corosync
2.4.0 available immediately from our website at
http://build.clusterlabs.org/corosync/releases/.

This release is mostly about long awaited QDevice feature and few rather 
small fixes.


Qdevice is complete rewrite of old cman qdisk using network arbiter 
(instead of disk) so it's very similar to Linux HA quorum daemon/HP 
Serviceguard Quorum Server/...


Qdevice currently consists of two daemons:

- corosync-qdevice is a daemon running on each node of a cluster. It 
provides a configured number of votes to the quorum subsystem based  on 
a third-party arbitrator’s decision. Its primary use is to allow a 
cluster to sustain more node failures than standard quorum rules allow. 
It is recommended  for  clusters  with an even number of nodes and 
highly recommended for 2 node clusters.


- corosync-qnetd is a daemon running outside of the cluster with the 
purpose of providing a vote to the  corosync-qdevice model net. It’s

designed to support multiple clusters and be almost configuration and
state free. New clusters are handled dynamically and  no  configuration
file exists. It’s also able to run as non-root user - which is 
recommended. Connection between the corosync-qdevice model net client 
can be optionally configured with TLS client certificate checking. The 
communication protocol between server and client is designed to be very 
simple and allow backwards compatibility.


To compile corosync-qdevice/corosync-qnetd, configure.sh has to be 
invoked with --enable-qdevices/--enable-qnetd switches.


To find out how to configure qdevice/qnetd take a look to man pages 
corosync-qdevice (8) and corosync-qnetd (8).


Please note that because of required changes in votequorum, 
libvotequorum is no longer binary compatible. This is reason for version 
bump.


Starting with this release 2.3 branch becomes unsupported and officially 
supported Needle is 2.4 branch. Just a note, this doesn't affect support 
of Flatiron where nothing changes and 1.4 branch is still supported.


Changelog for fixes for 2.4.0 (qdevices commits not included):

Ferenc Wágner (8):
  cmap_track_add.3.in: fix typo: bellow -> below
  Fix typo: funtion -> function
  Fix typo: interger -> integer
  Fix typo: Uknown -> Unknown
  Fix typo: aquire -> acquire
  Fix typo: retrive -> retrieve
  Fix typo: alocated -> allocated
  Fix typo: Diabled -> disabled

Jan Friesse (1):
  config: get_cluster_mcast_addr error is not fatal

bliu (1):
  low:typo fix in sam.h

Upgrade is (more than usually) highly recommended.

Thanks/congratulations to all people that contributed to achieve this
great milestone.


___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Corosync IPC unix socket path

[Jan Friesse]

Rytis,


Hi,

We are using corosync (currently v2.3.5) for group communication
services, and we would like to package it to separate container (all the
service-per-container stuff), thus requiring to forward IPC. Is it
possible to set unix socket path for corosync IPC?


Sadly nope. Corosync is using libqb IPC and libqb is using abstract 
sockets for Linux/Cygwin (for other OSes it is using regular named 
socket stored in /var/run).


I've created bug report https://github.com/ClusterLabs/libqb/issues/244.

Regards,
  Honza


Thanks in advance.

Best Regards,
Rytis Karpuška



___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org




___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Corosync maximum nodes

[Jan Friesse]

Hello!
I'm very sorry to disturb you with such question but I can't find
information if there is maximum nodes' limit in corosync? I've found a bug
report https://bugzilla.redhat.com/show_bug.cgi?id=905296#c5 with "Corosync
has hardcoded maximum number of nodes to 64" but it was posted 4 years
ago..


This limit is gone in 2.x, but it doesn't mean corosync is able to 
handle much more without finetuning.



If anybody knows how many nodes I can add to future HA cluster?


http://lists.clusterlabs.org/pipermail/users/2017-January/004764.html

But honestly, as Chrissie told, pacemaker-remote is usually better way 
to go (as long as you are not planning to use dlm on all nodes).


Honza



Any tips or links would be much appreciated.
Thank you!



___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org




___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Corosync ring shown faulty between healthy nodes & networks (rrp_mode: passive)

[Jan Friesse]

Martin Schlegel napsal(a):

Thanks for the confirmation Jan, but this sounds a bit scary to me !

Spinning this experiment a bit further ...

Would this not also mean that with a passive rrp with 2 rings it only takes 2
different nodes that are not able to communicate on different networks at the
same time to have all rings marked faulty on _every_node ... therefore all
cluster members loosing quorum immediately even though n-2 cluster members are
technically able to send and receive heartbeat messages through all 2 rings ?


Not exactly, but this situation causes corosync to start behaving really 
badly spending most of the time in "creating new membership" loop.


Yes, RRP is simply bad. If you can, use bonding. Improvement of RRP by 
replace it by knet is biggest TODO for 3.x.


Regards,
  Honza



I really hope the answer is no and the cluster still somehow has a quorum in
this case.

Regards,
Martin Schlegel



Jan Friesse <jfrie...@redhat.com> hat am 5. Oktober 2016 um 09:01 geschrieben:

Martin,


Hello all,

I am trying to understand why the following 2 Corosync heartbeat ring
failure
scenarios
I have been testing and hope somebody can explain why this makes any sense.

Consider the following cluster:

  * 3x Nodes: A, B and C
  * 2x NICs for each Node
  * Corosync 2.3.5 configured with "rrp_mode: passive" and
  udpu transport with ring id 0 and 1 on each node.
  * On each node "corosync-cfgtool -s" shows:
  [...] ring 0 active with no faults
  [...] ring 1 active with no faults

Consider the following scenarios:

  1. On node A only block all communication on the first NIC configured with
ring id 0
  2. On node A only block all communication on all NICs configured with
ring id 0 and 1

The result of the above scenarios is as follows:

  1. Nodes A, B and C (!) display the following ring status:
  [...] Marking ringid 0 interface  FAULTY
  [...] ring 1 active with no faults
  2. Node A is shown as OFFLINE - B and C display the following ring status:
  [...] ring 0 active with no faults
  [...] ring 1 active with no faults

Questions:
  1. Is this the expected outcome ?


Yes


2. In experiment 1. B and C can still communicate with each other over both
NICs, so why are
  B and C not displaying a "no faults" status for ring id 0 and 1 just like
in experiment 2.


Because this is how RRP works. RRP marks whole ring as failed so every
node sees that ring as failed.


when node A is completely unreachable ?


Because it's different scenario. In scenario 1 there are 3 nodes
membership where one of them has failed one ring -> whole ring is
failed. In scenario 2 there are 2 nodes membership where both rings
works as expected. Node A is completely unreachable and it's not in the
membership.

Regards,
  Honza


Regards,
Martin Schlegel

___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org






___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org




___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] corosync-quorum tool, output name key on Name column if set?

[Jan Friesse]

Thomas Lamprecht napsal(a):

On 09/20/2016 12:36 PM, Christine Caulfield wrote:

On 20/09/16 10:46, Thomas Lamprecht wrote:

Hi,

when I'm using corosync-quorumtool [-l] and have my ring0_addr set to a
IP address,
which does not resolve to a hostname, I get the nodes IP addresses for
the 'Name' column.

As I'm using the nodelist.node.X.name key to set the name of a node it
seems a bit confusing
to me that not this one gets preferred or at least also outputted. Its
quite a minor issue if
not nit picking but as I associate my nodes with there name I.

I'd be ready to assemble a patch and one possibility would be adapting
the output to something
like:


# corosync-quorumtool

Quorum information
--
Date: Tue Sep 20 11:12:14 2016
Quorum provider:  corosync_votequorum
Nodes:3
Node ID:  1
Ring ID:  1/1784
Quorate:  Yes

Votequorum information
--
Expected votes:   3
Highest expected: 3
Total votes:  3
Quorum:   2
Flags:Quorate

Membership information
--
 Nodeid  Votes Name ring0_addr
  1  1 uno  10.10.20.1 (local)
  2  1 due  10.10.20.2
  3  1 tre  10.10.20.3


And respective:


# corosync-quorumtool -l

Membership information
--
 Nodeid  Votes Name ring0_addr
  1  1 uno  10.10.20.1 (local)
  2  1 due  10.10.20.2
  3  1 tre  10.10.20.3

additional ring1_addr could be also outputted if set.

This would be just a general idea, if there are suggestions I'll gladly
hear them.

As such a change may be not ideal during a stable release, e.g as
corosync user could
parse the corosync-quorumtool output (I mean there are quite better
places to get the
info but there may be still user doing this)  another possibility would
be adding an
option flag to corosync similar to '-i' (show node IP addresses instead
of the resolved
name) which then shows the nodelist.node.X.name value instead of IP or
resolved name.

Another, third, option would be letting the output as is but if the '-i'
option is not
set prefer the nodelist.node.X.name over the resolved hostname and fall
back to IP if
both are not available.
I'd almost prefer this change the most, it lets the output as it is and
it seems logical
that the Name column outputs the name key if possible, imo.

Would such a patch be welcomed or is this just something I find a little
strange?

Hi Tomas,

I'd be happy to receive such a patch. The main reason it's not done this
way is that it's not always obvious how to resolve a name from it's IP
address. If corosync.conf has a nodelist then using that does seem like
the best option though (and bear in mind that more than 1 ring is
possible). If corosync.conf is set up to use multicast then we have no
choice to guess at what the name might be (as happens now).

Most of corosync-quorumtool was written when nodelist was not the
dominant way of configuring a cluster which is why it is the way it is
at the moment.

As to what should be the default and which options are most useful, I
would be interested to hear the views of the community as to what they
would like to see :)

Chrissie


Hi Chrissie,

Thanks for your answer!



Thomas,


OK, then I'll look into it a bit more and try to figure out which options
really could make sense, if earlier code hadn't the nodelist configuration
wasn't that dominant, I may find other places too where it could be used
for
outputs or as input parameter.

You mean if corosync.conf is setup without nodelist, just with multicast?


I believe so. Simply because nodelist is pretty new concept (2.x).


As with the nodelist section configured multicast is also possible, asking
just to understand you correctly.

Yes, would be nice to get some input here, I'll wait a bit, else I'll
send a
patch to get the discussion going. :)

I have also another, organizational question. I saw on the GitHub page from
corosync that pull request there are preferred, and also that the


True


communication should occur through GitHub, so should I use GitHub
instead of
the clusterlabs list for mails like my initial one from this thread?


Nope, not necessarily. Way "First discuss on ML then send PR" works very 
same as "Open issue, discuss and then send PR". Actually from time to 
time ML way is better because you get broader audience.


Regards,
  Honza



Thanks,
Thomas



___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org



___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: 

Re: [ClusterLabs] permissions under /etc/corosync/qnetd

[Jan Friesse]

Ferenc Wágner napsal(a):

Jan Friesse <jfrie...@redhat.com> writes:


Ferenc Wágner napsal(a):


Have you got any plans/timeline for 2.4.2 yet?


Yep, I'm going to release it in few minutes/hours.


Man, that was quick.  I've got a bunch of typo fixes queued..:) Please
consider announcing upcoming releases a couple of days in advance; as a
packager, I'd much appreciate it.  Maybe even tag release candidates...


We are tagging RC of big releases. This release was super small (eventho 
breaking compatibility). And actually, it's better to be released rather 
sooner than later. I will definitively think about RC of smaller 
releases (2.3.x -> 2.4.x).




Anyway, I've got a question concerning corosync-qnetd.  I run it as
user and group coroqnetd.  Is granting it read access to cert8.db and
key3.db enough for proper operation?  corosync-qnetd-certutil gives


Should be, but it's not tested.


write access to group coroqnetd to everything, which seems unintuitive


Yep. Idea is to allow scenario of qnetd administrator role. So basically 
regular (non-root) user within coroqnetd group without root passwd 
knowledge/sudo administering qnetd.




to me.  Please note that I've got zero experience with NSS.  But I don't
expect the daemon to change the certificate database.  Should I?


Nope it shouldn't.

Regards,
  Honza






___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

[ClusterLabs] Corosync 2.4.2 is available at corosync.org!

[Jan Friesse]

I am pleased to announce the latest maintenance release of Corosync
2.4.2 available immediately from our website at
http://build.clusterlabs.org/corosync/releases/.

This release is mainly because we forgot to bump libvotequorum.so major 
version number in 2.4.0. This is not that big deal because libvotequorum 
isn't used by 3rd party applications (pacemaker, ...). Still makes sense 
to have this issue fixed. Also thanks to Ferenc Wágner for notice.



Complete changelog for 2.4.2:
Christine Caulfield (1):
  man: mention qdevice incompatibilites in votequorum.5

Fabio M. Di Nitto (1):
  [build] Fix build on RHEL7.3 latest

Jan Friesse (3):
  Man: Fix corosync-qdevice-net-certutil link
  Qnetd LMS: Fix two partition use case
  libvotequorum: Bump version

Michael Jones (1):
  cfg: Prevents use of uninitialized buffer


Upgrade is (as usually) highly recommended.

Thanks/congratulations to all people that contributed to achieve this
great milestone.


___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Corosync 2.4.0 is available at corosync.org!

[Jan Friesse]

Ferenc Wágner napsal(a):

Jan Friesse <jfrie...@redhat.com> writes:


Jan Friesse <jfrie...@redhat.com> writes:


Please note that because of required changes in votequorum,
libvotequorum is no longer binary compatible. This is reason for
version bump.


Er, what version bump?  Corosync 2.4.1 still produces
libvotequorum.so.7.0.0 for me, just like Corosync 2.3.6.


Yep, you are right. Thanks for notice, this is something what should
have happened.


Thanks for confirming.


Anyway, 2.3.6 and 2.4.x votequorum are incompatible (there were both
API and ABI changes). Probably something to fix in 2.4.2.


Have you got any plans/timeline for 2.4.2 yet?


Yep, I'm going to release it in few minutes/hours.



Anyway, we're packaging 2.4.1 for Debian now, shall we ship it with

-7.0.0
+8.0.0

in lib/libvotequorum.verso?




___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Antw: Re: Antw: Re: Antw: Re: When the DC crmd is frozen, cluster decisions are delayed infinitely

[Jan Friesse]

On 10/14/2016 11:21 AM, renayama19661...@ybb.ne.jp wrote:

Hi Klaus,
Hi All,

I tried prototype of watchdog using WD service.
  - 
https://github.com/HideoYamauchi/pacemaker/commit/3ee97b76e0212b1790226864dfcacd1a327dbcc9

Please comment.

Thank you Hideo for providing the prototype.
Added the patch to my build and it seems to
be working as expected.

A few thoughts triggered by this approach:

- we have to alert the corosync-people as in
   a chat with Jan Friesse he pointed me to the
   fact that for corosync 3.x the wd-service was
   planned to be removed


Actually I didn't express myself correctly. What I wanted to say was 
"I'm considering idea of removing it", simply because it's disabled in 
downstream.


BUT keep in mind that removing functionality = ask community to find out 
if there is not somebody actively using it.


And because there is active users and future use case, removing of wd is 
not an option.





   especially delicate as the binding is very loose
   so that - as is - it builds against a corosync with
   disabled wd-service without any complaints...

- as of now if you enable wd-service in the
   corosync-build it is on by default and would
   be hogging the watchdog presumably
   (there is obviously a pull request that makes
   it default to off)

- with my thoughts about adding an API to
   sbd previously in the thread I was trying to
   target closer observation of pacemaker_remoted
   as well (remote-nodes don't have corosync
   running)

   I guess it would be possible to run corosync
   with a static config as single-node cluster
   bound to localhost for that purpose.

   I read the thread about corosync-remote and
   that happening might make the special-handling
   for pacemaker-remote obsolete anyway ...

- to enable the approach to live alongside
   sbd it would be possible to make sbd use
   the corosync-API as well for watchdog purposes
   instead of opening the watchdog directly

   This shouldn't be a big deal for sbd used to
   observe a pacemaker-node as cluster-watcher
   (the part of sbd that sends cpg-pings to corosync)
   already builds against corosync.
   The blockdevice-part of sbd being basically
   generic it might be an issue though.

Regards,
Klaus




Best Regards,
Hideo Yamauchi.


- Original Message -

From: "renayama19661...@ybb.ne.jp" <renayama19661...@ybb.ne.jp>
To: "users@clusterlabs.org" <users@clusterlabs.org>
Cc:
Date: 2016/10/11, Tue 17:58
Subject: Re: [ClusterLabs] Antw: Re: Antw: Re: Antw: Re: When the DC crmd is 
frozen, cluster decisions are delayed infinitely

Hi Klaus,

Thank you for comment.

I make the patch which is prototype using WD service.

Please wait a little.

Best Regards,
Hideo Yamauchi.




- Original Message -

  From: Klaus Wenninger <kwenn...@redhat.com>
  To: users@clusterlabs.org
  Cc:
  Date: 2016/10/10, Mon 21:03
  Subject: Re: [ClusterLabs] Antw: Re: Antw: Re: Antw: Re: When the DC crmd

is frozen, cluster decisions are delayed infinitely

  On 10/07/2016 11:10 PM, renayama19661...@ybb.ne.jp wrote:

   Hi All,

   Our user may not necessarily use sdb.

   I confirmed that there was a method using WD service of corosync as

one

  method not to use sdb.

   Pacemaker watches the process of pacemaker by WD service using CMAP

and can

  carry out watchdog.

  Have to have a look at that...
  But if we establish some in-between-layer in pacemaker we could have this
  as one of the possibilities besides e.g. sbd (with enhanced API), going for
  a watchdog-device directly, ...



   We can set up a patch of pacemaker.

  Always helpful to discuss/clarify an idea once some code is available ...


   Was the discussion of using WD service over so far?

  Not from my pov. Just a day off ;-)



   Best Regard,
   Hideo Yamauchi.


   - Original Message -

   From: Klaus Wenninger <kwenn...@redhat.com>
   To: Ulrich Windl <ulrich.wi...@rz.uni-regensburg.de>;

  users@clusterlabs.org

   Cc:
   Date: 2016/10/7, Fri 17:47
   Subject: Re: [ClusterLabs] Antw: Re: Antw: Re: Antw: Re: When the

DC

  crmd is frozen, cluster decisions are delayed infinitely

   On 10/07/2016 08:14 AM, Ulrich Windl wrote:

Klaus Wenninger <kwenn...@redhat.com>

schrieb am

   06.10.2016 um 18:03 in

Nachricht

<3980cfdd-ebd9-1597-f6bd-a1ca808f7...@redhat.com>:

On 10/05/2016 04:22 PM, renayama19661...@ybb.ne.jp wrote:

Hi All,


If a user uses sbd, can the cluster evade a

  problem of

   SIGSTOP of crmd?


As pointed out earlier, maybe crmd should feed a

  watchdog. Then

   stopping

crmd

will reboot the node (unless the watchdog fails).

Thank you for comment.

We examine watchdog of crmd, too.
In addition, I comment after examination advanced.

Was thinking of doing a small test implementation going
a little in the direction Lars Ellenberg had been

pointing

  out.

a couple of thoughts

Re: [ClusterLabs] cross DC cluster using public ip?

[Jan Friesse]

neeraj ch napsal(a):

Hello ,

We are testing out corosync and pacemaker for DB high availability on the
cloud. I was able to set up a cluster with in a DC using corosync 1.4 and
pacemaker 1.12. It works great and I wanted to try a cross DC cluster. I
was using unicast as multicast was disabled by default.

I was not sure how Corosync behaves with public IP's but I still went ahead
and tried it with both public IP's as well as DNS names. These DNS names
resolve as local IP when the other node is with in the same subnet.


Every node has to be able to see every other node. So mixing of public 
and private ips is not going to work (with exception of special case 
where all private ips are in the same network). Also keep in mind config 
file has to be same on all nodes.





while I was using public IP's both the node inside the same subnet as well
as outside were unable to connect, except for itself. While using DNS names
the membership information showed the nodes within same subnet being
connected to while the nodes outside were not connected


This is somehow expected.



My corosync config is as follows.

totem {
   version: 2
   secauth: off
   threads: 0
   interface {

member {
   memberaddr: 
}
   member {
   memberaddr: 
}
member {
   memberaddr: 
}
ringnumber: 0
bindnetaddr: 172.31.0.0
mcastport: 5405
ttl: 1
   }
   transport: udpu
}

logging {
   fileline: off
   to_stderr: no
   to_logfile: yes
   to_syslog: yes
   logfile: /var/log/cluster/corosync.log
   debug: on
   timestamp: on
   logger_subsys {
subsys: AMF
debug: on
   }
}

service {
 # Load the Pacemaker Cluster Resource Manager
 name: pacemaker
 ver: 1
}

amf {
   mode: disabled
}


I am checking membership information by using corosync-objctl. I have also
tried using public ip as the bind address , that makes the membership from


Just to make sure. This "public" ip is really ip of given machine?


1 to 0 as it doesn't add itself.

If any one has any suggestion / advice on how to debug or what I am doing
wrong . Any help would be very appreciated.

Thank you



___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org




___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] large cluster with corosync

[Jan Friesse]

Arne Jansen napsal(a):



On 04.01.2017 11:25, Kristoffer Grönlund wrote:

Arne Jansen  writes:


Hi,

I've built corosync for solaris and am trying to build a largish
cluster. I started corosync with default configuration on an
increasing number of nodes, one by one. At around 70 nodes the
cluster breaks down. Below is an excerpt from the logfile on the
first node.
When the cluster breaks down corosync seems to completely flood
the network. It doesn't recover by itself, I have to stop all nodes.
Things get worse if I start corosync on multiple nodes at once.
In this case it already breaks down around 40 nodes.

Is it supposed to work with such a setup? Does it just need tuning?
My goal is have a cluster of several hundred nodes.


Hi,

No, corosync has a limit to how many nodes the cluster can
contain and still function properly. According to SUSE, the limit is 32
nodes, according to Red Hat the limit is 16 (those are the limits for
enterprise support as far as I know) - but even having that number
of nodes will probably require some tweaking of timeouts in
corosync.conf, depending on what kind of network you have.

If I recall correctly, there are plans for overcoming this limit in the
future but right now that's the situation.

There is the pacemaker_remote project which allows for adding
additional, non-corosync nodes to a cluster which can run resources but
don't participate in quorum and rely on at least some of the core
cluster nodes still being available. Using pacemaker_remote could enable
a cluster of several hundred nodes.

My question would be, why do you need so many nodes in a high
availaibility cluster?


At least those limits doesn't seem to get enforced, as a 64 node cluster
seems to work, although a bit shaky.


No, they are not enforced. 16/32 are official supported number of nodes. 
Basically, this is number what was tested and known to work reliably. 
This doesn't mean corosync doesn't work with bigger number of nodes. 
Eventho I'm quite surprised that 64 nodes really works.


Variables you can try tweak.
- Definitively start with increase totem.config (default 1000, you can 
try 1)
- If it doesn't help, try increase totem.join (default is 50, 1000 may 
work) and consider increase totem.send_join (default is 0, 100 may be 
good idea).
- As a last variable, increase of totem.merge (default is 200, 2000 may 
do the job).


And definitively let us know about results. It's quite hard to test such 
a big amount of nodes so some of the variable may be sub-optimal. When 
we know which of variables are victims, we can change their defaults.


Regards,
  Honza


I want to use corosync + dlm to get a distributed lock manager to have
a uniform locking service throughout the platform as a basis for many
services.

Thanks,
Arne



Cheers,
Kristoffer



Thanks,
Arne

Jan 04 10:24:49 [4915] reniar corosync notice  [MAIN  ] Completed
service synchronization, ready to provide service.
Jan 04 10:25:00 [4915] reniar corosync notice  [TOTEM ] A new membership
(10.5.4.101:616) was formed. Members joined: 168101089
Jan 04 10:25:00 [4915] reniar corosync notice  [MAIN  ] Completed
service synchronization, ready to provide service.
Jan 04 10:25:11 [4915] reniar corosync notice  [TOTEM ] A new membership
(10.5.4.101:620) was formed. Members joined: 168101090
Jan 04 10:25:11 [4915] reniar corosync notice  [MAIN  ] Completed
service synchronization, ready to provide service.
Jan 04 10:25:21 [4915] reniar corosync notice  [TOTEM ] A new membership
(10.5.4.101:624) was formed. Members joined: 168101091
Jan 04 10:25:21 [4915] reniar corosync notice  [MAIN  ] Completed
service synchronization, ready to provide service.
Jan 04 10:25:32 [4915] reniar corosync notice  [TOTEM ] A new membership
(10.5.4.101:628) was formed. Members joined: 168101092
Jan 04 10:25:32 [4915] reniar corosync notice  [MAIN  ] Completed
service synchronization, ready to provide service.
Jan 04 10:26:09 [4915] reniar corosync notice  [TOTEM ] A new membership
(10.5.4.101:632) was formed. Members joined: 168101141
Jan 04 10:26:10 [4915] reniar corosync notice  [MAIN  ] Completed
service synchronization, ready to provide service.
Jan 04 10:26:20 [4915] reniar corosync notice  [TOTEM ] A new membership
(10.5.4.101:636) was formed. Members joined: 168101142
Jan 04 10:26:20 [4915] reniar corosync notice  [MAIN  ] Completed
service synchronization, ready to provide service.
Jan 04 10:26:31 [4915] reniar corosync notice  [TOTEM ] A new membership
(10.5.4.101:640) was formed. Members joined: 168101143
Jan 04 10:26:31 [4915] reniar corosync notice  [MAIN  ] Completed
service synchronization, ready to provide service.
Jan 04 10:26:42 [4915] reniar corosync notice  [TOTEM ] A new membership
(10.5.4.101:644) was formed. Members joined: 168101144
Jan 04 10:26:42 [4915] reniar corosync notice  [MAIN  ] Completed
service synchronization, ready to provide service.
Jan 04 10:26:52 [4915] reniar corosync notice  [TOTEM ] A new 

Re: [ClusterLabs] Buffer overflow (re-transmission list)

[Jan Friesse]

Hello,

I have a four node cluster. Each node connected with a centralized switch.
MTU size is default, 1500. On each node, a program continuously tries to
multi-cast as many messages as possible. With the default settings
(corosync.conf), buffer overflow does *not* occur till program runs on
three nodes. However as soon as fourth node start multi-casting, overflow
occur and significantly reduce the performance.

Why buffer overflow with just four nodes?

Is hardware topology, centralized switch, is not correct?


Centralized switch is ok.



Later, I reduced window_size and max_messages to 20 and 5 respectively. No
overflow but not sure whether performance is as expected.

I would like to better understand these two parameters. Lets say in a
cluster of four nodes, I have window_size 50 and max_messages also set to
50. Does that mean only one node will be able to multi-cast in a single
token rotation? If one node was not able to multi-cast because window_size


Not exactly. Token contains "backlog" field which is updated by all 
members. This backlog is then used to find out if there is not a member 
who wasn't able to send messages (whose backlog is too big). So let's 
say one node sent a lot messages in one round, and second node sent no 
messages. In next round, first node finds out this information and its 
number of allowed messages to sent is lowered (or in extreme case goes

to zero) so second node can send some messages.

Regards,
  Honza


is reached, when and how this node will get opportunity to send message?

Thanks,
Satish



___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org




___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] large cluster with corosync

[Jan Friesse]

Arne Jansen napsal(a):



On 04.01.2017 13:52, Jan Friesse wrote:


Variables you can try tweak.
- Definitively start with increase totem.config (default 1000, you can
try 1)


what does that do? Haven't found it in corosync.conf(5)


Sorry, typo. I meant totem.token.




- If it doesn't help, try increase totem.join (default is 50, 1000 may
work) and consider increase totem.send_join (default is 0, 100 may be
good idea).


The problem is indeed the flood of join messages overwhelming the
receive socket. I increased the socket size in source code from 160k


Ok. I'm thinking if it would make sense to make this define user 
configurable or probably even better somehow depending on number of 
configured nodes.



to 3MB and forming the cluster works fine for 80 nodes. send_join
looks like a very useful variable to avoid the need for excessively
large buffers, so I'll test with join=120ms and send_join=60ms.


Sounds good.

Honza



-Arne


- As a last variable, increase of totem.merge (default is 200, 2000 may
do the job).

And definitively let us know about results. It's quite hard to test such
a big amount of nodes so some of the variable may be sub-optimal. When
we know which of variables are victims, we can change their defaults.

Regards,
  Honza


I want to use corosync + dlm to get a distributed lock manager to have
a uniform locking service throughout the platform as a basis for many
services.

Thanks,
Arne



Cheers,
Kristoffer



Thanks,
Arne

Jan 04 10:24:49 [4915] reniar corosync notice  [MAIN  ] Completed
service synchronization, ready to provide service.
Jan 04 10:25:00 [4915] reniar corosync notice  [TOTEM ] A new
membership
(10.5.4.101:616) was formed. Members joined: 168101089
Jan 04 10:25:00 [4915] reniar corosync notice  [MAIN  ] Completed
service synchronization, ready to provide service.
Jan 04 10:25:11 [4915] reniar corosync notice  [TOTEM ] A new
membership
(10.5.4.101:620) was formed. Members joined: 168101090
Jan 04 10:25:11 [4915] reniar corosync notice  [MAIN  ] Completed
service synchronization, ready to provide service.
Jan 04 10:25:21 [4915] reniar corosync notice  [TOTEM ] A new
membership
(10.5.4.101:624) was formed. Members joined: 168101091
Jan 04 10:25:21 [4915] reniar corosync notice  [MAIN  ] Completed
service synchronization, ready to provide service.
Jan 04 10:25:32 [4915] reniar corosync notice  [TOTEM ] A new
membership
(10.5.4.101:628) was formed. Members joined: 168101092
Jan 04 10:25:32 [4915] reniar corosync notice  [MAIN  ] Completed
service synchronization, ready to provide service.
Jan 04 10:26:09 [4915] reniar corosync notice  [TOTEM ] A new
membership
(10.5.4.101:632) was formed. Members joined: 168101141
Jan 04 10:26:10 [4915] reniar corosync notice  [MAIN  ] Completed
service synchronization, ready to provide service.
Jan 04 10:26:20 [4915] reniar corosync notice  [TOTEM ] A new
membership
(10.5.4.101:636) was formed. Members joined: 168101142
Jan 04 10:26:20 [4915] reniar corosync notice  [MAIN  ] Completed
service synchronization, ready to provide service.
Jan 04 10:26:31 [4915] reniar corosync notice  [TOTEM ] A new
membership
(10.5.4.101:640) was formed. Members joined: 168101143
Jan 04 10:26:31 [4915] reniar corosync notice  [MAIN  ] Completed
service synchronization, ready to provide service.
Jan 04 10:26:42 [4915] reniar corosync notice  [TOTEM ] A new
membership
(10.5.4.101:644) was formed. Members joined: 168101144
Jan 04 10:26:42 [4915] reniar corosync notice  [MAIN  ] Completed
service synchronization, ready to provide service.
Jan 04 10:26:52 [4915] reniar corosync notice  [TOTEM ] A new
membership
(10.5.4.101:648) was formed. Members joined: 168101145
Jan 04 10:26:52 [4915] reniar corosync notice  [MAIN  ] Completed
service synchronization, ready to provide service.
Jan 04 10:27:03 [4915] reniar corosync notice  [TOTEM ] A new
membership
(10.5.4.101:652) was formed. Members joined: 168101146
Jan 04 10:27:03 [4915] reniar corosync notice  [MAIN  ] Completed
service synchronization, ready to provide service.
Jan 04 10:27:14 [4915] reniar corosync notice  [TOTEM ] A new
membership
(10.5.4.101:656) was formed. Members joined: 168101147
Jan 04 10:27:14 [4915] reniar corosync notice  [MAIN  ] Completed
service synchronization, ready to provide service.
Jan 04 10:27:25 [4915] reniar corosync notice  [TOTEM ] A new
membership
(10.5.4.101:660) was formed. Members joined: 168101148
Jan 04 10:27:25 [4915] reniar corosync notice  [MAIN  ] Completed
service synchronization, ready to provide service.
Jan 04 10:27:35 [4915] reniar corosync notice  [TOTEM ] A new
membership
(10.5.4.101:664) was formed. Members joined: 168101161
Jan 04 10:27:35 [4915] reniar corosync notice  [MAIN  ] Completed
service synchronization, ready to provide service.
Jan 04 10:27:46 [4915] reniar corosync notice  [TOTEM ] A new
membership
(10.5.4.101:668) was formed. Members joined: 168101162
Jan 04 10:27:46 [4915] reniar corosync notice  [MAIN  ] Completed
service

Re: [ClusterLabs] wireshark cannot recognize corosync packets

[Jan Friesse]

I have checked all the config files are the same, except bindnetaddr.
So I'm sending only logs.


I'm not sure if config files matches log files. Because config file 
contains nodes 200.201.162.(52|53|54), but log files contains ip 
200.201.162.(52|53|55).


Can you confirm node with ip 200.201.162.54 exists and it shouldn't be 
200.201.162.55 (or 200.201.162.55 shouldn't have ip 200.201.162.54)?


Honza








在2017年03月16 15时54分, "Jan Friesse"<jfrie...@redhat.com>写道:


corosync.conf and debug logs are in attachment.


Thanks for them. They look really interesting. As can be seen

Mar 14 11:37:28 [57827] node-132.acloud.vt corosync debug   [TOTEM ]
timer_function_orf_token_timeout The token was lost in the
  OPERATIONAL state.

corosync correctly detected token lost. Also

Mar 14 11:44:41 [57827] node-132.acloud.vt corosync debug   [TOTEM ]
memb_state_gather_enter entering GATHER state from 11(merg
e during join).

says it correctly detected merge. But since then it's becoming weird.
Mar 14 11:44:54 [57827] node-132.acloud.vt corosync debug   [TOTEM ]
memb_state_gather_enter entering GATHER state from 0(conse
nsus timeout).
Mar 14 11:45:06 [57827] node-132.acloud.vt corosync debug   [TOTEM ]
memb_state_gather_enter entering GATHER state from 0(conse
nsus timeout).
...
Mar 14 12:55:47 [154709] node-132.acloud.vt corosync debug   [TOTEM ]
memb_state_gather_enter entering GATHER state from 0(cons
ensus timeout)

So even after two other nodes merged, there is still something what
prevents corosync to reach consensus.

Would it be possible to attach also other nodes logs/configs?

For now I guess reason can be one ofe:
- ifdown on one of other nodes which made whole membership broken
- different node list in config between nodes
- "forget" node with node list containing one of the 200.201.162.x nodes

Regards,
   Honza


And two messages from kernel:

2017-03-14 11:37:20.097233 - info  e1000: eth0 NIC Link is Down

2017-03-14 11:44:41.032121 - info  e1000: eth0 NIC Link is Up 1000 Mbps
Full Duplex, Flow Control: RX


Thanks.


On 2017/3/15 16:29, Jan Friesse wrote:

Yesterday I found corosync took almost one hour to form a cluster(a
failed node came back online).


This for sure shouldn't happen (at least with default timeout settings).



So I captured some corosync packets, and opened the pcap file in
wireshark.

But wireshark only displayed raw udp, no totem.

Wireshark version is 2.2.5. I'm sure it supports corosync totem.

corosync is 2.4.0.


Wireshark has corosync dissector, but only for version 1.x. 2.x is not
supported yet.



And if corosync takes too long to form a cluster, how to diagnose it?

I read the logs, but could not figure it out.


Logs, specially when debug is enabled, has usually enough info. Can
paste your config + logs?

Regards,
   Honza



Thanks.



___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org



___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org




___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org




___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org



___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org




___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] wireshark cannot recognize corosync packets

[Jan Friesse]

Yesterday I found corosync took almost one hour to form a cluster(a
failed node came back online).


This for sure shouldn't happen (at least with default timeout settings).



So I captured some corosync packets, and opened the pcap file in wireshark.

But wireshark only displayed raw udp, no totem.

Wireshark version is 2.2.5. I'm sure it supports corosync totem.

corosync is 2.4.0.


Wireshark has corosync dissector, but only for version 1.x. 2.x is not 
supported yet.




And if corosync takes too long to form a cluster, how to diagnose it?

I read the logs, but could not figure it out.


Logs, specially when debug is enabled, has usually enough info. Can 
paste your config + logs?


Regards,
  Honza



Thanks.



___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org



___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Three node cluster becomes completely fenced if one node leaves

[Jan Friesse]

The original message has the logs from nodes 1 and 3. Node 2, the one that
got fenced in this test, doesn't really show much. Here are the logs from
it:

Mar 24 16:35:10 b014 ntpd[2318]: Deleting interface #5 enp6s0f0,
192.168.100.14#123, interface stats: received=0, sent=0, dropped=0,
active_time=3253 secs
Mar 24 16:35:10 b014 ntpd[2318]: Deleting interface #7 enp6s0f0,
fe80::a236:9fff:fe8a:6500%6#123, interface stats: received=0, sent=0,
dropped=0, active_time=3253 secs
Mar 24 16:35:13 b014 corosync[2166]: notice  [TOTEM ] A processor failed,
forming new configuration.
Mar 24 16:35:13 b014 corosync[2166]:  [TOTEM ] A processor failed, forming
new configuration.
Mar 24 16:35:13 b014 corosync[2166]: notice  [TOTEM ] The network interface
is down.


This is problem. Corosync handles ifdown really badly. If this was not 
intentional it may be caused by NetworkManager. Then please install 
equivalent of NetworkManager-config-server package (it's actually one 
file called 00-server.conf so you can extract it from, for example, 
Fedora package 
https://www.rpmfind.net/linux/RPM/fedora/devel/rawhide/x86_64/n/NetworkManager-config-server-1.8.0-0.1.fc27.noarch.html)


Regards,
  Honza


Mar 24 16:35:13 b014 corosync[2166]: notice  [TOTEM ] adding new UDPU
member {192.168.100.13}
Mar 24 16:35:13 b014 corosync[2166]: notice  [TOTEM ] adding new UDPU
member {192.168.100.14}
Mar 24 16:35:13 b014 corosync[2166]: notice  [TOTEM ] adding new UDPU
member {192.168.100.15}
Mar 24 16:35:13 b014 corosync[2166]:  [TOTEM ] The network interface is
down.
Mar 24 16:35:13 b014 corosync[2166]:  [TOTEM ] adding new UDPU member
{192.168.100.13}
Mar 24 16:35:13 b014 corosync[2166]:  [TOTEM ] adding new UDPU member
{192.168.100.14}
Mar 24 16:35:13 b014 corosync[2166]:  [TOTEM ] adding new UDPU member
{192.168.100.15}

---
Seth Reid



On Wed, Mar 29, 2017 at 7:17 AM, Bob Peterson  wrote:


- Original Message -
| I will try to install updated packages from ubuntu 16.10 or newer. It
can't
| get worse than not working.
|
| Can you think of any logs that might help? I've enabled debug on corosync
| log, but it really doesn't show anything else other than corosync
exiting.
| Any diagnostic tools you can recommend?
|
| ---
| Seth Reid


Hi Seth,

Can you post the pertinent messages from the consoles of all nodes in the
cluster? Hopefully you were monitoring them.

Regards,

Bob Peterson
Red Hat File Systems

___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org





___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org




___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Ubuntu 16.04 - 2 node setup

[Jan Friesse]

James,


Hey guys,


Apologies for burdening you with my issue, but I'm at my wits' end!


I'm trying to set up a 2-node cluster on two Ubuntu 16.04 VMs. I actually had 
this working earlier, but because I had tweaked a number of different settings 
(both corosync related and external settings), I reverted my VMs back to an 
earlier checkpoint to ensure I wasn't just running off a luck 'magic config' 
and I could replicate my setup... turns out, I can't!


The config for nodes is as follows:


```

totem {
   version: 2
   cluster_name: swarm
   transport: udpu
   interface {
 ringnumber: 0
 bindnetaddr: 10.172.0.0
   }


^^^ Try completely remove interface section


}

nodelist {
   node {
 ring0_addr: 10.172.0.81
 name: SWARM01


^^^ What name should do?


 nodeid: 1
   }
   node {
 ring0_addr: 10.172.0.82
 name: SWARM02
 nodeid: 2
   }
}

quorum {
   provider: corosync_votequorum
   two_node: 1
}

logging {
   to_logfile: yes
   to_syslog: yes
   logfile: /var/log/corosync/corosync.log
   timestamp: on
}
```



This time around, when I first run corosync with `systemctl start corosync` it 
comes up using 127.0.0.1.


Apr 12 20:28:37 SWARM01 corosync[6025]:   [TOTEM ] Initializing 
transmit/receive security (NSS
Apr 12 20:28:37 SWARM01 corosync[6025]:   [TOTEM ] The network interface 
[127.0.0.1] is now up
Apr 12 20:28:37 SWARM01 corosync[6025]:   [QB] server name: cmap
Apr 12 20:28:37 SWARM01 corosync[6025]:   [QB] server name: cfg
Apr 12 20:28:37 SWARM01 corosync[6025]:   [QB] server name: cpg
Apr 12 20:28:37 SWARM01 corosync[6025]:   [QB] server name: votequorum
Apr 12 20:28:37 SWARM01 corosync[6025]:   [QB] server name: quorum
Apr 12 20:28:37 SWARM01 corosync[6025]:   [TOTEM ] A new membership 
(127.0.0.1:4) was formed.

Results from `sudo corosync-quorumtool`

Quorum information
--
Date: Wed Apr 12 20:31:12 2017
Quorum provider:  corosync_votequorum
Nodes:1
Node ID:  2130706433
Ring ID:  4
Quorate:  No

Votequorum information
--
Expected votes:   2
Highest expected: 2
Total votes:  1
Quorum:   2 Activity blocked
Flags:

Membership information
--
 Nodeid  Votes Name
2130706433  1 localhost (local)


This really shouldn't happen



And results from `sudo corosync-cmapctl | grep members`
runtime.totem.pg.mrp.srp.members.2130706433.config_version (u64) = 0
runtime.totem.pg.mrp.srp.members.2130706433.ip (str) = r(0) ip(127.0.0.1)
runtime.totem.pg.mrp.srp.members.2130706433.join_count (u32) = 1
runtime.totem.pg.mrp.srp.members.2130706433.status (str) = joined

It's also not using the correct node number (Should be 1 or 2 depending on 
which node I try it on). Then if I try to restart the service, it just fails 
and doesn't log anything in /var/log/corosync/corosync.log.


This means corosync.conf is probably ignored.



Apr 12 20:52:01 SWARM01 systemd[1]: Starting Corosync Cluster Engine...
Apr 12 20:52:01 SWARM01 systemd[1]: corosync.service: Main process exited, 
code=exited, status=8/n/a
Apr 12 20:52:01 SWARM01 systemd[1]: Failed to start Corosync Cluster Engine.
Apr 12 20:52:01 SWARM01 systemd[1]: corosync.service: Unit entered failed state.
Apr 12 20:52:01 SWARM01 systemd[1]: corosync.service: Failed with result 
'exit-code'.

The only way I can get to test this again is by completely removing corosync 
(apt remove --purge corosync), reinstalling and trying again. I've tried 
disabling the firewall completely to see if that was interfering, but to me, 
it's as if corosync isn't respecting my config file this time around?


Yes it looks so.



Any guidance at all would be greatly appreciated!


Can you please attach your config file? Is corosync from ubuntu package 
or your own compiled one? can you please try to stop corosync service, 
login as root, add "to_stderr: yes" into logging section of config file, 
execute "corosync -f" and paste result?


Regards,
  Honza






James Booth
Senior ICT Technician
Email: james.bo...@primarytec.co.uk
Mobile: +44 07725817464

[http://www.primarytec.co.uk/wp-content/uploads/2016/11/signature.png]

Registered in England. No 04760864. Registered office as above

The information transmitted is intended only for the person or entity to which 
it is addressed and may contain confidential and/or privileged material. Any 
review, retransmission, dissemination or other use of, or taking of any action 
in reliance upon, this information by persons or entities other than the 
intended recipient is prohibited.
If you received this in error, please contact the sender and delete the 
material from any computer.



___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: 

Re: [ClusterLabs] corosync totem.token too long may cause pacemaker(cluster) unstable?

[Jan Friesse]

cys napsal(a):

At 2017-03-08 17:10:36, "Jan Friesse" <jfrie...@redhat.com> wrote:

Hi,
We changed totem.token from 3s to 60s. Then something strange were observed, 
such as unexpected node offline.
I read corosync.conf manpage, but still don't understand the reason.
Can anyone explain this? or maybe our conf is broken?


What corosync version are you using? Your config file is not exactly
"broken", but I'm not sure how exactly all the settings play together,
so maybe minimize it. Setting high token timeout shouldn't cause
problems, only failure detection takes long time.




Corosync Cluster Engine, version '2.4.0'
Copyright (c) 2006-2009 Red Hat, Inc.

Sometimes corosync took very long time to be in operational state. I'm not sure 
whether it's related with totem.token.


It can be caused by token timeout + consensus timeout.


Could you tell me how to diagnose it?


Basically try following config file:

 CUT HERE ---

quorum {
 provider: corosync_votequorum
 two_node: 0
}
totem {
version: 2

crypto_cipher: none
crypto_hash: none

transport: udpu
cluster_name: ClusterName
}

nodelist {
 node {
 ring0_addr: 39.39.0.4
 nodeid: 1
 }
 node {
 ring0_addr: 39.39.0.5
 nodeid: 2
 }
 node {
 ring0_addr: 39.39.0.6
 nodeid: 3
 }
}

--- CUT HERE ---

Replace ClusterName with something more clever and give it a try. This 
is almost minimal corosync.conf with default values and it usually works 
quite well. If you observe too much lost tokens, try increase token 
(timeout) but 60 seconds is too much. 10 should do the job.


Regards,
  Honza



Thanks.
___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org




___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] corosync totem.token too long may cause pacemaker(cluster) unstable?

[Jan Friesse]

Hi,
We changed totem.token from 3s to 60s. Then something strange were observed, 
such as unexpected node offline.
I read corosync.conf manpage, but still don't understand the reason.
Can anyone explain this? or maybe our conf is broken?


What corosync version are you using? Your config file is not exactly 
"broken", but I'm not sure how exactly all the settings play together, 
so maybe minimize it. Setting high token timeout shouldn't cause 
problems, only failure detection takes long time.





Our corosync.conf:

compatibility: whitetank
quorum {
 provider: corosync_votequorum
 two_node: 0
}
totem {
 version: 2
 token: 6
 token_retransmits_before_loss_const: 10
 join: 60
 consensus: 3600
 vsftype: none
 max_messages: 20
 clear_node_high_bit: yes
 rrp_mode: none
 secauth: off
 threads: 2
 transport: udpu
 interface {
 ringnumber: 0
 bindnetaddr: 39.39.0.5
 mcastport: 5405
}
}
amf {
 mode: disabled
}
aisexec {
 user: root
 group: root
}
nodelist {
 node {
 ring0_addr: 39.39.0.4
 nodeid: 1
 }
 node {
 ring0_addr: 39.39.0.5
 nodeid: 2
 }
 node {
 ring0_addr: 39.39.0.6
 nodeid: 3
 }
}
___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org




___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Updated attribute is not displayed in crm_mon

[Jan Friesse]

Ken Gaillot napsal(a):

On Mon, 2017-08-14 at 12:33 -0500, Ken Gaillot wrote:

On Wed, 2017-08-02 at 09:59 +, 井上 和徳 wrote:

Hi,

In Pacemaker-1.1.17, the attribute updated while starting pacemaker is not 
displayed in crm_mon.
In Pacemaker-1.1.16, it is displayed and results are different.

https://github.com/ClusterLabs/pacemaker/commit/fe44f400a3116a158ab331a92a49a4ad8937170d
This commit is the cause, but the following result (3.) is expected behavior?


This turned out to be an odd one. The sequence of events is:

1. When the node leaves the cluster, the DC (correctly) wipes all its
transient attributes from attrd and the CIB.

2. Pacemaker is newly started on the node, and a transient attribute is
set before the node joins the cluster.

3. The node joins the cluster, and its transient attributes (including
the new value) are sync'ed with the rest of the cluster, in both attrd
and the CIB. So far, so good.

4. Because this is the node's first join since its crmd started, its
crmd wipes all of its transient attributes again. The idea is that the
node may have restarted so quickly that the DC hasn't yet done it (step
1 here), so clear them now to avoid any problems with old values.
However, the crmd wipes only the CIB -- not attrd (arguably a bug).


Whoops, clarification: the node may have restarted so quickly that
corosync didn't notice it left, so the DC would never have gotten the


Corosync always notice left of node no matter if left is longer or 
within token timeout.



"peer lost" message that triggers wiping its transient attributes.

I suspect the crmd wipes only the CIB in this case because we assumed
attrd would be empty at this point -- missing exactly this case where a
value was set between start-up and first join.


5. With the older pacemaker version, both the joining node and the DC
would request a full write-out of all values from attrd. Because step 4
only wiped the CIB, this ends up restoring the new value. With the newer
pacemaker version, this step is no longer done, so the value winds up
staying in attrd but not in CIB (until the next write-out naturally
occurs).

I don't have a solution yet, but step 4 is clearly the problem (rather
than the new code that skips step 5, which is still a good idea
performance-wise). I'll keep working on it.


[test case]
1. Start pacemaker on two nodes at the same time and update the attribute 
during startup.
In this case, the attribute is displayed in crm_mon.

[root@node1 ~]# ssh -f node1 'systemctl start pacemaker ; attrd_updater -n 
KEY -U V-1' ; \
ssh -f node3 'systemctl start pacemaker ; attrd_updater -n 
KEY -U V-3'
[root@node1 ~]# crm_mon -QA1
Stack: corosync
Current DC: node3 (version 1.1.17-1.el7-b36b869) - partition with quorum

2 nodes configured
0 resources configured

Online: [ node1 node3 ]

No active resources


Node Attributes:
* Node node1:
+ KEY   : V-1
* Node node3:
+ KEY   : V-3


2. Restart pacemaker on node1, and update the attribute during startup.

[root@node1 ~]# systemctl stop pacemaker
[root@node1 ~]# systemctl start pacemaker ; attrd_updater -n KEY -U V-10


3. The attribute is registered in attrd but it is not registered in CIB,
so the updated attribute is not displayed in crm_mon.

[root@node1 ~]# attrd_updater -Q -n KEY -A
name="KEY" host="node3" value="V-3"
name="KEY" host="node1" value="V-10"

[root@node1 ~]# crm_mon -QA1
Stack: corosync
Current DC: node3 (version 1.1.17-1.el7-b36b869) - partition with quorum

2 nodes configured
0 resources configured

Online: [ node1 node3 ]

No active resources


Node Attributes:
* Node node1:
* Node node3:
+ KEY   : V-3


Best Regards

___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org







___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Two nodes cluster issue

[Jan Friesse]

Tomer Azran napsal(a):

I tend to agree with Klaus – I don't think that having a hook that
bypass stonith is the right way. It is better to not use stonith at all.
I think I will try to use an iScsi target on my qdevice and set SBD to
use it.
I still don't understand why qdevice can't take the place SBD with
shared storage; correct me if I'm wrong, but it looks like both of
them are there for the same reason.


Qdevice is there to be third side arbiter who decides which partition is
quorate. It can also be seen as a quorum only node. So for two node
cluster it can be viewed as a third node (eventho it is quite special
because it cannot run resources). It is not doing fencing.

SBD is fencing device. It is using disk as a third side arbiter.


I've talked with Klaus and he told me that 7.3 is not using disk as a 
third side arbiter so sorry for confusion.


You should however still be able to use sbd for checking if pacemaker is 
alive and if the partition has quorum - otherwise the watchdog kills the 
node. So qdevice will give you "3rd" node and sbd fences unquorate 
partition.


Or (as mentioned previously) you can use fabric fencing.

Regards,
  Honza






From: Klaus Wenninger [mailto:kwenn...@redhat.com]
Sent: Monday, July 24, 2017 9:01 PM
To: Cluster Labs - All topics related to open-source clustering
welcomed ; Prasad, Shashank 
Subject: Re: [ClusterLabs] Two nodes cluster issue

On 07/24/2017 07:32 PM, Prasad, Shashank wrote:
Sometimes IPMI fence devices use shared power of the node, and it
cannot be avoided.
In such scenarios the HA cluster is NOT able to handle the power
failure of a node, since the power is shared with its own fence device.
The failure of IPMI based fencing can also exist due to other reasons
also.

A failure to fence the failed node will cause cluster to be marked
UNCLEAN.
To get over it, the following command needs to be invoked on the
surviving node.

pcs stonith confirm  --force

This can be automated by hooking a recovery script, when the the
Stonith resource ‘Timed Out’ event.
To be more specific, the Pacemaker Alerts can be used for watch for
Stonith timeouts and failures.
In that script, all that’s essentially to be executed is the
aforementioned command.

If I get you right here you can disable fencing then in the first place.
Actually quorum-based-watchdog-fencing is the way to do this in a
safe manner. This of course assumes you have a proper source for
quorum in your 2-node-setup with e.g. qdevice or using a shared
disk with sbd (not directly pacemaker quorum here but similar thing
handled inside sbd).


Since the alerts are issued from ‘hacluster’ login, sudo permissions
for ‘hacluster’ needs to be configured.

Thanx.


From: Klaus Wenninger [mailto:kwenn...@redhat.com]
Sent: Monday, July 24, 2017 9:24 PM
To: Kristián Feldsam; Cluster Labs - All topics related to open-source
clustering welcomed
Subject: Re: [ClusterLabs] Two nodes cluster issue

On 07/24/2017 05:37 PM, Kristián Feldsam wrote:
I personally think that power off node by switched pdu is more safe,
or not?

True if that is working in you environment. If you can't do a physical
setup
where you aren't simultaneously loosing connection to both your node and
the switch-device (or you just want to cover cases where that happens)
you have to come up with something else.




S pozdravem Kristián Feldsam
Tel.: +420 773 303 353, +421 944 137 535
E-mail.: supp...@feldhost.cz

www.feldhost.cz - FeldHost™ – profesionální
hostingové a serverové služby za adekvátní ceny.

FELDSAM s.r.o.
V rohu 434/3
Praha 4 – Libuš, PSČ 142 00
IČ: 290 60 958, DIČ: CZ290 60 958
C 200350 vedená u Městského soudu v Praze

Banka: Fio banka a.s.
Číslo účtu: 2400330446/2010
BIC: FIOBCZPPXX
IBAN: CZ82 2010  0024 0033 0446

On 24 Jul 2017, at 17:27, Klaus Wenninger
> wrote:

On 07/24/2017 05:15 PM, Tomer Azran wrote:
I still don't understand why the qdevice concept doesn't help on this
situation. Since the master node is down, I would expect the quorum to
declare it as dead.
Why doesn't it happens?

That is not how quorum works. It just limits the decision-making to
the quorate subset of the cluster.
Still the unknown nodes are not sure to be down.
That is why I suggested to have quorum-based watchdog-fencing with sbd.
That would assure that within a certain time all nodes of the
non-quorate part
of the cluster are down.






On Mon, Jul 24, 2017 at 4:15 PM +0300, "Dmitri Maziuk"
> wrote:

On 2017-07-24 07:51, Tomer Azran wrote:


We don't have the ability to use it.



Is that the only solution?




No, but I'd recommend thinking about it first. Are you sure you will

care about your cluster working when your server room is on fire? 'Cause

unless you have halon suppression, your server room is a complete

write-off anyway. (Think water 

Re: [ClusterLabs] Two nodes cluster issue

[Jan Friesse]

Tomer Azran napsal(a):

I tend to agree with Klaus – I don't think that having a hook that bypass 
stonith is the right way. It is better to not use stonith at all.
I think I will try to use an iScsi target on my qdevice and set SBD to use it.
I still don't understand why qdevice can't take the place SBD with shared 
storage; correct me if I'm wrong, but it looks like both of them are there for 
the same reason.


Qdevice is there to be third side arbiter who decides which partition is 
quorate. It can also be seen as a quorum only node. So for two node 
cluster it can be viewed as a third node (eventho it is quite special 
because it cannot run resources). It is not doing fencing.


SBD is fencing device. It is using disk as a third side arbiter.




From: Klaus Wenninger [mailto:kwenn...@redhat.com]
Sent: Monday, July 24, 2017 9:01 PM
To: Cluster Labs - All topics related to open-source clustering welcomed 
; Prasad, Shashank 
Subject: Re: [ClusterLabs] Two nodes cluster issue

On 07/24/2017 07:32 PM, Prasad, Shashank wrote:
Sometimes IPMI fence devices use shared power of the node, and it cannot be 
avoided.
In such scenarios the HA cluster is NOT able to handle the power failure of a 
node, since the power is shared with its own fence device.
The failure of IPMI based fencing can also exist due to other reasons also.

A failure to fence the failed node will cause cluster to be marked UNCLEAN.
To get over it, the following command needs to be invoked on the surviving node.

pcs stonith confirm  --force

This can be automated by hooking a recovery script, when the the Stonith 
resource ‘Timed Out’ event.
To be more specific, the Pacemaker Alerts can be used for watch for Stonith 
timeouts and failures.
In that script, all that’s essentially to be executed is the aforementioned 
command.

If I get you right here you can disable fencing then in the first place.
Actually quorum-based-watchdog-fencing is the way to do this in a
safe manner. This of course assumes you have a proper source for
quorum in your 2-node-setup with e.g. qdevice or using a shared
disk with sbd (not directly pacemaker quorum here but similar thing
handled inside sbd).


Since the alerts are issued from ‘hacluster’ login, sudo permissions for 
‘hacluster’ needs to be configured.

Thanx.


From: Klaus Wenninger [mailto:kwenn...@redhat.com]
Sent: Monday, July 24, 2017 9:24 PM
To: Kristián Feldsam; Cluster Labs - All topics related to open-source 
clustering welcomed
Subject: Re: [ClusterLabs] Two nodes cluster issue

On 07/24/2017 05:37 PM, Kristián Feldsam wrote:
I personally think that power off node by switched pdu is more safe, or not?

True if that is working in you environment. If you can't do a physical setup
where you aren't simultaneously loosing connection to both your node and
the switch-device (or you just want to cover cases where that happens)
you have to come up with something else.




S pozdravem Kristián Feldsam
Tel.: +420 773 303 353, +421 944 137 535
E-mail.: supp...@feldhost.cz

www.feldhost.cz - FeldHost™ – profesionální hostingové 
a serverové služby za adekvátní ceny.

FELDSAM s.r.o.
V rohu 434/3
Praha 4 – Libuš, PSČ 142 00
IČ: 290 60 958, DIČ: CZ290 60 958
C 200350 vedená u Městského soudu v Praze

Banka: Fio banka a.s.
Číslo účtu: 2400330446/2010
BIC: FIOBCZPPXX
IBAN: CZ82 2010  0024 0033 0446

On 24 Jul 2017, at 17:27, Klaus Wenninger 
> wrote:

On 07/24/2017 05:15 PM, Tomer Azran wrote:
I still don't understand why the qdevice concept doesn't help on this 
situation. Since the master node is down, I would expect the quorum to declare 
it as dead.
Why doesn't it happens?

That is not how quorum works. It just limits the decision-making to the quorate 
subset of the cluster.
Still the unknown nodes are not sure to be down.
That is why I suggested to have quorum-based watchdog-fencing with sbd.
That would assure that within a certain time all nodes of the non-quorate part
of the cluster are down.






On Mon, Jul 24, 2017 at 4:15 PM +0300, "Dmitri Maziuk" 
> wrote:

On 2017-07-24 07:51, Tomer Azran wrote:


We don't have the ability to use it.



Is that the only solution?




No, but I'd recommend thinking about it first. Are you sure you will

care about your cluster working when your server room is on fire? 'Cause

unless you have halon suppression, your server room is a complete

write-off anyway. (Think water from sprinklers hitting rich chunky volts

in the servers.)



Dima



___

Users mailing list: Users@clusterlabs.org

http://lists.clusterlabs.org/mailman/listinfo/users



Project Home: http://www.clusterlabs.org

Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf

Bugs: 

Re: [ClusterLabs] Corosync CPU load slowly increasing if one node present

[Jan Friesse]

Stefan,


Hello everyone!

I am using Pacemaker (1.1.12), Corosync (2.3.0) and libqb (0.16.0) in 2-node 
clusters (virtualized in VMware infrastructure, OS: RHEL 6.7).
I noticed that if only one node is present, the CPU usage of Corosync (as seen 
with top) is slowly but steadily increasing (over days; in my setting about 1% 
per day). The node is basically idle, some Pacemaker managed resources are 
running but they are not contacted by any clients.
I upgraded a test stand-alone node to Corosync (2.4.2) and libqb (1.0.1) (which 
at least made the memleak go away), but the CPU usage is still increasing on 
the node.
When I add a second node to the cluster, the CPU load drops back down to a 
normal (low) CPU usage.
I haven't witnessed the increasing CPU load yet if two nodes were present in a 
cluster.

Even if running Pacemaker/Corosync as a massive-overkill-Monit-replacement is 
questionable, the observed CPU-load is not what I expect to happen.

What could be the reason for this CPU-load increase? Is there a rational behind 
this?


This is really interesting observation. I can talk about corosync and I 
must say no, there is no rationale behind. It simply shouldn't be 
happening. Also I don't see any reason why connection of other node(s) 
could help to remove CPU-load.



Is this a config thing or something in the binaries?


For sure not in corosync. Also your config file looks just ok.

Could you test single ring only and udpu if behavior stays same?

Regards,
  Honza



BR, Stefan

My corosync.conf:

# Please read the corosync.conf.5 manual page
compatibility: whitetank

aisexec {
 user:root
 group:root
}

totem {
 version: 2

 # Security configuration
 secauth: on
 threads: 0

 # Timeout for token
 token: 1000
 token_retransmits_before_loss_const: 4

 # Number of messages that may be sent by one processor on receipt of 
the token
 max_messages: 20

 # How long to wait for join messages in the membership protocol (ms)
 join: 50
 consensus: 1200

 # Turn off the virtual synchrony filter
 vsftype: none

 # Stagger sending the node join messages by 1..send_join ms
 send_join: 50

 # Limit generated nodeids to 31-bits (positive signed integers)
 clear_node_high_bit: yes

 # Interface configuration
 rrp_mode: passive
 interface {
 ringnumber: 0
 bindnetaddr: 10.20.30.0
 mcastaddr: 226.95.30.100
 mcastport: 5510
 }
 interface {
 ringnumber: 1
 bindnetaddr: 10.20.31.0
 mcastaddr: 226.95.31.100
 mcastport: 5510
 }
}

logging {
 fileline: off
 to_stderr: no
 to_logfile: no
 to_syslog: yes
 syslog_facility: local3
 debug: off
}

amf {
 mode: disabled
}

quorum {
 provider: corosync_votequorum
 expected_votes: 1
}

___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org




___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Digest does not match

[Jan Friesse]

Among two cases where I have seen this error messages I solved one.
On one cluster these dedicated interfaces were connected to a switch
instead of being connected directly.


Ok



Though I still don't know what caused these errors on another system
(the logs in the previous email).


Actually there is really nothing in the logs what could indicate that 
nodes ever merged. I would suggest to check corosync.conf and authkeys 
equality.


Honza




___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org




___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Two nodes cluster issue

[Jan Friesse]

I read the corosync-qdevice (8) man page couple of times, and also the RH 
documentation at 
https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/High_Availability_Add-On_Reference/s1-quorumdev-HAAR.html
I think it will be great if you will be able to add some examples that 
demonstrate the difference between the two, and give some use cases that 
explain what is the preferred algorithm to use in each case.


It's really hard to say which algorithm suits concrete situation 
"better" but yes, I will try to add some examples.


Regards,
  Honza




-Original Message-----
From: Jan Friesse [mailto:jfrie...@redhat.com]
Sent: Monday, August 7, 2017 2:38 PM
To: Cluster Labs - All topics related to open-source clustering welcomed 
<users@clusterlabs.org>; kwenn...@redhat.com; Prasad, Shashank 
<sspra...@vanu.com>
Subject: Re: [ClusterLabs] Two nodes cluster issue

Tomer Azran napsal(a):

Just updating that I added another level of fencing using watchdog-fencing.
With the quorum device and this combination works in case of power failure of 
both server and ipmi interface.
An important note is that the stonith-watchdog-timeout must be configured in 
order to work.
After reading the following great post: 
http://blog.clusterlabs.org/blog/2015/sbd-fun-and-profit , I choose the softdog 
watchdog since the I don't think ipmi watchdog will do no good in case the ipmi 
interface is down (If it is OK it will be used as a fencing method).

Just for documenting the solution (in case someone else needed that), the 
configuration I added is:
systemctl enable sbd
pcs property set no-quorum-policy=suicide pcs property set
stonith-watchdog-timeout=15 pcs quorum device add model net
host=qdevice algorithm=lms

I just can't decide if the qdevice algorithm should be lms or ffsplit. I 
couldn't determine the difference between them and I'm not sure which one is 
the best when using two node cluster with qdevice and watchdog fencing.

Can anyone advise on that?


I'm pretty sure you've read corosync-qdevice (8) man page where is quite 
detailed description of algorithms so if you were not able to determine the 
difference them there is something wrong and man page needs improvement. What 
exactly you were unable to understand?

Also for your use case with 2 nodes both algorithms behaves same way.

Honza



-Original Message-
From: Jan Friesse [mailto:jfrie...@redhat.com]
Sent: Tuesday, July 25, 2017 11:59 AM
To: Cluster Labs - All topics related to open-source clustering
welcomed <mailto:users@clusterlabs.org>; mailto:kwenn...@redhat.com; Prasad,
Shashank <mailto:sspra...@vanu.com>
Subject: Re: [ClusterLabs] Two nodes cluster issue


Tomer Azran napsal(a):

I tend to agree with Klaus – I don't think that having a hook that
bypass stonith is the right way. It is better to not use stonith at all.
I think I will try to use an iScsi target on my qdevice and set SBD
to use it.
I still don't understand why qdevice can't take the place SBD with
shared storage; correct me if I'm wrong, but it looks like both of
them are there for the same reason.


Qdevice is there to be third side arbiter who decides which partition
is quorate. It can also be seen as a quorum only node. So for two
node cluster it can be viewed as a third node (eventho it is quite
special because it cannot run resources). It is not doing fencing.

SBD is fencing device. It is using disk as a third side arbiter.


I've talked with Klaus and he told me that 7.3 is not using disk as a third 
side arbiter so sorry for confusion.

You should however still be able to use sbd for checking if pacemaker is alive and if the 
partition has quorum - otherwise the watchdog kills the node. So qdevice will give you 
"3rd" node and sbd fences unquorate partition.

Or (as mentioned previously) you can use fabric fencing.

Regards,
 Honza






From: Klaus Wenninger [mailto:kwenn...@redhat.com]
Sent: Monday, July 24, 2017 9:01 PM
To: Cluster Labs - All topics related to open-source clustering
welcomed <mailto:users@clusterlabs.org>; Prasad, Shashank
<mailto:sspra...@vanu.com>
Subject: Re: [ClusterLabs] Two nodes cluster issue

On 07/24/2017 07:32 PM, Prasad, Shashank wrote:
Sometimes IPMI fence devices use shared power of the node, and it
cannot be avoided.
In such scenarios the HA cluster is NOT able to handle the power
failure of a node, since the power is shared with its own fence device.
The failure of IPMI based fencing can also exist due to other
reasons also.

A failure to fence the failed node will cause cluster to be marked
UNCLEAN.
To get over it, the following command needs to be invoked on the
surviving node.

pcs stonith confirm  --force

This can be automated by hooking a recovery script, when the the
Stonith resource ‘Timed Out’ event.
To be more specific, the Pacemaker Alerts can be used for watch for
Stonith timeouts and failures.
In that script, all that’s essentially to 

Re: [ClusterLabs] Two nodes cluster issue

[Jan Friesse]

Tomer Azran napsal(a):

Just updating that I added another level of fencing using watchdog-fencing.
With the quorum device and this combination works in case of power failure of 
both server and ipmi interface.
An important note is that the stonith-watchdog-timeout must be configured in 
order to work.
After reading the following great post: 
http://blog.clusterlabs.org/blog/2015/sbd-fun-and-profit , I choose the softdog 
watchdog since the I don't think ipmi watchdog will do no good in case the ipmi 
interface is down (If it is OK it will be used as a fencing method).

Just for documenting the solution (in case someone else needed that), the 
configuration I added is:
systemctl enable sbd
pcs property set no-quorum-policy=suicide
pcs property set stonith-watchdog-timeout=15
pcs quorum device add model net host=qdevice algorithm=lms

I just can't decide if the qdevice algorithm should be lms or ffsplit. I 
couldn't determine the difference between them and I'm not sure which one is 
the best when using two node cluster with qdevice and watchdog fencing.

Can anyone advise on that?


I'm pretty sure you've read corosync-qdevice (8) man page where is quite 
detailed description of algorithms so if you were not able to determine 
the difference them there is something wrong and man page needs 
improvement. What exactly you were unable to understand?


Also for your use case with 2 nodes both algorithms behaves same way.

Honza



-Original Message-
From: Jan Friesse [mailto:jfrie...@redhat.com]
Sent: Tuesday, July 25, 2017 11:59 AM
To: Cluster Labs - All topics related to open-source clustering welcomed 
<users@clusterlabs.org>; kwenn...@redhat.com; Prasad, Shashank 
<sspra...@vanu.com>
Subject: Re: [ClusterLabs] Two nodes cluster issue


Tomer Azran napsal(a):

I tend to agree with Klaus – I don't think that having a hook that
bypass stonith is the right way. It is better to not use stonith at all.
I think I will try to use an iScsi target on my qdevice and set SBD
to use it.
I still don't understand why qdevice can't take the place SBD with
shared storage; correct me if I'm wrong, but it looks like both of
them are there for the same reason.


Qdevice is there to be third side arbiter who decides which partition
is quorate. It can also be seen as a quorum only node. So for two node
cluster it can be viewed as a third node (eventho it is quite special
because it cannot run resources). It is not doing fencing.

SBD is fencing device. It is using disk as a third side arbiter.


I've talked with Klaus and he told me that 7.3 is not using disk as a third 
side arbiter so sorry for confusion.

You should however still be able to use sbd for checking if pacemaker is alive and if the 
partition has quorum - otherwise the watchdog kills the node. So qdevice will give you 
"3rd" node and sbd fences unquorate partition.

Or (as mentioned previously) you can use fabric fencing.

Regards,
Honza






From: Klaus Wenninger [mailto:kwenn...@redhat.com]
Sent: Monday, July 24, 2017 9:01 PM
To: Cluster Labs - All topics related to open-source clustering
welcomed <users@clusterlabs.org>; Prasad, Shashank
<sspra...@vanu.com>
Subject: Re: [ClusterLabs] Two nodes cluster issue

On 07/24/2017 07:32 PM, Prasad, Shashank wrote:
Sometimes IPMI fence devices use shared power of the node, and it
cannot be avoided.
In such scenarios the HA cluster is NOT able to handle the power
failure of a node, since the power is shared with its own fence device.
The failure of IPMI based fencing can also exist due to other reasons
also.

A failure to fence the failed node will cause cluster to be marked
UNCLEAN.
To get over it, the following command needs to be invoked on the
surviving node.

pcs stonith confirm  --force

This can be automated by hooking a recovery script, when the the
Stonith resource ‘Timed Out’ event.
To be more specific, the Pacemaker Alerts can be used for watch for
Stonith timeouts and failures.
In that script, all that’s essentially to be executed is the
aforementioned command.

If I get you right here you can disable fencing then in the first place.
Actually quorum-based-watchdog-fencing is the way to do this in a
safe manner. This of course assumes you have a proper source for
quorum in your 2-node-setup with e.g. qdevice or using a shared disk
with sbd (not directly pacemaker quorum here but similar thing
handled inside sbd).


Since the alerts are issued from ‘hacluster’ login, sudo permissions
for ‘hacluster’ needs to be configured.

Thanx.


From: Klaus Wenninger [mailto:kwenn...@redhat.com]
Sent: Monday, July 24, 2017 9:24 PM
To: Kristián Feldsam; Cluster Labs - All topics related to
open-source clustering welcomed
Subject: Re: [ClusterLabs] Two nodes cluster issue

On 07/24/2017 05:37 PM, Kristián Feldsam wrote:
I personally think that power off node by switched pdu is more safe,
or not?

True if that is working in you environment. If you can't do a
physical setup where

Re: [ClusterLabs] ClusterLabs.Org Documentation Problem?

[Jan Friesse]

Thanks for the reply. Yes, it's a bit confusing. I did end up using the 
documentation for Corosync 2.X since that seemed newer, but it also assumed 
CentOS/RHEL7 and systemd-based commands. It also incorporates cman, pcsd, 
psmisc, and policycoreutils-pythonwhich, which are all new to me. If there is 
anything I can do to assist with getting the documentation cleaned up, I'd be 
more than glad to help.


Just a small correction.

Documentation shouldn't incorporate cman. Cman was used with corosync 
1.x as a configuration layer and (more important) quorum provider. With 
Corosync 2.x quorum provider is already in corosync so no need for cman.






--
Eric Robinson

-Original Message-
From: Ken Gaillot [mailto:kgail...@redhat.com]
Sent: Tuesday, August 22, 2017 2:08 PM
To: Cluster Labs - All topics related to open-source clustering welcomed 

Subject: Re: [ClusterLabs] ClusterLabs.Org Documentation Problem?

On Tue, 2017-08-22 at 19:40 +, Eric Robinson wrote:

The documentation located here…



http://clusterlabs.org/doc/



…is confusing because it offers two combinations:



Pacemaker 1.0 for Corosync 1.x

Pacemaker 1.1 for Corosync 2.x



According to the documentation, if you use Corosync 1.x you need
Pacemaker 1.0, but if you use Corosync 2.x then you need Pacemaker
1.1.



However, on my Centos 6.9 system, when I do ‘yum install pacemaker
corosync” I get the following versions:



pacemaker-1.1.15-5.el6.x86_64

corosync-1.4.7-5.el6.x86_64



What’s the correct answer? Does Pacemaker 1.1.15 work with Corosync
1.4.7? If so, is the documentation at ClusterLabs misleading?



--
Eric Robinson


The page actually offers a third option ... "Pacemaker 1.1 for CMAN or Corosync 
1.x". That's the configuration used by CentOS 6.

However, that's still a bit misleading; the documentation set for "Pacemaker 1.1 for 
Corosync 2.x" is the only one that is updated, and it's mostly independent of the 
underlying layer, so you should prefer that set.

I plan to reorganize that page in the coming months, so I'll try to make it 
clearer.

--
Ken Gaillot 





___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org
___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org




___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] vip is not removed after node lost connection with the other two nodes

[Jan Friesse]

Jan Pokorný napsal(a):

[Hui, no need to address us individually along with the list, we are
both subscribed to it since around the beginning]

On 26/06/17 16:10 +0800, Hui Xiang wrote:

Thanks guys!!

@Ken
I did "ifconfig ethx down" to make the cluster interface down.


That's what I suspected and what I tried to show as problematic to say
the least, based on the previous dismay.


@Jan

Do you know what is the "known bug" mentioned below:

"During ifdown corosync will rebind to localhost (this is long time
known bug) and behaves weirdly."

http://lists.clusterlabs.org/pipermail/users/2015-July/000878.html


There wasn't much investigation on your side, was it?

https://github.com/corosync/corosync/wiki/Corosync-and-ifdown-on-active-network-interface

Honza Friesse or Chrissie can comment more on this topic.


There is really nothing to comment. We are working on fix for Corosync 
3.x. Also even after fixing this bug it will be bad idea to test cluster 
recovery just by using ifdown.







___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org




___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] how to sync data using cmap between cluster

[Jan Friesse]

ok, but why the node status(left, join) can be sync to other node in
the cluster by CMAP? thanks!


It is not synced by cmap. Node status is essential property of totem 
protocol and it is just stored into local cmap mostly for 
diagnostics/monitoring. Also they are not so much in sync. You can try 
to create two node cluster, block traffic on one of the node and you 
will get node A which sees node B as a left and node B which see node A 
as left.


Honza



On Thu, May 25, 2017 at 11:26 PM, Christine Caulfield
 wrote:

On 25/05/17 15:48, Rui Feng wrote:

Hi,

   I have a test based on corosync 2.3.4, and find the data stored by
cmap( corosync-cmapctl -s test i8 1) which can't be sync to other
node.
   Could somebody give some comment or solution for it, thanks!




cmap isn't replicated across the cluster. If you need data replication
then you'll have to use some other method.

Chrissie

___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org


___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org




___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Is "Process pause detected" triggered too easily?

[Jan Friesse]

Jean,


Hi Jan,

On Tue, 3 Oct 2017, Jan Friesse wrote:


I hope this makes sense! :)


I would still have some questions :) but that is really not related to
the problem you have.


Questions are welcome! I am new to this stack, so there is certainly room
for learning and for improvement.


My personal favorite is consensus timeout. Because you've set (and I
must say according to doc correctly) consensus timeout to 3600 (= 1.2 *
token). Problem is, that result token timeout is not 3000, but with 5
nodes it is actually 3000 (base token) + (no_nodes - 2) * 650 ms = 4950
(as you can check by observing runtime.config.totem.token key). So it
may make sense to set consensus timeout to ~6000.


Could you clarify the formula for me? I don't see how "- 2" and "650" map
to this configuration.


Since Corosync 2.3.4 when nodelist is used, totem.token is used only as 
a basis for calculating real token timeout. You can check corosync.conf 
man page for more information and formula.




And I suppose that on our bigger system (20+5 servers) we need to greatly
increase the consensus timeout.


Consensus timeout reflects token value so if it is not defined in config 
file it's computed as token * 1.2. This is not reflected in manpage and 
needs to be fixed.




Overall, tuning the timeouts seems related to be Black Magic. ;) I liked


It is


the idea suggested in an old thread that there would be a spreadsheet (or
even just plain formulas) exposing the relation between the various knobs.


Idea is to compute it in the code directly. This is implemented for some 
parts, but sadly not for some other. Reason is mostly that it's quite 
hard to make these timeouts right, so failure detection is fast enough 
but there are as few false membership changes as possible.




One thing I wonder is: would it make sense to annotate the state machine
diagram in the Totem paper (page 15 of
http://www.cs.jhu.edu/~yairamir/tocs.ps.gz) with those tunables? Assuming
the paper still reflects the behavior of the current code.


Yes, code reflects paper (to some extend, some things are slightly 
different) and I really like idea of annotating it, or actually having 
wiki page with this diagram and slight documentation of totemsrp insides.





This doesn't change the fact that "bug" is reproducible even with
"correct" consensus, so I will continue working on this issue.


Great! Thanks for taking the time to investigate.


Yep, np.

Honza




Cheers,
JM




___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] corosync service not automatically started

[Jan Friesse]

Ken Gaillot napsal(a):

On Tue, 2017-10-10 at 12:24 +0200, Václav Mach wrote:

On 10/10/2017 11:40 AM, Valentin Vidic wrote:

On Tue, Oct 10, 2017 at 11:26:24AM +0200, Václav Mach wrote:

# The primary network interface
allow-hotplug eth0
iface eth0 inet dhcp
# This is an autoconfigured IPv6 interface
iface eth0 inet6 auto


allow-hotplug or dhcp could be causing problems.  You can try
disabling corosync and pacemaker so they don't start on boot
and start them manually after a few minutes when the network
is stable.  If it works than you have some kind of a timing
issue.  You can try using 'auto eth0' or a static IP address
to see if it helps...



It seems that static network configuration really solved this issue.
No
further modifications of services were necessary.

Thanks for help.


Yes, that would be it -- corosync doesn't play well with DHCP. The
lease renewals (even when keeping the same IP) can disrupt corosync
communication. (At least that was the case I last looked into it --


Really? Do you have reproducer for this behavior? Because it sound's 
quite weird. DHCP client shouldn't really remove IP on renewal if IP 
doesn't differ.



there may have been recent changes to work around it, or that may be
coming in the new kronosnet protocol.)


Kronosnet should solve this problem for sure.

Honza



___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org




___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] corosync service not automatically started

[Jan Friesse]

Václav Mach napsal(a):


On 10/10/2017 11:40 AM, Valentin Vidic wrote:

On Tue, Oct 10, 2017 at 11:26:24AM +0200, Václav Mach wrote:

# The primary network interface
allow-hotplug eth0
iface eth0 inet dhcp
# This is an autoconfigured IPv6 interface
iface eth0 inet6 auto


allow-hotplug or dhcp could be causing problems.  You can try
disabling corosync and pacemaker so they don't start on boot
and start them manually after a few minutes when the network
is stable.  If it works than you have some kind of a timing
issue.  You can try using 'auto eth0' or a static IP address
to see if it helps...



It seems that static network configuration really solved this issue. No
further modifications of services were necessary.


Actually main problem is/was mixing interface section with nodelist and 
unavailability of IP when corosync was started.


Without interface section corosync would just show error and properly 
exit. So debugging would be easy.


What is quite weird is the fact that whatever init system you use 
doesn't wait for dhcp. I believe systemd + NetworkManager with 
NetworkManager-server does that properly.


Honza



Thanks for help.



___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org




___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] corosync race condition when node leaves immediately after joining

[Jan Friesse]

Jonathan,




On 13/10/17 17:24, Jan Friesse wrote:

I've done a bit of digging and am getting closer to the root cause of
the race.

We rely on having votequorum_sync_init called twice -- once when node 1
joins (with member_list_entries=2) and once when node 1 leaves (with
member_list_entries=1). This is important because votequorum_sync_init
marks nodes as NODESTATE_DEAD if they are not in quorum_members[] -- so
it needs to have seen the node appear then disappear. This is important
because get_total_votes only counts votes from nodes in state
NODESTATE_MEMBER.


So there are basically two problems.

Actually first (main) problem is that votequorum_sync_init is ever
called when that node joins. It really shouldn't. And problem is
simply because calling api->shutdown_request() is not enough. Can you
try replace it with exit(1) (for testing) and reproduce the problem?
I'm pretty sure problem disappears.


No, the problem still happens :-(


Not good.



I am using the following patch:

diff --git a/exec/cmap.c b/exec/cmap.c
index de730d2..1125cef 100644
--- a/exec/cmap.c
+++ b/exec/cmap.c
@@ -406,7 +406,7 @@ static void cmap_sync_activate (void)
 log_printf(LOGSYS_LEVEL_ERROR,
 "Received config version (%"PRIu64") is different
than my config version (%"PRIu64")! Exiting",
 cmap_highest_config_version_received,
cmap_my_config_version);
-   api->shutdown_request();
+   exit(1);
 return ;
 }
  }
diff --git a/exec/main.c b/exec/main.c
index b0d5639..4fd3e68 100644
--- a/exec/main.c
+++ b/exec/main.c
@@ -627,6 +627,7 @@ static void deliver_fn (
 ((void *)msg);
 }

+   log_printf(LOGSYS_LEVEL_NOTICE, "executing '%s' exec_handler_fn
%p for node %d (fn %d)", corosync_service[service]->name,
corosync_service[service]->exec_engine[fn_id].exec_handler_fn, nodeid,
fn_id);
 corosync_service[service]->exec_engine[fn_id].exec_handler_fn
 (msg, nodeid);
  }
diff --git a/exec/votequorum.c b/exec/votequorum.c
index 1a97c6d..7c0f34f 100644
--- a/exec/votequorum.c
+++ b/exec/votequorum.c
@@ -2099,6 +2100,7 @@ static void
message_handler_req_exec_votequorum_nodeinfo (
 node->flags = req_exec_quorum_nodeinfo->flags;
 node->votes = req_exec_quorum_nodeinfo->votes;
 node->state = NODESTATE_MEMBER;
+   log_printf(LOGSYS_LEVEL_NOTICE,
"message_handler_req_exec_votequorum_nodeinfo (%p) marking node %d as
MEMBER", message_handler_req_exec_votequorum_nodeinfo, nodeid);

 if (node->flags & NODE_FLAGS_LEAVING) {
 node->state = NODESTATE_LEAVING;

When it's working correctly I see this:

1508151960.072927 notice  [TOTEM ] A new membership (10.71.218.17:2304)
was formed. Members joined: 1
1508151960.073082 notice  [SYNC  ] calling sync_init on service
'corosync configuration map access' (0) with my_member_list_entries = 2
1508151960.073150 notice  [MAIN  ] executing 'corosync configuration map
access' exec_handler_fn 0x55b5eb504ca0 for node 1 (fn 0)
1508151960.073197 notice  [MAIN  ] executing 'corosync configuration map
access' exec_handler_fn 0x55b5eb504ca0 for node 2 (fn 0)
1508151960.073238 notice  [SYNC  ] calling sync_init on service
'corosync cluster closed process group service v1.01' (1) with
my_member_list_entries = 2
1508151961.073033 notice  [TOTEM ] A processor failed, forming new
configuration.

When it's not working correctly I see this:

1508151908.447584 notice  [TOTEM ] A new membership (10.71.218.17:2292)
was formed. Members joined: 1
1508151908.447757 notice  [MAIN  ] executing 'corosync vote quorum
service v1.0' exec_handler_fn 0x558b39fbbaa0 for node 1 (fn 0)
1508151908.447866 notice  [VOTEQ ]
message_handler_req_exec_votequorum_nodeinfo (0x558b39fbbaa0) marking
node 1 as MEMBER
1508151908.447972 notice  [VOTEQ ] get_total_votes: node 1 is a MEMBER
so counting vote
1508151908.448045 notice  [VOTEQ ] get_total_votes: node 2 is a MEMBER
so counting vote
1508151908.448091 notice  [QUORUM] This node is within the primary
component and will provide service.
1508151908.448134 notice  [QUORUM] Members[1]: 2
1508151908.448175 notice  [SYNC  ] calling sync_init on service
'corosync configuration map access' (0) with my_member_list_entries = 2
1508151908.448205 notice  [MAIN  ] executing 'corosync configuration map
access' exec_handler_fn 0x558b39fb3ca0 for node 1 (fn 0)
1508151908.448247 notice  [MAIN  ] executing 'corosync configuration map
access' exec_handler_fn 0x558b39fb3ca0 for node 2 (fn 0)
1508151908.448307 notice  [SYNC  ] calling sync_init on service
'corosync cluster closed process group service v1.01' (1) with
my_member_list_entries = 2
1508151909.447182 notice  [TOTEM ] A processor failed, forming new
configuration.

... and at that point I already see "Total votes: 2" in the
corosync-quorum

Re: [ClusterLabs] corosync race condition when node leaves immediately after joining

[Jan Friesse]

Jonathan Davies napsal(a):



On 12/10/17 11:54, Jan Friesse wrote:

I'm on corosync-2.3.4 plus my patch


Finally noticed ^^^ 2.3.4 is really old and as long as it is not some
patched version, I wouldn't recommend to use it. Can you give a try to
current needle?


I was mistaken to think I was on 2.3.4. Actually I am on the version
from CentOS 7.4 which is 2.4.0+patches.


Ok, so this is basically like needle.



I will try to reproduce it with needle.


But often at this point, cluster1's disappearance is not reflected in
the votequorum info on cluster2:


... Is this permanent (= until new node join/leave it , or it will fix
itself over (short) time? If this is permanent, it's a bug. If it
fixes itself it's result of votequorum not being virtual synchronous.


Yes, it's permanent. After several minutes of waiting, votequorum still
reports "total votes: 2" even though there's only one member.



That's bad. I've tried following setup:

- Both nodes with current needle
- Your config
- Second node is just running corosync
- First node is running following command:
   while true;do corosync -f; ssh node2 'corosync-quorumtool | grep
Total | grep 1' || exit 1;done

Running it for quite a while and I'm unable to reproduce the bug.
Sadly I'm unable to reproduce the bug even with 2.3.4. Do you think
that reproducer is correct?


Yes, that's similar enough to what I'm doing. The bug happens about 50%
of the time for me, so I do it manually rather than needing a loop. So
I'm not sure why you can't reproduce it.


That's really strange. Chrissie is also unable to reproduce it so we may 
miss some part of puzzle.




I've done a bit of digging and am getting closer to the root cause of
the race.

We rely on having votequorum_sync_init called twice -- once when node 1
joins (with member_list_entries=2) and once when node 1 leaves (with
member_list_entries=1). This is important because votequorum_sync_init
marks nodes as NODESTATE_DEAD if they are not in quorum_members[] -- so
it needs to have seen the node appear then disappear. This is important
because get_total_votes only counts votes from nodes in state
NODESTATE_MEMBER.


So there are basically two problems.

Actually first (main) problem is that votequorum_sync_init is ever 
called when that node joins. It really shouldn't. And problem is simply 
because calling api->shutdown_request() is not enough. Can you try 
replace it with exit(1) (for testing) and reproduce the problem? I'm 
pretty sure problem disappears.




When it goes wrong, I see that votequorum_sync_init is only called
*once* (with member_list_entries=1) -- after node 1 has joined and left.
So it never sees node 1 in member_list, hence never marks it as
NODESTATE_DEAD. But message_handler_req_exec_votequorum_nodeinfo has
indepedently marked the node as NODESTATE_MEMBER, hence get_total_votes
counts it and quorate is set to 1.


This is second problem and it has to be also fixed. You have to be 
pretty lucky to reproduce it so often.


Anyway, here is theory:
- node 1 calls api->shutdown_request() and continue processing (this is 
problem 1)
- both nodes gets to a state where they should call 
votequorum_sync_init, but node 2 is now not scheduled (as I said, it 
must be pretty luck)
- node 1 calls votequorum_sync_init and votequorum_sync_process (sending 
nodeinfo) and it's shutdown

- node 2 gets nodeinfo
- node 2 sees node 1 shutdown
- node 2 calls votequorum_sync_init and votequorum_sync_process

If this theory is true, we must probably fix the sync.c to have 2 or 
ideally 3-4 barriers instead of 1.




So why is votequorum_sync_init sometimes only called once? It looks like
it's all down to whether we manage to iterate through all the calls to
schedwrk_processor before entering the OPERATIONAL state. I haven't yet
looked into exactly what controls the timing of these two things.

Adding the following patch helps me to demonstrate the problem more
clearly:

diff --git a/exec/sync.c b/exec/sync.c
index e7b71bd..a2fb06d 100644
--- a/exec/sync.c
+++ b/exec/sync.c
@@ -544,6 +545,7 @@ static int schedwrk_processor (const void *context)
 }

 if
(my_sync_callbacks_retrieve(my_service_list[my_processing_idx].service_id,
NULL) != -1) {
+   log_printf(LOGSYS_LEVEL_NOTICE, "calling
sync_init on service '%s' (%d) with my_member_list_entries = %d",
my_service_list[my_processing_idx].name, my_processing_idx,
my_member_list_entries);
 my_service_list[my_processing_idx].sync_init
(my_trans_list,
 my_trans_list_entries, my_member_list,
 my_member_list_entries,
diff --git a/exec/votequorum.c b/exec/votequorum.c
index d5f06c1..aab6c15 100644
--- a/exec/votequorum.c
+++ b/exec/votequorum.c
@@ -2336,6 +2353,8 @@ static void votequorum_sync_init (
 int left_nodes;
 struct cluster_node *node;

+   log_printf(LOGSYS_LEVEL_NOTICE, &q

Re: [ClusterLabs] Strange Corosync (TOTEM) logs, Pacemaker OK but DLM stuck

[Jan Friesse]

Ferenc,


wf...@niif.hu (Ferenc Wágner) writes:


Jan Friesse <jfrie...@redhat.com> writes:


wf...@niif.hu writes:


In a 6-node cluster (vhbl03-08) the following happens 1-5 times a day
(in August; in May, it happened 0-2 times a day only, it's slowly
ramping up):

vhbl08 corosync[3687]:   [TOTEM ] A processor failed, forming new configuration.
vhbl03 corosync[3890]:   [TOTEM ] A processor failed, forming new configuration.
vhbl07 corosync[3805]:   [MAIN  ] Corosync main process was not scheduled for 
4317.0054 ms (threshold is 2400. ms). Consider token timeout increase.


^^^ This is main problem you have to solve. It usually means that
machine is too overloaded. It is happening quite often when corosync
is running inside VM where host machine is unable to schedule regular
VM running.


After some extensive tracing, I think the problem lies elsewhere: my
IPMI watchdog device is slow beyond imagination.


Confirmed: setting watchdog_device: off cluster wide got rid of the
above warnings.



Yep, good you found the issue. This is perfectly possible if ioctl blocks.


Its ioctl operations can take seconds, starving all other functions.
At least, it seems to block the main thread of Corosync.  Is this a
plausible scenario?  Corosync has two threads, what are their roles?


First (main) thread is basically doing almost everything. There is a 
main loop (epoll) I've described in previous mail.


Second thread is created by libqb and it's used only for logging. This 
is to prevent blocking of corosync when syslog/file log write blocks for 
some reason. It means some messages may be lost but it's still better 
than blocking.


Back to problem you have. It's definitively HW issue but I'm thinking 
how to solve it in software. Right now, I can see two ways:
1. Set dog FD to be non blocking right at the end of setup_watchdog - 
This is proffered but I'm not sure if it's really going to work.

2. Create thread which makes sure to tackle wd regularly.

Regards,
  Honza

___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Strange Corosync (TOTEM) logs, Pacemaker OK but DLM stuck

[Jan Friesse]

Ferenc,

Jan Friesse <jfrie...@redhat.com> writes:


Back to problem you have. It's definitively HW issue but I'm thinking
how to solve it in software. Right now, I can see two ways:
1. Set dog FD to be non blocking right at the end of setup_watchdog -
This is proffered but I'm not sure if it's really going to work.


I'll run some test to see what works (if anything).  The keepalives can
be provided by write()s as well, but somehow I don't expect that to make
a difference.  We'll see.


Sounds good.




2. Create thread which makes sure to tackle wd regularly.


That would work, but maybe too well if entirely decoupled from the main
loop.



Of course this would need to be done very carefully. I believe killing 
tackle thread would work well with minimum risks.


Honza


___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Is "Process pause detected" triggered too easily?

[Jan Friesse]

On Wed, 27 Sep 2017, Jan Friesse wrote:


I don't think scheduling is the case. If scheduler would be the case
other message (Corosync main process was not scheduled for ...) would
kick in. This looks more like a something is blocked in totemsrp.


Ah, interesting!


Also, it looks like the side effect is that corosync drops important
messages (I think "join" messages?), and I fear that this can lead to


You mean membership join messages? Because there are a lot (327) of them
in log you've sent.


Yes. In my test setup I didn't see any issue where we lost membership join
messages, but the reason why I am looking into this is this:

We had one problem on a real deployment of DLM+corosync (5 voters and 20
non-voters, with dlm on those 20, for a specific application that uses


What you mean by voters and non-voters? There is 25 nodes in total and 
each of them is running corosync?



libdlm). On a reboot of one server running just corosync (which thus did
NOT run dlm), a large number of other servers got briefly evicted from the


This is kind of weird. AFAIK DLM is joining to CPG group and using CPG 
membership. So if DLM was not running on the node then other nodes 
joined to DLM CPG group should not even notice its leave.



corosync ring; and when rejoining, dlm complained about a "stateful merge"
which forces a reboot. Note, dlm fencing is disabled.

In that system, it was "legal" for corosync to kick out these servers
(they had zero vote), but it was highly unexpected (they were running
fine) and the impact is high (reboot).


What you mean by zero vote? You mean DLM vote or corosync number of 
votes (related to quorum)?




We did see "Process pause detected" in the logs on that system when the
incident happened, which is why I think could be a clue.


I've tried to reproduce the problem and I was not successful with 3 
nodes cluster using more or less default config (not changing 
join/consensus/...). I'll try 5 nodes possibly with totem values and see 
if problem appears.


Regards,
  Honza




I'll definitively try to reproduce this bug and let you know. I don't
think any message get lost, but it's better to be on a safe side.


Thanks!


Cheers,
JM




___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Is "Process pause detected" triggered too easily?

[Jan Friesse]

Jean,


On Mon, 2 Oct 2017, Jan Friesse wrote:


We had one problem on a real deployment of DLM+corosync (5 voters and 20
non-voters, with dlm on those 20, for a specific application that uses


What you mean by voters and non-voters? There is 25 nodes in total and
each of them is running corosync?


Yes, there are 25 servers running corosync:

- 5 are configured to have one vote for quorum, on these servers corosync
serves no other purpose

- 20 have zero vote for quorum, and these servers also run DLM and the
application that uses DLM

The intent with this configuration is:

- to avoid split brain in case of network partition: application servers
must be in the same partition as the quorum majority (so, 3 of the 5
"voters") to carry on their operations

- to allow independent failure of any number of application servers

I hope this makes sense! :)


I would still have some questions :) but that is really not related to 
the problem you have.





libdlm). On a reboot of one server running just corosync (which thus did
NOT run dlm), a large number of other servers got briefly evicted from the


This is kind of weird. AFAIK DLM is joining to CPG group and using CPG
membership. So if DLM was not running on the node then other nodes joined to
DLM CPG group should not even notice its leave.


Indeed, but we saw "Process pause detected" on all servers, and corosync
temporarily formed an operational cluster excluding most of the
"non-voters" (those with zero quorum vote). Then most servers joined back,
but then DLM complained about the "stateful merge".


Yep, when this was on all servers it's a huge problem and it explains a lot.




What you mean by zero vote? You mean DLM vote or corosync number of
votes (related to quorum)?


I mean the vote in the corosync quorum, I'm not aware of anything like
that with DLM (or maybe you could think of the per-server weight when one
manually defines servers that master locks in a lock space, but we don't
use that).



Got it.




I've tried to reproduce the problem and I was not successful with 3
nodes cluster using more or less default config (not changing
join/consensus/...). I'll try 5 nodes possibly with totem values and see
if problem appears.


I've tried again today, and first with just 3 servers (VMs), using the
same config I sent earlier (which has 3 nodes with 1 vote, 2 nodes with 0
vote), I could no longer reproduce either. Then I spawned 2 more VMs and
had them join the existing 3-node cluster (those I added were the 2
servers with 0 vote), and then I saw the "Process pause ..." log line. And
now I have stopped the last 2 servers, and I am back to just 3, and I keep
seeing that log line.

If you're still curious and if that's useful, I can try to reproduce on a
set of VMs where I could give you full ssh access.


So good news is that I was able to reproduce it. Even better news is, 
that I was able to reproduce it even without changing join/consensus/... 
parameters. What's even better that with parameters changed it's 
becoming much easier to reproduce the problem. So in theory if I will be 
able to identify that parameter, it may make sense to increase/decrease 
that parameter close to infinity/0 and debugging should then become easy.


My personal favorite is consensus timeout. Because you've set (and I 
must say according to doc correctly) consensus timeout to 3600 (= 1.2 * 
token). Problem is, that result token timeout is not 3000, but with 5 
nodes it is actually 3000 (base token) + (no_nodes - 2) * 650 ms = 4950 
(as you can check by observing runtime.config.totem.token key). So it 
may make sense to set consensus timeout to ~6000.


This doesn't change the fact that "bug" is reproducible even with 
"correct" consensus, so I will continue working on this issue.


Honza





Thanks!

Cheers,
JM




___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Strange Corosync (TOTEM) logs, Pacemaker OK but DLM stuck

[Jan Friesse]

Ferenc,


Hi,

In a 6-node cluster (vhbl03-08) the following happens 1-5 times a day
(in August; in May, it happened 0-2 times a day only, it's slowly
ramping up):

vhbl08 corosync[3687]:   [TOTEM ] A processor failed, forming new configuration.
vhbl03 corosync[3890]:   [TOTEM ] A processor failed, forming new configuration.
vhbl07 corosync[3805]:   [MAIN  ] Corosync main process was not scheduled for 
4317.0054 ms (threshold is 2400. ms). Consider token timeout increase.



^^^ This is main problem you have to solve. It usually means that 
machine is too overloaded. It is happening quite often when corosync is 
running inside VM where host machine is unable to schedule regular VM 
running.


As a start you can try what message say = Consider token timeout 
increase. Currently you have 3 seconds, in theory 6 second should be enough.


Honza


vhbl07 corosync[3805]:   [TOTEM ] A processor failed, forming new configuration.
vhbl04 corosync[3759]:   [TOTEM ] A new membership (10.0.6.9:3056) was formed. 
Members
vhbl05 corosync[3919]:   [TOTEM ] A new membership (10.0.6.9:3056) was formed. 
Members
vhbl06 corosync[3759]:   [TOTEM ] A new membership (10.0.6.9:3056) was formed. 
Members
vhbl07 corosync[3805]:   [TOTEM ] A new membership (10.0.6.9:3056) was formed. 
Members
vhbl08 corosync[3687]:   [TOTEM ] A new membership (10.0.6.9:3056) was formed. 
Members
vhbl03 corosync[3890]:   [TOTEM ] A new membership (10.0.6.9:3056) was formed. 
Members
vhbl07 corosync[3805]:   [QUORUM] Members[6]: 167773705 167773706 167773707 
167773708 167773709 167773710
vhbl08 corosync[3687]:   [QUORUM] Members[6]: 167773705 167773706 167773707 
167773708 167773709 167773710
vhbl06 corosync[3759]:   [QUORUM] Members[6]: 167773705 167773706 167773707 
167773708 167773709 167773710
vhbl07 corosync[3805]:   [MAIN  ] Completed service synchronization, ready to 
provide service.
vhbl04 corosync[3759]:   [QUORUM] Members[6]: 167773705 167773706 167773707 
167773708 167773709 167773710
vhbl08 corosync[3687]:   [MAIN  ] Completed service synchronization, ready to 
provide service.
vhbl06 corosync[3759]:   [MAIN  ] Completed service synchronization, ready to 
provide service.
vhbl04 corosync[3759]:   [MAIN  ] Completed service synchronization, ready to 
provide service.
vhbl05 corosync[3919]:   [QUORUM] Members[6]: 167773705 167773706 167773707 
167773708 167773709 167773710
vhbl03 corosync[3890]:   [QUORUM] Members[6]: 167773705 167773706 167773707 
167773708 167773709 167773710
vhbl05 corosync[3919]:   [MAIN  ] Completed service synchronization, ready to 
provide service.
vhbl03 corosync[3890]:   [MAIN  ] Completed service synchronization, ready to 
provide service.

The cluster is running Corosync 2.4.2 multicast.  Those lines really end
at Members, there are no joined of left nodes listed.  Pacemaker on top
reacts like:

[9982] vhbl03 pacemakerd: info: pcmk_quorum_notification:   Quorum retained 
| membership=3056 members=6
[9991] vhbl03   crmd: info: pcmk_quorum_notification:   Quorum retained 
| membership=3056 members=6
[9986] vhbl03cib: info: cib_process_request:Completed 
cib_modify operation for section nodes: OK (rc=0, origin=vhbl07/crmd/4477, 
version=0.1694.12)
[9986] vhbl03cib: info: cib_process_request:Completed 
cib_modify operation for section status: OK (rc=0, origin=vhbl07/crmd/4478, 
version=0.1694.12)
[9986] vhbl03cib: info: cib_process_ping:   Reporting our current 
digest to vhbl07: 85250f3039d269f96012750f13e232d9 for 0.1694.12 
(0x55ef057447d0 0)

on all nodes except for vhbl07, where it says:

[9886] vhbl07   crmd: info: pcmk_quorum_notification:   Quorum retained 
| membership=3056 members=6
[9877] vhbl07 pacemakerd: info: pcmk_quorum_notification:   Quorum retained 
| membership=3056 members=6
[9881] vhbl07cib: info: cib_process_request:Forwarding 
cib_modify operation for section nodes to all (origin=local/crmd/
[9881] vhbl07cib: info: cib_process_request:Forwarding 
cib_modify operation for section status to all (origin=local/crmd
[9881] vhbl07cib: info: cib_process_request:Completed 
cib_modify operation for section nodes: OK (rc=0, origin=vhbl07/cr
[9881] vhbl07cib: info: cib_process_request:Completed 
cib_modify operation for section status: OK (rc=0, origin=vhbl07/c
[9881] vhbl07cib: info: cib_process_ping:   Reporting our current 
digest to vhbl07: 85250f3039d269f96012750f13e232d9 for 0.1694.

So Pacemaker does nothing, basically, and I can't see any adverse effect
to resource management, but DLM seems to have some problem, which may or
may not be related.  When the TOTEM error appears, all nodes log this:

vhbl03 dlm_controld[3914]: 2801675 dlm:controld ring 167773705:3056 6 memb 
167773705 167773706 167773707 167773708 167773709 167773710
vhbl03 dlm_controld[3914]: 2801675 fence work wait for cluster ringid
vhbl03 dlm_controld[3914]: 

Re: [ClusterLabs] Strange Corosync (TOTEM) logs, Pacemaker OK but DLM stuck

[Jan Friesse]

Ferenc,

Jan Friesse <jfrie...@redhat.com> writes:


wf...@niif.hu writes:


In a 6-node cluster (vhbl03-08) the following happens 1-5 times a day
(in August; in May, it happened 0-2 times a day only, it's slowly
ramping up):

vhbl08 corosync[3687]:   [TOTEM ] A processor failed, forming new configuration.
vhbl03 corosync[3890]:   [TOTEM ] A processor failed, forming new configuration.
vhbl07 corosync[3805]:   [MAIN  ] Corosync main process was not scheduled for 
4317.0054 ms (threshold is 2400. ms). Consider token timeout increase.


^^^ This is main problem you have to solve. It usually means that
machine is too overloaded. [...]


Before I start tracing the scheduler, I'd like to ask something: what
wakes up the Corosync main process periodically?  The token making a
full circle?  (Please forgive my simplistic understanding of the TOTEM
protocol.)  That would explain the recommendation in the log message,
but does not fit well with the overload assumption: totally idle nodes
could just as easily produce such warnings if there are no other regular
wakeup sources.  (I'm looking at timer_function_scheduler_timeout but I
know too little of libqb to decide.)


Corosync main loop is based on epoll, so corosync is waked up ether by 
receiving data (network socket or unix socket for services) or when 
there are data to sent and socket is ready for non blocking write or 
after timeout. This timeout is exactly what you call other wakeup resource.


Timeout is used for scheduling periodical tasks inside corosync.

One of periodical tasks is scheduler pause detector. It is basically 
scheduled every (token_timeout / 3) msec and it computes diff between 
current and last time. If diff is larger than (token_timeout * 0.8) it 
displays warning.





As a start you can try what message say = Consider token timeout
increase. Currently you have 3 seconds, in theory 6 second should be
enough.


It was probably high time I realized that token timeout is scaled
automatically when one has a nodelist.  When you say Corosync should
work OK with default settings up to 16 nodes, you assume this scaling is
in effect, don't you?  On the other hand, I've got no nodelist in the
config, but token = 3000, which is less than the default 1000+4*650 with
six nodes, and this will get worse as the cluster grows.


This is described in corosync.conf man page (token_coefficient).

Final timeout is computed using totem.token as a base value. So if you 
set totem.token to 3000 it means that final totem timeout value is not 
3000 but (3000 + 4 * 650).



Regards,
  Honza



Comments on the above ramblings welcome!

I'm grateful for all the valuable input poured into this thread by all
parties: it's proven really educative in quite unexpected ways beyond
what I was able to ask in the beginning.




___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Strange Corosync (TOTEM) logs, Pacemaker OK but DLM stuck

[Jan Friesse]

Ferenc,


Jan Friesse <jfrie...@redhat.com> writes:


wf...@niif.hu writes:


In a 6-node cluster (vhbl03-08) the following happens 1-5 times a day
(in August; in May, it happened 0-2 times a day only, it's slowly
ramping up):

vhbl08 corosync[3687]:   [TOTEM ] A processor failed, forming new configuration.
vhbl03 corosync[3890]:   [TOTEM ] A processor failed, forming new configuration.
vhbl07 corosync[3805]:   [MAIN  ] Corosync main process was not scheduled for 
4317.0054 ms (threshold is 2400. ms). Consider token timeout increase.


^^^ This is main problem you have to solve. It usually means that
machine is too overloaded. It is happening quite often when corosync
is running inside VM where host machine is unable to schedule regular
VM running.


Hi Honza,

Corosync isn't running in a VM here, these nodes are 2x8 core servers
hosting VMs themselves as Pacemaker resources.  (Incidentally, some of
these VMs run Corosync to form a test cluster, but that should be
irrelevant now.)  And they aren't overloaded in any apparent way: Munin
reports 2900% CPU idle (out of 32 hyperthreads).  There's no swap, but
the corosync process is locked into memory anyway.  It's also running as
SCHED_RR prio 99, competing only with multipathd and the SCHED_FIFO prio
99 kernel threads (migration/* and watchdog/*) under Linux 4.9.  I'll
try to take a closer look at the scheduling of these.  Can you recommend
some indicators to check out?


No real hints. But one question. Are you 100% sure memory is locked? 
Because we had problem where mlockall was called in wrong place so 
corosync was actually not locked and it was causing similar issues.


This behavior is fixed by
https://github.com/corosync/corosync/commit/238e2e62d8b960e7c10bfa0a8281d78ec99f3a26



Are scheduling delays expected to generate TOTEM membership "changes"
without any leaving and joining nodes?


Yes it is




As a start you can try what message say = Consider token timeout
increase. Currently you have 3 seconds, in theory 6 second should be
enough.


OK, thanks for the tip.  Can I do this on-line, without shutting down
Corosync?



Corosync way is to just edit/copy corosync.conf on all nodes and call 
corosync-cfgtool -R on one of the nodes (crmsh/pcs may have better way).


Regards,
  Honza


___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] corosync race condition when node leaves immediately after joining

[Jan Friesse]

Jonathan,
I believe main "problem" is votequorum ability to work during sync phase 
(votequorum is only one service with this ability, see 
votequorum_overview.8 section VIRTUAL SYNCHRONY)...



Hi ClusterLabs,

I'm seeing a race condition in corosync where votequorum can have
incorrect membership info when a node joins the cluster then leaves very
soon after.

I'm on corosync-2.3.4 plus my patch
https://github.com/corosync/corosync/pull/248. That patch makes the
problem readily reproducible but the bug was already present.

Here's the scenario. I have two hosts, cluster1 and cluster2. The
corosync.conf on cluster2 is:

 totem {
   version: 2
   cluster_name: test
   config_version: 2
   transport: udpu
 }
 nodelist {
   node {
 nodeid: 1
 ring0_addr: cluster1
   }
   node {
 nodeid: 2
 ring0_addr: cluster2
   }
 }
 quorum {
   provider: corosync_votequorum
   auto_tie_breaker: 1
 }
 logging {
   to_syslog: yes
 }

The corosync.conf on cluster1 is the same except with "config_version: 1".

I start corosync on cluster2. When I start corosync on cluster1, it
joins and then immediately leaves due to the lower config_version.
(Previously corosync on cluster2 would also exit but with
https://github.com/corosync/corosync/pull/248 it remains alive.)

But often at this point, cluster1's disappearance is not reflected in
the votequorum info on cluster2:


... Is this permanent (= until new node join/leave it , or it will fix 
itself over (short) time? If this is permanent, it's a bug. If it fixes 
itself it's result of votequorum not being virtual synchronous.


Honza



 Quorum information
 --
 Date: Tue Oct 10 16:43:50 2017
 Quorum provider:  corosync_votequorum
 Nodes:1
 Node ID:  2
 Ring ID:  700
 Quorate:  Yes

 Votequorum information
 --
 Expected votes:   2
 Highest expected: 2
 Total votes:  2
 Quorum:   2
 Flags:Quorate AutoTieBreaker

 Membership information
 --
 Nodeid  Votes Name
  2  1 cluster2 (local)

The logs on cluster1 show:

 Oct 10 16:43:37 cluster1 corosync[15750]:  [CMAP  ] Received config
version (2) is different than my config version (1)! Exiting

The logs on cluster2 show:

 Oct 10 16:43:37 cluster2 corosync[5102]:  [TOTEM ] A new membership
(10.71.218.17:588) was formed. Members joined: 1
 Oct 10 16:43:37 cluster2 corosync[5102]:  [QUORUM] This node is
within the primary component and will provide service.
 Oct 10 16:43:37 cluster2 corosync[5102]:  [QUORUM] Members[1]: 2
 Oct 10 16:43:37 cluster2 corosync[5102]:  [TOTEM ] A new membership
(10.71.218.18:592) was formed. Members left: 1
 Oct 10 16:43:37 cluster2 corosync[5102]:  [QUORUM] Members[1]: 2
 Oct 10 16:43:37 cluster2 corosync[5102]:  [MAIN  ] Completed
service synchronization, ready to provide service.

It looks like QUORUM has seen cluster1's arrival but not its departure!

When it works as expected, the state is left consistent:

 Quorum information
 --
 Date: Tue Oct 10 16:58:14 2017
 Quorum provider:  corosync_votequorum
 Nodes:1
 Node ID:  2
 Ring ID:  604
 Quorate:  No

 Votequorum information
 --
 Expected votes:   2
 Highest expected: 2
 Total votes:  1
 Quorum:   2 Activity blocked
 Flags:AutoTieBreaker

 Membership information
 --
 Nodeid  Votes Name
  2  1 cluster2 (local)

Logs on cluster1:

 Oct 10 16:58:01 cluster1 corosync[16430]:  [CMAP  ] Received config
version (2) is different than my config version (1)! Exiting

Logs on cluster2 are either:

 Oct 10 16:58:01 cluster2 corosync[17835]:  [TOTEM ] A new
membership (10.71.218.17:600) was formed. Members joined: 1
 Oct 10 16:58:01 cluster2 corosync[17835]:  [QUORUM] This node is
within the primary component and will provide service.
 Oct 10 16:58:01 cluster2 corosync[17835]:  [QUORUM] Members[1]: 2
 Oct 10 16:58:01 cluster2 corosync[17835]:  [CMAP  ] Highest config
version (2) and my config version (2)
 Oct 10 16:58:01 cluster2 corosync[17835]:  [TOTEM ] A new
membership (10.71.218.18:604) was formed. Members left: 1
 Oct 10 16:58:01 cluster2 corosync[17835]:  [QUORUM] This node is
within the non-primary component and will NOT provide any services.
 Oct 10 16:58:01 cluster2 corosync[17835]:  [QUORUM] Members[1]: 2
 Oct 10 16:58:01 cluster2 corosync[17835]:  [MAIN  ] Completed
service synchronization, ready to provide service.

... in which it looks like QUORUM has seen cluster1's arrival *and* its
departure,

or:

 Oct 10 16:59:03 cluster2 

Re: [ClusterLabs] Is "Process pause detected" triggered too easily?

[Jan Friesse]

Jean,


Hello,

As the subject line suggests, I am wondering why I see so many of these
log lines (many means about 10 times per minute, usually several in the
same second):

Sep 26 19:56:24 [950] vm0 corosync notice  [TOTEM ] Process pause detected
for 2555 ms, flushing membership messages.
Sep 26 19:56:24 [950] vm0 corosync notice  [TOTEM ] Process pause detected
for 2558 ms, flushing membership messages.

Let me add some context:
- this is observed in 3 small VMs on my laptop
- the OS is CentOS 7.3, corosync is 2.4.0-9.el7_4.2
- these VMs only run corosync, nothing else
- the VM host (my laptop) is idle 60-80% of the time
- VMs are qemu-kvm guests, connected with tap interfaces
- AND the messages only appear when, on one of the VMs, I do stop/start
corosync in a tight loop, like this:

[root@vm2 ~]# while :; do echo $(date) stop; systemctl stop corosync ;
echo $(date) start;systemctl start corosync ; done
Tue Sep 26 19:50:19 CEST 2017 stop
Tue Sep 26 19:50:21 CEST 2017 start
Tue Sep 26 19:50:21 CEST 2017 stop
Tue Sep 26 19:50:22 CEST 2017 start
...

I understand that this kind of test is stressful (and quite articial), but
I'm still surprised to see these particular messages, because it seems to
me a bit unlikely that the corosync process is not properly scheduled for
seconds at a time so frequently (several times per minute).


I don't think scheduling is the case. If scheduler would be the case 
other message (Corosync main process was not scheduled for ...) would 
kick in. This looks more like a something is blocked in totemsrp.





So I wonder if maybe there could be other explanations?

Also, it looks like the side effect is that corosync drops important
messages (I think "join" messages?), and I fear that this can lead to


You mean membership join messages? Because there are a lot (327) of them 
in log you've sent.



bigger issues with DLM (which is why I'm looking into this in the first
place).

In case that's helpful, attached are 10 minutes of corosync log and the
config file I'm using (it has 5 nodes declared, but I reproduce even with
just 3 nodes).

Thanks in advance for any suggestion!


I'll definitively try to reproduce this bug and let you know. I don't 
think any message get lost, but it's better to be on a safe side.


Regards,
  Honza




Cheers,
JM



___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org




___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] corosync race condition when node leaves immediately after joining

[Jan Friesse]

Jonathan,




On 16/10/17 15:58, Jan Friesse wrote:

Jonathan,




On 13/10/17 17:24, Jan Friesse wrote:

I've done a bit of digging and am getting closer to the root cause of
the race.

We rely on having votequorum_sync_init called twice -- once when
node 1
joins (with member_list_entries=2) and once when node 1 leaves (with
member_list_entries=1). This is important because votequorum_sync_init
marks nodes as NODESTATE_DEAD if they are not in quorum_members[]
-- so
it needs to have seen the node appear then disappear. This is
important
because get_total_votes only counts votes from nodes in state
NODESTATE_MEMBER.


So there are basically two problems.

Actually first (main) problem is that votequorum_sync_init is ever
called when that node joins. It really shouldn't. And problem is
simply because calling api->shutdown_request() is not enough. Can you
try replace it with exit(1) (for testing) and reproduce the problem?
I'm pretty sure problem disappears.


No, the problem still happens :-(


Not good.



I am using the following patch:

diff --git a/exec/cmap.c b/exec/cmap.c
index de730d2..1125cef 100644
--- a/exec/cmap.c
+++ b/exec/cmap.c
@@ -406,7 +406,7 @@ static void cmap_sync_activate (void)
 log_printf(LOGSYS_LEVEL_ERROR,
 "Received config version (%"PRIu64") is different
than my config version (%"PRIu64")! Exiting",
 cmap_highest_config_version_received,
cmap_my_config_version);
-   api->shutdown_request();
+   exit(1);
 return ;
 }
  }
diff --git a/exec/main.c b/exec/main.c
index b0d5639..4fd3e68 100644
--- a/exec/main.c
+++ b/exec/main.c
@@ -627,6 +627,7 @@ static void deliver_fn (
 ((void *)msg);
 }

+   log_printf(LOGSYS_LEVEL_NOTICE, "executing '%s' exec_handler_fn
%p for node %d (fn %d)", corosync_service[service]->name,
corosync_service[service]->exec_engine[fn_id].exec_handler_fn, nodeid,
fn_id);
 corosync_service[service]->exec_engine[fn_id].exec_handler_fn
 (msg, nodeid);
  }
diff --git a/exec/votequorum.c b/exec/votequorum.c
index 1a97c6d..7c0f34f 100644
--- a/exec/votequorum.c
+++ b/exec/votequorum.c
@@ -2099,6 +2100,7 @@ static void
message_handler_req_exec_votequorum_nodeinfo (
 node->flags = req_exec_quorum_nodeinfo->flags;
 node->votes = req_exec_quorum_nodeinfo->votes;
 node->state = NODESTATE_MEMBER;
+   log_printf(LOGSYS_LEVEL_NOTICE,
"message_handler_req_exec_votequorum_nodeinfo (%p) marking node %d as
MEMBER", message_handler_req_exec_votequorum_nodeinfo, nodeid);

 if (node->flags & NODE_FLAGS_LEAVING) {
 node->state = NODESTATE_LEAVING;

When it's working correctly I see this:

1508151960.072927 notice  [TOTEM ] A new membership (10.71.218.17:2304)
was formed. Members joined: 1
1508151960.073082 notice  [SYNC  ] calling sync_init on service
'corosync configuration map access' (0) with my_member_list_entries = 2
1508151960.073150 notice  [MAIN  ] executing 'corosync configuration map
access' exec_handler_fn 0x55b5eb504ca0 for node 1 (fn 0)
1508151960.073197 notice  [MAIN  ] executing 'corosync configuration map
access' exec_handler_fn 0x55b5eb504ca0 for node 2 (fn 0)
1508151960.073238 notice  [SYNC  ] calling sync_init on service
'corosync cluster closed process group service v1.01' (1) with
my_member_list_entries = 2
1508151961.073033 notice  [TOTEM ] A processor failed, forming new
configuration.

When it's not working correctly I see this:

1508151908.447584 notice  [TOTEM ] A new membership (10.71.218.17:2292)
was formed. Members joined: 1
1508151908.447757 notice  [MAIN  ] executing 'corosync vote quorum
service v1.0' exec_handler_fn 0x558b39fbbaa0 for node 1 (fn 0)
1508151908.447866 notice  [VOTEQ ]
message_handler_req_exec_votequorum_nodeinfo (0x558b39fbbaa0) marking
node 1 as MEMBER
1508151908.447972 notice  [VOTEQ ] get_total_votes: node 1 is a MEMBER
so counting vote
1508151908.448045 notice  [VOTEQ ] get_total_votes: node 2 is a MEMBER
so counting vote
1508151908.448091 notice  [QUORUM] This node is within the primary
component and will provide service.
1508151908.448134 notice  [QUORUM] Members[1]: 2
1508151908.448175 notice  [SYNC  ] calling sync_init on service
'corosync configuration map access' (0) with my_member_list_entries = 2
1508151908.448205 notice  [MAIN  ] executing 'corosync configuration map
access' exec_handler_fn 0x558b39fb3ca0 for node 1 (fn 0)
1508151908.448247 notice  [MAIN  ] executing 'corosync configuration map
access' exec_handler_fn 0x558b39fb3ca0 for node 2 (fn 0)
1508151908.448307 notice  [SYNC  ] calling sync_init on service
'corosync cluster closed process group service v1.01' (1) with
my_member_list_entries = 2
1508151909.447182 notice  [TOTEM ] A processor failed, forming new
configuration.

... and at that point I already see &

Re: [ClusterLabs] corosync race condition when node leaves immediately after joining

[Jan Friesse]

Jonathan,



On 18/10/17 14:38, Jan Friesse wrote:

Can you please try to remove
"votequorum_exec_send_nodeinfo(us->node_id);" line from votequorum.c
in the votequorum_exec_init_fn function (around line 2306) and let me
know if problem persists?


Wow! With that change, I'm pleased to say that I'm not able to reproduce
the problem at all!


Sounds good.



Is this a legitimate fix, or do we still need the call to
votequorum_exec_send_nodeinfo for other reasons?


That is good question. Calling of votequorum_exec_send_nodeinfo should 
not be needed because it's called by sync_process only slightly later.


But to mark this as a legitimate fix, I would like to find out why is 
this happening and if it is legal or not. Basically because I'm not able 
to reproduce the bug at all (and I was really trying also with various 
usleeps/packet loss/...) I would like to have more information about 
notworking_cluster1.log. Because tracing doesn't work, we need to try 
blackbox. Could you please add


icmap_set_string("runtime.blackbox.dump_flight_data", "yes");

line before api->shutdown_request(); in cmap.c ?

It should trigger dumping blackbox in /var/lib/corosync. When you 
reproduce the nonworking_cluster1, could you please ether:

- compress the file pointed by /var/lib/corosync/fdata symlink
- or execute corosync-blackbox
- or execute qb-blackbox "/var/lib/corosync/fdata"

and send it?

Thank you for your help,
  Honza



Thanks,
Jonathan



___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] pcs authentication fails on Centos 7.0 & 7.1

[Jan Friesse]

Digimer napsal(a):

On 2017-11-12 04:20 AM, Aviran Jerbby wrote:


Hi Clusterlabs mailing list,



I'm having issues running pcs authentication on RH cent os 7.0/7.1
(Please see log below).



*_It's important to mention that pcs authentication with RH cent os
7.2/7.4 and with the same setup and packages is working._*



*[root@ufm-host42-014 tmp]# cat /etc/redhat-release *

*CentOS Linux release 7.0.1406 (Core) *

*[root@ufm-host42-014 tmp]# rpm -qa | grep openssl*

*openssl-libs-1.0.2k-8.el7.x86_64*

*openssl-devel-1.0.2k-8.el7.x86_64*

*openssl-1.0.2k-8.el7.x86_64*

*[root@ufm-host42-014 tmp]# rpm -qa | grep pcs*

*pcs-0.9.158-6.el7.centos.x86_64*

*pcsc-lite-libs-1.8.8-4.el7.x86_64*

*[root@ufm-host42-014 tmp]# pcs cluster auth
ufm-host42-012.rdmz.labs.mlnx ufm-host42-013.rdmz.labs.mlnx
ufm-host42-014.rdmz.labs.mlnx -u hacluster -p "" --debug*

*Running: /usr/bin/ruby -I/usr/lib/pcsd/ /usr/lib/pcsd/pcsd-cli.rb auth*

*Environment:*

*  DISPLAY=localhost:10.0*

*  GEM_HOME=/usr/lib/pcsd/vendor/bundle/ruby*

*  HISTCONTROL=ignoredups*

*  HISTSIZE=1000*

*  HOME=/root*

*  HOSTNAME=ufm-host42-014.rdmz.labs.mlnx*

*  KDEDIRS=/usr*

*  LANG=en_US.UTF-8*

*  LC_ALL=C*

*  LESSOPEN=||/usr/bin/lesspipe.sh %s*

*  LOGNAME=root*

*
LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=01;05;37;41:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.jpg=01;35:*.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.

v
ob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.axv=01;35:*.anx=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=01;36:*.au=01;36:*.flac=01;36:*.mid=01;36:*.midi=01;36:*.mka=01;36:*.mp3=01;36:*.mpc=01;36:*.ogg=01;36:*.ra=01;36:*.wav=01;36:*.axa=01;36:*.oga=01;36:*.spx=01;36:*.xspf=01;36:*


* MAIL=/var/spool/mail/root*

*  OLDPWD=/root*

*
PATH=/usr/lib64/qt-3.3/bin:/root/perl5/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin*

*  PCSD_DEBUG=true*

*  PCSD_NETWORK_TIMEOUT=60*

*  PERL5LIB=/root/perl5/lib/perl5:*

*  PERL_LOCAL_LIB_ROOT=:/root/perl5*

*  PERL_MB_OPT=--install_base /root/perl5*

*  PERL_MM_OPT=INSTALL_BASE=/root/perl5*

*  PWD=/tmp*

*  QTDIR=/usr/lib64/qt-3.3*

*  QTINC=/usr/lib64/qt-3.3/include*

*  QTLIB=/usr/lib64/qt-3.3/lib*

*  QT_GRAPHICSSYSTEM=native*

*  QT_GRAPHICSSYSTEM_CHECKED=1*

*  QT_PLUGIN_PATH=/usr/lib64/kde4/plugins:/usr/lib/kde4/plugins*

*  SHELL=/bin/bash*

*  SHLVL=1*

*  SSH_CLIENT=10.208.0.12 47232 22*

*  SSH_CONNECTION=10.208.0.12 47232 10.224.40.143 22*

*  SSH_TTY=/dev/pts/0*

*  TERM=xterm*

*  USER=root*

*  XDG_RUNTIME_DIR=/run/user/0*

*  XDG_SESSION_ID=6*

*  _=/usr/sbin/pcs*

*--Debug Input Start--*

*{"username": "hacluster", "local": false, "nodes":
["ufm-host42-014.rdmz.labs.mlnx", "ufm-host42-013.rdmz.labs.mlnx",
"ufm-host42-012.rdmz.labs.mlnx"], "password": "", "force": false}*

*--Debug Input End--*

* *

*Finished running: /usr/bin/ruby -I/usr/lib/pcsd/
/usr/lib/pcsd/pcsd-cli.rb auth*

*Return value: 0*

*--Debug Stdout Start--*

*{*

*  "status": "ok",*

*  "data": {*

*"auth_responses": {*

*  "ufm-host42-014.rdmz.labs.mlnx": {*

*"status": "noresponse"*

* },*

*  "ufm-host42-012.rdmz.labs.mlnx": {*

*"status": "noresponse"*

*  },*

*  "ufm-host42-013.rdmz.labs.mlnx": {*

*"status": "noresponse"*

*  }*

*},*

*"sync_successful": true,*

*"sync_nodes_err": [*

* *

*],*

*"sync_responses": {*

*}*

*  },*

*  "log": [*

*"I, [2017-11-07T19:52:27.434067 #25065]  INFO -- : PCSD Debugging
enabled\n",*

*"D, [2017-11-07T19:52:27.454014 #25065] DEBUG -- : Did not detect
RHEL 6\n",*

*"I, [2017-11-07T19:52:27.454076 #25065]  INFO -- : Running:
/usr/sbin/corosync-cmapctl totem.cluster_name\n",*

*"I, [2017-11-07T19:52:27.454127 #25065]  INFO -- : CIB USER:
hacluster, groups: \n",*

*"D, [2017-11-07T19:52:27.458142 #25065] DEBUG -- : []\n",*

*"D, [2017-11-07T19:52:27.458216 #25065] DEBUG -- : [\"Failed to
initialize the cmap API. Error CS_ERR_LIBRARY\\n\"]\n",*

*"D, 

Re: [ClusterLabs] corosync race condition when node leaves immediately after joining

[Jan Friesse]

Jonathan,
I've finished (I hope) proper fix for problem you've seen, so can you 
please try to test


https://github.com/corosync/corosync/pull/280

Thanks,
  Honza




On 31/10/17 10:41, Jan Friesse wrote:

Did you get a chance to confirm whether the workaround to remove the
final call to votequorum_exec_send_nodeinfo from votequorum_exec_init_fn
is safe?


I didn't had time to find out what exactly is happening, but I can
confirm you, that workaround is safe. It's just not a full fix and
there can still be situations when the bug appears.



The patch works well in our testing, but I'm keen to hear whether you
think this is likely to be safe for use in production.


It's safe but it's just a workaround.


Thanks for confirming.

Jonathan



___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] corosync race condition when node leaves immediately after joining

[Jan Friesse]

On 13/11/17 17:06, Jan Friesse wrote:

Jonathan,
I've finished (I hope) proper fix for problem you've seen, so can you
please try to test

https://github.com/corosync/corosync/pull/280

Thanks,
   Honza


Hi Honza,


Hi Jonathan,



Thanks very much for putting this fix together.

I'm happy to confirm that I do not see the problem with this fix.


That is a good news.



In my repro environment that normally triggers the problem once in every
2 attempts, I didn't see the problem at all after over 1000 attempts
with these patches.


Perfect. I'll wait for proper peer-review of code and merge after that.

Thanks again for your patience and big help with reproducing and testing 
of this bug.


Regards,
  Honza



Thanks!
Jonathan



___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

[ClusterLabs] Corosync 1.4.9 is available at corosync.org!

[Jan Friesse]

I am pleased to announce the latest maintenance of old stable release of 
Corosync

1.4.9 available immediately from our website at
http://build.clusterlabs.org/corosync/releases/.

This release contains just a few bugfixes.

Complete changelog for 1.4.9:
Jan Friesse (7):
  Revert "ipcc: Fix ERR_LIBRARY error if finalise called inside 
dispatch"

  ipcc: Fix ERR_LIBRARY on finalize call in dispatch
  Remove all links to old ML
  totempg: Fix memory leak
  totemconfig: Properly check transport key
  totemrrp: Fix situation when all rings are faulty
  corosync_ring_id_store: Use safer permissions


If your deployment still run on top of Corosync 1.x it's really highly 
recommended to upgrade.


Thanks/congratulations to all people that contributed to achieve this
great milestone.


___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Corosync 1.4.9 is available at corosync.org!

[Jan Friesse]

Digimer,


On 2017-12-08 11:46 AM, Jan Friesse wrote:

I am pleased to announce the latest maintenance of old stable release of
Corosync
1.4.9 available immediately from our website at
http://build.clusterlabs.org/corosync/releases/.

This release contains just a few bugfixes.

Complete changelog for 1.4.9:
Jan Friesse (7):
   Revert "ipcc: Fix ERR_LIBRARY error if finalise called inside
dispatch"
   ipcc: Fix ERR_LIBRARY on finalize call in dispatch
   Remove all links to old ML
   totempg: Fix memory leak
   totemconfig: Properly check transport key
   totemrrp: Fix situation when all rings are faulty
   corosync_ring_id_store: Use safer permissions


If your deployment still run on top of Corosync 1.x it's really highly
recommended to upgrade.

Thanks/congratulations to all people that contributed to achieve this
great milestone.


Is it recommended for RHEL 6.x users to upgrade? I assume there won't be
an official RPM update made available.


RHEL 6.x packages ether already contains most of the patches or will 
contain them in next version.


Regards,
  Honza


___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] corosync race condition when node leaves immediately after joining

[Jan Friesse]

Jonathan,


Hi Honza,

On 19/10/17 17:05, Jonathan Davies wrote:

On 19/10/17 16:56, Jan Friesse wrote:

Jonathan,




On 18/10/17 16:18, Jan Friesse wrote:

Jonathan,



On 18/10/17 14:38, Jan Friesse wrote:

Can you please try to remove
"votequorum_exec_send_nodeinfo(us->node_id);" line from votequorum.c
in the votequorum_exec_init_fn function (around line 2306) and
let me
know if problem persists?


Wow! With that change, I'm pleased to say that I'm not able to
reproduce
the problem at all!


Sounds good.



Is this a legitimate fix, or do we still need the call to
votequorum_exec_send_nodeinfo for other reasons?


That is good question. Calling of votequorum_exec_send_nodeinfo should
not be needed because it's called by sync_process only slightly later.

But to mark this as a legitimate fix, I would like to find out why is
this happening and if it is legal or not. Basically because I'm not
able to reproduce the bug at all (and I was really trying also with
various usleeps/packet loss/...) I would like to have more information
about notworking_cluster1.log. Because tracing doesn't work, we need
to try blackbox. Could you please add

icmap_set_string("runtime.blackbox.dump_flight_data", "yes");

line before api->shutdown_request(); in cmap.c ?

It should trigger dumping blackbox in /var/lib/corosync. When you
reproduce the nonworking_cluster1, could you please ether:
- compress the file pointed by /var/lib/corosync/fdata symlink
- or execute corosync-blackbox
- or execute qb-blackbox "/var/lib/corosync/fdata"

and send it?


Attached, along with the "debug: trace" log from cluster2.


Thanks a lot for the logs. I'm - finally - able to reproduce bug
(with the 2 artificial pauses - included at the end of the mail).
I'll try to fix the main bug (what may take some time, eventho I have
kind of idea what is happening) and let you know.


Glad to hear that the logs are useful and you're able to reproduce the
problem! I look forward to hearing what you come up with, and am happy
to test out patches if that would help.


Did you get a chance to confirm whether the workaround to remove the
final call to votequorum_exec_send_nodeinfo from votequorum_exec_init_fn
is safe?


I didn't had time to find out what exactly is happening, but I can 
confirm you, that workaround is safe. It's just not a full fix and there 
can still be situations when the bug appears.




The patch works well in our testing, but I'm keen to hear whether you
think this is likely to be safe for use in production.


It's safe but it's just a workaround.


Regards,
  Honza



Thanks,
Jonathan



___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

[ClusterLabs] Corosync-qdevice 3.0 - Alpha2 is available at GitHub!

[Jan Friesse]

I am pleased to announce the second testing (Alpha 2) release of 
Corosync-Qdevice 3.0 available immediately from GitHub at
https://github.com/corosync/corosync-qdevice/releases as 
corosync-qdevice-2.91.0.


This release contains mostly bugfixes.

Complete changelog for Alpha 2 (compared to Alpha 1):

Bin Liu (1):
  qdevice: optarg should be str in init_from_cmap

Ferenc Wágner (1):
  Fix typo: sucesfully -> successfully

Jan Friesse (12):
  spec: Modernize spec file a bit
  init: Quote subshell result properly
  Quote certutils scripts properly
  Fix NULL pointer dereference
  Nodelist is set into string not array
  Check if user_data can be dereferenced
  Add safer wrapper of strtoll
  qdevice: Replace strtol by strtonum
  qnetd: Replace strtol by strtonum
  msgio: Fix reading of msg longer than PR_INT32_MAX
  msgio: Remove unused code
  spec: Remove unused clean section


We did our best to test this release as best as we could, but still take 
it as an Alpha version.


Thanks/congratulations to all people that contributed to achieve this
great milestone.
___
Users mailing list: Users@clusterlabs.org
https://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] corosync 2.4 CPG config change callback

[Jan Friesse]

Hi Thomas,

Hi,

Am 04/25/2018 um 09:57 AM schrieb Jan Friesse:

Thomas Lamprecht napsal(a):

On 4/24/18 6:38 PM, Jan Friesse wrote:

On 4/6/18 10:59 AM, Jan Friesse wrote:

Thomas Lamprecht napsal(a):

Am 03/09/2018 um 05:26 PM schrieb Jan Friesse:
I've tested it too and yes, you are 100% right. Bug is there and 
it's
pretty easy to reproduce when node with lowest nodeid is paused. 
It's

slightly harder when node with higher nodeid is paused.



Do you were able to make some progress on this issue?


Ya, kind of. Sadly I had to work on different problem, but I'm 
expecting to sent patch next week.




I guess the different problems where the ones related to the issued 
CVEs :)


Yep.

Also I've spent quite a lot of the time thinking about best possible 
solution. CPG is quite old, it was full of weird bugs and risk of 
breakage is very high.


Anyway, I've decided to not to try hack what is apparently broken 
and just go for risky but proper solution (= needs a LOT more 
testing, but so far looks good).




I did not looked deep into how your revert plays out with the
mentioned commits of the heuristics approach, but this fix would
mean to bring corosync back to a state it had already, and thus
was already battle tested?


Yep, but not fully. Important change was to use joinlists as 
authoritative source of information about other node clients, so I 
believe that solved problems which should had been "solved" by 
downlist heuristics.





Patch and approach seems good to me, with my limited knowledge,
when looking at the various "bandaid" fix commits you mentioned.


Patch is in PR (needle): https://github.com/corosync/corosync/pull/347



Much thanks! First tests work well here.
I could not yet reproduce the problem with the patch applied in both,
testcpg and our cluster configuration file system.


That's good to hear :)



I'll let it run


Perfect.




Just wanted to give some quick feedback.
We deployed this to your community repository about a week ago (after
another week of successful testing), we had no negative feedback or
issues reported or seen yet, with (strong lower bound) > 10k systems
running the fix by now.


Thanks, that's exciting news.



I saw just now that you merged it into needle and master, so, while a 
bit late, this just backs the confidence into the fix up.


Definitively not late until it's released :)



Much thanks for your, and the reviewers, work!


Yep, you are welcomed.

Honza



cheers,
Thomas



___
Users mailing list: Users@clusterlabs.org
https://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

[ClusterLabs] Corosync 3.0 - Alpha3 is available at corosync.org!

[Jan Friesse]

I am pleased to announce the second testing (Alpha 3) release of 
Corosync 3.0 (codename Camelback) available immediately from our website at

http://build.clusterlabs.org/corosync/releases/ as corosync-2.99.2.

You can also download RPMs for various distributions from CI 
https://kronosnet.org/builds/.


This release contains very important fixes so it's "must to have" for 
everybody with previous alpha versions.


1) Fix Alpha-2 regression which may result in membership not created
2) Fix long time problem with CPG where paused node may not register 
other nodes leave


Complete changelog for Alpha 3 (compared to Alpha 2):
Chris Lamb (1):
  man: Make the manpages reproducible

Fabio M. Di Nitto (1):
  Drop all references to SECURITY file

Ferenc Wágner (4):
  Fix typo: sucesfully -> successfully
  Fix typo: defualt -> default
  NSS_NoDB_Init: the parameter is reserved, must be NULL
  tools: don't distribute what we can easily make

Jan Friesse (7):
  totemsrp: Implement sanity checks of received msgs
  totemsrp: Check join and leave msg length
  SECURITY: Remove SECURITY file
  totemsrp: Fix srp_addr_compare
  totemsrp: Log proc/fail lists in memb_join_process
  totemsrp: Fix leave message regression
  cpg: Inform clients about left nodes during pause

Rytis Karpuška (1):
  cpg: Handle fragmented message sending interrupt


We did our best to test this release as best as we could, but still take 
it as an Alpha version.


For anybody who migrates from stable version please consult 
http://people.redhat.com/ccaulfie/docs/KnetCorosync.pdf about changes in 
Corosync 3.


Thanks/congratulations to all people that contributed to achieve this
great milestone.
___
Users mailing list: Users@clusterlabs.org
https://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] corosync race condition when node leaves immediately after joining

[Jan Friesse]

Jonathan,




On 18/10/17 16:18, Jan Friesse wrote:

Jonathan,



On 18/10/17 14:38, Jan Friesse wrote:

Can you please try to remove
"votequorum_exec_send_nodeinfo(us->node_id);" line from votequorum.c
in the votequorum_exec_init_fn function (around line 2306) and let me
know if problem persists?


Wow! With that change, I'm pleased to say that I'm not able to reproduce
the problem at all!


Sounds good.



Is this a legitimate fix, or do we still need the call to
votequorum_exec_send_nodeinfo for other reasons?


That is good question. Calling of votequorum_exec_send_nodeinfo should
not be needed because it's called by sync_process only slightly later.

But to mark this as a legitimate fix, I would like to find out why is
this happening and if it is legal or not. Basically because I'm not
able to reproduce the bug at all (and I was really trying also with
various usleeps/packet loss/...) I would like to have more information
about notworking_cluster1.log. Because tracing doesn't work, we need
to try blackbox. Could you please add

icmap_set_string("runtime.blackbox.dump_flight_data", "yes");

line before api->shutdown_request(); in cmap.c ?

It should trigger dumping blackbox in /var/lib/corosync. When you
reproduce the nonworking_cluster1, could you please ether:
- compress the file pointed by /var/lib/corosync/fdata symlink
- or execute corosync-blackbox
- or execute qb-blackbox "/var/lib/corosync/fdata"

and send it?


Attached, along with the "debug: trace" log from cluster2.


Thanks a lot for the logs. I'm - finally - able to reproduce bug 
(with the 2 artificial pauses - included at the end of the mail). I'll 
try to fix the main bug (what may take some time, eventho I have kind of 
idea what is happening) and let you know.


Thanks a lot for all the logs and your super helpful cooperation,
  Honza





Thanks,
Jonathan


diff --git a/exec/cpg.c b/exec/cpg.c
index 78ac1e9..8a2ce6a 100644
--- a/exec/cpg.c
+++ b/exec/cpg.c
@@ -591,6 +591,14 @@ static void cpg_sync_init (
 static int cpg_sync_process (void)
 {
int res = -1;
+   static int pause = 0;
+
+   if (pause < 20) {
+   pause++;
+   return (-1);
+   } else {
+   pause = 0;
+   }

if (my_sync_state == CPGSYNC_DOWNLIST) {
res = cpg_exec_send_downlist();
diff --git a/exec/totemsrp.c b/exec/totemsrp.c
index 91c5423..1240229 100644
--- a/exec/totemsrp.c
+++ b/exec/totemsrp.c
@@ -2156,6 +2156,7 @@ static void memb_state_operational_enter (struct 
totemsrp_instance *instance)

memcpy (>my_old_ring_id, >my_ring_id,
sizeof (struct memb_ring_id));

+   poll(NULL, 0, 600);
return;
 }



___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

[ClusterLabs] Corosync 2.4.3 is available at corosync.org!

[Jan Friesse]

I am pleased to announce the latest maintenance release of Corosync
2.4.3 available immediately from our website at
http://build.clusterlabs.org/corosync/releases/.

This release contains a lot of fixes. New feature is support for 
heuristics in qdevice.


Complete changelog for 2.4.3:
Adrian Vondendriesch (1):
  doc: document watchdog_device parameter

Andrew Price (1):
  Main: Call mlockall after fork

Bin Liu (7):
  Totempg: remove duplicate memcpy in mcast_msg func
  Qdevice: fix spell errors in qdevice
  logconfig: Do not overwrite logger_subsys priority
  totemconfig: Prefer nodelist over bindnetaddr
  cpghum: Fix printf of size_t variable
  Qnetd lms: Use UTILS_PRI_RING_ID printf format str
  wd: Report error when close of wd fails

Christine Caulfield (6):
  votequorum: Don't update expected_votes display if value is too high
  votequorum: simplify reconfigure message handling
  quorumtool: Add option to show all node addresses
  main: Don't ask libqb to handle segv, it doesn't work
  man: Document -a option to corosync-quorumtool
  main: use syslog & printf directly for early log messages

Edwin Torok (1):
  votequorum: make atb consistent on nodelist reload

Ferenc Wágner (7):
  Fix typo: Destorying -> Destroying
  init: Add doc URIs to the systemd service files
  wd: fix typo
  corosync.conf.5: Fix watchdog documentation
  corosync.conf.5: add warning about slow watchdogs
  wd: remove extra capitalization typo
  corosync.conf.5: watchdog support is conditional

Hideo Yamauchi (1):
  notifyd: Add the community name to an SNMP trap

Jan Friesse (11):
  Logsys: Change logsys syslog_priority priority
  totemrrp: Fix situation when all rings are faulty
  main: Display reason why cluster cannot be formed
  totem: Propagate totem initialization failure
  totemcrypto: Refactor symmetric key importing
  totemcrypto: Use different method to import key
  main: Add option to set priority
  main: Add support for libcgroup
  totemcrypto: Fix compiler warning
  cmap: Remove noop highest config version check
  qdevice: Add support for heuristics

Jan Pokorný (2):
  Spec: drop unneeded dependency
  Spec: make internal dependencies arch-qualified

Jonathan Davies (1):
  cmap: don't shutdown highest config_version node

Kazunori INOUE (1):
  totemudp: Remove memb_join discarding

Keisuke MORI (1):
  Spec: fix arch-qualified dependencies

Khem Raj (1):
  Include fcntl.h for F_* and O_* defines

Masse Nicolas (1):
  totemudp: Retry if bind fails

Richard B Winters (1):
  Remove deprecated doxygen flags

Takeshi MIZUTA (3):
  man: Fix typos in man page
  man: Modify man-page according to command usage
  Remove redundant header file inclusion

yuusuke (1):
  upstart: Add softdog module loading example


Upgrade is (as usually) highly recommended.

Thanks/congratulations to all people that contributed to achieve this
great milestone.


___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] [IMPORTANT] Fatal, yet rare issue verging on libqb's design flaw and/or it's use corosync around daemon-forking

[Jan Friesse]

It was discovered that corosync exposes itself for a self-crash
under rare circumstance whereby corosync executable is run when there
is already a daemon instance around (does not apply to corosync serving
without any backgrounding, i.e. launched with "-f" switch).

Such a circumstance can be provoked unattendedly by the third party,
incl. "corosync -v" probe triggered internally by pcs (since 9e19af58
~ 0.9.145), which is what makes the root cause analysis of such
inflicted crash somewhat difficult to guess & analyze (the other
reason may be rather runaway core dump if produced at all due to
fencing coming, based on the few observed cases).

The problems comes from the fact that corosync is arranged such that
the logging is set up very early, even before the main control flow
of the program starts.  And part of this early enabling is also
starting "blackbox" recording, which uses mmap'd file stored in
/dev/shm that, moreover, only varies on PID that is part of the file
name -- and when corosync perform the fork so as to detach itself
from the environment it started it, such PID is free to be reused.
And against all odds, when that happens with this fresh new corosync
process, it happily mangles the file underneath the former daemon one,
leading to crashes indicated by SIGBUS, rarely also SIGFPE.

* * *

There are two quick mitigation techniques that can be readily applied:

1. make on-PATH corosync executable rather a "careful" wrapper:

cp -a /sbin/corosync /sbin/corosync.orig
> /sbin/corosync cat < /proc/sys/kernel/pid_max

* * *

The claim this problem is fixed, at least all three mentioned components
will have to do its part to limit the problem in the future:

- corosync (do something new after fork?)


Patch proposal:

https://github.com/corosync/corosync/pull/308

Also problem is really very rare and reproducing it is quite hard.



- libqb (be more careful about the crashing condition?)

- pcs (either find a different way to check "is-old-stack", or double
   check if the probe's PID doesn't happen to hit the one baked in
   existing files in /dev/shm?)

so as to cover the-counterpart-not-up2date cases, and also will likely
lead to augmenting and/or overloading semantics of libqb's API.
All is being worked on, stay tuned.



___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org




___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] What does these logs mean in corosync.log

[Jan Friesse]

lkxjtu,
I will just comment corosync log.


These logs are both print when system is abnormal, I am very confused what they 
mean. Does anyone know what they mean? Thank you very much
corosync version   2.4.0
pacemaker version  1.1.16

1)
Feb 01 10:57:58 [18927] paas-controller-192-167-0-2   crmd:  warning: 
find_xml_node:Could not find parameters in resource-agent.

2)
Feb 08 00:00:10 [32899] paas-controller-22-0-2-10 corosync notice  [TOTEM ] 
orf_token_rtr Retransmit List: 19f1bb
Feb 08 00:00:10 [32899] paas-controller-22-0-2-10 corosync notice  [TOTEM ] 
orf_token_rtr Retransmit List: 19f1bb
Feb 08 00:00:10 [32899] paas-controller-22-0-2-10 corosync notice  [TOTEM ] 
orf_token_rtr Retransmit List: 19f1bb
Feb 08 00:00:10 [32899] paas-controller-22-0-2-10 corosync notice  [TOTEM ] 
orf_token_rtr Retransmit List: 19f1bb
Feb 08 00:00:10 [32899] paas-controller-22-0-2-10 corosync notice  [TOTEM ] 
orf_token_rtr Retransmit List: 19f1bb
Feb 08 00:00:25 [32899] paas-controller-22-0-2-10 corosync notice  [TOTEM ] 
orf_token_rtr Retransmit List: 19f1cf
Feb 08 00:00:25 [32899] paas-controller-22-0-2-10 corosync notice  [TOTEM ] 
orf_token_rtr Retransmit List: 19f1cf
Feb 08 00:00:25 [32899] paas-controller-22-0-2-10 corosync notice  [TOTEM ] 
orf_token_rtr Retransmit List: 19f1cf
Feb 08 00:00:25 [32899] paas-controller-22-0-2-10 corosync notice  [TOTEM ] 
orf_token_rtr Retransmit List: 19f1cf



The question is. How often you get this lines? If there are only few of 
them, it's nothing to worry, it just means that corosync message(s) 
was/were lost and Corosync tries to resend them again.


But if you have a lot of these, followed by new membership forming, it 
means you ether:
- are using multicast, but messages got lost for some reason (usually 
switches) -> try UDPU
- MTU of network is smaller than 1500 bytes and fragmentation is not 
allowed -> try reduce totem.netmtu


Honza



3)
Feb 11 22:57:17 [5206] paas-controller-192-20-20-6cib: info: 
crm_compress_string:   Compressed 233922 bytes into 11533 (ratio 20:1) in 51ms
Feb 11 22:57:21 [5206] paas-controller-192-20-20-6cib: info: 
crm_compress_string:   Compressed 233922 bytes into 11522 (ratio 20:1) in 53ms
Feb 11 22:57:21 [5206] paas-controller-192-20-20-6cib: info: 
crm_compress_string:   Compressed 233922 bytes into 11537 (ratio 20:1) in 45ms
Feb 11 22:57:21 [5206] paas-controller-192-20-20-6cib: info: 
crm_compress_string:   Compressed 233922 bytes into 11514 (ratio 20:1) in 47ms
Feb 11 22:57:22 [5206] paas-controller-192-20-20-6cib: info: 
crm_compress_string:   Compressed 233922 bytes into 11536 (ratio 20:1) in 50ms
Feb 11 22:57:22 [5206] paas-controller-192-20-20-6cib: info: 
crm_compress_string:   Compressed 233922 bytes into 11551 (ratio 20:1) in 51ms
Feb 11 22:57:22 [5206] paas-controller-192-20-20-6cib: info: 
crm_compress_string:   Compressed 233922 bytes into 11524 (ratio 20:1) in 54ms
Feb 11 22:57:22 [5206] paas-controller-192-20-20-6cib: info: 
crm_compress_string:   Compressed 233922 bytes into 11545 (ratio 20:1) in 60ms
Feb 11 22:57:22 [5206] paas-controller-192-20-20-6cib: info: 
crm_compress_string:   Compressed 233922 bytes into 11536 (ratio 20:1) in 54ms
Feb 11 22:57:25 [5206] paas-controller-192-20-20-6cib: info: 
crm_compress_string:   Compressed 233922 bytes into 11522 (ratio 20:1) in 61ms



___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org



___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Does CMAN Still Not Support Multipe CoroSync Rings?

[Jan Friesse]

Eric,

General question. I tried to set up a cman + corosync + pacemaker cluster using two corosync rings. When I start the cluster, everything works fine, except when I do a 'corosync-cfgtool -s' it only shows one ring. I tried manually editing the /etc/cluster/cluster.conf file adding two  


AFAIK cluster.conf should be edited so altname is used. So something 
like in this example: 
https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/cluster_administration/s1-config-rrp-cli-ca


I don't think you have to add altmulticast.

Honza

sections, but then cman complained that I didn't have a multicast 
address specified, even though I did. I tried editing the 
/etc/corosdync/corosync.conf file, and then I could get two rings, but 
the nodes would not both join the cluster. Bah! I did some reading and 
saw that cman didn't support multiple rings years ago. Did it never get 
updated?


[sig]




___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org



___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Pacemaker 2.0.0-rc1 now available

[Jan Friesse]

Ken Gaillot napsal(a):

On Fri, 2018-02-16 at 15:06 -0600, Ken Gaillot wrote:

Source code for the first release candidate for Pacemaker version
2.0.0
is now available at:

   https://github.com/ClusterLabs/pacemaker/releases/tag/Pacemaker-2.0
.0
-rc1

The main goal of the change from Pacemaker 1 to 2 is to drop support
for deprecated legacy usage, in order to make the code base more
maintainable going into the future. As such, this release involves a
net drop of more than 20,000 lines of code!

Rolling (live) upgrades are possible only from Pacemaker 1.1.11 or
later, on top of corosync 2. Other setups can be upgraded with the
cluster stopped.

It is recommended to run "cibadmin --upgrade" (or the equivalent in
your higher-level tool of choice) both before and after the upgrade.

The final 2.0.0 release will automatically transform most of the
dropped older syntax to the newer form. However, this functionality
is
not yet complete in rc1.

The most significant changes in this release include:

* Support has been dropped for heartbeat and corosync 1 (whether
using
CMAN or plugin), and many legacy aliases for cluster options
(including
default-resource-stickiness, which should be set as resource-
stickiness
in rsc_defaults instead).

* The default location of the Pacemaker detail log is now
/var/log/pacemaker/pacemaker.log, and Pacemaker will no longer use
Corosync's logging preferences. Options are available in the
configure
script to change the default log locations.


Thank you a lot!



* The master XML tag is deprecated (though still supported) in favor
of
using the standard clone tag with a new "promotable" meta-attribute
set
to true. The "master-max" and "master-node-max" master meta-
attributes
are deprecated in favor of new "promoted-max" and "promoted-node-max"
clone meta-attributes. Documentation now refers to these as
promotable
clones rather than master/slave, stateful or multistate clones.

* The record-pending option now defaults to true, which means pending
actions will be shown in status displays.

* Three minor regressions introduced in 1.1.18, and one introduced in
1.1.17, have been fixed.

More details are available in the change log:

   https://github.com/ClusterLabs/pacemaker/blob/2.0/ChangeLog

and in a special wiki page for the 2.0 release:

   https://wiki.clusterlabs.org/wiki/Pacemaker_2.0_Changes

Everyone is encouraged to download, compile and test the new release.
We do many regression tests and simulations, but we can't cover all
possible use cases, so your feedback is important and appreciated.

Many thanks to all contributors of source code to this release,
including


Whoops, hit send too soon :)

Andrew Beekhof, Bin Liu, Gao,Yan, Jan Pokorný, and Ken Gaillot



___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] corosync.conf token configuration

[Jan Friesse]

Adrián,


Hello,

I was wondering if someone have some description of the parameters: token, 
token_retransmits, token_retransmits_before_loss_const and consensus. I have 
read about it in the man page of corosync.conf but trying some configuration of 
the cluster I realized that I did not control when the new configuration or 
stonith was going to happened. I have tried corosync 1.X and 2.X in several 
virtual servers (debian-9).


token timeout = time after which token is declared as lost by node and 
when node begins to form new membership = token + (number_of_nodes - 2) 
* token_coefficient. This applies for corosync 2.x, 1.x doesn't have the 
token_coefficient.


consensus is just maximum time to wait till all nodes agrees on new 
membership (so consensus timeout has effect after token timeout). On 
timeout, node tries to create different membership.


token_retransmits_before_loss_const is used only for computation of 
token_retransmit, formula is token_retransmits = token_timeout / 
(token_retransmits_before_loss_const + 0.2)


token_retransmit is used for making membership
more stable. If token is not received in given time, previous token is
retransmitted. So If the token was lost on the net (and because of UDP 
it's possible), it may be retransmitted.




corosync.conf:

# How long before declaring a token lost (ms)
token: 2

# Consensus, time before token lost to stonith the server (ms)
# consensus: 6

# Interval between tokens (ms)
# token_retransmit: 1

# How many token retransmited before forming a new configuration
token_retransmits_before_loss_const: 20

I expected to declare the token lost before 20s after the processor failed (for example, 
connection lost to the servers), then "token_retransmit_before_lost_const" should act 
(I don’t know how it works) and the stonith occurs 24s before the message of “new configuration” 
(default consensus = 1,2 * token -> 1,2 * 20s = 24s). In brief, the cluster is barely 44s 
awake without connection before reboot (stonith) is done, in contrast, I expect the parameter 
"token_retransmits_before_loss_const: 20” to delay the token lost whilel is trying to 
reconnect.


So it means 20 sec to detect the failure + (1.2 * 20 = 24) to form a new 
membership -> 44 sec.




I am right?

On the other hand, If I use the parameter consensus, I can calculate exactly 
when the stonith is going to happen.

Please, if someone knows the answer I will appreciate any help.
Thank you




Regards,
  Honza





___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org




___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org