Re: [ClusterLabs] why resources are restarted when a node rejoins a cluster?

[Kristián Feldsam]

It looks like stickiness not configured.

http://clusterlabs.org/doc/en-US/Pacemaker/1.1-plugin/html/Clusters_from_Scratch/ch05s03s02.html

S pozdravem Kristián Feldsam
Tel.: +420 773 303 353, +421 944 137 535
E-mail.: supp...@feldhost.cz

www.feldhost.cz - FeldHost™ – profesionální hostingové a serverové služby za 
adekvátní ceny.

FELDSAM s.r.o.
V rohu 434/3
Praha 4 – Libuš, PSČ 142 00
IČ: 290 60 958, DIČ: CZ290 60 958
C 200350 vedená u Městského soudu v Praze

Banka: Fio banka a.s.
Číslo účtu: 2400330446/2010
BIC: FIOBCZPPXX
IBAN: CZ82 2010  0024 0033 0446

> On 25 Jul 2017, at 16:06, Ken Gaillot  wrote:
> 
> On Mon, 2017-07-24 at 23:07 -0400, Digimer wrote:
>> On 2017-07-24 11:04 PM, ztj wrote:
>>> Hi all,
>>> I have 2 Centos nodes with heartbeat and pacemaker-1.1.13 installed,
>>> and almost everything is working fine, I have only apache configured
>>> for testing, when a node goes down the failover is done correctly,
>>> but there's a problem when a node failbacks.
>>> 
>>> For example, let's say that Node1 has the lead on apache resource,
>>> then I reboot Node1, so Pacemaker detect it goes down, then apache
>>> is promoted to the Node2 and it keeps there running fine, that's
>>> fine, but when Node1 recovers and joins the cluster again, apache is
>>> restarted on Node2 again.
>>> 
>>> Anyone knows, why resources are restarted when a node rejoins a
>>> cluster? thanks
> 
> That's not the default behavior, so something else is going on. Show
> your configuration (with any sensitive information removed) for more
> help.
> 
>> You sent this to the moderators, not the list.
>> 
>> Please don't use heartbeat, it is extremely deprecated. Please switch
>> to corosync.
> 
> Since it's CentOS, it has to be corosync, unless heartbeat was compiled
> locally.
> 
>> 
>> To offer any other advice, you need to share your config and the logs
>> from both nodes. Please respond to the list, not
>> developers-ow...@clusterlabs.org.
>> 
>> digimer
>> -- 
>> Digimer
>> Papers and Projects: https://alteeve.com/w/
>> "I am, somehow, less interested in the weight and convolutions of Einstein’s 
>> brain than in the near certainty that people of equal talent have lived and 
>> died in cotton fields and sweatshops." - Stephen Jay Gould
>> ___
>> Users mailing list: Users@clusterlabs.org
>> http://lists.clusterlabs.org/mailman/listinfo/users
>> 
>> Project Home: http://www.clusterlabs.org
>> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>> Bugs: http://bugs.clusterlabs.org
> 
> -- 
> Ken Gaillot 
> 
> 
> 
> 
> 
> ___
> Users mailing list: Users@clusterlabs.org
> http://lists.clusterlabs.org/mailman/listinfo/users
> 
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org

___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] why resources are restarted when a node rejoins a cluster?

[Ken Gaillot]

On Mon, 2017-07-24 at 23:07 -0400, Digimer wrote:
> On 2017-07-24 11:04 PM, ztj wrote:
> > Hi all,
> > I have 2 Centos nodes with heartbeat and pacemaker-1.1.13 installed,
> > and almost everything is working fine, I have only apache configured
> > for testing, when a node goes down the failover is done correctly,
> > but there's a problem when a node failbacks.
> > 
> > For example, let's say that Node1 has the lead on apache resource,
> > then I reboot Node1, so Pacemaker detect it goes down, then apache
> > is promoted to the Node2 and it keeps there running fine, that's
> > fine, but when Node1 recovers and joins the cluster again, apache is
> > restarted on Node2 again.
> > 
> > Anyone knows, why resources are restarted when a node rejoins a
> > cluster? thanks

That's not the default behavior, so something else is going on. Show
your configuration (with any sensitive information removed) for more
help.

> You sent this to the moderators, not the list.
> 
> Please don't use heartbeat, it is extremely deprecated. Please switch
> to corosync.

Since it's CentOS, it has to be corosync, unless heartbeat was compiled
locally.

> 
> To offer any other advice, you need to share your config and the logs
> from both nodes. Please respond to the list, not
> developers-ow...@clusterlabs.org.
> 
> digimer
> -- 
> Digimer
> Papers and Projects: https://alteeve.com/w/
> "I am, somehow, less interested in the weight and convolutions of Einstein’s 
> brain than in the near certainty that people of equal talent have lived and 
> died in cotton fields and sweatshops." - Stephen Jay Gould
> ___
> Users mailing list: Users@clusterlabs.org
> http://lists.clusterlabs.org/mailman/listinfo/users
> 
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org

-- 
Ken Gaillot 





___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Two nodes cluster issue

[Jan Friesse]

Tomer Azran napsal(a):

I tend to agree with Klaus – I don't think that having a hook that
bypass stonith is the right way. It is better to not use stonith at all.
I think I will try to use an iScsi target on my qdevice and set SBD to
use it.
I still don't understand why qdevice can't take the place SBD with
shared storage; correct me if I'm wrong, but it looks like both of
them are there for the same reason.


Qdevice is there to be third side arbiter who decides which partition is
quorate. It can also be seen as a quorum only node. So for two node
cluster it can be viewed as a third node (eventho it is quite special
because it cannot run resources). It is not doing fencing.

SBD is fencing device. It is using disk as a third side arbiter.


I've talked with Klaus and he told me that 7.3 is not using disk as a 
third side arbiter so sorry for confusion.


You should however still be able to use sbd for checking if pacemaker is 
alive and if the partition has quorum - otherwise the watchdog kills the 
node. So qdevice will give you "3rd" node and sbd fences unquorate 
partition.


Or (as mentioned previously) you can use fabric fencing.

Regards,
  Honza






From: Klaus Wenninger [mailto:kwenn...@redhat.com]
Sent: Monday, July 24, 2017 9:01 PM
To: Cluster Labs - All topics related to open-source clustering
welcomed ; Prasad, Shashank 
Subject: Re: [ClusterLabs] Two nodes cluster issue

On 07/24/2017 07:32 PM, Prasad, Shashank wrote:
Sometimes IPMI fence devices use shared power of the node, and it
cannot be avoided.
In such scenarios the HA cluster is NOT able to handle the power
failure of a node, since the power is shared with its own fence device.
The failure of IPMI based fencing can also exist due to other reasons
also.

A failure to fence the failed node will cause cluster to be marked
UNCLEAN.
To get over it, the following command needs to be invoked on the
surviving node.

pcs stonith confirm  --force

This can be automated by hooking a recovery script, when the the
Stonith resource ‘Timed Out’ event.
To be more specific, the Pacemaker Alerts can be used for watch for
Stonith timeouts and failures.
In that script, all that’s essentially to be executed is the
aforementioned command.

If I get you right here you can disable fencing then in the first place.
Actually quorum-based-watchdog-fencing is the way to do this in a
safe manner. This of course assumes you have a proper source for
quorum in your 2-node-setup with e.g. qdevice or using a shared
disk with sbd (not directly pacemaker quorum here but similar thing
handled inside sbd).


Since the alerts are issued from ‘hacluster’ login, sudo permissions
for ‘hacluster’ needs to be configured.

Thanx.


From: Klaus Wenninger [mailto:kwenn...@redhat.com]
Sent: Monday, July 24, 2017 9:24 PM
To: Kristián Feldsam; Cluster Labs - All topics related to open-source
clustering welcomed
Subject: Re: [ClusterLabs] Two nodes cluster issue

On 07/24/2017 05:37 PM, Kristián Feldsam wrote:
I personally think that power off node by switched pdu is more safe,
or not?

True if that is working in you environment. If you can't do a physical
setup
where you aren't simultaneously loosing connection to both your node and
the switch-device (or you just want to cover cases where that happens)
you have to come up with something else.




S pozdravem Kristián Feldsam
Tel.: +420 773 303 353, +421 944 137 535
E-mail.: supp...@feldhost.cz

www.feldhost.cz - FeldHost™ – profesionální
hostingové a serverové služby za adekvátní ceny.

FELDSAM s.r.o.
V rohu 434/3
Praha 4 – Libuš, PSČ 142 00
IČ: 290 60 958, DIČ: CZ290 60 958
C 200350 vedená u Městského soudu v Praze

Banka: Fio banka a.s.
Číslo účtu: 2400330446/2010
BIC: FIOBCZPPXX
IBAN: CZ82 2010  0024 0033 0446

On 24 Jul 2017, at 17:27, Klaus Wenninger
> wrote:

On 07/24/2017 05:15 PM, Tomer Azran wrote:
I still don't understand why the qdevice concept doesn't help on this
situation. Since the master node is down, I would expect the quorum to
declare it as dead.
Why doesn't it happens?

That is not how quorum works. It just limits the decision-making to
the quorate subset of the cluster.
Still the unknown nodes are not sure to be down.
That is why I suggested to have quorum-based watchdog-fencing with sbd.
That would assure that within a certain time all nodes of the
non-quorate part
of the cluster are down.






On Mon, Jul 24, 2017 at 4:15 PM +0300, "Dmitri Maziuk"
> wrote:

On 2017-07-24 07:51, Tomer Azran wrote:


We don't have the ability to use it.



Is that the only solution?




No, but I'd recommend thinking about it first. Are you sure you will

care about your cluster working when your server room is on fire? 'Cause

unless you have halon suppression, your server room is a complete

write-off anyway. (Think water 

Re: [ClusterLabs] Two nodes cluster issue

[Jan Friesse]

Tomer Azran napsal(a):

I tend to agree with Klaus – I don't think that having a hook that bypass 
stonith is the right way. It is better to not use stonith at all.
I think I will try to use an iScsi target on my qdevice and set SBD to use it.
I still don't understand why qdevice can't take the place SBD with shared 
storage; correct me if I'm wrong, but it looks like both of them are there for 
the same reason.


Qdevice is there to be third side arbiter who decides which partition is 
quorate. It can also be seen as a quorum only node. So for two node 
cluster it can be viewed as a third node (eventho it is quite special 
because it cannot run resources). It is not doing fencing.


SBD is fencing device. It is using disk as a third side arbiter.




From: Klaus Wenninger [mailto:kwenn...@redhat.com]
Sent: Monday, July 24, 2017 9:01 PM
To: Cluster Labs - All topics related to open-source clustering welcomed 
; Prasad, Shashank 
Subject: Re: [ClusterLabs] Two nodes cluster issue

On 07/24/2017 07:32 PM, Prasad, Shashank wrote:
Sometimes IPMI fence devices use shared power of the node, and it cannot be 
avoided.
In such scenarios the HA cluster is NOT able to handle the power failure of a 
node, since the power is shared with its own fence device.
The failure of IPMI based fencing can also exist due to other reasons also.

A failure to fence the failed node will cause cluster to be marked UNCLEAN.
To get over it, the following command needs to be invoked on the surviving node.

pcs stonith confirm  --force

This can be automated by hooking a recovery script, when the the Stonith 
resource ‘Timed Out’ event.
To be more specific, the Pacemaker Alerts can be used for watch for Stonith 
timeouts and failures.
In that script, all that’s essentially to be executed is the aforementioned 
command.

If I get you right here you can disable fencing then in the first place.
Actually quorum-based-watchdog-fencing is the way to do this in a
safe manner. This of course assumes you have a proper source for
quorum in your 2-node-setup with e.g. qdevice or using a shared
disk with sbd (not directly pacemaker quorum here but similar thing
handled inside sbd).


Since the alerts are issued from ‘hacluster’ login, sudo permissions for 
‘hacluster’ needs to be configured.

Thanx.


From: Klaus Wenninger [mailto:kwenn...@redhat.com]
Sent: Monday, July 24, 2017 9:24 PM
To: Kristián Feldsam; Cluster Labs - All topics related to open-source 
clustering welcomed
Subject: Re: [ClusterLabs] Two nodes cluster issue

On 07/24/2017 05:37 PM, Kristián Feldsam wrote:
I personally think that power off node by switched pdu is more safe, or not?

True if that is working in you environment. If you can't do a physical setup
where you aren't simultaneously loosing connection to both your node and
the switch-device (or you just want to cover cases where that happens)
you have to come up with something else.




S pozdravem Kristián Feldsam
Tel.: +420 773 303 353, +421 944 137 535
E-mail.: supp...@feldhost.cz

www.feldhost.cz - FeldHost™ – profesionální hostingové 
a serverové služby za adekvátní ceny.

FELDSAM s.r.o.
V rohu 434/3
Praha 4 – Libuš, PSČ 142 00
IČ: 290 60 958, DIČ: CZ290 60 958
C 200350 vedená u Městského soudu v Praze

Banka: Fio banka a.s.
Číslo účtu: 2400330446/2010
BIC: FIOBCZPPXX
IBAN: CZ82 2010  0024 0033 0446

On 24 Jul 2017, at 17:27, Klaus Wenninger 
> wrote:

On 07/24/2017 05:15 PM, Tomer Azran wrote:
I still don't understand why the qdevice concept doesn't help on this 
situation. Since the master node is down, I would expect the quorum to declare 
it as dead.
Why doesn't it happens?

That is not how quorum works. It just limits the decision-making to the quorate 
subset of the cluster.
Still the unknown nodes are not sure to be down.
That is why I suggested to have quorum-based watchdog-fencing with sbd.
That would assure that within a certain time all nodes of the non-quorate part
of the cluster are down.






On Mon, Jul 24, 2017 at 4:15 PM +0300, "Dmitri Maziuk" 
> wrote:

On 2017-07-24 07:51, Tomer Azran wrote:


We don't have the ability to use it.



Is that the only solution?




No, but I'd recommend thinking about it first. Are you sure you will

care about your cluster working when your server room is on fire? 'Cause

unless you have halon suppression, your server room is a complete

write-off anyway. (Think water from sprinklers hitting rich chunky volts

in the servers.)



Dima



___

Users mailing list: Users@clusterlabs.org

http://lists.clusterlabs.org/mailman/listinfo/users



Project Home: http://www.clusterlabs.org

Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf

Bugs: 

[ClusterLabs] Antw: Re: epic fail

[Ulrich Windl]

>>> Dimitri Maziuk  schrieb am 24.07.2017 um 19:24 in
Nachricht :
> OK, how about this:

[...]
> 
> this last bit: "kernel: block drbd 0"x3 and "drbd(drbd_storage)"x3 goes
> on until the power-cycle.

So you would have time to investigate with lsof who's using your filesystem or 
device.
In my experience either some root user interactively changed to the filesystem 
(hopefully not you ;-)), or some process uses the mountpoint as working 
directory, or some resource says it stopped while it did not.

Regards,
Ulrich

> 
> -- 
> Dimitri Maziuk
> Programmer/sysadmin
> BioMagResBank, UW-Madison -- http://www.bmrb.wisc.edu 





___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

[ClusterLabs] Antw: Re: epic fail

[Ulrich Windl]

>>> Jan Pokorný  schrieb am 24.07.2017 um 16:40 in
Nachricht
<20170724144039.gc31...@redhat.com>:
> On 23/07/17 14:40 +0200, Valentin Vidic wrote:
>> On Sun, Jul 23, 2017 at 07:27:03AM -0500, Dmitri Maziuk wrote:
>>> So yesterday I ran yum update that puled in the new pacemaker and tried
to
>>> restart it. The node went into its usual "can't unmount drbd because
kernel
>>> is using it" and got stonith'ed in the middle of yum transaction. The end
>>> result: DRBD reports split brain, HA daemons don't start on boot, RPM
>>> database is FUBAR. I've had enough. I'm rebuilding this cluster as centos
6
>>> + heartbeat R1.
>> 
>> It seems you did not put the node into standby before the upgrade as it
>> still had resources running.  What was the old/new pacemaker version
there?
> 
> Thinking out loud, it shouldn't be too hard to deliver an RPM
> plugin[1] with RPM-shipped pacemaker (it doesn't make much sense
> otherwise) that will hook into RPM transactions, putting the node
> into standby first so to cover the corner case one updates the
> live cluster.  Something akin to systemd_inhibit.so.

While possible, the simple objection is: If someone who is installing software
on a cluster node has no idea how the cluster works, he should lear the hard
way.
Also putting a node in standby can take significant time where you don't see
any progress on the console. People who have no idea how the cluster works
might think the process is hanging or waiting for some input. And finally: I
perefer to stop the whole node before updating the cluster software, instead of
setting the node to standby. So please let the user decide what to do, not the
RPM. And at the very last: If the RPM puts the node into standby (if it isn't
already!), it should also put the node online again (if it had been online
before) to support users that have no idea how the cluster works.

So obviously I don't like the idea. Maybe do it the HP-UX Service Guard way:
Refuse to install/update cluster software if the node is active.

Regards,
Ulrich

> 
> Would there be an interest, though?  And would that be meaningful?
> 
> [1] http://rpm.org/devel_doc/plugins.html 
> 
> -- 
> Jan (Poki)




___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org