Re: [Pacemaker] WG: configuration of stonith

[Masopust, Christian]

Hi Emmanuel,

thanks for the hint, reading (again) the chapter about resource stickiness, I 
see and understand
the difference :)

br,
christian


Von: emmanuel segura [mailto:emi2f...@gmail.com]
Gesendet: Montag, 09. Dezember 2013 16:12
An: m...@sys4.de; The Pacemaker cluster resource manager
Betreff: Re: [Pacemaker] WG: configuration of stonith

I think they sould be

pcs constraint location ipmi-fencing-sv2837 prefers sv2837=-INFINITY
pcs constraint location ipmi-fencing-sv2836 prefers sv2836=-INFINITY



2013/12/9 Michael Schwartzkopff mailto:m...@sys4.de>>
Am Montag, 9. Dezember 2013, 14:58:13 schrieben Sie:
> > > pcs stonith create ipmi-fencing-sv2837 fence_ipmilan
> >
> > pcmk_host_list="sv2837"
> >
> > > ipaddr=10.110.28.37 action="off" login=ipmi passwd=abc
> >
> > delay=15 op monitor
> >
> > > interval=60s pcs stonith create ipmi-fencing-sv2836 fence_ipmilan
> > > pcmk_host_list="sv2836" ipaddr=10.110.28.36 action="off" login=ipmi
> > > passwd=abc delay=15 op monitor interval=60s
> > >
> > > pcs property set stonith-enabled=true
> > >
> > > pcs constraint location ipmi-fencing-sv2837 prefers sv2836=INFINITY
> > > pcs constraint location ipmi-fencing-sv2836 prefers sv2837=INFINITY
> > >
> > > pcs status
> > > 
> > >
> > > Full list of resources:
> > >  ClusterIP(ocf::heartbeat:IPaddr2):   Started sv2836
> > >  FIXRoute (ocf::baader:FIXRoute): Started sv2836
> >
> > >  ipmi-fencing-sv2837  (stonith:fence_ipmilan):
> > Started sv2836
> >
> > >  ipmi-fencing-sv2836  (stonith:fence_ipmilan):
> > Started sv2837
> >
> > This is not optimal. Nothing prevents the resource, that can
> > fence node sv2837
> > to run on host sv2837. You just say, that it should run on
> > node sv2836.
> >
> > Better would be something like
> >
> > crm configure location place-fencing-sv2837 -inf: sv2837
> >
> > or the equivalent in pcs.
> >
> > Greetings,
>
> Hi Michael,
>
> I thought that the lines above will do that:
> > > pcs constraint location ipmi-fencing-sv2837 prefers sv2836=INFINITY
> > > pcs constraint location ipmi-fencing-sv2836 prefers sv2837=INFINITY
>
> Don't they?

I don't  know pcs in depth and I could not find any detailed doc. So I stick
with crmsh.

As far as I can judge your lines above your tell the cluster, that the
resource, that can fence node sv2836, gets INF points if it runs on node
sv2837. But what happens if node sv2837 is down? Nothing prevents the resource
starting on the node that it should fence. So if one node is down both fencing
resources will run on the remaining node. No very nice.

I suggest to assign -INF points to the resource that can fence node sv2836 if
it runs on node sv2836. So it will run on sv2837. If that node is not
available the resource cannot run. On the node (remaining) node sv2836 only
the resource that can fence sv2837.

For details see: http://clusterlabs.org/doc/crm_fencing.html

Mit freundlichen Grüßen,

Michael Schwartzkopff

--
[*] sys4 AG

http://sys4.de, +49 (89) 30 90 46 64, 
+49 (162) 165 0044
Franziskanerstraße 15, 81669 München

Sitz der Gesellschaft: München, Amtsgericht München: HRB 199263
Vorstand: Patrick Ben Koetter, Axel von der Ohe, Marc Schiffbauer
Aufsichtsratsvorsitzender: Florian Kirstein

___
Pacemaker mailing list: 
Pacemaker@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org




--
esta es mi vida e me la vivo hasta que dios quiera
___
Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [Pacemaker] WG: configuration of stonith

[Masopust, Christian]

> > > 
> > > This is not optimal. Nothing prevents the resource, that can
> > > fence node sv2837
> > > to run on host sv2837. You just say, that it should run on
> > > node sv2836.
> > > 
> > > Better would be something like
> > > 
> > > crm configure location place-fencing-sv2837 -inf: sv2837
> > > 
> > > or the equivalent in pcs.
> > > 
> > > Greetings,
> > 
> > Hi Michael,
> > 
> > I thought that the lines above will do that:
> > > > pcs constraint location ipmi-fencing-sv2837 prefers 
> sv2836=INFINITY
> > > > pcs constraint location ipmi-fencing-sv2836 prefers 
> sv2837=INFINITY
> > 
> > Don't they?
> 
> I don't  know pcs in depth and I could not find any detailed 
> doc. So I stick 
> with crmsh.
> 
> As far as I can judge your lines above your tell the cluster, 
> that the 
> resource, that can fence node sv2836, gets INF points if it 
> runs on node 
> sv2837. But what happens if node sv2837 is down? Nothing 
> prevents the resource 
> starting on the node that it should fence. So if one node is 
> down both fencing 
> resources will run on the remaining node. No very nice.
> 
> I suggest to assign -INF points to the resource that can 
> fence node sv2836 if 
> it runs on node sv2836. So it will run on sv2837. If that node is not 
> available the resource cannot run. On the node (remaining) 
> node sv2836 only 
> the resource that can fence sv2837.
> 
> For details see: http://clusterlabs.org/doc/crm_fencing.html
> 
> Mit freundlichen Grüßen,
> 
> Michael Schwartzkopff

Aaaah I see... thanks a lot for the explanation!

br,
christian
___
Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [Pacemaker] WG: configuration of stonith

[Masopust, Christian]

> > pcs stonith create ipmi-fencing-sv2837 fence_ipmilan 
> pcmk_host_list="sv2837"
> > ipaddr=10.110.28.37 action="off" login=ipmi passwd=abc 
> delay=15 op monitor
> > interval=60s pcs stonith create ipmi-fencing-sv2836 fence_ipmilan
> > pcmk_host_list="sv2836" ipaddr=10.110.28.36 action="off" login=ipmi
> > passwd=abc delay=15 op monitor interval=60s
> > 
> > pcs property set stonith-enabled=true
> > 
> > pcs constraint location ipmi-fencing-sv2837 prefers sv2836=INFINITY
> > pcs constraint location ipmi-fencing-sv2836 prefers sv2837=INFINITY
> > 
> > pcs status
> > 
> > Full list of resources:
> >  ClusterIP  (ocf::heartbeat:IPaddr2):   Started sv2836
> >  FIXRoute   (ocf::baader:FIXRoute): Started sv2836
> >  ipmi-fencing-sv2837(stonith:fence_ipmilan):
> Started sv2836
> >  ipmi-fencing-sv2836(stonith:fence_ipmilan):
> Started sv2837
> 
> This is not optimal. Nothing prevents the resource, that can 
> fence node sv2837 
> to run on host sv2837. You just say, that it should run on 
> node sv2836.
> 
> Better would be something like
> 
> crm configure location place-fencing-sv2837 -inf: sv2837
> 
> or the equivalent in pcs.
> 
> Greetings,

Hi Michael,

I thought that the lines above will do that:

> > pcs constraint location ipmi-fencing-sv2837 prefers sv2836=INFINITY
> > pcs constraint location ipmi-fencing-sv2836 prefers sv2837=INFINITY

Don't they?

br,
christian

___
Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [Pacemaker] WG: configuration of stonith

[Masopust, Christian]

Hi Andreas,

as far as I can say (as a pacemaker novice), everything fine besides
the equal delays for both stoninths... I would suggest to configure
only one with a delay of 15s.

br,
christian 

> -Ursprüngliche Nachricht-
> Von: Dvorak Andreas [mailto:andreas.dvo...@baaderbank.de] 
> Gesendet: Montag, 09. Dezember 2013 14:40
> An: 'pacemaker@oss.clusterlabs.org'
> Betreff: [Pacemaker] WG: configuration of stonith
> 
> Dear all
> 
> My problem with stonith is solved.
> 
> Here is what I did:
> 
> pcs stonith create ipmi-fencing-sv2837 fence_ipmilan 
> pcmk_host_list="sv2837" ipaddr=10.110.28.37 action="off" 
> login=ipmi passwd=abc delay=15 op monitor interval=60s
> pcs stonith create ipmi-fencing-sv2836 fence_ipmilan 
> pcmk_host_list="sv2836" ipaddr=10.110.28.36 action="off" 
> login=ipmi passwd=abc delay=15 op monitor interval=60s
> 
> pcs property set stonith-enabled=true
> 
> pcs constraint location ipmi-fencing-sv2837 prefers sv2836=INFINITY
> pcs constraint location ipmi-fencing-sv2836 prefers sv2837=INFINITY
> 
> pcs status
> 
> Full list of resources:
>  ClusterIP(ocf::heartbeat:IPaddr2):   Started sv2836 
>  FIXRoute (ocf::baader:FIXRoute): Started sv2836 
>  ipmi-fencing-sv2837  (stonith:fence_ipmilan):Started sv2836 
>  ipmi-fencing-sv2836  (stonith:fence_ipmilan):Started sv2837
> 
> Best regards,
> Andreas
> 
> -Ursprüngliche Nachricht-
> Von: Dvorak Andreas 
> Gesendet: Montag, 9. Dezember 2013 09:55
> An: 'The Pacemaker cluster resource manager'
> Betreff: Re: [Pacemaker] configuration of stonith
> 
> Dear all,
> 
> thank you for the answers.
> 
> Now I created to stonith resources
> pcs stonith create ipmi-fencing-sv2837 fence_ipmilan 
> pcmk_host_list="sv2837" ipaddr=10.110.28.37 action="reboot" 
> login=abc passwd=abc123 delay=15 op monitor interval=60s pcs 
> stonith create ipmi-fencing-sv2836 fence_ipmilan 
> pcmk_host_list="sv2836" ipaddr=10.110.28.36 action="reboot" 
> login=abc passwd=abc123 delay=15 op monitor interval=60s
> 
> the current status is:
> pcs status
> Cluster name: fix-prod
> Last updated: Mon Dec  9 09:41:48 2013
> Last change: Mon Dec  9 09:40:03 2013 via cibadmin on sv2836
> Stack: cman
> Current DC: sv2837 - partition with quorum
> Version: 1.1.10-1.el6_4.4-368c726
> 2 Nodes configured
> 4 Resources configured
> 
> Online: [ sv2836 sv2837 ]
> 
> Full list of resources:
> 
>  ClusterIP(ocf::heartbeat:IPaddr2):   Started sv2836 
>  FIXRoute (ocf::baader:FIXRoute): Started sv2836 
>  ipmi-fencing-sv2837  (stonith:fence_ipmilan):Stopped 
>  ipmi-fencing-sv2836  (stonith:fence_ipmilan):Stopped 
> 
> Failed actions:
> ipmi-fencing-sv2837_start_0 on sv2837 'unknown error' 
> (1): call=276, status=Error, last-rc-change='Mon Dec  9 
> 09:39:55 2013', queued=17090ms, exec=0ms
> ipmi-fencing-sv2836_start_0 on sv2837 'unknown error' 
> (1): call=286, status=Error, last-rc-change='Mon Dec  9 
> 09:40:13 2013', queued=17085ms, exec=0ms
> ipmi-fencing-sv2837_start_0 on sv2836 'unknown error' 
> (1): call=369, status=Error, last-rc-change='Mon Dec  9 
> 09:40:13 2013', queued=17085ms, exec=0ms
> ipmi-fencing-sv2836_start_0 on sv2836 'unknown error' 
> (1): call=375, status=Error, last-rc-change='Mon Dec  9 
> 09:40:31 2013', queued=17090ms, exec=0ms
> 
> Do I need to tell the stonith resource where to run and how 
> can I do that?
> In the parameter pcmk_host_list I have the hostname of the other node.
> 
> Best regards,
> Andreas
> 
> ___
> Pacemaker mailing list: Pacemaker@oss.clusterlabs.org 
> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
> 
> Project Home: http://www.clusterlabs.org Getting started: 
> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org
> 
> ___
> Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
> 
> Project Home: http://www.clusterlabs.org
> Getting started: 
> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org
> 
___
Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [Pacemaker] configuration of stonith

[Masopust, Christian]

> >>
> >> If you're using 1.1.10+,
> >>
> >> pcs stonith create fence_pcmk1_ipmi fence_ipmilan \
> >> pcmk_host_list="pcmk-1" ipaddr="pcmk-1.ipmi" \
> >> action="reboot" login="admin" passwd="secret" delay=15 \
> >> op monitor interval=60s
> >>
> >> pcs stonith create fence_pcmk2_ipmi fence_ipmilan \
> >> pcmk_host_list="pcmk-2" ipaddr="pcmk-2.ipmi" \
> >> action="reboot" login="admin" passwd="secret" delay=15 \
> >> op monitor interval=60s
> >>
> >> is sufficient.
> >>
> > 
> > Hi,
> > 
> > just two questions about setting these stonith:
> > 
> > - shouldn't the delay's be different to avoid a stonith-battle?
> 
> As Emmanuel said, yes, it is needed to avoid dual-fencing in two-node
> clusters, though the issue is not restricted to rhcs (or any HA
> clustering that allows two nodes).
> 
> The node with the 'delay="15"' will have a 15 second 
> head-start, so in a
> network partition triggered fence, the node with the delay 
> should always
> live and the node without the delay will be immediately fenced.
> 
> > - when creating these stonith I see them both started on one single
> >   node. Don't I need some location constraints?  Such that 
> "fence_pcmk1"
> >   only runs on pcmk2 and vice versa?
> 
> What version of pacemaker are you using?
> 

Hi Digimer,

first when seeing this behaviour there was version 1.1.8. This weekend
I've updated to 1.1.10 (latest available with CentOS 6.5) and now I see
that fence_pcmk1 is started at pcmk1 and fence_pcmk2 at pcmk2.
Is that correct? To my (probably wrong) understanding it should be vice-versa,
shouldn't it?

br,
christian
___
Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [Pacemaker] configuration of stonith

[Masopust, Christian]

 

> -Ursprüngliche Nachricht-
> Von: Digimer [mailto:li...@alteeve.ca] 
> Gesendet: Freitag, 06. Dezember 2013 17:20
> An: m...@sys4.de; The Pacemaker cluster resource manager
> Betreff: Re: [Pacemaker] configuration of stonith
> 
> 
> If you're using 1.1.10+,
> 
> pcs stonith create fence_pcmk1_ipmi fence_ipmilan \
> pcmk_host_list="pcmk-1" ipaddr="pcmk-1.ipmi" \
> action="reboot" login="admin" passwd="secret" delay=15 \
> op monitor interval=60s
> 
> pcs stonith create fence_pcmk2_ipmi fence_ipmilan \
> pcmk_host_list="pcmk-2" ipaddr="pcmk-2.ipmi" \
> action="reboot" login="admin" passwd="secret" delay=15 \
> op monitor interval=60s
> 
> is sufficient.
> 

Hi,

just two questions about setting these stonith:

- shouldn't the delay's be different to avoid a stonith-battle?

- when creating these stonith I see them both started on one single
  node. Don't I need some location constraints?  Such that "fence_pcmk1"
  only runs on pcmk2 and vice versa?

Thanks,
christian
___
Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [Pacemaker] some questions about STONITH

[Masopust, Christian]

Hi Lars,

at this point I'd like to jump in as I'm completely new to fencing :)

My question is: which node exactly does the fencing? 

Thanks a lot,
christian 

> -Ursprüngliche Nachricht-
> Von: Lars Marowsky-Bree [mailto:l...@suse.com] 
> Gesendet: Dienstag, 19. November 2013 20:10
> An: The Pacemaker cluster resource manager
> Betreff: Re: [Pacemaker] some questions about STONITH
> 
> On 2013-11-19T23:06:04, Andrey Groshev  wrote:
> 
> > > First, like digimer wrote, clearly stonith-by-ssh is useless for
> > > production since you can't fence nodes that are having 
> problems. But for
> > > testing, it's worth a try.
> > Maybe I do not quite understand correctly the term "fence"
> 
> A "fence" request is executed when a node is deemed to be in an
> untrustworthy state - when a stop has failed, or when a network error
> occurs. Note that in the last case, login via ssh is 
> obviously no longer
> possible at all.
> 
> With the new fence-topology, you could try ssh first before escalating
> to a real fencing mechanism, but why bother?
> 
> > > Note that cluster-glue actually does include an 
> external/ssh script.
> > > You're reinventing the wheel ;-)
> > I've seen your script, thanks for the example
> > But my wheels are hard! :)
> > I need authorization by key, but but I do not want to mix 
> them with /root/.ssh/...
> 
> Why not extend the existing agent rather than writing your own?
> 
> > I am indifferent what server reboot if the key matches.
> > I exactly know that the server was rebooted.
> 
> I'm not sure about the first sentence; clearly you care which 
> server is
> rebooted, namely the one the cluster wants to have rebooted 
> (or powered
> off), right? That must be a misunderstanding.
> 
> Regards,
> Lars
> 
> -- 
> Architect Storage/HA
> SUSE LINUX Products GmbH, GF: Jeff Hawn, Jennifer Guild, 
> Felix Imendörffer, HRB 21284 (AG Nürnberg)
> "Experience is the name everyone gives to their mistakes." -- 
> Oscar Wilde
> 
> 
> ___
> Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
> 
> Project Home: http://www.clusterlabs.org
> Getting started: 
> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org
> 
___
Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [Pacemaker] Fencing of KVM virtual machines

[Masopust, Christian]

Hi Michael,

thank's a lot for that link! Will give it a try!!

br,
christian 

> -Ursprüngliche Nachricht-
> Von: Michael Schwartzkopff [mailto:mi...@clusterbau.com] 
> Gesendet: Dienstag, 02. Oktober 2012 08:22
> An: The Pacemaker cluster resource manager
> Betreff: Re: [Pacemaker] Fencing of KVM virtual machines
> 
> > Hi all,
> > 
> > I've running several pacemaker clusters in KVM virtual 
> machines (everything
> > based on Debian 6) and now it's up to configure fencing...
> > 
> > I've found that I have to use "fence-virt" for that task
> > (http://www.clusterlabs.org/wiki/Guest_Fencing) but it 
> seems that it only
> > will work in case my VMs are on a single host system.
> > 
> > Is anybody of you using fence-virt when the VMs are on 
> different hosts?
> > And can anybody explain how to compile and install 
> fence-virt on a Debian 6
> > system?
> > 
> > Thanks a lot!
> > Christian
> 
> Hi,
> 
> ever tried external/libvirt
> 
> See: 
> http://www.hastexo.com/resources/hints-and-kinks/fencing-libvirtkvm-
> virtualized-cluster-nodes
> 
> -- 
> Dr. Michael Schwartzkopff
> Guardinistr. 63
> 81375 München
> 
> Tel: (0163) 172 50 98
> 
___
Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

[Pacemaker] Fencing of KVM virtual machines

[Masopust, Christian]

Hi all,

I've running several pacemaker clusters in KVM virtual machines (everything 
based on
Debian 6) and now it's up to configure fencing...

I've found that I have to use "fence-virt" for that task 
(http://www.clusterlabs.org/wiki/Guest_Fencing)
but it seems that it only will work in case my VMs are on a single host system.

Is anybody of you using fence-virt when the VMs are on different hosts?
And can anybody explain how to compile and install fence-virt on a Debian 6 
system?

Thanks a lot!
Christian

___
Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org