[ClusterLabs] Question about MySQL agent behaviour, in particular _mysql_master_IP

[Les Green]

Hi All,

I've recently had an issue when I've needed to add
_mysql_master_IP to hosts in a MySQL HA cluster as I
needed to move a running cluster onto a VPN.

It seems _REPL_INFO is only set when a resource is
promoted, so as a test, I:
* Put the system in maintenance mode
* Manually removed the _REPL_INFO from the CIB
* Added the appropriate _mysql_master_IP attributes for
each node.
* Manually configured MySQL to be using IP based replication
* Then took the cluster out of maintenance mode.

The replication continued ok, but as predicted the
_REPL_INFO was not populated. When putting the current
master into standby, the slave was promoted correctly and the
_REPL_INFO populated correctly.

HOWEVER, while doing this, I noticed in the code, if the master hostname
changes, the replication for the slave is restarted at the master
position, not the slave position. See:
https://github.com/ClusterLabs/resource-agents/blob/master/heartbeat/mysql#L535

Why on earth would we ever do this? This seems rife for a disaster of
lost writes. I feel we should only ever be using the current slave
position as the start position for a CHANGE MASTER command.

Can somebody set me straight on this matter? I feel maybe I'm missing
something significant.

Cheers, Les


___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

[ClusterLabs] SOLVED: Antw: Re: Antw: can't live migrate VirtualDomain which is part of a group

[Lentes, Bernd]

- On Apr 25, 2017, at 1:37 PM, Ulrich Windl 
ulrich.wi...@rz.uni-regensburg.de wrote:

 "Lentes, Bernd"  schrieb am 25.04.2017 
 um
> 11:02 in Nachricht
> <406563603.26964612.1493110931994.javamail.zim...@helmholtz-muenchen.de>:
> 
>> 
>> - On Apr 25, 2017, at 8:08 AM, Ulrich Windl
>> ulrich.wi...@rz.uni-regensburg.de
>> wrote:
>> 
>>> Berdn
>>> 
>>> you are long enough on this list to know that the reason for your failure is
>>> most likely to be found in the logs which you did not provide. Couldn't you
>>> find out yourself from the logs?
>>> 
>>> Regards,
>>> Ulrich
>>> 
>>> 
>> 
>> Hi Ulrich,
>> 
>> 
>> if i had found something in the log i would not have asked.
>> From what i understand from Ken is that the error is the resource IPaddr
>> which is by default not able to live-migrate.
>> 
>> Just a few minutes ago i tried again to live migrate the VirtualDomain
>> resource, and again it shutted down on one node
>> and rebooted on the other.
>> 
>> Here is the respective excerpt from the log. Maybe you can point out to me
>> where i can find the reason for the problem:
>> 
> 
> Usually there is a kind of action summary that is logged before the first 
> action
> is executed. If any of these actions fail, the outcome could be different from
> what was intended. In your case there does not seem to be an error in any
> action, so the outcome is what was planned (by crm). So (as we learned) the
> plans have to be changed.
> I see a migration of prim_vnc_ip_mausdb via restart and some operation with
> prim_vnc_ip_mousdb is already in progress...
> 
>> Apr 25 10:54:18 ha-idg-2 crmd[8587]:   notice: te_rsc_command: Initiating
>> action 52: stop prim_vnc_ip_mausdb_stop_0 on ha-idg-1
>> Apr 25 10:54:18 ha-idg-2 crmd[8587]:   notice: te_rsc_command: Initiating
>> action 53: start prim_vnc_ip_mausdb_start_0 on ha-idg-2 (local)
>> Apr 25 10:54:18 ha-idg-2 IPaddr(prim_vnc_ip_mausdb)[25724]: INFO: Using
>> calculated netmask for 146.107.235.161: 255.255.255.0
>> Apr 25 10:54:18 ha-idg-2 IPaddr(prim_vnc_ip_mausdb)[25724]: INFO: eval
>> ifconfig br0:0 146.107.235.161 netmask 255.255.255.0 broadcast
>> 146.107.235.255
>> Apr 25 10:54:18 ha-idg-2 crmd[8587]:   notice: process_lrm_event: Operation
>> prim_vnc_ip_mausdb_start_0: ok (node=ha-idg-2, call=283, rc=0, 
>> cib-update=1567,
>> confirmed=true)
>> Apr 25 10:54:18 ha-idg-2 crmd[8587]:   notice: te_rsc_command: Initiating
>> action 55: start prim_vm_mausdb_start_0 on ha-idg-2 (local)
>> Apr 25 10:54:19 ha-idg-2 kernel: [583994.652325] device vnet0 entered
>> promiscuous mode
>> Apr 25 10:54:19 ha-idg-2 kernel: [583994.718044] br0: port 2(vnet0) entering
>> forwarding state
>> Apr 25 10:54:19 ha-idg-2 kernel: [583994.718049] br0: port 2(vnet0) entering
>> forwarding state
>> Apr 25 10:54:20 ha-idg-2 crmd[8587]:   notice: handle_request: Current ping
>> state: S_TRANSITION_ENGINE
>> Apr 25 10:54:21 ha-idg-2 crmd[8587]:   notice: handle_request: Current ping
>> state: S_TRANSITION_ENGINE
>> Apr 25 10:54:22 ha-idg-2 crmd[8587]:   notice: process_lrm_event: Operation
>> prim_vm_mausdb_start_0: ok (node=ha-idg-2, call=284, rc=0, cib-update=1568,
>> confirmed=true)
>> Apr 25 10:54:22 ha-idg-2 crmd[8587]:   notice: te_rsc_command: Initiating
>> action 56: monitor prim_vm_mausdb_monitor_3 on ha-idg-2 (local)
>> Apr 25 10:54:22 ha-idg-2 crmd[8587]:   notice: process_lrm_event: Operation
>> prim_vm_mausdb_monitor_3: ok (node=ha-idg-2, call=285, rc=0,
>> cib-update=1569, confirmed=false)
>> Apr 25 10:54:22 ha-idg-2 crmd[8587]:   notice: run_graph: Transition 817
>> (Complete=10, Pending=0, Fired=0, Skipped=0, Incomplete=0,
>> Source=/var/lib/pacemaker/pengine/pe-input-1601.bz2): Complete
>> Apr 25 10:54:22 ha-idg-2 crmd[8587]:   notice: do_state_transition: State
>> transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS
>> cause=C_FSA_INTERNAL origin=notify_crmd ]
>> Apr 25 10:54:24 ha-idg-2 crmd[8587]:   notice: handle_request: Current ping
>> state: S_IDLE
>> Apr 25 10:54:25 ha-idg-2 crmd[8587]:   notice: handle_request: Current ping
>> state: S_IDLE
>> 
>> 
>> 


for the sake of completeness:

i changed the RA, but still couldn't live migrate the complete group.
What i found out then is that i related the start/stop of the primitive IPaddr 
wrong to the actions migrate_to and migrate_from.
First i related migrate_to with ip_start and migrate_from with ip_stop.
But then i could just live migrate in one direction, not vice versa.
When i related migrate_to with ip_stop and migrate_from with ip_start 
everything went fine.
And i forgot to set the monitor operations in the definition of the resource.
I thought they are added by default. They aren't ! And they are very important 
:-)


This is now my resource:

primitive prim_vnc_ip_mausdb ocf:lentes:IPaddr \
params ip=146.107.235.161 nic=br0 cidr_netmask=24 \
op migrate_from interval=0 timeout=30 \
op migrate_to interval=0 timeout=30 \
op monitor interval=10 timeout=

[ClusterLabs] crm_mon -h (writing to a html-file) not showing all desired information and having trouble with the -d option

[Lentes, Bernd]

Hi,

playing around with my cluster i always have a shell with crm_mon running 
because it provides me a lot of useful and current information concerning 
cluster, nodes, resources ...
Normally i have a "crm_mon -nrfRAL" running.
I'd like to have that output as a web page too.
So i tried the option -h.
I have crm_mon from pacemaker 1.1.12 on a SLES 11 SP4 box. I'm writing the file 
to /srv/www/hawk/public/crm_mon.html.
I have hawk running, so i don't need an extra webserver for that.

First, i was very astonished when i used the option -d (daemonize). Using that 
hawk does not find the html-file, although i see it in the fs, and it's looking 
good.
Hawk (or lighttpd) throws an error 404. Without -d lighttpd finds the files and 
presents it via browser !?!

This is the file without -d:

ha-idg-2:/srv/www/hawk/public # stat crm_mon.html
  File: `crm_mon.html'
  Size: 1963Blocks: 8  IO Block: 4096   regular file
Device: 1fh/31d Inode: 7082Links: 1
Access: (0644/-rw-r--r--)  Uid: (0/root)   Gid: (0/root)
Access: 2017-05-08 18:03:25.695754151 +0200
Modify: 2017-05-08 18:03:20.875680374 +0200
Change: 2017-05-08 18:03:20.875680374 +0200


Same file with crm_mon -d:

ha-idg-2:/srv/www/hawk/public # stat crm_mon.html
  File: `crm_mon.html'
  Size: 1963Blocks: 8  IO Block: 4096   regular file
Device: 1fh/31d Inode: 7084Links: 1
Access: (0640/-rw-r-)  Uid: (0/root)   Gid: (0/root)
Access: 2017-05-08 18:04:16.048524856 +0200
Modify: 2017-05-08 18:04:16.048524856 +0200
Change: 2017-05-08 18:04:16.048524856 +0200
 Birth: -

I see no important difference, just the different inode.

This is the access.log from lighttpd:

10.35.34.70 ha-idg-2:7630 - [08/May/2017:18:04:10 +0200] "GET /crm_mon.html 
HTTP/1.1" 200 563 "https://ha-idg-2:7630/crm_mon.html"; "Mozilla/5.0 (Windows NT 
6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/57.0.2987.133 
Safa
ri/537.36"
10.35.34.70 ha-idg-2:7630 - [08/May/2017:18:04:15 +0200] "GET /crm_mon.html 
HTTP/1.1" 200 563 "https://ha-idg-2:7630/crm_mon.html"; "Mozilla/5.0 (Windows NT 
6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/57.0.2987.133 
Safa
ri/537.36"
10.35.34.70 ha-idg-2:7630 - [08/May/2017:18:04:20 +0200] "GET /crm_mon.html 
HTTP/1.1" 404 1163 "https://ha-idg-2:7630/crm_mon.html"; "Mozilla/5.0 (Windows 
NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/57.0.2987.133 
Saf
ari/537.36"

It simply changes from http status code 200 to 404. Why ? 

And using "crm_mon -nfrotRALV -h /srv/www/hawk/public/crm_mon.html" i get the 
following output:


Cluster summary

Last updated: Mon May 8 18:08:58 2017
Current DC: ha-idg-2
2 Nodes configured.
14 Resources configured.
Config Options

STONITH of failed nodes :   enabled
Cluster is  :   symmetric
No Quorum Policy:   Ignore
Node List

Node: ha-idg-1: online
prim_clvmd  (ocf::lvm2:clvmd):  Started 
prim_stonith_ipmi_ha-idg-2  (stonith:external/ipmi):Started 
prim_ocfs2  (ocf::ocfs2:o2cb):  Started 
prim_vm_mausdb  (ocf::heartbeat:VirtualDomain): Started 
prim_vg_cluster_01  (ocf::heartbeat:LVM):   Started 
prim_fs_lv_xml_vm   (ocf::heartbeat:Filesystem):Started 
prim_dlm(ocf::pacemaker:controld):  Started 
prim_vnc_ip_mausdb  (ocf::lentes:IPaddr):   Started 
Node: ha-idg-2: online
prim_clvmd  (ocf::lvm2:clvmd):  Started 
prim_stonith_ipmi_ha-idg-1  (stonith:external/ipmi):Started 
prim_ocfs2  (ocf::ocfs2:o2cb):  Started 
prim_vg_cluster_01  (ocf::heartbeat:LVM):   Started 
prim_fs_lv_xml_vm   (ocf::heartbeat:Filesystem):Started 
prim_dlm(ocf::pacemaker:controld):  Started 
Inactive Resources

I'm missing the constraints, operations and timing details. How can i get them ?


Bernd

-- 
Bernd Lentes 

Systemadministration 
institute of developmental genetics 
Gebäude 35.34 - Raum 208 
HelmholtzZentrum München 
bernd.len...@helmholtz-muenchen.de 
phone: +49 (0)89 3187 1241 
fax: +49 (0)89 3187 2294 

Erst wenn man sich auf etwas festlegt kann man Unrecht haben 
Scott Adams
 

Helmholtz Zentrum Muenchen
Deutsches Forschungszentrum fuer Gesundheit und Umwelt (GmbH)
Ingolstaedter Landstr. 1
85764 Neuherberg
www.helmholtz-muenchen.de
Aufsichtsratsvorsitzende: MinDir'in Baerbel Brumme-Bothe
Geschaeftsfuehrer: Prof. Dr. Guenther Wess, Heinrich Bassler, Dr. Alfons Enhsen
Registergericht: Amtsgericht Muenchen HRB 6466
USt-IdNr: DE 129521671


___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] crm_mon -h (writing to a html-file) not showing all desired information and having trouble with the -d option

[Ken Gaillot]

On 05/08/2017 11:13 AM, Lentes, Bernd wrote:
> Hi,
> 
> playing around with my cluster i always have a shell with crm_mon running 
> because it provides me a lot of useful and current information concerning 
> cluster, nodes, resources ...
> Normally i have a "crm_mon -nrfRAL" running.
> I'd like to have that output as a web page too.
> So i tried the option -h.
> I have crm_mon from pacemaker 1.1.12 on a SLES 11 SP4 box. I'm writing the 
> file to /srv/www/hawk/public/crm_mon.html.
> I have hawk running, so i don't need an extra webserver for that.
> 
> First, i was very astonished when i used the option -d (daemonize). Using 
> that hawk does not find the html-file, although i see it in the fs, and it's 
> looking good.
> Hawk (or lighttpd) throws an error 404. Without -d lighttpd finds the files 
> and presents it via browser !?!
> 
> This is the file without -d:
> 
> ha-idg-2:/srv/www/hawk/public # stat crm_mon.html
>   File: `crm_mon.html'
>   Size: 1963Blocks: 8  IO Block: 4096   regular file
> Device: 1fh/31d Inode: 7082Links: 1
> Access: (0644/-rw-r--r--)  Uid: (0/root)   Gid: (0/root)
> Access: 2017-05-08 18:03:25.695754151 +0200
> Modify: 2017-05-08 18:03:20.875680374 +0200
> Change: 2017-05-08 18:03:20.875680374 +0200
> 
> 
> Same file with crm_mon -d:
> 
> ha-idg-2:/srv/www/hawk/public # stat crm_mon.html
>   File: `crm_mon.html'
>   Size: 1963Blocks: 8  IO Block: 4096   regular file
> Device: 1fh/31d Inode: 7084Links: 1
> Access: (0640/-rw-r-)  Uid: (0/root)   Gid: (0/root)

The "other" bit is gone, is that it?

> Access: 2017-05-08 18:04:16.048524856 +0200
> Modify: 2017-05-08 18:04:16.048524856 +0200
> Change: 2017-05-08 18:04:16.048524856 +0200
>  Birth: -
> 
> I see no important difference, just the different inode.
> 
> This is the access.log from lighttpd:
> 
> 10.35.34.70 ha-idg-2:7630 - [08/May/2017:18:04:10 +0200] "GET /crm_mon.html 
> HTTP/1.1" 200 563 "https://ha-idg-2:7630/crm_mon.html"; "Mozilla/5.0 (Windows 
> NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) 
> Chrome/57.0.2987.133 Safa
> ri/537.36"
> 10.35.34.70 ha-idg-2:7630 - [08/May/2017:18:04:15 +0200] "GET /crm_mon.html 
> HTTP/1.1" 200 563 "https://ha-idg-2:7630/crm_mon.html"; "Mozilla/5.0 (Windows 
> NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) 
> Chrome/57.0.2987.133 Safa
> ri/537.36"
> 10.35.34.70 ha-idg-2:7630 - [08/May/2017:18:04:20 +0200] "GET /crm_mon.html 
> HTTP/1.1" 404 1163 "https://ha-idg-2:7630/crm_mon.html"; "Mozilla/5.0 (Windows 
> NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) 
> Chrome/57.0.2987.133 Saf
> ari/537.36"
> 
> It simply changes from http status code 200 to 404. Why ? 
> 
> And using "crm_mon -nfrotRALV -h /srv/www/hawk/public/crm_mon.html" i get the 
> following output:
> 
> 
> Cluster summary
> 
> Last updated: Mon May 8 18:08:58 2017
> Current DC: ha-idg-2
> 2 Nodes configured.
> 14 Resources configured.
> Config Options
> 
> STONITH of failed nodes   :   enabled
> Cluster is:   symmetric
> No Quorum Policy  :   Ignore
> Node List
> 
> Node: ha-idg-1: online
> prim_clvmd(ocf::lvm2:clvmd):  Started 
> prim_stonith_ipmi_ha-idg-2(stonith:external/ipmi):Started 
> prim_ocfs2(ocf::ocfs2:o2cb):  Started 
> prim_vm_mausdb(ocf::heartbeat:VirtualDomain): Started 
> prim_vg_cluster_01(ocf::heartbeat:LVM):   Started 
> prim_fs_lv_xml_vm (ocf::heartbeat:Filesystem):Started 
> prim_dlm  (ocf::pacemaker:controld):  Started 
> prim_vnc_ip_mausdb(ocf::lentes:IPaddr):   Started 
> Node: ha-idg-2: online
> prim_clvmd(ocf::lvm2:clvmd):  Started 
> prim_stonith_ipmi_ha-idg-1(stonith:external/ipmi):Started 
> prim_ocfs2(ocf::ocfs2:o2cb):  Started 
> prim_vg_cluster_01(ocf::heartbeat:LVM):   Started 
> prim_fs_lv_xml_vm (ocf::heartbeat:Filesystem):Started 
> prim_dlm  (ocf::pacemaker:controld):  Started 
> Inactive Resources
> 
> I'm missing the constraints, operations and timing details. How can i get 
> them ?
> 
> 
> Bernd

The crm_mon HTML code doesn't get many reports/requests/submissions from
users, so it doesn't get a lot of attention. I wouldn't be too surprised
if there are some loose ends.

I'm not sure why those sections wouldn't appear. The code for it seems
to be there.



___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] crm_mon -h (writing to a html-file) not showing all desired information and having trouble with the -d option

[Greg Woods]

On Mon, May 8, 2017 at 10:13 AM, Lentes, Bernd <
bernd.len...@helmholtz-muenchen.de> wrote:

> Access: (0644/-rw-r--r--)  Uid: (0/root)   Gid: (0/root)
>
> Access: (0640/-rw-r-)  Uid: (0/root)   Gid: (0/root)
>
> I see no important difference, just the different inode.
>

The permissions are also different, this is most likely the problem. The
web server doesn't have permission to read the file.

--Greg
___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] crm_mon -h (writing to a html-file) not showing all desired information and having trouble with the -d option

[Lentes, Bernd]

- On May 8, 2017, at 6:44 PM, Ken Gaillot kgail...@redhat.com wrote:


>> 
>> This is the file without -d:
>> 
>> ha-idg-2:/srv/www/hawk/public # stat crm_mon.html
>>   File: `crm_mon.html'
>>   Size: 1963Blocks: 8  IO Block: 4096   regular file
>> Device: 1fh/31d Inode: 7082Links: 1
>> Access: (0644/-rw-r--r--)  Uid: (0/root)   Gid: (0/root)
>> Access: 2017-05-08 18:03:25.695754151 +0200
>> Modify: 2017-05-08 18:03:20.875680374 +0200
>> Change: 2017-05-08 18:03:20.875680374 +0200
>> 
>> 
>> Same file with crm_mon -d:
>> 
>> ha-idg-2:/srv/www/hawk/public # stat crm_mon.html
>>   File: `crm_mon.html'
>>   Size: 1963Blocks: 8  IO Block: 4096   regular file
>> Device: 1fh/31d Inode: 7084Links: 1
>> Access: (0640/-rw-r-)  Uid: (0/root)   Gid: (0/root)
> 
> The "other" bit is gone, is that it?
> 
>> Access: 2017-05-08 18:04:16.048524856 +0200
>> Modify: 2017-05-08 18:04:16.048524856 +0200
>> Change: 2017-05-08 18:04:16.048524856 +0200
>>  Birth: -
>> 

You are right. I oversaw that. That must be the reason.
lighttpd is running as user lighttpd, so it needs the "other" permission.

Bernd
 

Helmholtz Zentrum Muenchen
Deutsches Forschungszentrum fuer Gesundheit und Umwelt (GmbH)
Ingolstaedter Landstr. 1
85764 Neuherberg
www.helmholtz-muenchen.de
Aufsichtsratsvorsitzende: MinDir'in Baerbel Brumme-Bothe
Geschaeftsfuehrer: Prof. Dr. Guenther Wess, Heinrich Bassler, Dr. Alfons Enhsen
Registergericht: Amtsgericht Muenchen HRB 6466
USt-IdNr: DE 129521671


___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

[ClusterLabs] how to set a dedicated fence delay for a stonith agent ?

[Lentes, Bernd]

Hi,

i remember that digimer often campaigns for a fence delay in a 2-node  cluster.
E.g. here: http://oss.clusterlabs.org/pipermail/pacemaker/2013-July/019228.html
In my eyes it makes sense, so i try to establish that. I have two HP servers, 
each with an ILO card.
I have to use the stonith:external/ipmi agent, the stonith:external/riloe 
refused to work.

But i don't have a delay parameter there.
crm ra info stonith:external/ipmi:

...
pcmk_delay_max (time, [0s]): Enable random delay for stonith actions and 
specify the maximum of random delay
This prevents double fencing when using slow devices such as sbd.
Use this to enable random delay for stonith actions and specify the maximum 
of random delay.
...

This is the only delay parameter i can use. But a random delay does not seem to 
be a reliable solution.

The stonith:ipmilan agent also provides just a random delay. Same with the 
riloe agent.

How did anyone solve this problem ?

Or do i have to edit the RA (I will get practice in that :-))?


Bernd


-- 
Bernd Lentes 

Systemadministration 
institute of developmental genetics 
Gebäude 35.34 - Raum 208 
HelmholtzZentrum München 
bernd.len...@helmholtz-muenchen.de 
phone: +49 (0)89 3187 1241 
fax: +49 (0)89 3187 2294 

Erst wenn man sich auf etwas festlegt kann man Unrecht haben 
Scott Adams
 

Helmholtz Zentrum Muenchen
Deutsches Forschungszentrum fuer Gesundheit und Umwelt (GmbH)
Ingolstaedter Landstr. 1
85764 Neuherberg
www.helmholtz-muenchen.de
Aufsichtsratsvorsitzende: MinDir'in Baerbel Brumme-Bothe
Geschaeftsfuehrer: Prof. Dr. Guenther Wess, Heinrich Bassler, Dr. Alfons Enhsen
Registergericht: Amtsgericht Muenchen HRB 6466
USt-IdNr: DE 129521671


___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Instant service restart during failback

[Ken Gaillot]

If you look in the logs when the node comes back, there should be some
"pengine:" messages noting that the restarts will be done, and then a
"saving inputs in " message. If you can attach that file (both
with and without the constraint changes would be ideal), I'll take a
look at it.

On 04/21/2017 05:26 AM, Euronas Support wrote:
> Seems that replacing inf: with 0: in some colocation constraints fixes the 
> problem, but still cannot understand why it worked for one node and not for 
> the other.
> 
> On 20.4.2017 12:16:02 Klechomir wrote:
>> Hi Klaus,
>> It would have been too easy if it was interleave.
>> All my cloned resoures have interlave=true, of course.
>> What bothers me more is that the behaviour is asymmetrical.
>>
>> Regards,
>> Klecho
>>
>> On 20.4.2017 10:43:29 Klaus Wenninger wrote:
>>> On 04/20/2017 10:30 AM, Klechomir wrote:
 Hi List,
 Been investigating the following problem recently:

 Have two node cluster with 4 cloned (2 on top of 2) + 1 master/slave
 services on it (corosync+pacemaker 1.1.15)
 The failover works properly for both nodes, i.e. when one node is
 restarted/turned in standby, the other properly takes over, but:

 Every time when node2 has been in standby/turned off and comes back,
 everything recovers propery.
 Every time when node1 has been in standby/turned off and comes back,
 part
 of the cloned services on node2 are getting instantly restarted, at the
 same second when node1 re-appeares, without any apparent reason (only
 the
 stop/start messages in the debug).

 Is there some known possible reason for this?
>>>
>>> That triggers some deja-vu feeling...
>>> Did you have a similar issue a couple of weeks ago?
>>> I remember in that particular case 'interleave=true' was not the
>>> solution to the problem but maybe here ...
>>>
>>> Regards,
>>> Klaus
>>>
 Best regards,
 Klecho

___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] pacemaker daemon shutdown time with lost remote node

[Ken Gaillot]

On 04/28/2017 02:22 PM, Radoslaw Garbacz wrote:
> Hi,
> 
> I have a question regarding pacemaker daemon shutdown
> procedure/configuration.
> 
> In my case, when a remote node is lost pacemaker needs exactly 10minutes
> to shutdown, during which there is nothing logged.
> So my questions:
> 1. What is pacemaker doing at this time?
> 2. How to make it shorter?

The logs from the other nodes will be helpful. One of the nodes will be
the DC, and will have all the scheduled commands.

Generally, in a shutdown, pacemaker first tries to stop all resources.
If one of those stops is either taking a long time or timing out, that
might explain it.

> Changed Pacemaker Configuration:
> - cluster-delay
> - dc-deadtime
> 
> 
> Pacemaker Logs:
> Apr 28 17:38:08 [17689] ip-10-41-177-183 pacemakerd:   notice:
> crm_signal_dispatch: Caught 'Terminated' signal | 15 (invoking handler)
> Apr 28 17:38:08 [17689] ip-10-41-177-183 pacemakerd:   notice:
> pcmk_shutdown_worker:Shutting down Pacemaker
> Apr 28 17:38:08 [17689] ip-10-41-177-183 pacemakerd:   notice:
> stop_child:  Stopping crmd | sent signal 15 to process 17698
> Apr 28 17:48:07 [17695] ip-10-41-177-183   lrmd: info:
> cancel_recurring_action: Cancelling ocf operation
> monitor_head_monitor_191000
> Apr 28 17:48:07 [17695] ip-10-41-177-183   lrmd: info:
> log_execute: executing - rsc:monitor_head action:stop call_id:130
> [...]
> Apr 28 17:48:07 [17689] ip-10-41-177-183 pacemakerd: info: main:   
> Exiting pacemakerd
> Apr 28 17:48:07 [17689] ip-10-41-177-183 pacemakerd: info:
> crm_xml_cleanup: Cleaning up memory from libxml2
> 
> 
> Pacemaker built from github: 1.16
> 
> 
> Help greatly appreciated.
> 
> -- 
> Best Regards,
> 
> Radoslaw Garbacz
> XtremeData Incorporated

___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] Antw: Antw: notice: throttle_handle_load: High CPU load detected

[Ken Gaillot]

On 05/05/2017 12:37 AM, jitendra.jaga...@dell.com wrote:
>  
> 
> Hello All,
> 
>  
> 
> Sorry for resurrecting old thread.
> 
>  
> 
> I am also observing “High CPU load detected" messages in the logs
> 
>  
> 
> In this email chain, I see everyone is suggesting to change
> "load-threshold" settings
> 
>  
> 
> But I am not able to find any good information about “load-threshold”
> except this https://www.mankier.com/7/crmd
> 
>  
> 
> Even in Pacemaker document
> “http://clusterlabs.org/doc/en-US/Pacemaker/1.1-pcs/pdf/Pacemaker_Explained/Pacemaker-1.1-Pacemaker_Explained-en-US.pdf”
> 
>  
> 
> There is not much detail about “load-threshold”.
> 
>  
> 
> Please can someone share steps or any commands to modify “load-threshold”.
> 
>  
> 
> Thanks
> 
> Jitendra

Hi Jitendra,

Those messages indicate there is a real issue with the CPU load. When
the cluster notices high load, it reduces the number of actions it will
execute at the same time. This is generally a good idea, to avoid making
the load worse.

The messages don't hurt anything, they just let you know that there is
something worth investigating.

If you've investigated the load and it's not something to be concerned
about, you can change load-threshold to adjust what the cluster
considers "high". The load-threshold works like this:

* It defaults to 0.8 (which means pacemaker should try to avoid
consuming more than 80% of the system's resources).

* On a single-core machine, load-threshold is multiplied by 0.6 (because
with only one core you *really* don't want to consume too many
resources); on a multi-core machine, load-threshold is multiplied by the
number of cores (to normalize the system load per core).

* That number is then multiplied by 1.2 to get the "Noticeable CPU load
detected" message (debug level), by 1.6 to get the "Moderate CPU load"
message, and 2.0 to get the "High CPU load" message. These are measured
against the 1-minute system load average (the same number you would get
with top, uptime, etc.).

So, if you raise load-threshold above 0.8, you won't see the log
messages until the load gets even higher. But, that doesn't do anything
about the actual load problem.

> *From:*Kostiantyn Ponomarenko [mailto:konstantin.ponomare...@gmail.com]
> *Sent:* Tuesday, April 5, 2016 8:37 AM
> *To:* kgail...@redhat.com
> *Cc:* Cluster Labs - All topics related to open-source clustering
> welcomed 
> *Subject:* Re: [ClusterLabs] Antw: Antw: notice: throttle_handle_load:
> High CPU load detected
> 
>  
> 
> Thank you, Ken.
> 
> This helps a lot.
> 
> Now I am sure that my current approach fits best for me =)
> 
> 
> Thank you,
> 
> Kostia
> 
>  
> 
> On Wed, Mar 30, 2016 at 11:10 PM, Ken Gaillot  > wrote:
> 
> On 03/29/2016 08:22 AM, Kostiantyn Ponomarenko wrote:
> > Ken, thank you for the answer.
> >
> > Every node in my cluster under normal conditions has "load average" of
> > about 420. It is mainly connected to the high disk IO on the system.
> > My system is designed to use almost 100% of its hardware
> (CPU/RAM/disks),
> > so the situation when the system consumes almost all HW resources is
> > normal.
> 
> 420 suggests that HW resources are outstripped -- anything above the
> system's number of cores means processes are waiting for some resource.
> (Although with an I/O-bound workload like this, the number of cores
> isn't very important -- most will be sitting idle despite the high
> load.) And if that's during normal conditions, what will happen during a
> usage spike? It sounds like a recipe for less-than-HA.
> 
> Under high load, there's a risk of negative feedback, where monitors
> time out, causing pacemaker to schedule recovery actions, which cause
> load to go higher and more monitors to time out, etc. That's why
> throttling is there.
> 
> > I would like to get rid of "High CPU load detected" messages in the
> > log, because
> > they flood corosync.log as well as system journal.
> >
> > Maybe you can give an advice what would be the best way do to it?
> >
> > So far I came up with the idea of setting "load-threshold" to 1000% ,
> > because of:
> > 420(load average) / 24 (cores) = 17.5 (adjusted_load);
> > 2 (THROTLE_FACTOR_HIGH) * 10 (throttle_load_target) = 20
> >
> > if(adjusted_load > THROTTLE_FACTOR_HIGH * throttle_load_target) {
> > crm_notice("High %s detected: %f", desc, load);
> 
> That should work, as far as reducing the log messages, though of course
> it also reduces the amount of throttling pacemaker will do.
> 
> > In this case do I need to set "node-action-limit" to something
> less than "2
> > x cores" (which is default).
> 
> It's not necessary, but it would help compensate for the reduced
> throttling by imposing a maximum number of actions run at one time.
> 
> I usually wouldn't recommend reducing log verbosi

Re: [ClusterLabs] Antw: Behavior after stop action failure with the failure-timeout set and STONITH disabled

[Ken Gaillot]

On 05/05/2017 07:49 AM, Jan Wrona wrote:
> On 5.5.2017 08:15, Ulrich Windl wrote:
> Jan Wrona  schrieb am 04.05.2017 um 16:41 in
> Nachricht
>> :
>>> I hope I'll be able to explain the problem clearly and correctly.
>>>
>>> My setup (simplified): I have two cloned resources, a filesystem mount
>>> and a process which writes to that filesystem. The filesystem is Gluster
>>> so its OK to clone it. I also have a mandatory ordering constraint
>>> "start gluster-mount-clone then start writer-process-clone". I don't
>>> have a STONITH device, so I've disable STONITH by setting
>>> stonith-enabled=false.
>>>
>>> The problem: Sometimes the Gluster freezes for a while, which causes the
>>> gluster-mount resource's monitor with the OCF_CHECK_LEVEL=20 to timeout
>>> (it is unable to write the status file). When this happens, the cluster

Have you tried increasing the monitor timeouts?

>> Actually I would do two things:
>>
>> 1) Find out why Gluster freezes, and what to do to avoid that
> 
> It freezes when one of the underlying MD RAIDs starts its regular check.
> I've decreased its speed limit (from the default 200 MB/s to the 50
> MB/s, I cannot go any lower), but it helped only a little, the mount
> still tends to freeze for a few seconds during the check.
> 
>>
>> 2) Implement stonith
> 
> Currently I can't. But AFAIK Pacemaker should work properly even with
> disabled STONITH and the state I've run into doesn't seem right to me at
> all. I was asking for clarification of what the cluster is trying to do
> in such situation, I don't understand the "Ignoring expired calculated
> failure" log messages and I don't understand why the crm_mon was showing
> that the writer-process is started even though it was not.

Pacemaker can work without stonith, but there are certain failure
situations that can't be recovered any other way, so whether that's
working "properly" is a matter of opinion. :-) In this particular case,
stonith doesn't make the situation much better -- you want to prevent
the need for stonith to begin with (hopefully increasing the monitor
timeouts is sufficient). But stonith is still good to have for other
situations.

The cluster shows the service as started because it determines the state
by the service's operation history:

   successful start at time A = started
   successful start at time A + failed stop at time B = started (failed)
   after failure expires, back to: successful start at time A = started

If the service is not actually running at that point, the next recurring
monitor should detect that.

>> Regards,
>> Ulrich
>>
>>
>>> tries to recover by restarting the writer-process resource. But the
>>> writer-process is writing to the frozen filesystem which makes it
>>> uninterruptable, not even SIGKILL works. Then the stop operation times
>>> out and on-fail with disabled STONITH defaults to block (don’t perform
>>> any further operations on the resource):
>>> warning: Forcing writer-process-clone away from node1.example.org after
>>> 100 failures (max=100)
>>> After that, the cluster continues with the recovery process by
>>> restarting the gluster-mount resource on that node and it usually
>>> succeeds. As a consequence of that remount, the uninterruptable system
>>> call in the writer process fails, signals are finally delivered and the
>>> writer-process is terminated. But the cluster doesn't know about that!
>>>
>>> I thought I can solve this by setting the failure-timeout meta attribute
>>> to the writer-process resource, but it only made things worse. The
>>> documentation states: "Stop failures are slightly different and crucial.
>>> ... If a resource fails to stop and STONITH is not enabled, then the
>>> cluster has no way to continue and will not try to start the resource
>>> elsewhere, but will try to stop it again after the failure timeout.",

The documentation is silently making the assumption that the condition
that led to the initial stop is still true. In this case, if the gluster
failure has long since been cleaned up, there is no reason to try to
stop the writer-process.

>>> but I'm seeing something different. When the policy engine is launched
>>> after the nearest cluster-recheck-interval, following lines are written
>>> to the syslog:
>>> crmd[11852]: notice: State transition S_IDLE -> S_POLICY_ENGINE
>>> pengine[11851]:  notice: Clearing expired failcount for writer-process:1
>>> on node1.example.org
>>> pengine[11851]:  notice: Clearing expired failcount for writer-process:1
>>> on node1.example.org
>>> pengine[11851]:  notice: Ignoring expired calculated failure
>>> writer-process_stop_0 (rc=1,
>>> magic=2:1;64:557:0:2169780b-ca1f-483e-ad42-118b7c7c1a7d) on
>>> node1.example.org
>>> pengine[11851]:  notice: Clearing expired failcount for writer-process:1
>>> on node1.example.org
>>> pengine[11851]:  notice: Ignoring expired calculated failure
>>> writer-process_stop_0 (rc=1,
>>> magic=2:1;64:557:0:2169780b-ca1f-483e-ad42-118b7c7c1a7d) on
>>> node1

[ClusterLabs] Pacemaker 1.1.17-rc1 now available

[Ken Gaillot]

Source code for the first release candidate for Pacemaker version 1.1.17
is now available at:

https://github.com/ClusterLabs/pacemaker/releases/tag/Pacemaker-1.1.17-rc1

The most significant enhancements in this release are:

* A new "bundle" resource type simplifies launching resources inside
Docker containers. This feature is considered experimental for this
release. It was discussed in detail previously:

  http://lists.clusterlabs.org/pipermail/users/2017-April/005380.html

A walk-through is available on the ClusterLabs wiki for anyone who wants
to experiment with the feature:

  http://wiki.clusterlabs.org/wiki/Bundle_Walk-Through

* A new environment variable PCMK_node_start_state can specify that a
node should start in standby mode. It was also discussed previously:

  http://lists.clusterlabs.org/pipermail/users/2017-April/005607.html

* The "crm_resource --cleanup" and "crm_failcount" commands can now
operate on a single operation type (previously, they could only operate
on all operations at once). This is part of an underlying switch to
tracking failure counts per operation, also discussed previously:

  http://lists.clusterlabs.org/pipermail/users/2017-April/005391.html

* Several command-line tools have new options, including "crm_resource
--validate" to run a resource agent's validate-all action,
"stonith_admin --list-targets" to list all potential targets of a fence
device, and "crm_attribute --pattern" to update or delete all node
attributes matching a regular expression

* The cluster's handling of fence failures has been improved. Among the
changes, a new "stonith-max-attempts" cluster option specifies how many
times fencing can fail for a target before the cluster will no longer
immediately re-attempt it (previously hard-coded at 10).

* Location constraints using rules may now compare a node attribute
against a resource parameter, using the new "value-source" field.
Previously, node attributes could only be compared against literal
values. This is most useful in combination with rsc-pattern to apply the
constraint to multiple resources.

As usual, to support the new features, the CRM feature set has been
incremented. This means that mixed-version clusters are supported only
during a rolling upgrade -- nodes with an older version will not be
allowed to rejoin once they shut down.

For a more detailed list of bug fixes and other changes, see the change log:

https://github.com/ClusterLabs/pacemaker/blob/1.1/ChangeLog

Everyone is encouraged to download, compile and test the new release. We
do many regression tests and simulations, but we can't cover all
possible use cases, so your feedback is important and appreciated.

Many thanks to all contributors of source code to this release,
including Alexandra Zhuravleva, Andrew Beekhof, Aravind Kumar, Eric
Marques, Ferenc Wágner, Yan Gao, Hayley Swimelar, Hideo Yamauchi, Igor
Tsiglyar, Jan Pokorný, Jehan-Guillaume de Rorthais, Ken Gaillot, Klaus
Wenninger, Kristoffer Grönlund, Michal Koutný, Nate Clark, Patrick
Hemmer, Sergey Mishin, Vladislav Bogdanov, and Yusuke Iida. Apologies if
I have overlooked anyone.
-- 
Ken Gaillot 

___
Users mailing list: Users@clusterlabs.org
http://lists.clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org