Re: [ClusterLabs] large cluster - failure recovery

[Ken Gaillot]

On 11/04/2015 12:55 PM, Digimer wrote:
> On 04/11/15 01:50 PM, Radoslaw Garbacz wrote:
>> Hi,
>>
>> I have a cluster of 32 nodes, and after some tuning was able to have it
>> started and running,
> 
> This is not supported by RH for a reasons; it's hard to get the timing
> right. SUSE supports up to 32 nodes, but they must be doing some serious
> magic behind the scenes.
> 
> I would *strongly* recommend dividing this up into a few smaller
> clusters... 8 nodes per cluster would be max I'd feel comfortable with.
> You need your cluster to solve more problems than it causes...

Hi Radoslaw,

RH supports up to 16. 32 should be possible with recent
pacemaker+corosync versions and careful tuning, but it's definitely
leading-edge.

An alternative with pacemaker 1.1.10+ (1.1.12+ recommended) is Pacemaker
Remote, which easily scales to dozens of nodes:
http://clusterlabs.org/doc/en-US/Pacemaker/1.1-pcs/html-single/Pacemaker_Remote/index.html

Pacemaker Remote is a really good approach once you start pushing the
limits of cluster nodes. Probably better than trying to get corosync to
handle more nodes. (There are long-term plans for improving corosync's
scalability, but that doesn't help you now.)

>> but it does not recover from a node disconnect-connect failure.
>> It regains quorum, but CIB does not recover to a synchronized state and
>> "cibadmin -Q" times out.
>>
>> Is there anything with corosync or pacemaker parameters I can do to make
>> it recover from such a situation
>> (everything works for smaller clusters).
>>
>> In my case it is OK for a node to disconnect (all the major resources
>> are shutdown)
>> and later reconnect the cluster (the running monitoring agent will
>> cleanup and restart major resources if needed),
>> so I do not have STONITH configured.
>>
>> Details:
>> OS: CentOS 6
>> Pacemaker: Pacemaker 1.1.9-1512.el6
> 
> Upgrade.

If you can upgrade to the latest CentOS 6.7, you can get a much newer
Pacemaker. But Pacemaker is probably not limiting your cluster nodes;
the newer version's main benefit would be Pacemaker Remote support. (Of
course there are plenty of bug fixes and new features as well.)

>> Corosync: Corosync Cluster Engine, version '2.3.2'
> 
> This is not supported on EL6 at all. Please stick with corosync 1.4 and
> use the cman pluging as the quorum provider.

CentOS is self-supported anyway, so if you're willing to handle your own
upgrades and such, nothing wrong with compiling. But corosync is up to
2.3.5 so you're already behind. :) I'd recommend compiling libqb 0.17.2
if you're compiling recent corosync and/or pacemaker.

Alternatively, CentOS 7 will have recent versions of everything.

>> Corosync configuration:
>> token: 1
>> #token_retransmits_before_loss_const: 10
>> consensus: 15000
>> join: 1000
>> send_join: 80
>> merge: 1000
>> downcheck: 2000
>> #rrp_problem_count_timeout: 5000
>> max_network_delay: 150 # for azure
>>
>>
>> Some logs:
>>
>> [...]
>> Nov 04 17:50:18 [7985] ip-10-142-181-98 stonith-ng:   notice:
>> cib_process_diff: Diff 1.9254.1 -> 1.9255.1 from local not
>> applied to 1.9275.1: current "epoch" is greater than required
>> Nov 04 17:50:18 [7985] ip-10-142-181-98 stonith-ng:   notice:
>> update_cib_cache_cb:  [cib_diff_notify] Patch aborted: Application
>> of an update diff failed (-1006)
>> Nov 04 17:50:18 [7985] ip-10-142-181-98 stonith-ng:   notice:
>> cib_process_diff: Diff 1.9255.1 -> 1.9256.1 from local not
>> applied to 1.9275.1: current "epoch" is greater than required
>> Nov 04 17:50:18 [7985] ip-10-142-181-98 stonith-ng:   notice:
>> update_cib_cache_cb:  [cib_diff_notify] Patch aborted: Application
>> of an update diff failed (-1006)
>> Nov 04 17:50:18 [7985] ip-10-142-181-98 stonith-ng:   notice:
>> cib_process_diff: Diff 1.9256.1 -> 1.9257.1 from local not
>> applied to 1.9275.1: current "epoch" is greater than required
>> Nov 04 17:50:18 [7985] ip-10-142-181-98 stonith-ng:   notice:
>> update_cib_cache_cb:  [cib_diff_notify] Patch aborted: Application
>> of an update diff failed (-1006)
>> Nov 04 17:50:18 [7985] ip-10-142-181-98 stonith-ng:   notice:
>> cib_process_diff: Diff 1.9257.1 -> 1.9258.1 from local not
>> applied to 1.9275.1: current "epoch" is greater than required
>> Nov 04 17:50:18 [7985] ip-10-142-181-98 stonith-ng:   notice:
>> update_cib_cache_cb:  [cib_diff_notify] Patch aborted: Application
>> of an update diff failed (-1006)
>> [...]
>>
>> [...]
>> Nov 04 17:43:24 [12176] ip-10-109-145-175crm_mon:error:
>> cib_native_perform_op_delegate: Couldn't perform cib_query
>> operation (timeout=120s): Operation already in progress (-114)
>> Nov 04 17:43:24 [12176] ip-10-109-145-175crm_mon:error:
>> get_cib_copy:   Couldnt retrieve the CIB
>> Nov 04 17:43:24 [12176] ip-10-109-145-175crm_mon:error:
>> cib_native_perform_op_delegate: Couldn't perform cib_query
>> 

Re: [ClusterLabs] large cluster - failure recovery

[Radoslaw Garbacz]

Thank you Ken and Digimer for all your suggestions.

On Wed, Nov 4, 2015 at 2:32 PM, Ken Gaillot  wrote:

> On 11/04/2015 12:55 PM, Digimer wrote:
> > On 04/11/15 01:50 PM, Radoslaw Garbacz wrote:
> >> Hi,
> >>
> >> I have a cluster of 32 nodes, and after some tuning was able to have it
> >> started and running,
> >
> > This is not supported by RH for a reasons; it's hard to get the timing
> > right. SUSE supports up to 32 nodes, but they must be doing some serious
> > magic behind the scenes.
> >
> > I would *strongly* recommend dividing this up into a few smaller
> > clusters... 8 nodes per cluster would be max I'd feel comfortable with.
> > You need your cluster to solve more problems than it causes...
>
> Hi Radoslaw,
>
> RH supports up to 16. 32 should be possible with recent
> pacemaker+corosync versions and careful tuning, but it's definitely
> leading-edge.
>
> An alternative with pacemaker 1.1.10+ (1.1.12+ recommended) is Pacemaker
> Remote, which easily scales to dozens of nodes:
>
> http://clusterlabs.org/doc/en-US/Pacemaker/1.1-pcs/html-single/Pacemaker_Remote/index.html
>
> Pacemaker Remote is a really good approach once you start pushing the
> limits of cluster nodes. Probably better than trying to get corosync to
> handle more nodes. (There are long-term plans for improving corosync's
> scalability, but that doesn't help you now.)
>
> >> but it does not recover from a node disconnect-connect failure.
> >> It regains quorum, but CIB does not recover to a synchronized state and
> >> "cibadmin -Q" times out.
> >>
> >> Is there anything with corosync or pacemaker parameters I can do to make
> >> it recover from such a situation
> >> (everything works for smaller clusters).
> >>
> >> In my case it is OK for a node to disconnect (all the major resources
> >> are shutdown)
> >> and later reconnect the cluster (the running monitoring agent will
> >> cleanup and restart major resources if needed),
> >> so I do not have STONITH configured.
> >>
> >> Details:
> >> OS: CentOS 6
> >> Pacemaker: Pacemaker 1.1.9-1512.el6
> >
> > Upgrade.
>
> If you can upgrade to the latest CentOS 6.7, you can get a much newer
> Pacemaker. But Pacemaker is probably not limiting your cluster nodes;
> the newer version's main benefit would be Pacemaker Remote support. (Of
> course there are plenty of bug fixes and new features as well.)
>
> >> Corosync: Corosync Cluster Engine, version '2.3.2'
> >
> > This is not supported on EL6 at all. Please stick with corosync 1.4 and
> > use the cman pluging as the quorum provider.
>
> CentOS is self-supported anyway, so if you're willing to handle your own
> upgrades and such, nothing wrong with compiling. But corosync is up to
> 2.3.5 so you're already behind. :) I'd recommend compiling libqb 0.17.2
> if you're compiling recent corosync and/or pacemaker.
>
> Alternatively, CentOS 7 will have recent versions of everything.
>
> >> Corosync configuration:
> >> token: 1
> >> #token_retransmits_before_loss_const: 10
> >> consensus: 15000
> >> join: 1000
> >> send_join: 80
> >> merge: 1000
> >> downcheck: 2000
> >> #rrp_problem_count_timeout: 5000
> >> max_network_delay: 150 # for azure
> >>
> >>
> >> Some logs:
> >>
> >> [...]
> >> Nov 04 17:50:18 [7985] ip-10-142-181-98 stonith-ng:   notice:
> >> cib_process_diff: Diff 1.9254.1 -> 1.9255.1 from local not
> >> applied to 1.9275.1: current "epoch" is greater than required
> >> Nov 04 17:50:18 [7985] ip-10-142-181-98 stonith-ng:   notice:
> >> update_cib_cache_cb:  [cib_diff_notify] Patch aborted: Application
> >> of an update diff failed (-1006)
> >> Nov 04 17:50:18 [7985] ip-10-142-181-98 stonith-ng:   notice:
> >> cib_process_diff: Diff 1.9255.1 -> 1.9256.1 from local not
> >> applied to 1.9275.1: current "epoch" is greater than required
> >> Nov 04 17:50:18 [7985] ip-10-142-181-98 stonith-ng:   notice:
> >> update_cib_cache_cb:  [cib_diff_notify] Patch aborted: Application
> >> of an update diff failed (-1006)
> >> Nov 04 17:50:18 [7985] ip-10-142-181-98 stonith-ng:   notice:
> >> cib_process_diff: Diff 1.9256.1 -> 1.9257.1 from local not
> >> applied to 1.9275.1: current "epoch" is greater than required
> >> Nov 04 17:50:18 [7985] ip-10-142-181-98 stonith-ng:   notice:
> >> update_cib_cache_cb:  [cib_diff_notify] Patch aborted: Application
> >> of an update diff failed (-1006)
> >> Nov 04 17:50:18 [7985] ip-10-142-181-98 stonith-ng:   notice:
> >> cib_process_diff: Diff 1.9257.1 -> 1.9258.1 from local not
> >> applied to 1.9275.1: current "epoch" is greater than required
> >> Nov 04 17:50:18 [7985] ip-10-142-181-98 stonith-ng:   notice:
> >> update_cib_cache_cb:  [cib_diff_notify] Patch aborted: Application
> >> of an update diff failed (-1006)
> >> [...]
> >>
> >> [...]
> >> Nov 04 17:43:24 [12176] ip-10-109-145-175crm_mon:error:
> >> cib_native_perform_op_delegate: Couldn't 

[ClusterLabs] Antw: Re: [Question] Question about mysql RA.

[Ulrich Windl]

>>> Ken Gaillot  schrieb am 04.11.2015 um 16:44 in 
>>> Nachricht
<563a27c2.5090...@redhat.com>:
> On 11/04/2015 04:36 AM, renayama19661...@ybb.ne.jp wrote:
[...]
>> pid=`cat $OCF_RESKEY_pid 2> /dev/null `
>> /bin/kill $pid > /dev/null
> 
> I think before this line, the RA should do a "kill -0" to check whether
> the PID is running, and return $OCF_SUCCESS if not. That way, we can
> still return an error if the real kill fails.

And remove the stale PID file if there is no such pid. For very busy systems 
one could use ps for that PID to see whether the PID belongs to the expected 
process. There is a small chance that a PID exists, but does not belong to the 
expected process...

> 
>> rc=$?
>> if [ $rc != 0 ]; then
>> ocf_exit_reason "MySQL couldn't be stopped"
>> return $OCF_ERR_GENERIC
>> fi
>> (snip)
>> -
>> 
>> The mysql RA does such a code from old days.
>>  * http://hg.linux-ha.org/agents/file/67234f982ab7/heartbeat/mysql 
>> 
>> Does mysql RA know the reason becoming this made?
>> Possibly is it a factor to be conscious of mysql cluster?
>> 
>> I think about a patch of this movement of mysql RA.
>> I want to know the detailed reason.
>> 
>> Best Regards,
>> Hideo Yamauchi.
> 
> 
> ___
> Users mailing list: Users@clusterlabs.org 
> http://clusterlabs.org/mailman/listinfo/users 
> 
> Project Home: http://www.clusterlabs.org 
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf 
> Bugs: http://bugs.clusterlabs.org 





___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

[ClusterLabs] how to fence in a two node cluster.If haretbeat network is down between the two nodes, which node will fence the other node?

[Shilu]

[cid:image002.png@01D1091A.AEFC8740]
In HA, if a node is declared dead, it needs to be fenced/stonith'ed before its 
services are recovered. Not doing this can lead to a split-brain.

If haretbeat network is down between the two nodes,which node will fence the 
other node?

If two nodes were fenced respectively,the service will be not available.

I want to know if IPMI fence is appropriate to two nodes cluster. Is there a 
better fence way?



-



???
This e-mail and its attachments contain confidential information from H3C, 
which is
intended only for the person or entity whose address is listed above. Any use 
of the
information contained herein in any way (including, but not limited to, total 
or partial
disclosure, reproduction, or dissemination) by persons other than the intended
recipient(s) is prohibited. If you receive this e-mail in error, please notify 
the sender
by phone or email immediately and delete it!
___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] how to fence in a two node cluster.If haretbeat network is down between the two nodes, which node will fence the other node?

[Digimer]

On 05/11/15 02:09 AM, Shilu wrote:
> cid:image002.png@01D1091A.AEFC8740
> 
> In HA, if a node is declared dead, it needs to be fenced/stonith'ed
> before its services are recovered. Not doing this can lead to a split-brain.
> 
> If haretbeat network is down between the two nodes,which node will fence
> the other node?
> 
> If two nodes were fenced respectively,the service will be not available.
> 
> I want to know if IPMI fence is appropriate to two nodes cluster. Is
> there a better fence way?

You will choose a node to be the survivor in this case, and the set
'delay="15"' in the IPMI fence attribute list.

If you set the delay against node 1, for example, then in a comm break
like you describe, both will declare the other dead and start fencing at
the same time. Node 1 looks up how to fence node 2, sees no delay and
immediately starts to fence. Meanwhile, node 2 looks up how to fence
node 1, sees a delay and pauses for 15 seconds. Node 1 will kill node 2
long before the delay expires.

Also, disable acpid. You want the node to react to the power button
being pressed by shutting down immediately.

Also, this doesn't solve all problems. Pull all power to a node. This
will cause IPMI to fail, so the fence call will fail. This is why I use
IPMI *and* a pair of switched PDUs (IPMI on one switch, PDUs on another).

-- 
Digimer
Papers and Projects: https://alteeve.ca/w/
What if the cure for cancer is trapped in the mind of a person without
access to education?

___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [ClusterLabs] large cluster - failure recovery

[Digimer]

On 04/11/15 01:50 PM, Radoslaw Garbacz wrote:
> Hi,
> 
> I have a cluster of 32 nodes, and after some tuning was able to have it
> started and running,

This is not supported by RH for a reasons; it's hard to get the timing
right. SUSE supports up to 32 nodes, but they must be doing some serious
magic behind the scenes.

I would *strongly* recommend dividing this up into a few smaller
clusters... 8 nodes per cluster would be max I'd feel comfortable with.
You need your cluster to solve more problems than it causes...

> but it does not recover from a node disconnect-connect failure.
> It regains quorum, but CIB does not recover to a synchronized state and
> "cibadmin -Q" times out.
> 
> Is there anything with corosync or pacemaker parameters I can do to make
> it recover from such a situation
> (everything works for smaller clusters).
> 
> In my case it is OK for a node to disconnect (all the major resources
> are shutdown)
> and later reconnect the cluster (the running monitoring agent will
> cleanup and restart major resources if needed),
> so I do not have STONITH configured.
> 
> Details:
> OS: CentOS 6
> Pacemaker: Pacemaker 1.1.9-1512.el6

Upgrade.

> Corosync: Corosync Cluster Engine, version '2.3.2'

This is not supported on EL6 at all. Please stick with corosync 1.4 and
use the cman pluging as the quorum provider.

> Corosync configuration:
> token: 1
> #token_retransmits_before_loss_const: 10
> consensus: 15000
> join: 1000
> send_join: 80
> merge: 1000
> downcheck: 2000
> #rrp_problem_count_timeout: 5000
> max_network_delay: 150 # for azure
> 
> 
> Some logs:
> 
> [...]
> Nov 04 17:50:18 [7985] ip-10-142-181-98 stonith-ng:   notice:
> cib_process_diff: Diff 1.9254.1 -> 1.9255.1 from local not
> applied to 1.9275.1: current "epoch" is greater than required
> Nov 04 17:50:18 [7985] ip-10-142-181-98 stonith-ng:   notice:
> update_cib_cache_cb:  [cib_diff_notify] Patch aborted: Application
> of an update diff failed (-1006)
> Nov 04 17:50:18 [7985] ip-10-142-181-98 stonith-ng:   notice:
> cib_process_diff: Diff 1.9255.1 -> 1.9256.1 from local not
> applied to 1.9275.1: current "epoch" is greater than required
> Nov 04 17:50:18 [7985] ip-10-142-181-98 stonith-ng:   notice:
> update_cib_cache_cb:  [cib_diff_notify] Patch aborted: Application
> of an update diff failed (-1006)
> Nov 04 17:50:18 [7985] ip-10-142-181-98 stonith-ng:   notice:
> cib_process_diff: Diff 1.9256.1 -> 1.9257.1 from local not
> applied to 1.9275.1: current "epoch" is greater than required
> Nov 04 17:50:18 [7985] ip-10-142-181-98 stonith-ng:   notice:
> update_cib_cache_cb:  [cib_diff_notify] Patch aborted: Application
> of an update diff failed (-1006)
> Nov 04 17:50:18 [7985] ip-10-142-181-98 stonith-ng:   notice:
> cib_process_diff: Diff 1.9257.1 -> 1.9258.1 from local not
> applied to 1.9275.1: current "epoch" is greater than required
> Nov 04 17:50:18 [7985] ip-10-142-181-98 stonith-ng:   notice:
> update_cib_cache_cb:  [cib_diff_notify] Patch aborted: Application
> of an update diff failed (-1006)
> [...]
> 
> [...]
> Nov 04 17:43:24 [12176] ip-10-109-145-175crm_mon:error:
> cib_native_perform_op_delegate: Couldn't perform cib_query
> operation (timeout=120s): Operation already in progress (-114)
> Nov 04 17:43:24 [12176] ip-10-109-145-175crm_mon:error:
> get_cib_copy:   Couldnt retrieve the CIB
> Nov 04 17:43:24 [12176] ip-10-109-145-175crm_mon:error:
> cib_native_perform_op_delegate: Couldn't perform cib_query
> operation (timeout=120s): Operation already in progress (-114)
> Nov 04 17:43:24 [12176] ip-10-109-145-175crm_mon:error:
> get_cib_copy:   Couldnt retrieve the CIB
> Nov 04 17:47:40 [10599] ip-10-109-145-175 corosync notice  [QUORUM]
> Members[32]: 3 27 11 29 23 21 24 9 17 12 32 13 2 10 16 15 6 28 19 1 22 26 5\
> Nov 04 17:47:40 [10599] ip-10-109-145-175 corosync notice  [QUORUM]
> Members[32]: 14 20 31 30 8 25 18 7 4
> Nov 04 17:47:40 [10599] ip-10-109-145-175 corosync notice  [MAIN  ]
> Completed service synchronization, ready to provide service.
> Nov 04 18:06:55 [10599] ip-10-109-145-175 corosync notice  [QUORUM]
> Members[32]: 3 27 11 29 23 21 24 9 17 12 32 13 2 10 16 15 6 28 19 1 22 26 5\
> Nov 04 18:06:55 [10599] ip-10-109-145-175 corosync notice  [QUORUM]
> Members[32]: 14 20 31 30 8 25 18 7 4
> [...]
> 
> [...]
> Nov 04 18:21:15 [17749] ip-10-178-149-131 stonith-ng:   notice:
> update_cib_cache_cb:[cib_diff_notify] Patch aborted: Application of
> an update diff failed (-1006)
> Nov 04 18:21:15 [17749] ip-10-178-149-131 stonith-ng: info:
> apply_xml_diff: Digest mis-match: expected
> 01192e5118739b7c33c23f7645da3f45, calculated
> f8028c0c98526179ea5df0a2ba0d09de
> Nov 04 18:21:15 [17749] ip-10-178-149-131 stonith-ng:  warning:
> cib_process_diff:   Diff 1.15046.2 -> 1.15046.3 from local not
> applied to 1.15046.2: 

Re: [ClusterLabs] பதில்: Re: crm_mon memory leak

[Karthikeyan Ramasamy]

Hi Ken,
  Both CPU and this SNMP notification were interrelated, as the number of 
retries was infinite.  As a service was down, for every retry it was trying to 
send a notification as well and that stalled Pacemaker.

Thanks,
Karthik.
-Original Message-
From: Karthikeyan Ramasamy 
Sent: 02 நவம்பர் 2015 21:22
To: 'kgail...@redhat.com'; users@clusterlabs.org
Subject: RE: பதில்: Re: [ClusterLabs] crm_mon memory leak

Many thanks.  Will check and get back.

-Original Message-
From: Ken Gaillot [mailto:kgail...@redhat.com]
Sent: 02 நவம்பர் 2015 21:21
To: Karthikeyan Ramasamy; users@clusterlabs.org
Subject: Re: பதில்: Re: [ClusterLabs] crm_mon memory leak

On 11/02/2015 09:39 AM, Karthikeyan Ramasamy wrote:
> Yes, Ken.  There were multiple instances of the external script also running. 
>  What do you think could possibly be wrong with the script that triggers the 
> crm_mon process everytime?

It's the other way around, crm_mon spawns the script. So if the script doesn't 
exit properly, neither will crm_mon.

Easy test: put an "exit" at the top of your script. If the problem goes away, 
then it's in the script somewhere. Mostly you want to make sure the script 
completes within your monitoring interval.

> We are on RHEL 6.5.  I am not sure what's the plan for RHEL 6.7 and 7.1.  
> 
> Thanks,
> Karthik.
> 
> -Original Message-
> From: Ken Gaillot [mailto:kgail...@redhat.com]
> Sent: 02 நவம்பர் 2015 21:04
> To: Karthikeyan Ramasamy; users@clusterlabs.org
> Subject: Re: பதில்: Re: [ClusterLabs] crm_mon memory leak
> 
> On 10/31/2015 12:38 AM, Karthikeyan Ramasamy wrote:
>> Thanks, Mr.Gaillot.
>>
>> Yes, we trigger the snmp notification with an external script.  From your 
>> response, I understand that the issue wouldn't occur with 1.1.14, as it 
>> wouldn't require the crm_mon process.  Is this understanding correct?
> 
> Correct, crm_mon is not spawned with the new method. However if the problem 
> originates in the external script, then it still might not work properly, but 
> with the new method Pacemaker will kill it after a timeout.
> 
>> We have been given 1.1.10 as the supported version from RedHat.  If I raise 
>> a ticket to RedHat, would they be able to provide us a patch for 1.1.10?
> 
> If you're using RHEL 6.7, you should be able to simply "yum update" to get 
> 1.1.12. If you're using RHEL 7.1, you should be able to get 1.1.13.
> That would give you more bugfixes, which may or may not help your issue.
> If you're using an older version, there may not be updates any longer.
> 
> If you open a ticket, support can help you isolate where the problem is.
> 
> When you saw many crm_mon processes running, did you also see many copies of 
> the external script running?
> 
>> Many thanks for your response.
>>
>> Thanks,
>> Karthik.
>>
>>
>>  Ken Gaillot எழுதியது 
>>
>> On 10/30/2015 05:29 AM, Karthikeyan Ramasamy wrote:
>>> Dear Pacemaker support,
>>> We are using pacemaker1.1.10-14 to implement a service management 
>>> framework, with high availability on the road-map.  This pacemaker 
>>> version was available through redhat for our environments
>>>
>>>   We are running into an issue where pacemaker causes a node to crash.  The 
>>> last feature we integrated was SNMP notification.  While listing out the 
>>> processes we found that crm_mon processes occupying 58GB of available 64GB, 
>>> when the node crashed.  When we removed that feature, pacemaker was stable 
>>> again.
>>>
>>> Section 7.1 of the pacemaker document details that SNMP notification agent 
>>> triggers a crm_mon process at regular intervals.  On checking clusterlabs 
>>> for list of known issues, we found this crm_mon memory leak issue.  
>>> Although not related, we think that there is some problem with the crm_mon 
>>> process.
>>>
>>> http://clusterlabs.org/pipermail/users/2015-August/001084.html
>>>
>>> Can you please let us know if there are issues with SNMP notification in 
>>> Pacemaker or if there is anything that we could be wrong.  Also, any 
>>> workarounds for this issue if available, would be very helpful for us.  
>>> Please help.
>>>
>>> Thanks,
>>> Karthik.
>>
>> Are you using ClusterMon with Pacemaker's built-in SNMP capability, 
>> or an external script that generates the SNMP trap?
>>
>> If you're using the built-in capability, that has to be explicitly 
>> enabled when Pacemaker is compiled. Many distributions (including
>> RHEL) do not enable it. Run "crm_mon --help"; if it shows a "-S" 
>> option, you have it enabled, otherwise not.
>>
>> If you're using an external script to generate the SNMP trap, please 
>> post it (with any sensitive info taken out of course).
>>
>> The ClusterMon resource will generate a crm_mon at regular intervals, 
>> but it should exit quickly. It sounds like it's not exiting at all, 
>> which is why you see this problem.
>>
>> If you have a RHEL subscription, you can open a support ticket with 
>> Red Hat. Note that stonith must be enabled before Red Hat 

Re: [ClusterLabs] Pacemaker build error

[Ken Gaillot]

On 11/03/2015 11:10 PM, Jim Van Oosten wrote:
> 
> 
> I am getting a compile error when building Pacemaker on Linux version
> 2.6.32-431.el6.x86_64.
> 
> The build commands:
> 
> git clone git://github.com/ClusterLabs/pacemaker.git
> cd pacemaker
> ./autogen.sh && ./configure --prefix=/usr --sysconfdir=/etc
> make
> make install
> 
> The compile error:
> 
> Making install in services
> gmake[2]: Entering directory
> `/tmp/software/HA_linux/pacemaker/lib/services'
>   CC   libcrmservice_la-services.lo
> services.c: In function 'resources_action_create':
> services.c:153: error: 'svc_action_private_t' has no member named 'pending'
> services.c: In function 'services_action_create_generic':
> services.c:340: error: 'svc_action_private_t' has no member named 'pending'
> gmake[2]: *** [libcrmservice_la-services.lo] Error 1
> gmake[2]: Leaving directory `/tmp/software/HA_linux/pacemaker/lib/services'
> gmake[1]: *** [install-recursive] Error 1
> gmake[1]: Leaving directory `/tmp/software/HA_linux/pacemaker/lib'
> make: *** [install-recursive] Error 1
> 
> 
> The pending field that services.c is attenpting to set is conditioned on
> the SUPPORT_DBUS flag in services_private.h.
> 
> pacemaker/lib/services/services_private.h
> 
>
>  #if SUPPORT_DBUS  
>
>
>
> 
> 
> 
> 
> 
>  DBusPendingCall*   
>  pending;   
> 
> 
> 
> 
> 
>  unsigned timerid;  
> 
> 
> 
> 
> 
> 
>  #endif 
> 
> 
> 
> 
> 
> Am I building Pacemaker incorrectly or should I open an defect for this
> problem?
> 
> Jim VanOosten
> jimvo at  us.ibm.com

This report is enough; I'll do up a quick fix today. Thanks for catching it.

___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

[ClusterLabs] large cluster - failure recovery

[Radoslaw Garbacz]

Hi,

I have a cluster of 32 nodes, and after some tuning was able to have it
started and running,
but it does not recover from a node disconnect-connect failure.
It regains quorum, but CIB does not recover to a synchronized state and
"cibadmin -Q" times out.

Is there anything with corosync or pacemaker parameters I can do to make it
recover from such a situation
(everything works for smaller clusters).

In my case it is OK for a node to disconnect (all the major resources are
shutdown)
and later reconnect the cluster (the running monitoring agent will cleanup
and restart major resources if needed),
so I do not have STONITH configured.

Details:
OS: CentOS 6
Pacemaker: Pacemaker 1.1.9-1512.el6
Corosync: Corosync Cluster Engine, version '2.3.2'


Corosync configuration:
token: 1
#token_retransmits_before_loss_const: 10
consensus: 15000
join: 1000
send_join: 80
merge: 1000
downcheck: 2000
#rrp_problem_count_timeout: 5000
max_network_delay: 150 # for azure


Some logs:

[...]
Nov 04 17:50:18 [7985] ip-10-142-181-98 stonith-ng:   notice:
cib_process_diff: Diff 1.9254.1 -> 1.9255.1 from local not applied
to 1.9275.1: current "epoch" is greater than required
Nov 04 17:50:18 [7985] ip-10-142-181-98 stonith-ng:   notice:
update_cib_cache_cb:  [cib_diff_notify] Patch aborted: Application of
an update diff failed (-1006)
Nov 04 17:50:18 [7985] ip-10-142-181-98 stonith-ng:   notice:
cib_process_diff: Diff 1.9255.1 -> 1.9256.1 from local not applied
to 1.9275.1: current "epoch" is greater than required
Nov 04 17:50:18 [7985] ip-10-142-181-98 stonith-ng:   notice:
update_cib_cache_cb:  [cib_diff_notify] Patch aborted: Application of
an update diff failed (-1006)
Nov 04 17:50:18 [7985] ip-10-142-181-98 stonith-ng:   notice:
cib_process_diff: Diff 1.9256.1 -> 1.9257.1 from local not applied
to 1.9275.1: current "epoch" is greater than required
Nov 04 17:50:18 [7985] ip-10-142-181-98 stonith-ng:   notice:
update_cib_cache_cb:  [cib_diff_notify] Patch aborted: Application of
an update diff failed (-1006)
Nov 04 17:50:18 [7985] ip-10-142-181-98 stonith-ng:   notice:
cib_process_diff: Diff 1.9257.1 -> 1.9258.1 from local not applied
to 1.9275.1: current "epoch" is greater than required
Nov 04 17:50:18 [7985] ip-10-142-181-98 stonith-ng:   notice:
update_cib_cache_cb:  [cib_diff_notify] Patch aborted: Application of
an update diff failed (-1006)
[...]

[...]
Nov 04 17:43:24 [12176] ip-10-109-145-175crm_mon:error:
cib_native_perform_op_delegate: Couldn't perform cib_query
operation (timeout=120s): Operation already in progress (-114)
Nov 04 17:43:24 [12176] ip-10-109-145-175crm_mon:error:
get_cib_copy:   Couldnt retrieve the CIB
Nov 04 17:43:24 [12176] ip-10-109-145-175crm_mon:error:
cib_native_perform_op_delegate: Couldn't perform cib_query
operation (timeout=120s): Operation already in progress (-114)
Nov 04 17:43:24 [12176] ip-10-109-145-175crm_mon:error:
get_cib_copy:   Couldnt retrieve the CIB
Nov 04 17:47:40 [10599] ip-10-109-145-175 corosync notice  [QUORUM]
Members[32]: 3 27 11 29 23 21 24 9 17 12 32 13 2 10 16 15 6 28 19 1 22 26 5\
Nov 04 17:47:40 [10599] ip-10-109-145-175 corosync notice  [QUORUM]
Members[32]: 14 20 31 30 8 25 18 7 4
Nov 04 17:47:40 [10599] ip-10-109-145-175 corosync notice  [MAIN  ]
Completed service synchronization, ready to provide service.
Nov 04 18:06:55 [10599] ip-10-109-145-175 corosync notice  [QUORUM]
Members[32]: 3 27 11 29 23 21 24 9 17 12 32 13 2 10 16 15 6 28 19 1 22 26 5\
Nov 04 18:06:55 [10599] ip-10-109-145-175 corosync notice  [QUORUM]
Members[32]: 14 20 31 30 8 25 18 7 4
[...]

[...]
Nov 04 18:21:15 [17749] ip-10-178-149-131 stonith-ng:   notice:
update_cib_cache_cb:[cib_diff_notify] Patch aborted: Application of an
update diff failed (-1006)
Nov 04 18:21:15 [17749] ip-10-178-149-131 stonith-ng: info:
apply_xml_diff: Digest mis-match: expected
01192e5118739b7c33c23f7645da3f45, calculated
f8028c0c98526179ea5df0a2ba0d09de
Nov 04 18:21:15 [17749] ip-10-178-149-131 stonith-ng:  warning:
cib_process_diff:   Diff 1.15046.2 -> 1.15046.3 from local not applied
to 1.15046.2: Failed application of an update diff
Nov 04 18:21:15 [17749] ip-10-178-149-131 stonith-ng:   notice:
update_cib_cache_cb:[cib_diff_notify] Patch aborted: Application of an
update diff failed (-1006)
Nov 04 18:21:15 [17749] ip-10-178-149-131 stonith-ng:   notice:
cib_process_diff:   Diff 1.15046.2 -> 1.15046.3 from local not applied
to 1.15046.3: current "num_updates" is greater than required
[...]


ps. Sorry if should posted on corosync newsgroup, just the CIB
synchronization fails, so this group seemed to me the right place.

-- 
Best Regards,

Radoslaw Garbacz
___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: 

Re: [ClusterLabs] Pacemaker build error

[Ken Gaillot]

On 11/04/2015 09:31 AM, Ken Gaillot wrote:
> On 11/03/2015 11:10 PM, Jim Van Oosten wrote:
>>
>>
>> I am getting a compile error when building Pacemaker on Linux version
>> 2.6.32-431.el6.x86_64.
>>
>> The build commands:
>>
>> git clone git://github.com/ClusterLabs/pacemaker.git
>> cd pacemaker
>> ./autogen.sh && ./configure --prefix=/usr --sysconfdir=/etc
>> make
>> make install
>>
>> The compile error:
>>
>> Making install in services
>> gmake[2]: Entering directory
>> `/tmp/software/HA_linux/pacemaker/lib/services'
>>   CC   libcrmservice_la-services.lo
>> services.c: In function 'resources_action_create':
>> services.c:153: error: 'svc_action_private_t' has no member named 'pending'
>> services.c: In function 'services_action_create_generic':
>> services.c:340: error: 'svc_action_private_t' has no member named 'pending'
>> gmake[2]: *** [libcrmservice_la-services.lo] Error 1
>> gmake[2]: Leaving directory `/tmp/software/HA_linux/pacemaker/lib/services'
>> gmake[1]: *** [install-recursive] Error 1
>> gmake[1]: Leaving directory `/tmp/software/HA_linux/pacemaker/lib'
>> make: *** [install-recursive] Error 1
>>
>>
>> The pending field that services.c is attenpting to set is conditioned on
>> the SUPPORT_DBUS flag in services_private.h.
>>
>> pacemaker/lib/services/services_private.h
>>
>>
>>  #if SUPPORT_DBUS  
>>
>>
>>
>>
>>
>> 
>> 
>> 
>>  DBusPendingCall*   
>>  pending;   
>> 
>> 
>> 
>> 
>> 
>>  unsigned timerid;  
>> 
>> 
>> 
>>
>>
>> 
>>  #endif 
>> 
>>
>>
>>
>>
>> Am I building Pacemaker incorrectly or should I open an defect for this
>> problem?
>>
>> Jim VanOosten
>> jimvo at  us.ibm.com
> 
> This report is enough; I'll do up a quick fix today. Thanks for catching it.

The fix is upstream. Pull the latest commit and you should be good to go.


___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

[ClusterLabs] [Question] Question about mysql RA.

[renayama19661014]

Hi All,

I contributed a patch several times about mysql, too.

I did not mind it very much before, but mysql RA makes next move.

Step1) Constitute a cluster using mysql in Pacemaker.
Step2) The mysql process kill by signal SIGKILL.
Step3) Stop Pacemaker before monitor error occurs and stop mysql

As a result, mysql RA causes trouble in stop.
By this trouble, Pacemaker does not stop until escalation.


The cause is processing unlike pgsql RA.
When a process of pid does not exist, in the case of a stop not to go by way of 
monitor trouble, mysql RA produces ERR_GENERIC.
When a process of pid does not exist, pgsql becomes the success of the stop.

-
* mysql
(snip)
mysql_monitor() {
    local rc
    local status_loglevel="err"

    # Set loglevel to info during probe
    if ocf_is_probe; then
        status_loglevel="info"
    fi
 
    mysql_common_status $status_loglevel

    rc=$?

    # TODO: check max connections error

    # If status returned an error, return that immediately
    if [ $rc -ne $OCF_SUCCESS ]; then
        return $rc
    fi
(snip)
mysql_stop() {
    if ocf_is_ms; then
        # clear preference for becoming master
        $CRM_MASTER -D

        # Remove VIP capability
        set_reader_attr 0
    fi

    mysql_common_stop
}
(snip)

* mysql-common.sh
(snip)
mysql_common_status() {
    local loglevel=$1
    local pid=$2
    if [ -z "$pid" ]; then
        if [ ! -e $OCF_RESKEY_pid ]; then
            ocf_log $loglevel "MySQL is not running"
            return $OCF_NOT_RUNNING;
        fi

        pid=`cat $OCF_RESKEY_pid`;
    fi
    if [ -d /proc -a -d /proc/1 ]; then
        [ "u$pid" != "u" -a -d /proc/$pid ]
    else
        kill -s 0 $pid >/dev/null 2>&1
    fi

    if [ $? -eq 0 ]; then
        return $OCF_SUCCESS;
    else
        ocf_log $loglevel "MySQL not running: removing old PID file"
        rm -f $OCF_RESKEY_pid
        return $OCF_NOT_RUNNING;
    fi
}
(snip)
mysql_common_stop()
{
    local pid
    local rc

    if [ ! -f $OCF_RESKEY_pid ]; then
        ocf_log info "MySQL is not running"
        return $OCF_SUCCESS
    fi

    pid=`cat $OCF_RESKEY_pid 2> /dev/null `
    /bin/kill $pid > /dev/null
    rc=$?
    if [ $rc != 0 ]; then
        ocf_exit_reason "MySQL couldn't be stopped"
        return $OCF_ERR_GENERIC
    fi
(snip)
-

The mysql RA does such a code from old days.
 * http://hg.linux-ha.org/agents/file/67234f982ab7/heartbeat/mysql

Does mysql RA know the reason becoming this made?
Possibly is it a factor to be conscious of mysql cluster?

I think about a patch of this movement of mysql RA.
I want to know the detailed reason.

Best Regards,
Hideo Yamauchi.

___
Users mailing list: Users@clusterlabs.org
http://clusterlabs.org/mailman/listinfo/users

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org