Re: [Pacemaker] SLES 11 SP3 boothd behaviour

[Rainer Brestan]

Booth version 0.1.0 has no retry method for packets, one single packet loss and the election does not work anymore.

Also it has a stupid checking of ballot values against promised values, in which case it communicates but does not do things right.

Booth version 0.1.0 is very sensitive about start order and saved ballot values as ticket attributes. The fault might come from your start procedure and the state it remembers from previous runs.

And it communicates through two different ways, one is the UDP port you configure in booth.conf used for renew and the other is a fixed TCP port somewhere in 23000 used for catchup.

I might give you a hint if you attach the complete log from Aug 25 10:07:10  until Aug 25 10:08:10 for all three sites and attach the snipplet of booth.conf to see the order of lines with site and arbitrator.

Anyhow booth 0.1.0 is really unusable for productive environment, it has so many errors in it, you cant rely on it.

The booth version from GIT after 4th august 2014 behaves very well even in case of double fault (restart during network down) and builds under SLES and RHEL quite well now.

Rainer

 

Gesendet: Dienstag, 26. August 2014 um 14:58 Uhr
Von: "Sutherland, Rob" 
An: "pacemaker@oss.clusterlabs.org" 
Betreff: Re: [Pacemaker] SLES 11 SP3 boothd behaviour




All nodes in question NTP from the same time source (yes, we have run into synchronicity issues in the past).

 

Interestingly, increasing the lease from 60 seconds to 120 seconds did not affect the behaviour.

 

Rob

 



From: John Lauro [mailto:john.la...@covenanteyes.com]
Sent: Monday, August 25, 2014 6:17 PM
To: Sutherland, Rob
Subject: Re: [Pacemaker] SLES 11 SP3 boothd behaviour



 


You probably already checked this, but just in case...

No experience at all with geo-redundancy, but this sounds suspiciously like it could be a time sync problem.  Have you tried something like "ntpq -np" on all 3 nodes and verify the offsets are all low (ie: < +/- 10) and times are in sync?
(Assuming you are running ntpd, and the process didn't stop.)
 





From: "Rob Sutherland" 
To: pacemaker@oss.clusterlabs.org
Sent: Monday, August 25, 2014 3:43:34 PM
Subject: [Pacemaker] SLES 11 SP3 boothd behaviour


Hello all,



 



We’re in the process of implementing geo-redundancy on SLES 11 SP3 (version 0.1.0). We are seeing behavior in which site 2 in a geo-cluster decides that the ticket has expired long before actual expiry. Here’s an example time-line:



 



1 - All sites (site 1, site 2 and arbitrator) agree on ticket owner and expiry. i.e. site 2 has the ticket with a 60-second expiry:



Aug 25 10:07:10 linux-4i31 booth-arbitrator: [22526]: info: command: 'crm_ticket -t geo-ticket -S expires -v 1408975690' was executed



Aug 25 10:07:10 bb5Btas0 booth-site: [27782]: info: command: 'crm_ticket -t geo-ticket -S expires -v 1408975690' was executed



Aug 25 10:07:10 bb5Atas1 booth-site: [7826]: info: command: 'crm_ticket -t geo-ticket -S expires -v 1408975690' was executed



 



2 - After 48 seconds (80% into lease), all three nodes are still in agreement:



Site 2: 



Aug 25 10:07:58 bb5Btas0 booth-site: [27782]: info: command: 'crm_ticket -t geo-ticket -S owner -v 2' was executed  



Aug 25 10:07:58 bb5Btas0 booth-site: [27782]: info: command: 'crm_ticket -t geo-ticket -S expires -v 1408975738' was executed



 



The arbitrator: 



Aug 25 10:07:58 linux-4i31 crm_ticket[23836]:   notice: crm_log_args: Invoked: crm_ticket -t geo-ticket -S owner -v 2



Aug 25 10:07:58 linux-4i31 booth-arbitrator: [22526]: info: command: 'crm_ticket -t geo-ticket -S expires -v 1408975738' was executed



 



Site 1:



Aug 25 10:07:58 bb5Atas1 booth-site: [7826]: info: command: 'crm_ticket -t geo-ticket -S owner -v 2' was executed



Aug 25 10:07:58 bb5Atas1 booth-site: [7826]: info: command: 'crm_ticket -t geo-ticket -S expires -v 1408975738' was executed



 



3 - Site 2 decides that the ticket has expired (at the  expiry time set in step 1)



Aug 25 10:08:10 bb5Btas0 booth-site: [27782]: debug: lease expires ...



 



4 - At 10:08:58, both site 1 and the arbitrator expire the lease and pick a new master.



 



I presume that there was some missed communication between site 2 and the rest of the geo-cluster. There is nothing in the logs to help debug this, though. Any hints on debugging this?



 



BTW: we only ever see this on a site 2 – never a site 1. This is consistent across several labs. Is there a bias towards site 1?



 



Thanks in advance,



 



Rob



 



 



___
Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

Project Home: http://www.clusterlabs.org
Getting started:  http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org


 


___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.

Re: [Pacemaker] Interval-origin in monitor operations does not work

[Rainer Brestan]

When do we need negative durations ?

Is it because iso8601 must be able to handle negative durations ?

What is the expected reaction on setting a negative interval in the monitor operation ?

 

Just to know what i can expect from the test.

 

Rainer

Gesendet: Montag, 05. Mai 2014 um 06:37 Uhr
Von: "Andrew Beekhof" 
An: "The Pacemaker cluster resource manager" 
Betreff: Re: [Pacemaker] Interval-origin in monitor operations does not work


On 2 May 2014, at 4:55 pm, Andrew Beekhof  wrote:

>
> On 15 Apr 2014, at 4:12 am, Rainer Brestan  wrote:
>
>> Of course, I can.
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>>
>> Yes, the origin is in the future, but consider above monitor configuration.
>> The monitor operation shall run every hour at 34 minutes.
>> If i would specifiy a full date in the past then pengine has to run a number of while(rc<0) loops in unpack_operation.
>> One year after full date exactly 8760 and this for every call of unpack_operation.
>> Thats why i specified the first possible run time every day and then they are maximum of 23 while loop runs.
>>
>> If unpack_operation is called between 00:00 and 00:34 the described situation happens.
>> Origin is later than now.
>>
>> Applying this patch will help.
>
> It will, but as I suspected it will also cause:
>
> iso8601 -d '2014-01-01 00:00:30Z' -D P-1D -E '2013-12-31 00:00:30Z'
>
> to fail with:
>
> Date: 2014-01-01 00:00:30Z
> Duration: -01--01 00:00:00Z
> Duration ends at: 2014-01-00 00:00:30Z
>
> which isn't right :)
>
> I'm working on a true fix now...

These are the resulting patches in https://github.com/beekhof/pacemaker:

+ Andrew Beekhof (14 seconds ago) 44af669: Test: PE: Correctly handle origin offsets in the future (HEAD, master)
+ Andrew Beekhof (27 minutes ago) 3f20485: Fix: PE: Correctly handle origin offsets in the future
+ Andrew Beekhof (4 minutes ago) d39bad6: Test: iso8601: Improved logging of durations
+ Andrew Beekhof (29 minutes ago) afb6c16: Fix: iso8601: Different logic is needed when logging and calculating durations

>
>
>> diff --git a/lib/common/iso8601.c b/lib/common/iso8601.c
>> index 7dc2495..742de70 100644
>> --- a/lib/common/iso8601.c
>> +++ b/lib/common/iso8601.c
>> @@ -1137,7 +1137,7 @@ crm_time_add_days(crm_time_t * a_time, int extra)
>> ydays = crm_time_leapyear(a_time->years) ? 366 : 365;
>> }
>> - while (a_time->days <= 0) {
>> + while (a_time->days < 0) {
>> a_time->years--;
>> a_time->days += crm_time_leapyear(a_time->years) ? 366 : 365;
>> }
>>
>> Rainer
>> Gesendet: Mittwoch, 09. April 2014 um 08:57 Uhr
>> Von: "Andrew Beekhof" 
>> An: "The Pacemaker cluster resource manager" 
>> Betreff: Re: [Pacemaker] Interval-origin in monitor operations does not work
>>
>> On 1 Apr 2014, at 5:10 am, Rainer Brestan  wrote:
>>
>>> Using interval-origin in monitor operation definition does not work any more.
>>> Veryfied on Pacemaker 1.1.10, but we think it does not work since 1.1.8 until now.
>>>
>>> Pengine calculates start delay in function unpack_operation and calls there crm_time_subtract.
>>>
>>> The call to crm_time_subtract with
>>> origin=2014-03-31 19:20:00Z
>>> date_set->now=2014-03-31 17:31:04Z
>>> result in
>>> delay=-0001-12-31 01:48:56Z
>>> delay_s=31456136
>>> start_delay=31456136000
>>> which is almost a year in the future.
>>
>> To be fair, the origin was also in the future.
>> I don't think that was expected.
>>
>> Can you supply your cib so I can experiment?
>>
>>>
>>> The function crm_time_subtract calculates this by the crm_time_add_* functions.
>>>
>>> The buggy statement is in crm_time_add_days.
>>> If the calculated number of days is zero, it subtracts one year and add the number of days, in this case 365.
>>> But if a_time->days is zero, it must not do anything.
>>>
>>> The function crm_time_get_seconds, which is called by unpack_operation cannot handle negative years, so it ignores the year -1 but adds 365 days.
>>>
>>> There are two solutions.
>>> One is the add handling on negative years to crm_time_get_seconds.
>>> The other is to exchange line 1140 in iso8601.c
>>> while (a_time->days <= 0) {
>>> by
>>> while (a_time->days < 0) {
>>>
>>> Second solution is verified to bring the expected result, start-delay of little less than two hours

Re: [Pacemaker] Interval-origin in monitor operations does not work

[Rainer Brestan]

Of course, I can.


  
    
  
  
  
    
    
  
  
    
  

 

Yes, the origin is in the future, but consider above monitor configuration.

The monitor operation shall run every hour at 34 minutes.

If i would specifiy a full date in the past then pengine has to run a number of while(rc<0) loops in unpack_operation.

One year after full date exactly 8760 and this for every call of unpack_operation.

Thats why i specified the first possible run time every day and then they are maximum of 23 while loop runs.



 

If unpack_operation is called between 00:00 and 00:34 the described situation happens.

Origin is later than now.

 

Applying this patch will help.


diff --git a/lib/common/iso8601.c b/lib/common/iso8601.c
index 7dc2495..742de70 100644
--- a/lib/common/iso8601.c
+++ b/lib/common/iso8601.c
@@ -1137,7 +1137,7 @@ crm_time_add_days(crm_time_t * a_time, int extra)
 ydays = crm_time_leapyear(a_time->years) ? 366 : 365;
 }

-    while (a_time->days <= 0) {
+    while (a_time->days < 0) {
 a_time->years--;
 a_time->days += crm_time_leapyear(a_time->years) ? 366 : 365;
 }

 

Rainer



Gesendet: Mittwoch, 09. April 2014 um 08:57 Uhr
Von: "Andrew Beekhof" 
An: "The Pacemaker cluster resource manager" 
Betreff: Re: [Pacemaker] Interval-origin in monitor operations does not work


On 1 Apr 2014, at 5:10 am, Rainer Brestan  wrote:

> Using interval-origin in monitor operation definition does not work any more.
> Veryfied on Pacemaker 1.1.10, but we think it does not work since 1.1.8 until now.
>
> Pengine calculates start delay in function unpack_operation and calls there crm_time_subtract.
>
> The call to crm_time_subtract with
> origin=2014-03-31 19:20:00Z
> date_set->now=2014-03-31 17:31:04Z
> result in
> delay=-0001-12-31 01:48:56Z
> delay_s=31456136
> start_delay=31456136000
> which is almost a year in the future.

To be fair, the origin was also in the future.
I don't think that was expected.

Can you supply your cib so I can experiment?

>
> The function crm_time_subtract calculates this by the crm_time_add_* functions.
>
> The buggy statement is in crm_time_add_days.
> If the calculated number of days is zero, it subtracts one year and add the number of days, in this case 365.
> But if a_time->days is zero, it must not do anything.
>
> The function crm_time_get_seconds, which is called by unpack_operation cannot handle negative years, so it ignores the year -1 but adds 365 days.
>
> There are two solutions.
> One is the add handling on negative years to crm_time_get_seconds.
> The other is to exchange line 1140 in iso8601.c
> while (a_time->days <= 0) {
> by
> while (a_time->days < 0) {
>
> Second solution is verified to bring the expected result, start-delay of little less than two hours.
> Handling of negative years in crm_time_get_seconds might not be a proper solution as the return value of the function is unsigned long long and what to report if the complete calculation gives a negative number of seconds.
>
> Rainer
>
> ___
> Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org

___
Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org




___
Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

[Pacemaker] Interval-origin in monitor operations does not work

[Rainer Brestan]

Using interval-origin in monitor operation definition does not work any more.

Veryfied on Pacemaker 1.1.10, but we think it does not work since 1.1.8 until now.

 

Pengine calculates start delay in function unpack_operation and calls there crm_time_subtract.

 

The call to crm_time_subtract with

origin=2014-03-31 19:20:00Z

date_set->now=2014-03-31 17:31:04Z

result in

delay=-0001-12-31 01:48:56Z

delay_s=31456136

start_delay=31456136000

which is almost a year in the future.

 

The function crm_time_subtract calculates this by the crm_time_add_* functions.

 

The buggy statement is in crm_time_add_days.

If the calculated number of days is zero, it subtracts one year and add the number of days, in this case 365.

But if a_time->days is zero, it must not do anything.

 

The function crm_time_get_seconds, which is called by unpack_operation cannot handle negative years, so it ignores the year -1 but adds 365 days.

 

There are two solutions.

One is the add handling on negative years to crm_time_get_seconds.

The other is to exchange line 1140 in iso8601.c

while (a_time->days <= 0) {


by

while (a_time->days < 0) {

 

Second solution is verified to bring the expected result, start-delay of little less than two hours.

Handling of negative years in crm_time_get_seconds might not be a proper solution as the return value of the function is unsigned long long and what to report if the complete calculation gives a negative number of seconds.

 

Rainer

 


___
Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [Pacemaker] Kamailio managed by Pacemaker

[Rainer Brestan]

Correct.

If we are using syntax which is only valid in bash extensions, we should point to bash and not to sh.

Rainer

 

Gesendet: Montag, 03. Februar 2014 um 18:16 Uhr
Von: "Sherwood McGowan" 
An: pacemaker@oss.clusterlabs.org
Betreff: Re: [Pacemaker] Kamailio managed by Pacemaker

I found that on Ubuntu servers, the /bin/sh designation fails, but if
you use /bin/bash, it works fine

On 1/30/2014 3:25 AM, Rainer Brestan wrote:
> The resource agent was developed by Stefan Wenk an me.
> Plan is to include it into GIT Repo resource-agents by pull request
> after some short testing period outside or own labs.
> Rainer
> *Gesendet:* Donnerstag, 30. Januar 2014 um 00:25 Uhr
> *Von:* "Vladimir Broz" 
> *An:* "The Pacemaker cluster resource manager"
> 
> *Betreff:* Re: [Pacemaker] Kamailio managed by Pacemaker
> Hi,
>
> try kamailio mailing list. The OCF script for kamailio was posted and
> discussed in this thread:
>
> [sr-dev] kamailio OCF resource agent for pacemaker - Jan 7 10:12:58 CET
> 2014
> http://lists.sip-router.org/pipermail/sr-dev/2014-January/022639.html
>
> hope that helps,
> -Vlada
> On 29.1.2014 23:35, Sherwood McGowan wrote:
>
> Greetings all,
>
> I am curious to know if any of you have ever set up a cluster in
> which Pacemaker was managing Kamailio? I'd be very interested to
> discuss it with you.
>
> I'm currently working on a project that involves 2 nodes, with MySQL
> databases synchronized via DRBD, and a virtual ip...pretty standard
> stuff... Now, the problem is, I also need to manage Kamailio as
> well. Of course, I tried the LSB resource and the upstart resource,
> and have not gotten anywhere.
>
> I'd be happy to post configuration details and log snippets if
> someone would like.
>
>
>
> ___
> Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org
>
>
> ___ Pacemaker mailing list:
> Pacemaker@oss.clusterlabs.org
> http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home:
> http://www.clusterlabs.org Getting started:
> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs:
> http://bugs.clusterlabs.org
>
>
> ___
> Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org
>

--
Sherwood McGowan
sherwood.mcgo...@gmail.com

___
Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org




___
Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [Pacemaker] Kamailio managed by Pacemaker

[Rainer Brestan]

The resource agent was developed by Stefan Wenk an me.

Plan is to include it into GIT Repo resource-agents by pull request after some short testing period outside or own labs.

Rainer

Gesendet: Donnerstag, 30. Januar 2014 um 00:25 Uhr
Von: "Vladimir Broz" 
An: "The Pacemaker cluster resource manager" 
Betreff: Re: [Pacemaker] Kamailio managed by Pacemaker


Hi,

try kamailio mailing list. The OCF script for kamailio was posted and discussed in this thread:

[sr-dev] kamailio OCF resource agent for pacemaker -  Jan 7 10:12:58 CET 2014
http://lists.sip-router.org/pipermail/sr-dev/2014-January/022639.html

hope that helps,
-Vlada
 
On 29.1.2014 23:35, Sherwood McGowan wrote:

Greetings all,

I am curious to know if any of you have ever set up a cluster in which Pacemaker was managing Kamailio? I'd be very interested to discuss it with you.

I'm currently working on a project that involves 2 nodes, with MySQL databases synchronized via DRBD, and a virtual ip...pretty standard stuff... Now, the problem is, I also need to manage Kamailio as well. Of course, I tried the LSB resource and the upstart resource, and have not gotten anywhere.

I'd be happy to post configuration details and log snippets if someone would like.



___
Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org





___
Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [Pacemaker] Announce: SNMP agent for pacemaker

[Rainer Brestan]

I have got the SNMP subagent from pacemaker-mgmt 2.1.2 working with corosync 2.3 and pacemaker 1.1.10.

Some modification are implemented because of wrong attach method to CIB and one nasty bug, where hbagent crashes, when it does not find an operation on parsing a change.

As for all versions of hbagent with corosync it provides only the resource table of LINUX-HA MIB, but not the node tabel

Also i have created a very simple resource agent for hbagent to manage it as cluster resource (monitor method looks for SNMP result, so it can detect hanging hbagent, but still needs improvement).

When attaching a cluster IP address to this resource, cluster resources can be monitored through this address as long as resource is running anywhere.

The plan is (when i find time to do) to integrate the SNMP table part (which works quite well) of this agent into crm_mon with an option, to let crm_mon (when running as daemon) attaching to SNMP through AgentX.

Rainer

 

Gesendet: Mittwoch, 22. Januar 2014 um 11:39 Uhr
Von: "Michael Schwartzkopff" 
An: "Andrey Groshev" , pacemaker@oss.clusterlabs.org
Betreff: Re: [Pacemaker] Announce: SNMP agent for pacemaker

Am Mittwoch, 22. Januar 2014, 14:31:57 schrieben Sie:
> 22.01.2014, 12:43, "Michael Schwartzkopff" :
> > Hi,
> >
> > I am working on a SNMP agent for pacemaker. it is written in perl. At the
> > moment it is in an alpha stadium.
>
> On each node local call crm_mon/crm_resource ?

Partly yes. But in most cases I read the CIB and parse the config and status
part. I included memshare caching to minimize impact.

Mit freundlichen Grüßen,

Michael Schwartzkopff

--
[*] sys4 AG

http://sys4.de, +49 (89) 30 90 46 64, +49 (162) 165 0044
Franziskanerstraße 15, 81669 München

Sitz der Gesellschaft: München, Amtsgericht München: HRB 199263
Vorstand: Patrick Ben Koetter, Marc Schiffbauer
Aufsichtsratsvorsitzender: Florian Kirstein___
Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org




___
Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [Pacemaker] crmsh: New syntax for location constraints, suggestions / comments

[Rainer Brestan]

Hi Lars,

maybe a little off topic.

 

What i really miss in crmsh is the possibility to specify resource parameters which are different on different nodes, so the parameter is node dependant.

In XML syntax this is existing, Andrew gave me the hint as answer to an discussion how to deal with different node parameters.

But, if specified in XML syntax, crmsh cannot interpret it any more and print the resource in XML syntax.

Therefore, this is not usable with crmsh, as there is no single syntax display with "crm configure show".

 

Rainer

Gesendet: Freitag, 13. Dezember 2013 um 23:20 Uhr
Von: "Lars Marowsky-Bree" 
An: "The Pacemaker cluster resource manager" 
Betreff: Re: [Pacemaker] crmsh: New syntax for location constraints, suggestions / comments

On 2013-12-14T01:11:17, Vladislav Bogdanov  wrote:

> > The idea was to offer an additional construct that provides both
> > properties, since *most of the time*, that's what users want. In the
> > interest of clarity and brevity in the configuration, this would be
> > quite useful.
> group?

groups can't be used with resource sets, template resources, resources
can only be in one group, they don't support roles, mix badly between
primitive/clone/m/s resources, they aggregate resources into a container
object, etc.

It's not the same as a "proper" constraint, even if indeed groups are
also a form of shorthand.


Regards,
Lars

--
Architect Storage/HA
SUSE LINUX Products GmbH, GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 21284 (AG Nürnberg)
"Experience is the name everyone gives to their mistakes." -- Oscar Wilde


___
Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org




___
Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [Pacemaker] crmsh: New syntax for location constraints, suggestions / comments

[Rainer Brestan]

Please do not merge colocation and order together in a way that only none or both is present.

 

Example 1: Resource A communicates with resource B over network but A must run before B.

In this case only order is needed without colocation.

 

Example 2: Resource A and B share a local directory structure.

In this case they must run on the same node with colocation, but order is not important.

 


chain A B

does neither fit example 1 nor 2.

 

It is OK to have chain additional, but dont remove order and/or colocation.

Or chain is extended to allow specify only order or only colocation.

 

Rainer


Gesendet: Freitag, 13. Dezember 2013 um 11:57 Uhr
Von: "Lars Marowsky-Bree" 
An: "The Pacemaker cluster resource manager" 
Betreff: Re: [Pacemaker] crmsh: New syntax for location constraints, suggestions / comments

On 2013-12-13T11:46:05, Kristoffer Grönlund  wrote:

> This worries me as well, however the current syntax for constraints is
> confusing and error-prone.

Right. At least the { } would make it clear to users that it's now a
resource set and not merely more than 2 in the same sequence.

> It would be great to be able to do something
> to make this easier to use, but exactly what it would be is hard to
> say. Making a change that would silently invert the functionality of
> existing configurations is, I agree, not a good idea. However, maybe it
> would be acceptable if a "version: 2" header is required in the
> document to enable the new syntax?

Yeah. It's one of those "I wish we had done it differently" moments, but
I guess we're stuck here.

But another thing we discussed is hiding ids for dependencies by
default, since except for resource objects they really don't matter in
99% of the cases. That would condense the configuration significantly.

> Yet another option is to come up with some entirely new construct to
> supplement colocation and order which does what I think most people
> intuitively expects by default, which is enforces both colocation and
> ordering, so that 'foo depends on bar' means foo will start after bar,
> and only where bar is running.

chain a b c

(specifying the id or a score would be optional; score would default to
mandatory.)

I'm all in favor. ;-) I'd love it if this had backend support (so pcs
could use it too), but if we can't get it, we may just merge colocation
and order internally to crmsh.


Regards,
Lars

--
Architect Storage/HA
SUSE LINUX Products GmbH, GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 21284 (AG Nürnberg)
"Experience is the name everyone gives to their mistakes." -- Oscar Wilde


___
Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org




___
Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [Pacemaker] Announce: Pacemaker 1.1.10 now available

[Rainer Brestan]

I can also agree patch is working.

 

To be sure, that it had to do with notify, i have created a clone resource with notify=true and it happened to same way, after notify monitor was not called again.

 

With patch applied it works also for clone resources.

And from my output of the modified resource agents i can see on the timestamps of applied calls that there is no interruption of monitor operation.


Thu Aug 1 10:34:53 CEST 2013 resABC: operation monitor, type , operation
Thu Aug 1 10:34:59 CEST 2013 resABC: operation notify, type pre, operation start
Thu Aug 1 10:34:59 CEST 2013 resABC: operation notify, type post, operation start
Thu Aug 1 10:35:13 CEST 2013 resABC: operation monitor, type , operation

Monitor interval is set to 20 seconds and it is called at this intervals even if notify is in between.


 

Some hint for the check of sufficency:

On the original 1.1.10 version (without patch) i have tried some resource configuration change on clone resource with notify=true, which result in a "reload" call of the resource agent.

After logging reload, monitor starts again on both nodes.



Thu Aug 1 09:28:31 CEST 2013 resX: operation monitor, type , operation
Thu Aug 1 09:28:48 CEST 2013 resX: operation notify, type pre, operation start
Thu Aug 1 09:28:48 CEST 2013 resX: operation notify, type post, operation start
Thu Aug 1 09:38:47 CEST 2013 resX: operation reload, type , operation
Thu Aug 1 09:38:47 CEST 2013 resX: operation monitor, type , operation

 


Will there be a new tag (like 1.1.10-2) for version 1.1.10 with applied patch ?

 

Rainer


Gesendet: Donnerstag, 01. August 2013 um 05:56 Uhr
Von: "Takatoshi MATSUO" 
An: "The Pacemaker cluster resource manager" 
Betreff: Re: [Pacemaker] Announce: Pacemaker 1.1.10 now available

Hi Andrew

This patch works fine.

2013/8/1 Andrew Beekhof :
>
> On 01/08/2013, at 10:18 AM, Takatoshi MATSUO  wrote:
>
>> Hi Andrew
>>
>> I'm about to collect logs of crm_report,
>> but Rainer already provides it.
>>
>> Could you see his reports ?
>
> I had just written:
>
> "I can but they're insufficiently helpful."
>
> when a thought struck me
>
> Can you try the following patch?
> It would explain why I couldn't reproduce it locally earlier today.
>
> diff --git a/crmd/lrm.c b/crmd/lrm.c
> index d6b0dd0..4bce39a 100644
> --- a/crmd/lrm.c
> +++ b/crmd/lrm.c
> @@ -1744,7 +1744,9 @@ do_lrm_rsc_op(lrm_state_t * lrm_state, lrmd_rsc_info_t * rsc, const char *operat
> CRM_CHECK(op != NULL, return);
>
> /* stop any previous monitor operations before changing the resource state */
> - if (op->interval == 0 && strcmp(operation, CRMD_ACTION_STATUS) != 0) {
> + if (op->interval == 0
> + && strcmp(operation, CRMD_ACTION_STATUS) != 0
> + && strcmp(operation, CRMD_ACTION_NOTIFY) != 0) {
> guint removed = 0;
> struct stop_recurring_action_s data;
>
>
>
>>
>> Thanks,
>> Takatoshi MATSUO
>>
>>
>> 2013/8/1 Rainer Brestan :
>>> Base situation for the logs:
>>> Pacemaker stop on int2node1 and int2node2
>>> Master/slave resource msABC already configured.
>>> Included in the crm_report is also per node a file "a", this is the one,
>>> which the modified Stateful RA writes to log each action performed.
>>>
>>> 1.) 19:22:25 start Pacemaker on int2node1
>>> https://www.dropbox.com/s/ftbdl71ol2iyi42/step1.log.tar.bz2
>>> monitor on master is called
>>>
>>> 2.) 19:32:14 start Pacemaker on int2node2
>>> https://www.dropbox.com/s/s3jnxqvod9mlyz1/step2.log.tar.bz2
>>> monitor on master is not called any more
>>>
>>> 3.) 19:37:14 stop Pacemaker on int2node2
>>> https://www.dropbox.com/s/w75myab6fxh7mak/step3.log.tar.bz2
>>> monitor on master is still not called any more
>>>
>>> 4.) 19:42:14 start Pacemaker on in2node2
>>> https://www.dropbox.com/s/p00wl9kx4vwhilh/step4.log.tar.bz2
>>> monitor on master is called normally
>>>
>>> Hope this gives a clearer picture which component has forgotten the monitor
>>> action.
>>>
>>> Rainer
>>> Gesendet: Mittwoch, 31. Juli 2013 um 14:19 Uhr
>>>
>>> Von: "Andrew Beekhof" 
>>> An: "The Pacemaker cluster resource manager" 
>>> Betreff: Re: [Pacemaker] Announce: Pacemaker 1.1.10 now available
>>>
>>> On 31/07/2013, at 5:17 PM, Rainer Brestan  wrote:
>>>
>>>> Modified the RA to log each action call performed and from this log there
>>>> is no call of monitor action.
>>>>
>>>> From the logs i do not think it is the policy en

Re: [Pacemaker] Announce: Pacemaker 1.1.10 now available

[Rainer Brestan]

Base situation for the logs:

Pacemaker stop on int2node1 and int2node2

Master/slave resource msABC already configured.

Included in the crm_report is also per node a file "a", this is the one, which the modified Stateful RA writes to log each action performed.

 

1.) 19:22:25 start Pacemaker on int2node1

https://www.dropbox.com/s/ftbdl71ol2iyi42/step1.log.tar.bz2

monitor on master is called

 

2.) 19:32:14 start Pacemaker on int2node2

https://www.dropbox.com/s/s3jnxqvod9mlyz1/step2.log.tar.bz2

monitor on master is not called any more

 

3.) 19:37:14 stop Pacemaker on int2node2

https://www.dropbox.com/s/w75myab6fxh7mak/step3.log.tar.bz2

monitor on master is still not called any more

 

4.) 19:42:14 start Pacemaker on in2node2

https://www.dropbox.com/s/p00wl9kx4vwhilh/step4.log.tar.bz2

monitor on master is called normally

 

Hope this gives a clearer picture which component has forgotten the monitor action.


 

Rainer


Gesendet: Mittwoch, 31. Juli 2013 um 14:19 Uhr
Von: "Andrew Beekhof" 
An: "The Pacemaker cluster resource manager" 
Betreff: Re: [Pacemaker] Announce: Pacemaker 1.1.10 now available


On 31/07/2013, at 5:17 PM, Rainer Brestan  wrote:

> Modified the RA to log each action call performed and from this log there is no call of monitor action.
>
> From the logs i do not think it is the policy engine, it might be the LRM part of crmd (the is the only relevant change be seen after git diff between 1.1.10-rc7 and 1.1.10).

Ok. Can you still send me a crm_report though?
Even if the PE isn't at fault, it shows me what the cib looked like at the time which can be surprisingly helpful.
And it would have all the logs...

>
> Explanation of the below log:
> primitive resABC ocf:heartbeat:Stateful \
> op start interval="0s" timeout="60s" on-fail="restart" \
> op monitor interval="30s" timeout="60s" on-fail="restart" \
> op promote interval="0s" timeout="60s" on-fail="restart" \
> op demote interval="0" timeout="60s" on-fail="restart" \
> op stop interval="0" timeout="60s" on-fail="restart" \
> op monitor interval="20" role="Master" timeout="60"
> ms msABC resABC \
> meta master-max="1" master-node-max="1" clone-max="2" clone-node-max="1" notify="true"
> crm_mon at begin of log:
> Last updated: Wed Jul 31 08:30:57 2013
> Last change: Tue Jul 30 13:01:36 2013 via crmd on int2node1
> Stack: corosync
> Current DC: int2node1 (1743917066) - partition with quorum
> Version: 1.1.10-1.el6-368c726
> 2 Nodes configured
> 5 Resources configured
> Online: [ int2node1 int2node2 ]
> Master/Slave Set: msABC [resABC]
> Masters: [ int2node1 ]
> Slaves: [ int2node2 ]
> crm_mon at end of log:
> Last updated: Wed Jul 31 08:55:29 2013
> Last change: Tue Jul 30 13:01:36 2013 via crmd on int2node1
> Stack: corosync
> Current DC: int2node1 (1743917066) - partition with quorum
> Version: 1.1.10-1.el6-368c726
> 2 Nodes configured
> 5 Resources configured
> Online: [ int2node1 ]
> OFFLINE: [ int2node2 ]
> Master/Slave Set: msABC [resABC]
> Masters: [ int2node1 ]
>
> int2node1 is running, int2node2 is started
> 2013-07-31T08:30:52.631+02:00 int2node1 pengine[16443] notice: notice: LogActions: Start resABC:1 (int2node2)
> 2013-07-31T08:30:52.638+02:00 int2node1 crmd[16444] notice: notice: te_rsc_command: Initiating action 9: monitor resABC:1_monitor_0 on int2node2
> 2013-07-31T08:30:52.638+02:00 int2node1 crmd[16444] notice: notice: te_rsc_command: Initiating action 54: notify resABC_pre_notify_start_0 on int2node1 (local)
> 2013-07-31T08:30:52.681+02:00 int2node1 crmd[16444] notice: notice: process_lrm_event: LRM operation resABC_notify_0 (call=64, rc=0, cib-update=0, confirmed=true) ok
> 2013-07-31T08:30:52.780+02:00 int2node1 crmd[16444] notice: notice: te_rsc_command: Initiating action 25: start resABC:1_start_0 on int2node2
> 2013-07-31T08:30:52.940+02:00 int2node1 crmd[16444] notice: notice: te_rsc_command: Initiating action 55: notify resABC_post_notify_start_0 on int2node1 (local)
> 2013-07-31T08:30:52.943+02:00 int2node1 crmd[16444] notice: notice: te_rsc_command: Initiating action 56: notify resABC:1_post_notify_start_0 on int2node2
> 2013-07-31T08:30:52.982+02:00 int2node1 crmd[16444] notice: notice: process_lrm_event: LRM operation resABC_notify_0 (call=67, rc=0, cib-update=0, confirmed=true) ok
> 2013-07-31T08:30:52.992+02:00 int2node1 crmd[16444] notice: notice: te_rsc_command: Initiating action 24: monitor resABC_monitor_2 on int2node1 (local)
> 2013-07-31T08:30:52.996+02:00 int2node1 crmd[16444] notice: notice: te_rsc_command: Initiating action 26: monitor resABC:1_monitor_300

Re: [Pacemaker] Announce: Pacemaker 1.1.10 now available

[Rainer Brestan]

stop resABC_stop_0 on int2node2
2013-07-31T08:37:51.654+02:00 int2node1 crmd[16444] notice:   notice: te_rsc_command: Initiating action 57: notify resABC_post_notify_stop_0 on int2node1 (local)
2013-07-31T08:37:51.699+02:00 int2node1 crmd[16444] notice:   notice: process_lrm_event: LRM operation resABC_notify_0 (call=78, rc=0, cib-update=0, confirmed=true) ok
2013-07-31T08:37:51.699+02:00 int2node1 crmd[16444] notice:   notice: run_graph: Transition 86 (Complete=13, Pending=0, Fired=0, Skipped=2, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-125.bz2): Stopped
2013-07-31T08:37:51.705+02:00 int2node1 pengine[16443] notice:   notice: unpack_config: On loss of CCM Quorum: Ignore
2013-07-31T08:37:51.705+02:00 int2node1 pengine[16443] notice:   notice: stage6: Scheduling Node int2node2 for shutdown
2013-07-31T08:37:51.706+02:00 int2node1 pengine[16443] notice:   notice: process_pe_message: Calculated Transition 87: /var/lib/pacemaker/pengine/pe-input-126.bz2
2013-07-31T08:37:51.707+02:00 int2node1 crmd[16444] notice:   notice: run_graph: Transition 87 (Complete=1, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-126.bz2): Complete
2013-07-31T08:37:51.707+02:00 int2node1 crmd[16444] notice:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
2013-07-31T08:37:51.720+02:00 int2node1 crmd[16444] notice:   notice: peer_update_callback: do_shutdown of int2node2 (op 45) is complete

 


Output from RA on int2node1:


Wed Jul 31 08:30:52 CEST 2013 resABC: operation notify, type pre, operation start
Wed Jul 31 08:30:52 CEST 2013 resABC: operation notify, type post, operation start
Wed Jul 31 08:30:53 CEST 2013 resABC: operation monitor, type , operation
Wed Jul 31 08:31:13 CEST 2013 resABC: operation monitor, type , operation
Wed Jul 31 08:31:33 CEST 2013 resABC: operation monitor, type , operation
Wed Jul 31 08:31:53 CEST 2013 resABC: operation monitor, type , operation
Wed Jul 31 08:32:13 CEST 2013 resABC: operation monitor, type , operation
Wed Jul 31 08:32:33 CEST 2013 resABC: operation monitor, type , operation
Wed Jul 31 08:32:53 CEST 2013 resABC: operation monitor, type , operation
Wed Jul 31 08:33:13 CEST 2013 resABC: operation monitor, type , operation
Wed Jul 31 08:33:33 CEST 2013 resABC: operation monitor, type , operation
Wed Jul 31 08:33:53 CEST 2013 resABC: operation monitor, type , operation
Wed Jul 31 08:34:13 CEST 2013 resABC: operation monitor, type , operation
Wed Jul 31 08:34:33 CEST 2013 resABC: operation monitor, type , operation
Wed Jul 31 08:34:53 CEST 2013 resABC: operation monitor, type , operation
Wed Jul 31 08:35:13 CEST 2013 resABC: operation monitor, type , operation
Wed Jul 31 08:35:33 CEST 2013 resABC: operation monitor, type , operation
Wed Jul 31 08:35:53 CEST 2013 resABC: operation monitor, type , operation
Wed Jul 31 08:36:13 CEST 2013 resABC: operation monitor, type , operation
Wed Jul 31 08:36:33 CEST 2013 resABC: operation monitor, type , operation
Wed Jul 31 08:36:53 CEST 2013 resABC: operation monitor, type , operation
Wed Jul 31 08:37:13 CEST 2013 resABC: operation monitor, type , operation
Wed Jul 31 08:37:33 CEST 2013 resABC: operation monitor, type , operation
Wed Jul 31 08:37:51 CEST 2013 resABC: operation notify, type pre, operation stop
Wed Jul 31 08:37:51 CEST 2013 resABC: operation notify, type post, operation stop

 

After 08:37:51 no log output from Pacemaker for resABC, nor any output from RA on int2node1.

 



Gesendet: Mittwoch, 31. Juli 2013 um 02:10 Uhr
Von: "Andrew Beekhof" 
An: "The Pacemaker cluster resource manager" 
Betreff: Re: [Pacemaker] Announce: Pacemaker 1.1.10 now available


On 30/07/2013, at 9:13 PM, Rainer Brestan  wrote:

> I can agree, Master monitor operation is broken in 1.1.10 release.
> When the slave monitor action is started, the master monitor action is not called any more.

Based on?

>
> I have created a setup with Stateful resource with two nodes.
> Then the Pacemaker installation is changed to different versions without changing the configuration part of the CIB.
>
> Result:
> 1.1.10-rc5, 1.1.10-rc6 and 1.1.10-rc7 does not have this error
> 1.1.10-1 release has the error
>
> Installation order (just that anybody know how it was done):
> 1.1.10-1 -> error
> 1.1.10-rc5 -> no error
> 1.1.10-rc6 -> no error
> 1.1.10-rc7 -> no error
> 1.1.10-1 -> error
>
> Rainer
> Gesendet: Freitag, 26. Juli 2013 um 09:32 Uhr
> Von: "Takatoshi MATSUO" 
> An: "The Pacemaker cluster resource manager" 
> Betreff: Re: [Pacemaker] Announce: Pacemaker 1.1.10 now available
> Hi
>
> I used Stateful RA and caught a same issue.
>
> 1. before starting slave
>
> # crm_simulate -VVV -S -x /var/lib/pacemaker/pengine/pe-input-1543.bz2
> | grep "Resource action"
> * Resource ac

Re: [Pacemaker] Announce: Pacemaker 1.1.10 now available

[Rainer Brestan]

I can agree, Master monitor operation is broken in 1.1.10 release.

When the slave monitor action is started, the master monitor action is not called any more.

 

I have created a setup with Stateful resource with two nodes.

Then the Pacemaker installation is changed to different versions without changing the configuration part of the CIB.


 

Result:

1.1.10-rc5, 1.1.10-rc6 and 1.1.10-rc7 does not have this error

1.1.10-1 release has the error

 

Installation order (just that anybody know how it was done):

1.1.10-1 -> error

1.1.10-rc5 -> no error

1.1.10-rc6 -> no error

1.1.10-rc7 -> no error

1.1.10-1 -> error

 

Rainer


Gesendet: Freitag, 26. Juli 2013 um 09:32 Uhr
Von: "Takatoshi MATSUO" 
An: "The Pacemaker cluster resource manager" 
Betreff: Re: [Pacemaker] Announce: Pacemaker 1.1.10 now available

Hi

I used Stateful RA and caught a same issue.

1. before starting slave

# crm_simulate -VVV -S -x /var/lib/pacemaker/pengine/pe-input-1543.bz2
| grep "Resource action"
* Resource action: stateful monitor=2000 on 16-sl6

2. starting slave
# crm_simulate -VVV -S -x /var/lib/pacemaker/pengine/pe-input-1544.bz2
| grep "Resource action"
* Resource action: stateful monitor on 17-sl6
* Resource action: stateful notify on 16-sl6
* Resource action: stateful start on 17-sl6
* Resource action: stateful notify on 16-sl6
* Resource action: stateful notify on 17-sl6
* Resource action: stateful monitor=3000 on 17-sl6

3. after
# crm_simulate -VVV -S -x /var/lib/pacemaker/pengine/pe-input-1545.bz2
| grep "Resource action"
* Resource action: stateful monitor=3000 on 17-sl6

Monitor=2000 is deleted.
Is this correct ?


My setting

property \
no-quorum-policy="ignore" \
stonith-enabled="false"

rsc_defaults \
resource-stickiness="INFINITY" \
migration-threshold="1"

ms msStateful stateful \
meta \
master-max="1" \
master-node-max="1" \
clone-max="2" \
clone-node-max="1" \
notify="true"

primitive stateful ocf:heartbeat:Stateful \
op start timeout="60s" interval="0s" on-fail="restart" \
op monitor timeout="60s" interval="3s" on-fail="restart" \
op monitor timeout="60s" interval="2s" on-fail="restart" role="Master" \
op promote timeout="60s" interval="0s" on-fail="restart" \
op demote timeout="60s" interval="0s" on-fail="stop" \
op stop timeout="60s" interval="0s" on-fail="block"


Regards,
Takatoshi MATSUO

2013/7/26 Takatoshi MATSUO :
> Hi
>
> My report is late for 1.1.10 :(
>
> I am using pacemaker 1.1.10-0.1.ab2e209.git.
> It seems that master's monitor is stopped when slave is started.
>
> Does someone encounter same problem ?
> I attach a log and settings.
>
>
> Thanks,
> Takatoshi MATSUO
>
> 2013/7/26 Digimer :
>> Congrats!! I know this was a long time in the making.
>>
>> digimer
>>
>>
>> On 25/07/13 20:43, Andrew Beekhof wrote:
>>>
>>> Announcing the release of Pacemaker 1.1.10
>>>
>>> https://github.com/ClusterLabs/pacemaker/releases/Pacemaker-1.1.10
>>>
>>> There were three changes of note since rc7:
>>>
>>> + Bug cl#5161 - crmd: Prevent memory leak in operation cache
>>> + cib: Correctly read back archived configurations if the primary is
>>> corrupted
>>> + cman: Do not pretend we know the state of nodes we've never seen
>>>
>>> Along with assorted bug fixes, the major topics for this release were:
>>>
>>> - stonithd fixes
>>> - fixing memory leaks, often caused by incorrect use of glib reference
>>> counting
>>> - supportability improvements (code cleanup and deduplication,
>>> standardized error codes)
>>>
>>> Release candidates for the next Pacemaker release (1.1.11) can be
>>> expected some time around Novemeber.
>>>
>>> A big thankyou to everyone that spent time testing the release
>>> candidates and/or contributed patches. However now that Pacemaker is
>>> perfect, anyone reporting bugs will be shot :-)
>>>
>>> To build `rpm` packages:
>>>
>>> 1. Clone the current sources:
>>>
>>> # git clone --depth 0 git://github.com/ClusterLabs/pacemaker.git
>>> # cd pacemaker
>>>
>>> 1. Install dependancies (if you haven't already)
>>>
>>> [Fedora] # sudo yum install -y yum-utils
>>> [ALL] # make rpm-dep
>>>
>>> 1. Build Pacemaker
>>>
>>> # make release
>>>
>>> 1. Copy and deploy as needed
>>>
>>> ## Details - 1.1.10 - final
>>>
>>> Changesets: 602
>>> Diff: 143 files changed, 8162 insertions(+), 5159 deletions(-)
>>>
>>> ## Highlights
>>>
>>> ### Features added since Pacemaker-1.1.9
>>>
>>> + Core: Convert all exit codes to positive errno values
>>> + crm_error: Add the ability to list and print error symbols
>>> + crm_resource: Allow individual resources to be reprobed
>>> + crm_resource: Allow options to be set recursively
>>> + crm_resource: Implement --ban for moving resources away from nodes
>>> and --clear (replaces --unmove)
>>> + crm_resource: Support OCF tracing when using
>>> --force-(check|start|stop)
>>> + PE: Allow active nodes in our current membership to be fenced without
>>> quorum
>>> + PE: Suppress meaningless IDs when displaying anonymous clone status
>>> + Turn off auto-r

Re: [Pacemaker] crm subshell 1.2.4 incompatible to pacemaker 1.1.9?

[Rainer Brestan]

The bug is in the function is_normal_node.

This function checks the attribute "type" for state "normal".

But this attribute is not used any more.

 

CIB output from Pacemaker 1.1.8


    
  
    
  
  
  
    
  
  
    


CIB output from Pacemaker 1.1.7


    
  
  
  
  
    

 

Therefore, function listnodes will not return any node and function standby will use the current node as node and the first argument as lifetime.

In case of specified both (node and lifetime) it works because of other else path.

Rainer



 

Gesendet: Mittwoch, 15. Mai 2013 um 21:31 Uhr
Von: "Lars Ellenberg" 
An: pacemaker@oss.clusterlabs.org
Betreff: Re: [Pacemaker] crm subshell 1.2.4 incompatible to pacemaker 1.1.9?

On Wed, May 15, 2013 at 03:34:14PM +0200, Dejan Muhamedagic wrote:
> On Tue, May 14, 2013 at 10:03:59PM +0200, Lars Ellenberg wrote:
> > On Tue, May 14, 2013 at 09:59:50PM +0200, Lars Ellenberg wrote:
> > > On Mon, May 13, 2013 at 01:53:11PM +0200, Michael Schwartzkopff wrote:
> > > > Hi,
> > > >
> > > > crm tells me it is version 1.2.4
> > > > pacemaker tell me it is verison 1.1.9
> > > >
> > > > So it should work since incompatibilities are resolved in crm higher that
> > > > version 1.2.1. Anywas crm tells me nonsense:
> > > >
> > > > # crm
> > > > crm(live)# node
> > > > crm(live)node# standby node1
> > > > ERROR: bad lifetime: node1
> > >
> > > Your node is not named node1.
> > > check: crm node list
> > >
> > > Maybe a typo, maybe some case-is-significant nonsense,
> > > maybe you just forgot to use the fqdn.
> > > maybe the check for "is this a known node name" is (now) broken?
> > >
> > >
> > > standby with just one argument checks if that argument
> > > happens to be a known node name,
> > > and assumes that if it is not,
> > > it "has to be" a lifetime,
> > > and the current node is used as node name...
> > >
> > > Maybe we should invert that logic, and instead compare the single
> > > argument against allowed lifetime values (reboot, forever), and assume
> > > it is supposed to be a node name otherwise?
> > >
> > > Then the error would become
> > > ERROR: unknown node name: node1
> > >
> > > Which is probably more useful most of the time.
> > >
> > > Dejan?
> >
> > Something like this maybe:
> >
> > diff --git a/modules/ui.py.in b/modules/ui.py.in
> > --- a/modules/ui.py.in
> > +++ b/modules/ui.py.in
> > @@ -1185,7 +1185,7 @@ class NodeMgmt(UserInterface):
> > if not args:
> > node = vars.this_node
> > if len(args) == 1:
> > - if not args[0] in listnodes():
> > + if args[0] in ("reboot", "forever"):
>
> Yes, I wanted to look at it again. Another complication is that
> the lifetime can be just about anything in that date ISO format.

That may well be, but right now those would be rejected by crmsh
anyways:

if lifetime not in (None,"reboot","forever"):
common_err("bad lifetime: %s" % lifetime)
return False

--
: Lars Ellenberg
: LINBIT | Your Way to High Availability
: DRBD/HA support and consulting http://www.linbit.com

___
Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org




___
Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [Pacemaker] crm subshell 1.2.4 incompatible to pacemaker 1.1.9?

[Rainer Brestan]

Seems that it requires now the lifetime

crm node standby node1 forever

The error message is just nonsense.


Rainer

 


Gesendet: Montag, 13. Mai 2013 um 13:53 Uhr
Von: "Michael Schwartzkopff" 
An: pacemaker@oss.clusterlabs.org
Betreff: [Pacemaker] crm subshell 1.2.4 incompatible to pacemaker 1.1.9?



Hi,

 

crm tells me it is version 1.2.4

pacemaker tell me it is verison 1.1.9

 

So it should work since incompatibilities are resolved in crm higher that version 1.2.1. Anywas crm tells me nonsense:

 

# crm

crm(live)# node

crm(live)node# standby node1

ERROR: bad lifetime: node1

 

Any ideas?

 

--

Dr. Michael Schwartzkopff

Guardinistr. 63

81375 München

 

Tel: (0163) 172 50 98






___
Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [Pacemaker] 1.1.8 not compatible with 1.1.7?

[Rainer Brestan]

Hi Andrew,

yes, this clarifies a lot.

Seems that it is really time to throw away the plugin.

The CMAN solution wont be able (at least from the documentation) to attach new nodes without reconfiguration and restart CMAN on the existing nodes.

The alternative is corosync 2.x.

ClusterLabs has a quite long list of corosync versions from branch 2.0, 2.1, 2.2 und 2.3.

Beside the current reported issue of version 2.3, which version does ClusterLabs use for its regression test.

I found somewhere a note for 2.1.x, is this true ?

Rainer

 

Gesendet: Donnerstag, 09. Mai 2013 um 04:31 Uhr
Von: "Andrew Beekhof" 
An: "The Pacemaker cluster resource manager" 
Betreff: Re: [Pacemaker] 1.1.8 not compatible with 1.1.7?


On 08/05/2013, at 4:53 PM, Andrew Beekhof  wrote:

>
> On 08/05/2013, at 4:08 PM, Andrew Beekhof  wrote:
>
>>
>> On 03/05/2013, at 8:46 PM, Rainer Brestan  wrote:
>>
>>> Now i have all the logs for some combinations.
>>>
>>> Corosync: 1.4.1-7 for all the tests on all nodes
>>> Base is always fresh installation of each node with all packages equal except pacemaker version.
>>> int2node1 node id: 1743917066
>>> int2node2 node id: 1777471498
>>>
>>> In each ZIP file log from both nodes and the status output of crm_mon and cibadmin -Q is included.
>>>
>>> 1.) 1.1.8-4 attaches to running 1.1.7-6 cluster
>>> https://www.dropbox.com/s/06oyrle4ny47uv9/attach_1.1.8-4_to_1.1.7-6.zip
>>> Result: join outstanding
>>>
>>> 2.) 1.1.9-2 attaches to running 1.1.7-6 cluster
>>> https://www.dropbox.com/s/fv5kcm2yb5jz56z/attach_1.1.9-2_to_1.1.7-6.zip
>>> Result: join outstanding
>>
>> Neither side is seeing anything from the other, which is very unexpected.
>> I notice you're using the plugin... which acts as a message router.
>>
>> So I suspect something in there has changed (though I'm at a loss to say what) and that cman based clusters are unaffected.
>
> Confirmed, cman clusters are unaffected.
> I'm yet to work out what changed in the plugin.

I worked it out...

The Red Hat changelog for 1.1.8-2 originally contained

+- Cman is the only supported membership & quorum provider, do not ship the corosync plugin

When this decision was reversed (when I realised no-one was seeing the ERROR logs indicating it was going away), I neglected to re-instate the following distro specific patch (which avoided conflicts between the ID used by CMAN and Pacemaker):

diff --git a/configure.ac b/configure.ac
index a3784d5..dafa9e2 100644
--- a/configure.ac
+++ b/configure.ac
@@ -1133,7 +1133,7 @@ AC_MSG_CHECKING(for native corosync)
COROSYNC_LIBS=""
CS_USES_LIBQB=0

-PCMK_SERVICE_ID=9
+PCMK_SERVICE_ID=10
LCRSODIR="$libdir"

if test $SUPPORT_CS = no; then


So Pacemaker < 6.4 is talking on slot 10, while Pacemaker == 6.4 is using slot 9.
This is why the two versions cannot see each other :-(
I'm very sorry.


___
Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org




___
Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

[Pacemaker] [Patch] pacemaker-mgmt/hbagent avoid coredump with pacemaker>=1.1.8/corosync

[Rainer Brestan]

SNMP agent hbagent from pacemaker-mgmt produces segmentation fault if used with pacemaker>=1.1.8 and corosync.

 

The reason is function get_cib_fd in file hbagentv2.c.

It tries to get the file descriptor with function pointer inputfd, which is not initialized any more since change of IPC to libqb.

 

The patch uses the already present conditional define HAVE_CRM_IPC_NEW to get the file descriptor from function signon_raw and passes this as return value of get_cib_fd.

 

Rainer

 

hbagentv2.c.patch
Description: Binary data
___
Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [Pacemaker] 1.1.8 not compatible with 1.1.7?

[Rainer Brestan]

Now i have all the logs for some combinations.

 

Corosync: 1.4.1-7 for all the tests on all nodes

Base is always fresh installation of each node with all packages equal except pacemaker version.

int2node1 node id: 1743917066
int2node2 node id: 1777471498


 

In each ZIP file log from both nodes and the status output of crm_mon and cibadmin -Q is  included.

 

1.) 1.1.8-4 attaches to running 1.1.7-6 cluster

https://www.dropbox.com/s/06oyrle4ny47uv9/attach_1.1.8-4_to_1.1.7-6.zip

Result: join outstanding

 

2.) 1.1.9-2 attaches to running 1.1.7-6 cluster

https://www.dropbox.com/s/fv5kcm2yb5jz56z/attach_1.1.9-2_to_1.1.7-6.zip

Result: join outstanding

 

3.) 1.1.9-2 attaches to running 1.1.8-4 cluster

https://www.dropbox.com/s/y9o4yo8g8ahwjga/attach_1.1.9-2_to_1.1.8-4.zip

Result: join successful

 

Rainer


Gesendet: Freitag, 03. Mai 2013 um 01:30 Uhr
Von: "Andrew Beekhof" 
An: "The Pacemaker cluster resource manager" 
Betreff: Re: [Pacemaker] 1.1.8 not compatible with 1.1.7?


On 03/05/2013, at 4:46 AM, Rainer Brestan  wrote:

> Hi Lars,
> i have tried 1.1.9-2 from download area at clusterlabs for RHEL6 with corosync 1.4.1-17, also running with 1.1.7-6 at the other node.
> I have to go deeper in details later on (with logs), but the first try was worse than 1.1.8-7.
> When the node with 1.1.9-2 joins the cluster, it could not even decode the ais_message to get the node name of the node running 1.1.7-6.

Logs?

> It states a new node has joined with the correct node id, but as name it could only decode (null) as node name.
>
> Just as first impression.
>
> Rainer
>
> Gesendet: Dienstag, 30. April 2013 um 17:16 Uhr
> Von: "Lars Marowsky-Bree" 
> An: pacemaker@oss.clusterlabs.org
> Betreff: Re: [Pacemaker] 1.1.8 not compatible with 1.1.7?
> On 2013-04-24T11:44:57, Rainer Brestan  wrote:
>
> > Current DC: int2node2 - partition WITHOUT quorum
> > Version: 1.1.8-7.el6-394e906
>
> This may not be the answer you want, since it is fairly unspecific. But
> I think we noticed something similar when we pulled in 1.1.8, I don't
> recall the bug number, but I *think* it worked out with a later git
> version.
>
> Can you try a newer build than 1.1.8?
>
>
> Regards,
> Lars
>
> --
> Architect Storage/HA
> SUSE LINUX Products GmbH, GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 21284 (AG Nürnberg)
> "Experience is the name everyone gives to their mistakes." -- Oscar Wilde
>
>
> ___
> Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org
> ___
> Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org


___
Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org




___
Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [Pacemaker] 1.1.8 not compatible with 1.1.7?

[Rainer Brestan]

Hi Lars,

i have tried 1.1.9-2 from download area at clusterlabs for RHEL6 with corosync 1.4.1-17, also running with 1.1.7-6 at the other node.

I have to go deeper in details later on (with logs), but the first try was worse than 1.1.8-7.

When the node with 1.1.9-2 joins the cluster, it could not even decode the ais_message to get the node name of the node running 1.1.7-6.

It states a new node has joined with the correct node id, but as name it could only decode (null) as node name.

 

Just as first impression.

 

Rainer


 

Gesendet: Dienstag, 30. April 2013 um 17:16 Uhr


Von: "Lars Marowsky-Bree" 
An: pacemaker@oss.clusterlabs.org
Betreff: Re: [Pacemaker] 1.1.8 not compatible with 1.1.7?

On 2013-04-24T11:44:57, Rainer Brestan  wrote:

> Current DC: int2node2 - partition WITHOUT quorum
> Version: 1.1.8-7.el6-394e906

This may not be the answer you want, since it is fairly unspecific. But
I think we noticed something similar when we pulled in 1.1.8, I don't
recall the bug number, but I *think* it worked out with a later git
version.

Can you try a newer build than 1.1.8?


Regards,
Lars

--
Architect Storage/HA
SUSE LINUX Products GmbH, GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 21284 (AG Nürnberg)
"Experience is the name everyone gives to their mistakes." -- Oscar Wilde


___
Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org




___
Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [Pacemaker] 1.1.8 not compatible with 1.1.7?

[Rainer Brestan]

I have tried to make this test, because I had the same problem.

 

Origin:

One node cluster, node int2node1 running with IP address 10.16.242.231, quorum ignore, DC int2node1

 


[root@int2node1 sysconfig]# crm_mon -1

Last updated: Wed Apr 24 09:49:32 2013
Last change: Wed Apr 24 09:44:55 2013 via crm_resource on int2node1
Stack: openais
Current DC: int2node1 - partition WITHOUT quorum
Version: 1.1.7-6.el6-148fccfd5985c5590cc601123c6c16e966b85d14
1 Nodes configured, 2 expected votes
1 Resources configured.


Online: [ int2node1 ]

 Clone Set: cloneSysInfo [resSysInfo]
 Started: [ int2node1 ]

 

Next step:

Node int2node2 with IP address 10.16.242.233 joins the cluster.

 

Result:

 


[root@int2node1 sysconfig]# crm_mon -1

Last updated: Wed Apr 24 10:14:18 2013
Last change: Wed Apr 24 10:05:20 2013 via crmd on int2node1
Stack: openais
Current DC: int2node1 - partition WITHOUT quorum
Version: 1.1.7-6.el6-148fccfd5985c5590cc601123c6c16e966b85d14
2 Nodes configured, 2 expected votes
1 Resources configured.


Online: [ int2node1 ]
OFFLINE: [ int2node2 ]

 Clone Set: cloneSysInfo [resSysInfo]
 Started: [ int2node1 ]

 


[root@int2node1 sysconfig]# corosync-objctl | grep member
runtime.totem.pg.mrp.srp.members.1743917066.ip=r(0) ip(10.16.242.231)
runtime.totem.pg.mrp.srp.members.1743917066.join_count=1
runtime.totem.pg.mrp.srp.members.1743917066.status=joined
runtime.totem.pg.mrp.srp.members.1777471498.ip=r(0) ip(10.16.242.233)
runtime.totem.pg.mrp.srp.members.1777471498.join_count=1
runtime.totem.pg.mrp.srp.members.1777471498.status=joined

 


[root@int2node1 sysconfig]# crm_node -l
1743917066 int2node1 member

 


[root@int2node2 ~]# crm_mon -1
Last updated: Wed Apr 24 11:27:39 2013
Last change: Wed Apr 24 10:07:45 2013 via crm_resource on int2node2
Stack: classic openais (with plugin)
Current DC: int2node2 - partition WITHOUT quorum
Version: 1.1.8-7.el6-394e906
2 Nodes configured, 2 expected votes
1 Resources configured.


Online: [ int2node2 ]
OFFLINE: [ int2node1 ]

 Clone Set: cloneSysInfo [resSysInfo]
 Started: [ int2node2 ]

 


[root@int2node2 ~]# corosync-objctl | grep member
runtime.totem.pg.mrp.srp.members.1743917066.ip=r(0) ip(10.16.242.231)
runtime.totem.pg.mrp.srp.members.1743917066.join_count=1
runtime.totem.pg.mrp.srp.members.1743917066.status=joined
runtime.totem.pg.mrp.srp.members.1777471498.ip=r(0) ip(10.16.242.233)
runtime.totem.pg.mrp.srp.members.1777471498.join_count=1
runtime.totem.pg.mrp.srp.members.1777471498.status=joined

 


[root@int2node2 ~]# crm_node -l
1777471498 int2node2 member

 

Pacemaker log of int2node2 with trace setting.

https://www.dropbox.com/s/04ciy2g6dfbauxy/pacemaker.log?n=165978094

On int2node1 (1.1.7) the trace setting did not create the pacemaker.log file.









 

Below the excerpt of cib with node information from int2node2.

[root@int2node2 ~]# cibadmin -Q

  
    
  
  ...
  
    
    
  
  
    
    
    ...
    
    
    ...
    
  
  
    
  
    
    ...
    
  
  
    
    ...
    
  
    
    
  

 

On int2node2 the node state in the cib is different.


  
    
  


  
  
    


    ...
    
  
    
    
  

 




Rainer


Gesendet: Mittwoch, 17. April 2013 um 07:32 Uhr
Von: "Andrew Beekhof" 
An: "The Pacemaker cluster resource manager" 
Betreff: Re: [Pacemaker] 1.1.8 not compatible with 1.1.7?


On 15/04/2013, at 7:08 PM, Pavlos Parissis  wrote:

> Hoi,
>
> I upgraded 1st node and here are the logs
> https://dl.dropboxusercontent.com/u/1773878/pacemaker-issue/node1.debuglog
> https://dl.dropboxusercontent.com/u/1773878/pacemaker-issue/node2.debuglog
>
> Enabling tracing on the mentioned functions didn't give at least to me any more information.

10:22:08 pacemakerd[53588]: notice: crm_add_logfile: Additional logging available in /var/log/pacemaker.log

Thats the file(s) we need :)

>
> Cheers,
> Pavlos
>
>
> On 15 April 2013 01:42, Andrew Beekhof  wrote:
>
> On 15/04/2013, at 7:31 AM, Pavlos Parissis  wrote:
>
> > On 12/04/2013 09:37 μμ, Pavlos Parissis wrote:
> >> Hoi,
> >>
> >> As I wrote to another post[1] I failed to upgrade to 1.1.8 for a 2 node
> >> cluster.
> >>
> >> Before the upgrade process both nodes are using CentOS 6.3, corosync
> >> 1.4.1-7 and pacemaker-1.1.7.
> >>
> >> I followed the rolling upgrade process, so I stopped pacemaker and then
> >> corosync on node1 and upgraded to CentOS 6.4. The OS upgrade upgrades
> >> also pacemaker to 1.1.8-7 and corosync to 1.4.1-15.
> >> The upgrade of rpms went smoothly as I knew about the crmsh issue so I
> >> made sure I had crmsh rpm on my repos.
> >>
> >> Corosync started without any problems and both nodes could see each
> >> other[2]. But for some reason node2 failed to receive a reply on join
> >> offer from node1 and node1 never joined the cluster. Node1 formed a new
> >> cluster as it never got an reply from node2, so I ended up 

Re: [Pacemaker] attrd waits one second before doing update

[Rainer Brestan]

OK, and where is the difference between 1.1.8 and 1.1.7.

I am currently testing this on a one node cluster, so attrd wait for the message come back from himself.

This cant take one second, or is attrd waiting this time anyhow to be sure to get it from all nodes back?

Rainer

 

Gesendet: Freitag, 12. April 2013 um 02:03 Uhr
Von: "Andrew Beekhof" 
An: "The Pacemaker cluster resource manager" 
Betreff: Re: [Pacemaker] attrd waits one second before doing update


On 12/04/2013, at 7:17 AM, Rainer Brestan  wrote:

> In pacemaker 1.1.7-6 with corosync 1.4.1-7 update of attributes works almost online.
> Used with SysInfo resource agent and manual commands like "attrd_updater -U 4 -n test".
>
> In the logfile there is one line
> attrd[...] notice: attrd_trigger_update: Sending flush up to all hosts for: ...
> and a few milliseconds later
> attrd[...] notice: attrd_perform_update: Sent update ...
> with the same content.
>
> After upgrade to version 1.1.8-6 there is always nearly exact one second between trigger and perform.
> 2013-04-11T22:51:55.389+02:00 int2node2 attrd[28370] notice: notice: attrd_trigger_update: Sending flush op to all hosts for: text (81)
> 2013-04-11T22:51:56.397+02:00 int2node2 attrd[28370] notice: notice: attrd_perform_update: Sent update 5814: text=81
>
> And what i found out having several updates running, they have a single queue.
> All attrd_updater processes are waiting for the next to be finished, so there cant be more than one update per second any more.
>
> Has this something to do with
> attrd: Have single-shot clients wait for an ack before disconnecting
> stated in the Changelog for 1.1.8 ?

No, nothing at all.

>
> If yes, is it intended to have a single queue ?

More like unavoidable, since we need to talk to the other nodes and messages between them are ordered.

> And is this 1 second fixed ?
> From where does this 1 second come, i dont think that it takes one second to get the ack.

When the timer expires, attrd sends a cluster message to all nodes (including itself) telling them to update the CIB with their current value.
The delay comes from waiting for the cluster message we sent to arrive back again before sending our own updates, this helps ensure all the updates arrive in the CIB at almost the same time.

>
> This can run into heavy delays (and therefore timeouts) for monitor functions of RA performing attribute updates.
>
> Rainer
> ___
> Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org


___
Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org




___
Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

[Pacemaker] attrd waits one second before doing update

[Rainer Brestan]

In pacemaker 1.1.7-6 with corosync 1.4.1-7 update of attributes works almost online.

Used with SysInfo resource agent and manual commands like "attrd_updater -U 4 -n test".

 

In the logfile there is one line

attrd[...] notice: attrd_trigger_update: Sending flush up to all hosts for: ...

and a few milliseconds later

attrd[...] notice: attrd_perform_update: Sent update  ...

with the same content.

 

After upgrade to version 1.1.8-6 there is always nearly exact one second between trigger and perform.


2013-04-11T22:51:55.389+02:00 int2node2 attrd[28370] notice:   notice: attrd_trigger_update: Sending flush op to all hosts for: text (81)
2013-04-11T22:51:56.397+02:00 int2node2 attrd[28370] notice:   notice: attrd_perform_update: Sent update 5814: text=81

 

And what i found out having several updates running, they have a single queue.

All attrd_updater processes are waiting for the next to be finished, so there cant be more than one update per second any more.

 

Has this something to do with

attrd: Have single-shot clients wait for an ack before disconnecting

stated in the Changelog for 1.1.8 ?

 

If yes, is it intended to have a single queue ?

And is this 1 second fixed ?

From where does this 1 second come, i dont think that it takes one second to get the ack.

 

This can run into heavy delays (and therefore timeouts) for monitor functions of RA performing attribute updates.

 

Rainer


___
Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [Pacemaker] Master Slave Resource Agent won't promote

[Rainer Brestan]

Hi Felix,

maybe my hint is worthless, but have you implemented the crm_master calls in your RA ?

See Stateful RA demo $CRM_MASTER calls.

Rainer

Gesendet: Mittwoch, 10. April 2013 um 09:58 Uhr
Von: "Felix Zachlod" 
An: "'The Pacemaker cluster resource manager'" 
Betreff: [Pacemaker] Master Slave Resource Agent won't promote

Hello!

I have another problem with my resource agent which should run in a master
slave fashion.
I successfully tested the RA with ocf-tester and it completes any promote or
demote action:

fctarget[14659]: DEBUG: Resource is running
fctarget[14659]: DEBUG: Resource is currently running as Slave
fctarget[14659]: DEBUG: fctargettest demote : 0
Checking for promote action
fctarget[14677]: DEBUG: Resource is running
fctarget[14677]: DEBUG: Resource is currently running as Slave
fctarget[14677]: DEBUG: Resource is running MASTER
fctarget[14677]: DEBUG: Resource promoted
fctarget[14677]: DEBUG: fctargettest promote : 0
Testing: demotion of started resource
fctarget[14717]: DEBUG: Resource is running MASTER
fctarget[14717]: DEBUG: Resource is currently running as Master
fctarget[14717]: DEBUG: Resource is running
fctarget[14717]: DEBUG: Resource demoted
fctarget[14717]: DEBUG: fctargettest demote : 0
Testing: promote
fctarget[14746]: DEBUG: Resource is running
fctarget[14746]: DEBUG: Resource is currently running as Slave
fctarget[14746]: DEBUG: Resource is running MASTER
fctarget[14746]: DEBUG: Resource promoted
fctarget[14746]: DEBUG: fctargettest promote : 0
Testing: demote
fctarget[14790]: DEBUG: Resource is running MASTER
fctarget[14790]: DEBUG: Resource is currently running as Master
fctarget[14790]: DEBUG: Resource is running
fctarget[14790]: DEBUG: Resource demoted
fctarget[14790]: DEBUG: fctargettest demote : 0
Testing: demotion of demoted resource
fctarget[14819]: DEBUG: Resource is running
fctarget[14819]: DEBUG: Resource is currently running as Slave
fctarget[14819]: DEBUG: fctargettest demote : 0
[...]

fctarget passed all tests

I added the resource to the crm and it successfully starts on both nodes.
But it does not promote anywhere although I have configured target-role
"Master"
The Cluster Manager does not even try to promote as I don't get any log
output about something failing.

It seems the crm does not like my RA for some reason. I can successfully
promote a Stateful Dummy resource or a DRBD device with the CRM.

Configuration of resource looks like the following:

primitive p_fctarget_7701 ocf:onesty:fctarget \
params wwnn="77:00:00:00:00:00:01:00" wwpn="77:00:00:00:00:00:01:01"
parentwwpns="storage-test-a;21:00:00:e0:8b:86:0e:cf
storage-test-b;21:00:00:1b:32:88:66:c5" \
op stop interval="0" timeout="40s"

ms ms_fctarget_7701 p_fctarget_7701 \
meta master-max="1" clone-max="2" clone-node-max="1"
target-role="Master"

My resource agent does support the promote and demote action and advertises
them in the meta data:








role="Master"/>

role="Slave"/>




case $__OCF_ACTION in
start) fctarget_start;;
stop) fctarget_stop;;
promote) fctarget_promote;;
demote) fctarget_demote;;
monitor|status) fctarget_monitor;;
validate-all) fctarget_validate_all;;
*) fctarget_usage
exit $OCF_ERR_UNIMPLEMENTED
;;

This ist he output of crm status:

Master/Slave Set: ms_fctarget_7701 [p_fctarget_7701]
Slaves: [ storage-test-a storage-test-b ]

I tried manually promoting and demoting the resource which simply seems to
do NOTHING.

Can anyone give me a hint what I am missing?


Thank you all in advance


___
Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org




___
Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [Pacemaker] Clone Resources Individual Configuration per Node

[Rainer Brestan]

Hi Felix,

thats exactly the reason why I took the meta attribute variant.

It is currently available neither via crm_resource nor via crm.


 

Maybe a good point to submit an request to Dejan about extension of crmsh.

 

Rainer

 


Gesendet: Dienstag, 09. April 2013 um 15:32 Uhr
Von: "Felix Zachlod" 
An: "'The Pacemaker cluster resource manager'" 
Betreff: Re: [Pacemaker] Clone Resources Individual Configuration per Node

Hi again,

> I'd be interested to hear your conclusions.

I am currently trying your suggestion with using rules. Can anyone tell how
to apply such a config via the crm shell? I usually do not edit the xml
directly. But I cant find a way to do via the crm shell and did not find any
documentation how rules can be applied to attributes via the shell.

Thanks in advance


___
Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org




___
Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [Pacemaker] Clone Resources Individual Configuration per Node

[Rainer Brestan]

Hi Felix,

 

basically you have three option to provide information to the resource agent.

- Resource parameters

- Resource meta attributes

- Node attributes


 

Let me assume some information for an example.

Your nodes are named nodeA and nodeB.

The hardware address for nodeA shall be 0x3 and for nodeB 0x6.

 

Resource parameter should be defined in the meta information part of the resource agent. As long as you have always a defined number of nodes, this can work as follows.

crm configure ... param node1name=nodeA node1addr=0x3 node2name=nodeB node2addr=0x6

The resource agent can go through all nodeXname parameter and pick the number out with its own hostname and read the address with the same number.

Be aware, that all those parameters have to be defined in the mta-data.

 

Resource meta attributes are more relaxed, as you can assign every name and every value without defining them in the meta-data.

crm configure ... meta node1name=nodeA node1addr=0x3 node2name=nodeB node2addr=0x6

Read the meta data with crm_resource inside the resource agent.

 

Node attributes are set and read with crm_attribute and can be persistent.

Set the attribute with

crm_attribute -n hwaddr -N nodeA -l forever -v 0x3


crm_attribute -n hwaddr -N nodeB -l forever -v 0x6

Inside the resource agent read the value with

crm_attribute -n hwaddr -N $HOSTNAME -l forever -G -q

and you get the value for the current host.

 

I had a similar approach to do and used resource meta attributes mainly for two reasons.

- Resource configuration is bound to the resource and does not influence anybody else.

- Using "crm configure show" also shows the configuration of the resource, using crm_attribute would not.

 

Rainer



Gesendet: Montag, 08. April 2013 um 09:34 Uhr
Von: "Felix Zachlod" 
An: Pacemaker@oss.clusterlabs.org
Betreff: [Pacemaker] Clone Resources Individual Configuration per Node

Hello List,

I am up to writing a resource agent for Pacemaker which shall be used in a
Master/Slave manner. So the cluster shall always run two instances and one
of them shall be Master. My clone instances need a different configuration
depending on the node that they are running on. I know I could accomplish
this by having some kind of configuration file on each node but I want to
manage as much as possible through the cluster manager. So what I want is a
Master / Slave (or clone) resource running with instance attributes tied to
the cluster node which they are running on.

Let me explain what the purpose is:

The resource is dependent on specific hardware on the cluster node, which
has different hardware addresses on each node. So resource instance A DOES
deliver the same service as resource instance B at least when being in the
master state each but anyways has to be started with a different parameter.

I understood that I can configure a location constraint for a resource so
that it can be run only on one node Lets say my RA expects a param
hwaddr={string}. I can set up a resource for Node a with hwaddr="a" and one
for node b with hwaddr="b" but how can I tell pacemaker that these two
resources now shall be treated as a Master Slave set? Or am I thinking about
this too complicated?

Thank you for any hint in advance,
Felix


___
Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org




___
Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [Pacemaker] PGSQL resource promotion issue

[Rainer Brestan]

Look at the order.

Master/replication VIP ist started after promote and stopped after demote.

listen_addresses = "*" means dont take care about specific interfaces, so it will listen on all interfaces not matter if they are coming up or switching down after postgres start.


 

The master either sets the promotion score to CAN_PROMOTE=100 or CAN_NOT_PROMOTE=-INF.

For every slave which is in sync it sets 100 and for each missing sync -INF.

When the master is away all slaves fall back to the funtion have_master_right. Now the normal election begins. Every slave writes its own xlog location and the one with the highest value set promotion score to 1000.

All slaves which are not in sync dont participate in the master election any more. This is the starting code of function have_master_right.

 

Rainer

 


Gesendet: Freitag, 29. März 2013 um 12:33 Uhr
Von: "Steven Bambling" 
An: "The Pacemaker cluster resource manager" 
Betreff: Re: [Pacemaker] PGSQL resource promotion issue


 

On Mar 28, 2013, at 8:13 AM, Rainer Brestan <rainer.bres...@gmx.net> wrote:
 




Hi Steve,

i think, you have misunderstood how ip addresses are used with this setup, PGVIP should start after promotion.

Take a look at Takatoshi´s Wiki.

https://github.com/t-matsuo/resource-agents/wiki/Resource-Agent-for-PostgreSQL-9.1-streaming-replication




 

I see that he has the master/replication VIPs with a resource order to force promotion before moving the VIPs to the new master. 


 

  I don't get how the postgres service is going to listen on those interfaces if they have not already migrated to the new master.  Even with setting the listen_addresses = "*"

 



 

The promotion sequency is very simple.

When no master is existing, all slaves write their current replay xlog into the node attribute PGSQL-xlog-loc during monitor call.




Does this also hold true if a Master fails?  

 

From the looks of it, if there was a Master before the failure that the master score is set from the function that grabs the data_status from the master (STREAMING|SYNC, STREAMING|ASYNC, STREAMING|POTENTIAL, etc ).  

 

The reason I ask is if the master fails and the slaves don't then compare their xlog location, there is a potential for data loss if the incorrect slave is promoted.

 
 




You can see all them with crm_mon -A1f.

Each slave gets these attributes from all node configured in parameter node_list (hopefully your node names in Pacemaker are the same as in node_list) and compares them to get the highest.


If the highest is this list is the own one, it sets the master-score to 1000, on other nodes to 100.

Pacemaker then selects the node with the highest master score and promote this.

 

Rainer


Gesendet: Mittwoch, 27. März 2013 um 14:37 Uhr
Von: "Steven Bambling" <smbambl...@arin.net>
An: "The Pacemaker cluster resource manager" <pacemaker@oss.clusterlabs.org>
Betreff: Re: [Pacemaker] PGSQL resource promotion issue


In talking with andreask from IRC, I  miss understood the need to include the op monitor.  I figured it was pulled from the resource script by default.
 

I used pcs to add the new attributes and one was then promoted to master 

 

pcs resource add_operation PGSQL monitor interval=5s role=Master

pcs resource add_operation PGSQL monitor interval=7s

 

v/r

 

STEVE

 

On Mar 27, 2013, at 7:08 AM, Steven Bambling <smbambl...@arin.net> wrote:


 


 
I've built and installed the lastest resource-agents from github on Centos 6 and configured two resources 


 

1 primitive PGVIP:

pcs resource create PGVIP ocf:heartbeat:IPaddr2 ip=10.1.22.48 cidr_netmask=25 op monitor interval=1

 

Before setting up the PGSQL resource I manually configured sync/streaming replication on the three nodes with p1.example.com as the master and verified that replication was working.  I think removed the synchronous_standby_name from my postgresql.conf and stop all postgres services on all nodes

 

1 master/slave PGSQL: -- I've the resource to use sync replication.  Also I am using PGSQL 9.2.3

 

pcs resource create PGSQL ocf:heartbeat:pgsql params pgctl="/usr/pgsql-9.2/bin/pg_ctl" pgdata="/var/lib/pgsql/9.2/data" config="/var/lib/pgsql/9.2/data/postgresql.conf" stop_escalate="5" rep_mode="sync" node_list="p1.example.com p2.example.com p3.example.com" restore_command='cp /var/lib/pgsql/9.2/archive/%f "%p"' master_ip="10.1.22.48" repuser="postgres" restart_on_promote="true" tmpdir="/var/lib/pgsql/9.2/tmpdir" xlog_check_count="3" crm_attr_timeout="5" check_wal_receiver="true" --master

 

I'm able to successfully get all the nodes in the cluster started and the PGVIP resource starts on the 1st node and the PGSQL:[012] resource start 

Re: [Pacemaker] issues when installing on pxe booted environment

[Rainer Brestan]

Puh, i haven´t thought the discussion became this direction.

 

Corosync is not the only software, which need shared memory, so it is part of the OS startup to provide it, not part of Corosync or Pacemaker.

And yes, it is too late to mount it in Pacemaker startup.

Even in most embedded Linux installations share memory is available, just with smaller size.

So, there is no need to include anything in Corosync or Pacemaker startup directly.

 

My answer about shared memory is valid and neccessary only for anaconda based installations.

When you install a RedHat system, the installation process is done by anaconda with a running miniroot (install.img) as ram disk.

Within this miniroot shared memory is enabled, but they (RedHat) missed the mount.

If somebody wants to correct it, RedHat (more specific the maintainer of install.img) is the correct address for this gap.

 

To have world read access to CRM_CORE_DIR should be enough in case a core pattern set explicit to another directory (not tested yet).

When calling resource agents, CRM_CORE_DIR is the current PWD (tested with echo all environment variables to a file inside monitor call).

If this directory is not readable during switch user inside resource agent, stupid things could happen. Anyhow, it is not a good method of writing resource agents, switching user without setting the environment of the switched user. So, this is no fault of Pacemaker, more of loose programmed resource agents.

When the resource agent or any of its childs produces a core dump, it goes either to the current directory or to the directory specified with kernel core pattern.

If the core will go into the current directory and it is written by a switched user, who is not member of pcmk_gid, the core get lost.

 

As i am not sure every resource agent is written with proper switch user environment and not rewriting my core pattern, enable world write on CRM_CCORE_DIR was the easy work around for it.

 

Rainer


Gesendet: Freitag, 29. März 2013 um 09:51 Uhr
Von: "Jacek Konieczny" 
An: pacemaker@oss.clusterlabs.org
Betreff: Re: [Pacemaker] issues when installing on pxe booted environment

On Fri, 29 Mar 2013 11:37:37 +1100
Andrew Beekhof  wrote:

> On Thu, Mar 28, 2013 at 10:43 PM, Rainer Brestan
>  wrote:
> > Hi John,
> > to get Corosync/Pacemaker running during anaconda installation, i
> > have created a configuration RPM package which does a few actions
> > before starting Corosync and Pacemaker.
> >
> > An excerpt of the post install of this RPM.
> > # mount /dev/shm if not already existing, otherwise openais cannot
> > work if [ ! -d /dev/shm ]; then
> > mkdir /dev/shm
> > mount /dev/shm
> > fi
>
> Perhaps mention this to the corosync guys, it should probably go into
> their init script.

I don't think so. It is just a part of modern Linux system environment.
corosync is not supposed to mount the root filesystem or /proc –
mounting /dev/shm is not its responsibility either.

BTW The excerpt above assumes there is a /dev/shm entry in /etc/fstab.
Should this be added there by the corosync init script too?

Greets,
Jacek

___
Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org




___
Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [Pacemaker] OCF Resource agent promote question

[Rainer Brestan]

Hi Steve,

when pre promote notify is called, Pacemaker has already selected a node to promote, you cannot stop this sequency any more.

As the node with the highest score should have been promoted, this is a code to fail slaves coming up after promote node selection.

If you try to force another node to promote, the running master must be first demoted (and then unavailable). But to make this part of the resource agent can be dangerous as you dont know which state a new slave is joining the cluster comes from.

If you start all nodes at the same time and not the latest xlog becomes master, then your settings for monitor interval and xlog_check_count is too small.

Master promotion score will be set after slave monitor interval * xlog_check_count starting the second of the first slave resource comes up.

As it is not predictable any more, which server comes up how fast and when the first slave instance comes up, this time should be not smaller then 30 seconds.

Rainer

 

Gesendet: Donnerstag, 28. März 2013 um 12:41 Uhr
Von: "Steven Bambling" 
An: "The Pacemaker cluster resource manager" 
Betreff: Re: [Pacemaker] OCF Resource agent promote question



I'm reading the additions that you added to the pgsql resource agent to allow for streaming replication in Postgres 9.1+.  I'm trying to determine if your resource agent will compensate if node promoted ( new master ) does not have the newest data.  

 

From the looks of the pgsql_pre_promote function it seems that it will just fail other replicas (slaves) that have newer data, but will continue with the promotion of the new master even though it does not have the latest data. 

 

If this is correct is there a way to force the promotion of the node with the newest data?

 

v/r

 

STEVE

 

 


On Mar 26, 2013, at 8:19 AM, Steven Bambling <smbambl...@arin.net> wrote:
 


Excellent thanks so much for the clarification.  I'll drop this new RA in and see if I can get things working.
 

STEVE

 

 

On Mar 26, 2013, at 7:38 AM, Rainer Brestan <rainer.bres...@gmx.net>

 wrote:
 



 

Hi Steve,

pgsql RA does the same, it compares the last_xlog_replay_location of all nodes for master promotion.

Doing a promote as a restart instead of promote command to conserve timeline id is also on configurable option (restart_on_promote) of the current RA.

And the RA is definitely capable of having more than two instances. It goes through the parameter node_list and doing its actions for every member in the node list.

Originally it might be planned only to have only one slave, but the current implementation does not have this limitation. It has code for sync replication of more than two nodes, when some of them fall back into async to not promote them.

 

Of course, i will share the extension with the community, when they are ready for use. And the feature of having more than two instances is not removed. I am not running more than two instances on one site, current usage is to have two instances on one site and having two sites and manage master by booth. But it also under discussion to have more than two instances on one site, just to have no availability interruption in case of one server down and the other promote with restart.

The implementation is nearly finished, then begins the stress tests of failure scenarios.

 

Rainer


Gesendet: Dienstag, 26. März 2013 um 11:55 Uhr
Von: "Steven Bambling" <smbambl...@arin.net>
An: "The Pacemaker cluster resource manager" <pacemaker@oss.clusterlabs.org>
Betreff: Re: [Pacemaker] OCF Resource agent promote question


 

On Mar 26, 2013, at 6:32 AM, Rainer Brestan <rainer.bres...@gmx.net> wrote:
 



 

Hi Steve,

when Pacemaker does promotion, it has already selected a specific node to become master.

It is far too late in this state to try to update master scores.

 

But there is another problem with xlog in PostgreSQL.

 

According to some discussion on PostgreSQL mailing lists, not relevant xlog entries dont go into the xlog counter during redo and/or start. This is specially true for CHECKPOINT xlog records, where this situation can be easely reproduced.

This can lead to the situation, where the replication is up to date, but the slave shows an lower xlog value.

This issue was solved in 9.2.3, where wal receiver always counts the end of applied records.





 
We are currently testing with 9.2.3.  I'm using the functions http://www.databasesoup.com/2012/10/determining-furthest-ahead-replica.html along with tweaking a function to get the replay_lag in bytes to have a more accurate measurement.





 

There is also a second boring issue. The timeline change is replicated to the slaves, but they do not save it anywhere. In case slave starts up again and do not have access to the WAL archive, it cannot start any more. This was also addressed as patch in 9.2 branch, but i havent test if also fixed in 9.2.3.





After talking with 

Re: [Pacemaker] PGSQL resource promotion issue

[Rainer Brestan]

Hi Steve,

i think, you have misunderstood how ip addresses are used with this setup, PGVIP should start after promotion.

Take a look at Takatoshi´s Wiki.

https://github.com/t-matsuo/resource-agents/wiki/Resource-Agent-for-PostgreSQL-9.1-streaming-replication

 

The promotion sequency is very simple.

When no master is existing, all slaves write their current replay xlog into the node attribute PGSQL-xlog-loc during monitor call.

You can see all them with crm_mon -A1f.

Each slave gets these attributes from all node configured in parameter node_list (hopefully your node names in Pacemaker are the same as in node_list) and compares them to get the highest.


If the highest is this list is the own one, it sets the master-score to 1000, on other nodes to 100.

Pacemaker then selects the node with the highest master score and promote this.

 

Rainer


Gesendet: Mittwoch, 27. März 2013 um 14:37 Uhr
Von: "Steven Bambling" 
An: "The Pacemaker cluster resource manager" 
Betreff: Re: [Pacemaker] PGSQL resource promotion issue


In talking with andreask from IRC, I  miss understood the need to include the op monitor.  I figured it was pulled from the resource script by default.
 

I used pcs to add the new attributes and one was then promoted to master 

 

pcs resource add_operation PGSQL monitor interval=5s role=Master

pcs resource add_operation PGSQL monitor interval=7s

 

v/r

 

STEVE

 

On Mar 27, 2013, at 7:08 AM, Steven Bambling  wrote:


 


 
I've built and installed the lastest resource-agents from github on Centos 6 and configured two resources 


 

1 primitive PGVIP:

pcs resource create PGVIP ocf:heartbeat:IPaddr2 ip=10.1.22.48 cidr_netmask=25 op monitor interval=1

 

Before setting up the PGSQL resource I manually configured sync/streaming replication on the three nodes with p1.example.com as the master and verified that replication was working.  I think removed the synchronous_standby_name from my postgresql.conf and stop all postgres services on all nodes

 

1 master/slave PGSQL: -- I've the resource to use sync replication.  Also I am using PGSQL 9.2.3

 

pcs resource create PGSQL ocf:heartbeat:pgsql params pgctl="/usr/pgsql-9.2/bin/pg_ctl" pgdata="/var/lib/pgsql/9.2/data" config="/var/lib/pgsql/9.2/data/postgresql.conf" stop_escalate="5" rep_mode="sync" node_list="p1.example.com p2.example.com p3.example.com" restore_command='cp /var/lib/pgsql/9.2/archive/%f "%p"' master_ip="10.1.22.48" repuser="postgres" restart_on_promote="true" tmpdir="/var/lib/pgsql/9.2/tmpdir" xlog_check_count="3" crm_attr_timeout="5" check_wal_receiver="true" --master

 

I'm able to successfully get all the nodes in the cluster started and the PGVIP resource starts on the 1st node and the PGSQL:[012] resource start on each node in the cluster.  The one thing I don't understand is why none of the slaves is taking over the master role.


 

Also how would I go about force promoting one of the slaves into the master role via the PCS command line utility. 

 

v/r

 

STEVE

___
Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org








___
Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

Re: [Pacemaker] issues when installing on pxe booted environment

[Rainer Brestan]

Hi John,

to get Corosync/Pacemaker running during anaconda installation, i have created a configuration RPM package which does a few actions before starting Corosync and Pacemaker.

 

An excerpt of the post install of this RPM.


# mount /dev/shm if not already existing, otherwise openais cannot work
if [ ! -d /dev/shm ]; then
    mkdir /dev/shm
    mount /dev/shm
fi


# resource agents might run as different user
chmod -R go+rwx /var/lib/heartbeat/cores

 

Rainer

 



Gesendet: Donnerstag, 28. März 2013 um 00:46 Uhr
Von: "Andrew Beekhof" 
An: "The Pacemaker cluster resource manager" 
Betreff: Re: [Pacemaker] issues when installing on pxe booted environment

What about /dev/shm ?
Libqb tries to create some shared memory in that location by default.

On Thu, Mar 28, 2013 at 8:50 AM, John White  wrote:
> Yup:
> -bash-4.1$ cd /var/run/crm/
> -bash-4.1$ ls
> lost+found pcmk pengine st_callback st_command
> -bash-4.1$ touch blah
> -bash-4.1$ ls -l
> total 16
> -rw-r--r-- 1 hacluster haclient 0 Mar 27 14:50 blah
> drwx-- 2 root root 16384 Mar 14 15:00 lost+found
> srwxrwxrwx 1 root root 0 Mar 22 11:25 pcmk
> srwxrwxrwx 1 hacluster root 0 Mar 22 11:25 pengine
> srwxrwxrwx 1 root root 0 Mar 22 11:25 st_callback
> srwxrwxrwx 1 root root 0 Mar 22 11:25 st_command
> -bash-4.1$ ls -l /var/run/| grep crm
> drwxr-xr-x 3 hacluster haclient 4096 Mar 27 14:50 crm
> -bash-4.1$ whoami
> hacluster
> -bash-4.1$
> 
> John White
> HPC Systems Engineer
> (510) 486-7307
> One Cyclotron Rd, MS: 50C-3209C
> Lawrence Berkeley National Lab
> Berkeley, CA 94720
>
> On Mar 25, 2013, at 4:21 PM, Andreas Kurz  wrote:
>
>> On 2013-03-22 19:31, John White wrote:
>>> Hello Folks,
>>> We're trying to get a corosync/pacemaker instance going on a 4 node cluster that boots via pxe. There have been a number of state/file system issues, but those appear to be *mostly* taken care of thus far. We're running into an issue now where cib just isn't staying up with errors akin to the following (sorry for the lengthy dump, note the attrd and cib connection errors). Any ideas would be greatly appreciated:
>>>
>>> Mar 22 11:25:18 n0014 cib: [25839]: info: validate_with_relaxng: Creating RNG parser context
>>> Mar 22 11:25:18 n0014 attrd: [25841]: info: Invoked: /usr/lib64/heartbeat/attrd
>>> Mar 22 11:25:18 n0014 attrd: [25841]: info: crm_log_init_worker: Changed active directory to /var/lib/heartbeat/cores/hacluster
>>> Mar 22 11:25:18 n0014 attrd: [25841]: info: main: Starting up
>>> Mar 22 11:25:18 n0014 attrd: [25841]: info: get_cluster_type: Cluster type is: 'corosync'
>>> Mar 22 11:25:18 n0014 attrd: [25841]: notice: crm_cluster_connect: Connecting to cluster infrastructure: corosync
>>> Mar 22 11:25:18 n0014 attrd: [25841]: ERROR: init_cpg_connection: Could not connect to the Cluster Process Group API: 2
>>> Mar 22 11:25:18 n0014 attrd: [25841]: ERROR: main: HA Signon failed
>>> Mar 22 11:25:18 n0014 attrd: [25841]: info: main: Cluster connection active
>>> Mar 22 11:25:18 n0014 attrd: [25841]: info: main: Accepting attribute updates
>>> Mar 22 11:25:18 n0014 attrd: [25841]: ERROR: main: Aborting startup
>>> Mar 22 11:25:18 n0014 pengine: [25842]: info: Invoked: /usr/lib64/heartbeat/pengine
>>> Mar 22 11:25:18 n0014 pengine: [25842]: info: crm_log_init_worker: Changed active directory to /var/lib/heartbeat/cores/hacluster
>>> Mar 22 11:25:18 n0014 pengine: [25842]: debug: main: Checking for old instances of pengine
>>> Mar 22 11:25:18 n0014 pengine: [25842]: debug: init_client_ipc_comms_nodispatch: Attempting to talk on: /var/run/crm/pengine
>>
>> That "/var/run/crm" directory is available and owned by
>> hacluster.haclient ... and writable by at least the hacluster user?
>>
>> Regards,
>> Andreas
>>
>> --
>> Need help with Pacemaker?
>> http://www.hastexo.com/now
>>
>>> Mar 22 11:25:18 n0014 pacemakerd: [25834]: ERROR: pcmk_child_exit: Child process attrd exited (pid=25841, rc=100)
>>> Mar 22 11:25:18 n0014 pacemakerd: [25834]: notice: pcmk_child_exit: Child process attrd no longer wishes to be respawned
>>> Mar 22 11:25:18 n0014 pacemakerd: [25834]: info: update_node_processes: Node n0014.lustre now has process list: 00110312 (was 00111312)
>>> Mar 22 11:25:18 n0014 pengine: [25842]: debug: init_client_ipc_comms_nodispatch: Could not init comms on: /var/run/crm/pengine
>>> Mar 22 11:25:18 n0014 pengine: [25842]: debug: main: Init server comms
>>> Mar 22 11:25:18 n0014 pengine: [25842]: info: main: Starting pengine
>>> Mar 22 11:25:18 n0014 stonith-ng: [25838]: debug: init_cpg_connection: Adding fd=4 to mainloop
>>> Mar 22 11:25:18 n0014 stonith-ng: [25838]: info: init_ais_connection_once: Connection to 'corosync': established
>>> Mar 22 11:25:18 n0014 stonith-ng: [25838]: debug: crm_new_peer: Creating entry for node n0014.lustre/247988234
>>> Mar 22 11:25:18 n0014 stonith-ng: [25838]: info: crm_new_peer: Node n0014.lustre now has id: 247988234
>>> Mar 22 11:25:18 n0014 stonit

Re: [Pacemaker] OCF Resource agent promote question

[Rainer Brestan]

 

Hi Steve,

pgsql RA does the same, it compares the last_xlog_replay_location of all nodes for master promotion.

Doing a promote as a restart instead of promote command to conserve timeline id is also on configurable option (restart_on_promote) of the current RA.

And the RA is definitely capable of having more than two instances. It goes through the parameter node_list and doing its actions for every member in the node list.

Originally it might be planned only to have only one slave, but the current implementation does not have this limitation. It has code for sync replication of more than two nodes, when some of them fall back into async to not promote them.

 

Of course, i will share the extension with the community, when they are ready for use. And the feature of having more than two instances is not removed. I am not running more than two instances on one site, current usage is to have two instances on one site and having two sites and manage master by booth. But it also under discussion to have more than two instances on one site, just to have no availability interruption in case of one server down and the other promote with restart.

The implementation is nearly finished, then begins the stress tests of failure scenarios.

 

Rainer


Gesendet: Dienstag, 26. März 2013 um 11:55 Uhr
Von: "Steven Bambling" 
An: "The Pacemaker cluster resource manager" 
Betreff: Re: [Pacemaker] OCF Resource agent promote question


 

On Mar 26, 2013, at 6:32 AM, Rainer Brestan <rainer.bres...@gmx.net> wrote:
 



 

Hi Steve,

when Pacemaker does promotion, it has already selected a specific node to become master.

It is far too late in this state to try to update master scores.

 

But there is another problem with xlog in PostgreSQL.

 

According to some discussion on PostgreSQL mailing lists, not relevant xlog entries dont go into the xlog counter during redo and/or start. This is specially true for CHECKPOINT xlog records, where this situation can be easely reproduced.

This can lead to the situation, where the replication is up to date, but the slave shows an lower xlog value.

This issue was solved in 9.2.3, where wal receiver always counts the end of applied records.





 
We are currently testing with 9.2.3.  I'm using the functions http://www.databasesoup.com/2012/10/determining-furthest-ahead-replica.html along with tweaking a function to get the replay_lag in bytes to have a more accurate measurement.





 

There is also a second boring issue. The timeline change is replicated to the slaves, but they do not save it anywhere. In case slave starts up again and do not have access to the WAL archive, it cannot start any more. This was also addressed as patch in 9.2 branch, but i havent test if also fixed in 9.2.3.





After talking with one of the Postgres guys it was recommended that we look at an alternative solution to the built in trigger file that will make the master jump to a new timeline.  We are in place moving the recovery.conf to recovery.done via the resource agent and then restarting the the postgresql service on the "new" master so that it maintains the original timeline that the slaves will recognize.   




 

For data replication, no matter if PostgreSQL or any other database, you have always two choices of work.

- Data consistency is the top most priority. Dont go in operation, unless everything fine.

- Availability is the top most priority. Always try to have at least one running instance, even if data might not be latest.

 

The current pgsql RA does quite a good job for the first choice.

 

It currently has some limitations.

- After switchover, no matter of manual/automatic, it needs some work from maintenance personnel.

- Some failure scenarios of fault series lead to a non existing master without manual work.

- Geo-redundant replication with multi-site cluster ticket system (booth) does not work.

- If availability or unattended work is the priority, it cannot be used out of the box.

 

But it has a very good structure to be extended for other needs.

 

And this is what i currently implement.

Extend the RA to support both choices of work and prepare it for a multi-site cluster ticket system.





 
Would you be willing to share your extended RA?  Also do you run a cluster with more then 2 nodes ?

 

v/r

 

STEVE

 

 




 

Regards, Rainer


Gesendet: Dienstag, 26. März 2013 um 00:01 Uhr
Von: "Andreas Kurz" <andr...@hastexo.com>
An: pacemaker@oss.clusterlabs.org
Betreff: Re: [Pacemaker] OCF Resource agent promote question

Hi Steve,

On 2013-03-25 18:44, Steven Bambling wrote:
> All,
>
> I'm trying to work on a OCF resource agent that uses postgresql
> streaming replication. I'm running into a few issues that I hope might
> be answered or at least some pointers given to steer me in the right
> direction.

Why are you not using the existing pgsql RA? It is capable of doing
synchron

Re: [Pacemaker] OCF Resource agent promote question

[Rainer Brestan]

 

Hi Steve,

when Pacemaker does promotion, it has already selected a specific node to become master.

It is far too late in this state to try to update master scores.

 

But there is another problem with xlog in PostgreSQL.

 

According to some discussion on PostgreSQL mailing lists, not relevant xlog entries dont go into the xlog counter during redo and/or start. This is specially true for CHECKPOINT xlog records, where this situation can be easely reproduced.

This can lead to the situation, where the replication is up to date, but the slave shows an lower xlog value.

This issue was solved in 9.2.3, where wal receiver always counts the end of applied records.

 

There is also a second boring issue. The timeline change is replicated to the slaves, but they do not save it anywhere. In case slave starts up again and do not have access to the WAL archive, it cannot start any more. This was also addressed as patch in 9.2 branch, but i havent test if also fixed in 9.2.3.

 

For data replication, no matter if PostgreSQL or any other database, you have always two choices of work.

- Data consistency is the top most priority. Dont go in operation, unless everything fine.

- Availability is the top most priority. Always try to have at least one running instance, even if data might not be latest.

 

The current pgsql RA does quite a good job for the first choice.

 

It currently has some limitations.

- After switchover, no matter of manual/automatic, it needs some work from maintenance personnel.

- Some failure scenarios of fault series lead to a non existing master without manual work.

- Geo-redundant replication with multi-site cluster ticket system (booth) does not work.

- If availability or unattended work is the priority, it cannot be used out of the box.

 

But it has a very good structure to be extended for other needs.

 

And this is what i currently implement.

Extend the RA to support both choices of work and prepare it for a multi-site cluster ticket system.

 

Regards, Rainer


Gesendet: Dienstag, 26. März 2013 um 00:01 Uhr
Von: "Andreas Kurz" 
An: pacemaker@oss.clusterlabs.org
Betreff: Re: [Pacemaker] OCF Resource agent promote question

Hi Steve,

On 2013-03-25 18:44, Steven Bambling wrote:
> All,
>
> I'm trying to work on a OCF resource agent that uses postgresql
> streaming replication. I'm running into a few issues that I hope might
> be answered or at least some pointers given to steer me in the right
> direction.

Why are you not using the existing pgsql RA? It is capable of doing
synchronous and asynchronous replication and it is known to work fine.

Best regards,
Andreas

--
Need help with Pacemaker?
http://www.hastexo.com/now


>
> 1. A quick way of obtaining a list of "Online" nodes in the cluster
> that a resource will be able to migrate to. I've accomplished it with
> some grep and see but its not pretty or fast.
>
> # time pcs status | grep Online | sed -e "s/.*\[\(.*\)\]/\1/" | sed 's/ //'
> p1.example.net  p2.example.net
> 
>
> real0m2.797s
> user0m0.084s
> sys0m0.024s
>
> Once I get a list of active/online nodes in the cluster my thinking was
> to use PSQL to get the current xlog location and lag or each of the
> remaining nodes and compare them. If the node has a greater log
> position and/or less lag it will be given a greater master preference.
>
> 2. How to force a monitor/probe before a promote is run on ALL nodes to
> make sure that the master preference is up to date before
> migrating/failing over the resource.
> - I was thinking that maybe during the promote call it could get the log
> location and lag from each of the nodes via an psql call ( like above)
> and then force the resource to a specific node. Is there a way to do
> this and does it sound like a sane idea ?
>
>
> The start of my RA is located here suggestions and comments 100%
> welcome https://github.com/smbambling/pgsqlsr/blob/master/pgsqlsr
>
> v/r
>
> STEVE
>
>
> ___
> Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org
>




___
Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org




___
Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

[Pacemaker] Patch: Extend Tomcat RA with status regex

[Rainer Brestan]

The resource agent for tomcat is extended to allow the response of the status 
url to be checked against a regex.

The RA includes a new parameter "statusurlregex".

If this parameter is not present, it behaves as now.
If present, it checks against the regex.

Therefore, it is possible not just check tomcat returning something, but also 
correct response.

The patch contains a little change of the monitor function.

The current monitor function first checks the existence of tomcat processes and 
return OCF_NOT_RUNNING if not.
The next check is the response of the statusurl. If tomcat does not return 
anything, the monitor function return OCF_NOT_RUNNING, which is improper.
On returning OCF_NOT_RUNNING, pacemake will start tomcat immediately and not 
call stop. This might cause situations running several tomcat processes and 
none of them is working.

The patched monitor function will return OCF_ERR_GENERIC if no or not correct 
response on statusurl call. This let pacemaker call stop and then start.

Rainer 
-- 
Empfehlen Sie GMX DSL Ihren Freunden und Bekannten und wir
belohnen Sie mit bis zu 50,- Euro! https://freundschaftswerbung.gmx.de


tomcat.diff
Description: Binary data
___
Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org

[Pacemaker] Change in meta clone-max result in resource restart everywhere

[Rainer Brestan]

When updating the meta attribute clone-max all instances of the clone are 
terminated and immediately restarted.

Following configuration (not symmetric cluster):
primitive resMux_gw ocf:heartbeat:Dummy op start interval="0" timeout="10" op 
stop interval="0" timeout="10" op monitor interval="10" timeout="3" 
on-fail="restart" start-delay="10" meta failure-timeout="15m" 
migration-threshold="3"
clone cloneMux_gw resMux_gw meta clone-max="2" target-role="Started" 
is-managed="true"
location locMux_gwmanagement1 cloneMux_gw 1000: management1

crm resource status cloneMux_gw shows
resource cloneMux_gw is running on: management1
which is correct, because there is location information only present for node 
management1.

When clone-max is now updated by
crm resource meta cloneMux_gw set clone-max 1
resMux_gw is immediately restarted on management1. I see in pacemaker log a 
stop call to the resource agent and after a few milliseconds a start.

My question, is there any reason for stopping all instances during update of 
clone-max ?
After update of clone-max in the above case, the same resources run on the same 
nodes as before.

Pacemaker version is 1.1.5

Thanks, Rainer
-- 
NEU: FreePhone 3-fach-Flat mit kostenlosem Smartphone!  

Jetzt informieren: http://mobile.1und1.de/?ac=OM.PW.PW003K20328T7073a

___
Pacemaker mailing list: Pacemaker@oss.clusterlabs.org
http://oss.clusterlabs.org/mailman/listinfo/pacemaker

Project Home: http://www.clusterlabs.org
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
Bugs: http://bugs.clusterlabs.org