Re: [ClusterLabs] Antw: Pacemaker 1.1.16 - Release Candidate 1
On Mon, 7 Nov 2016 12:39:32 -0600 Ken Gaillotwrote: > On 11/07/2016 12:03 PM, Jehan-Guillaume de Rorthais wrote: > > On Mon, 7 Nov 2016 09:31:20 -0600 > > Ken Gaillot wrote: > > > >> On 11/07/2016 03:47 AM, Klaus Wenninger wrote: > >>> On 11/07/2016 10:26 AM, Jehan-Guillaume de Rorthais wrote: > On Mon, 7 Nov 2016 10:12:04 +0100 > Klaus Wenninger wrote: > > > On 11/07/2016 08:41 AM, Ulrich Windl wrote: > > Ken Gaillot schrieb am 04.11.2016 um 22:37 in > > Nachricht > >> <27c2ca20-c52c-8fb4-a60f-5ae12f7ff...@redhat.com>: > >>> On 11/04/2016 02:29 AM, Ulrich Windl wrote: > >>> Ken Gaillot schrieb am 03.11.2016 um 17:08 > >>> in > Nachricht > <8af2ff98-05fd-a2c7-f670-58d0ff68e...@redhat.com>: > ... > >>> Another possible use would be for a cron that needs to know whether a > >>> particular resource is running, and an attribute query is quicker and > >>> easier than something like parsing crm_mon output or probing the > >>> service. > >> crm_mon reads parts of the CIB; crm_attribute also does, I guess, so > >> besides of lacking options and inefficient implementation, why should > >> one be faster than the other? > > attrd_updater doesn't go for the CIB > AFAIK, attrd_updater actually goes to the CIB, unless you set "--private" > since 1.1.13: > https://github.com/ClusterLabs/pacemaker/blob/master/ChangeLog#L177 > >>> That prevents values being stored in the CIB. attrd_updater should > >>> always talk to attrd as I got it ... > >> > >> It's a bit confusing: Both crm_attribute and attrd_updater will > >> ultimately affect both attrd and the CIB in most cases, but *how* they > >> do so is different. crm_attribute modifies the CIB, and lets attrd pick > >> up the change from there; attrd_updater notifies attrd, and lets attrd > >> modify the CIB. > >> > >> The difference is subtle. > >> > >> With corosync 2, attrd only modifies "transient" node attributes (which > >> stay in effect till the next reboot), not "permanent" attributes. > > > > So why "--private" is not compatible with corosync 1.x as attrd_updater > > only set "transient" attributes anyway? > > Corosync 1 does not support certain reliability guarantees required by > the current attrd, so when building against the corosync 1 libraries, > pacemaker will install "legacy" attrd instead. The difference is mainly > that the current attrd can guarantee atomic updates to attribute values. > attrd_updater actually can set permanent attributes when used with > legacy attrd. OK, I understand now. > > How and where private attributes are stored? > > They are kept in memory only, in attrd. Of course, attrd is clustered, > so they are kept in sync across all nodes. OK, that was my guess. > >> So crm_attribute must be used if you want to set a permanent attribute. > >> crm_attribute also has the ability to modify cluster properties and > >> resource defaults, as well as node attributes. > >> > >> On the other hand, by contacting attrd directly, attrd_updater can > >> change an attribute's "dampening" (how often it is flushed to the CIB), > >> and it can (as mentioned above) set "private" attributes that are never > >> written to the CIB (and thus never cause the cluster to re-calculate > >> resource placement). > > > > Interesting, thank you for the clarification. > > > > As I understand it, it resumes to: > > > > crm_attribute -> CIB <-(poll/notify?) attrd > > attrd_updater -> attrd -> CIB > > Correct. On startup, attrd registers with CIB to be notified of all changes. > > > Just a quick question about this, is it possible to set a "dampening" high > > enough so attrd never flush it to the CIB (kind of private attribute too)? > > I'd expect that to work, if the dampening interval was higher than the > lifetime of the cluster being up. Interesting. > It's also possible to abuse attrd to create a kind of private attribute > by using a node name that doesn't exist and never will. :) This ability > is intentionally allowed, so you can set attributes for nodes that the > current partition isn't aware of, or nodes that are planned to be added > later, but only attributes for known nodes will be written to the CIB. Again, interesting. I'll do some test on my RA as I need clustered private attributes and was not able to get them under old stack (Debian < 8 or RHEL < 7). Thank you very much for your answers! Regards, ___ Users mailing list: Users@clusterlabs.org http://clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [ClusterLabs] Antw: Pacemaker 1.1.16 - Release Candidate 1
On 11/07/2016 12:03 PM, Jehan-Guillaume de Rorthais wrote: > On Mon, 7 Nov 2016 09:31:20 -0600 > Ken Gaillotwrote: > >> On 11/07/2016 03:47 AM, Klaus Wenninger wrote: >>> On 11/07/2016 10:26 AM, Jehan-Guillaume de Rorthais wrote: On Mon, 7 Nov 2016 10:12:04 +0100 Klaus Wenninger wrote: > On 11/07/2016 08:41 AM, Ulrich Windl wrote: > Ken Gaillot schrieb am 04.11.2016 um 22:37 in > Nachricht >> <27c2ca20-c52c-8fb4-a60f-5ae12f7ff...@redhat.com>: >>> On 11/04/2016 02:29 AM, Ulrich Windl wrote: >>> Ken Gaillot schrieb am 03.11.2016 um 17:08 >>> in Nachricht <8af2ff98-05fd-a2c7-f670-58d0ff68e...@redhat.com>: ... >>> Another possible use would be for a cron that needs to know whether a >>> particular resource is running, and an attribute query is quicker and >>> easier than something like parsing crm_mon output or probing the >>> service. >> crm_mon reads parts of the CIB; crm_attribute also does, I guess, so >> besides of lacking options and inefficient implementation, why should one >> be faster than the other? > attrd_updater doesn't go for the CIB AFAIK, attrd_updater actually goes to the CIB, unless you set "--private" since 1.1.13: https://github.com/ClusterLabs/pacemaker/blob/master/ChangeLog#L177 >>> That prevents values being stored in the CIB. attrd_updater should >>> always talk to attrd as I got it ... >> >> It's a bit confusing: Both crm_attribute and attrd_updater will >> ultimately affect both attrd and the CIB in most cases, but *how* they >> do so is different. crm_attribute modifies the CIB, and lets attrd pick >> up the change from there; attrd_updater notifies attrd, and lets attrd >> modify the CIB. >> >> The difference is subtle. >> >> With corosync 2, attrd only modifies "transient" node attributes (which >> stay in effect till the next reboot), not "permanent" attributes. > > So why "--private" is not compatible with corosync 1.x as attrd_updater only > set > "transient" attributes anyway? Corosync 1 does not support certain reliability guarantees required by the current attrd, so when building against the corosync 1 libraries, pacemaker will install "legacy" attrd instead. The difference is mainly that the current attrd can guarantee atomic updates to attribute values. attrd_updater actually can set permanent attributes when used with legacy attrd. > How and where private attributes are stored? They are kept in memory only, in attrd. Of course, attrd is clustered, so they are kept in sync across all nodes. >> So crm_attribute must be used if you want to set a permanent attribute. >> crm_attribute also has the ability to modify cluster properties and >> resource defaults, as well as node attributes. >> >> On the other hand, by contacting attrd directly, attrd_updater can >> change an attribute's "dampening" (how often it is flushed to the CIB), >> and it can (as mentioned above) set "private" attributes that are never >> written to the CIB (and thus never cause the cluster to re-calculate >> resource placement). > > Interesting, thank you for the clarification. > > As I understand it, it resumes to: > > crm_attribute -> CIB <-(poll/notify?) attrd > attrd_updater -> attrd -> CIB Correct. On startup, attrd registers with CIB to be notified of all changes. > Just a quick question about this, is it possible to set a "dampening" high > enough so attrd never flush it to the CIB (kind of private attribute too)? I'd expect that to work, if the dampening interval was higher than the lifetime of the cluster being up. It's also possible to abuse attrd to create a kind of private attribute by using a node name that doesn't exist and never will. :) This ability is intentionally allowed, so you can set attributes for nodes that the current partition isn't aware of, or nodes that are planned to be added later, but only attributes for known nodes will be written to the CIB. ___ Users mailing list: Users@clusterlabs.org http://clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [ClusterLabs] Antw: Pacemaker 1.1.16 - Release Candidate 1
On Mon, 7 Nov 2016 09:31:20 -0600 Ken Gaillotwrote: > On 11/07/2016 03:47 AM, Klaus Wenninger wrote: > > On 11/07/2016 10:26 AM, Jehan-Guillaume de Rorthais wrote: > >> On Mon, 7 Nov 2016 10:12:04 +0100 > >> Klaus Wenninger wrote: > >> > >>> On 11/07/2016 08:41 AM, Ulrich Windl wrote: > >>> Ken Gaillot schrieb am 04.11.2016 um 22:37 in > >>> Nachricht > <27c2ca20-c52c-8fb4-a60f-5ae12f7ff...@redhat.com>: > > On 11/04/2016 02:29 AM, Ulrich Windl wrote: > > Ken Gaillot schrieb am 03.11.2016 um 17:08 > > in > >> Nachricht > >> <8af2ff98-05fd-a2c7-f670-58d0ff68e...@redhat.com>: > >> ... > > Another possible use would be for a cron that needs to know whether a > > particular resource is running, and an attribute query is quicker and > > easier than something like parsing crm_mon output or probing the > > service. > crm_mon reads parts of the CIB; crm_attribute also does, I guess, so > besides of lacking options and inefficient implementation, why should one > be faster than the other? > >>> attrd_updater doesn't go for the CIB > >> AFAIK, attrd_updater actually goes to the CIB, unless you set "--private" > >> since 1.1.13: > >> https://github.com/ClusterLabs/pacemaker/blob/master/ChangeLog#L177 > > That prevents values being stored in the CIB. attrd_updater should > > always talk to attrd as I got it ... > > It's a bit confusing: Both crm_attribute and attrd_updater will > ultimately affect both attrd and the CIB in most cases, but *how* they > do so is different. crm_attribute modifies the CIB, and lets attrd pick > up the change from there; attrd_updater notifies attrd, and lets attrd > modify the CIB. > > The difference is subtle. > > With corosync 2, attrd only modifies "transient" node attributes (which > stay in effect till the next reboot), not "permanent" attributes. So why "--private" is not compatible with corosync 1.x as attrd_updater only set "transient" attributes anyway? How and where private attributes are stored? > So crm_attribute must be used if you want to set a permanent attribute. > crm_attribute also has the ability to modify cluster properties and > resource defaults, as well as node attributes. > > On the other hand, by contacting attrd directly, attrd_updater can > change an attribute's "dampening" (how often it is flushed to the CIB), > and it can (as mentioned above) set "private" attributes that are never > written to the CIB (and thus never cause the cluster to re-calculate > resource placement). Interesting, thank you for the clarification. As I understand it, it resumes to: crm_attribute -> CIB <-(poll/notify?) attrd attrd_updater -> attrd -> CIB Just a quick question about this, is it possible to set a "dampening" high enough so attrd never flush it to the CIB (kind of private attribute too)? Regards, ___ Users mailing list: Users@clusterlabs.org http://clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [ClusterLabs] Antw: Pacemaker 1.1.16 - Release Candidate 1
On 11/07/2016 03:47 AM, Klaus Wenninger wrote: > On 11/07/2016 10:26 AM, Jehan-Guillaume de Rorthais wrote: >> On Mon, 7 Nov 2016 10:12:04 +0100 >> Klaus Wenningerwrote: >> >>> On 11/07/2016 08:41 AM, Ulrich Windl wrote: >>> Ken Gaillot schrieb am 04.11.2016 um 22:37 in >>> Nachricht <27c2ca20-c52c-8fb4-a60f-5ae12f7ff...@redhat.com>: > On 11/04/2016 02:29 AM, Ulrich Windl wrote: > Ken Gaillot schrieb am 03.11.2016 um 17:08 in >> Nachricht >> <8af2ff98-05fd-a2c7-f670-58d0ff68e...@redhat.com>: >> ... > Another possible use would be for a cron that needs to know whether a > particular resource is running, and an attribute query is quicker and > easier than something like parsing crm_mon output or probing the service. > crm_mon reads parts of the CIB; crm_attribute also does, I guess, so besides of lacking options and inefficient implementation, why should one be faster than the other? >>> attrd_updater doesn't go for the CIB >> AFAIK, attrd_updater actually goes to the CIB, unless you set "--private" >> since 1.1.13: >> https://github.com/ClusterLabs/pacemaker/blob/master/ChangeLog#L177 > That prevents values being stored in the CIB. attrd_updater should > always talk to attrd as I got it ... It's a bit confusing: Both crm_attribute and attrd_updater will ultimately affect both attrd and the CIB in most cases, but *how* they do so is different. crm_attribute modifies the CIB, and lets attrd pick up the change from there; attrd_updater notifies attrd, and lets attrd modify the CIB. The difference is subtle. With corosync 2, attrd only modifies "transient" node attributes (which stay in effect till the next reboot), not "permanent" attributes. So crm_attribute must be used if you want to set a permanent attribute. crm_attribute also has the ability to modify cluster properties and resource defaults, as well as node attributes. On the other hand, by contacting attrd directly, attrd_updater can change an attribute's "dampening" (how often it is flushed to the CIB), and it can (as mentioned above) set "private" attributes that are never written to the CIB (and thus never cause the cluster to re-calculate resource placement). ___ Users mailing list: Users@clusterlabs.org http://clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [ClusterLabs] Antw: Pacemaker 1.1.16 - Release Candidate 1
On 11/07/2016 10:26 AM, Jehan-Guillaume de Rorthais wrote: > On Mon, 7 Nov 2016 10:12:04 +0100 > Klaus Wenningerwrote: > >> On 11/07/2016 08:41 AM, Ulrich Windl wrote: >> Ken Gaillot schrieb am 04.11.2016 um 22:37 in >> Nachricht >>> <27c2ca20-c52c-8fb4-a60f-5ae12f7ff...@redhat.com>: On 11/04/2016 02:29 AM, Ulrich Windl wrote: Ken Gaillot schrieb am 03.11.2016 um 17:08 in > Nachricht > <8af2ff98-05fd-a2c7-f670-58d0ff68e...@redhat.com>: > ... Another possible use would be for a cron that needs to know whether a particular resource is running, and an attribute query is quicker and easier than something like parsing crm_mon output or probing the service. >>> crm_mon reads parts of the CIB; crm_attribute also does, I guess, so >>> besides of lacking options and inefficient implementation, why should one >>> be faster than the other? >> attrd_updater doesn't go for the CIB > AFAIK, attrd_updater actually goes to the CIB, unless you set "--private" > since 1.1.13: > https://github.com/ClusterLabs/pacemaker/blob/master/ChangeLog#L177 That prevents values being stored in the CIB. attrd_updater should always talk to attrd as I got it ... ___ Users mailing list: Users@clusterlabs.org http://clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [ClusterLabs] Antw: Pacemaker 1.1.16 - Release Candidate 1
On Mon, 7 Nov 2016 10:12:04 +0100 Klaus Wenningerwrote: > On 11/07/2016 08:41 AM, Ulrich Windl wrote: > Ken Gaillot schrieb am 04.11.2016 um 22:37 in > Nachricht > > <27c2ca20-c52c-8fb4-a60f-5ae12f7ff...@redhat.com>: > >> On 11/04/2016 02:29 AM, Ulrich Windl wrote: > >> Ken Gaillot schrieb am 03.11.2016 um 17:08 in > >>> Nachricht > >>> <8af2ff98-05fd-a2c7-f670-58d0ff68e...@redhat.com>: ... > >> Another possible use would be for a cron that needs to know whether a > >> particular resource is running, and an attribute query is quicker and > >> easier than something like parsing crm_mon output or probing the service. > > crm_mon reads parts of the CIB; crm_attribute also does, I guess, so > > besides of lacking options and inefficient implementation, why should one > > be faster than the other? > > attrd_updater doesn't go for the CIB AFAIK, attrd_updater actually goes to the CIB, unless you set "--private" since 1.1.13: https://github.com/ClusterLabs/pacemaker/blob/master/ChangeLog#L177 ___ Users mailing list: Users@clusterlabs.org http://clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [ClusterLabs] Antw: Pacemaker 1.1.16 - Release Candidate 1
On 11/07/2016 08:41 AM, Ulrich Windl wrote: Ken Gaillotschrieb am 04.11.2016 um 22:37 in Nachricht > <27c2ca20-c52c-8fb4-a60f-5ae12f7ff...@redhat.com>: >> On 11/04/2016 02:29 AM, Ulrich Windl wrote: >> Ken Gaillot schrieb am 03.11.2016 um 17:08 in >>> Nachricht >>> <8af2ff98-05fd-a2c7-f670-58d0ff68e...@redhat.com>: ClusterLabs is happy to announce the first release candidate for Pacemaker version 1.1.16. Source code is available at: https://github.com/ClusterLabs/pacemaker/releases/tag/Pacemaker-1.1.16-rc1 The most significant enhancements in this release are: * rsc-pattern may now be used instead of rsc in location constraints, to allow a single location constraint to apply to all resources whose names match a regular expression. Sed-like %0 - %9 backreferences let submatches be used in node attribute names in rules. * The new ocf:pacemaker:attribute resource agent sets a node attribute according to whether the resource is running or stopped. This may be useful in combination with attribute-based rules to model dependencies that simple constraints can't handle. >>> I don't quite understand this: Isn't the state of a resource in the CIB >> status >>> section anyway? If not, why not add it? So it would be readily available for >>> anyone (rules, constraints, etc.). >> This (hopefully) lets you model more complicated relationships. >> >> For example, someone recently asked whether they could make an ordering >> constraint apply only at "start-up" -- the first time resource A starts, >> it does some initialization that B needs, but once that's done, B can be >> independent of A. > Is "at start-up" before start of the resource, after start of the resource, > or parallel to the start of the resource ;-) > Probably a "hook" in the corresponding RA is the better approach, unless you > can really model all of the above. > >> For that case, you could group A with an ocf:pacemaker:attribute >> resource. The important part is that the attribute is not set if A has >> never run on a node. So, you can make a rule that B can run only where >> the attribute is set, regardless of the value -- even if A is later >> stopped, the attribute will still be set. > If a resource is not running on a node,, it is "stopped"; isn't it? > >> Another possible use would be for a cron that needs to know whether a >> particular resource is running, and an attribute query is quicker and >> easier than something like parsing crm_mon output or probing the service. > crm_mon reads parts of the CIB; crm_attribute also does, I guess, so besides > of lacking options and inefficient implementation, why should one be faster > than the other? attrd_updater doesn't go for the CIB > >> It's all theoretical at this point, and I'm not entirely sure those >> examples would be useful :) but I wanted to make the agent available for >> people to experiment with. > A good product manager should resist the attempt to provide any feature the > customers ask for, avoiding bloat-ware. That is to protect the customer from > their own bad decisions. In most cases there is a better, more universal > solution to the specific problem. > > * Pacemaker's existing "node health" feature allows resources to move off nodes that become unhealthy. Now, when using node-health-strategy=progressive, a new cluster property node-health-base will be used as the initial health score of newly joined nodes (defaulting to 0, which is the previous behavior). This allows cloned and multistate resource instances to start on a node even if it has some "yellow" health attributes. >>> So the node health is more or less a "node score"? I don't understand the >> last >>> sentence. Maybe give an example? >> Yes, node health is a score that's added when deciding where to place a >> resource. It does get complicated ... >> >> Node health monitoring is optional, and off by default. >> >> Node health attributes are set to red, yellow or green (outside >> pacemaker itself -- either by a resource agent, or some external >> process). As an example, let's say we have three node health attributes >> for CPU usage, CPU temperature, and SMART error count. >> >> With a progressive strategy, red and yellow are assigned some negative >> score, and green is 0. In our example, let's say yellow gets a -10 score. >> >> If any of our attributes are yellow, resources will avoid the node >> (unless they have higher positive scores from something like stickiness >> or a location constraint). >> > I understood so far. > >> Normally, this is what you want, but if your resources are cloned on all >> nodes, maybe you don't care if some attributes are yellow. In that case, >> you can set node-health-base=20, so even if two attributes are yellow, >> it won't prevent resources from running (20 + -10 + -10 = 0). > I don't understand
Re: [ClusterLabs] Antw: Pacemaker 1.1.16 - Release Candidate 1
On 11/04/2016 02:53 AM, Jan Pokorný wrote: > On 04/11/16 08:29 +0100, Ulrich Windl wrote: >> Ken Gaillotschrieb am 03.11.2016 um 17:08 in >> Nachricht <8af2ff98-05fd-a2c7-f670-58d0ff68e...@redhat.com>: >>> ClusterLabs is happy to announce the first release candidate for >>> Pacemaker version 1.1.16. Source code is available at: >>> >>> https://github.com/ClusterLabs/pacemaker/releases/tag/Pacemaker-1.1.16-rc1 >>> >>> The most significant enhancements in this release are: >>> >>> [...] >>> >>> * Previously, the OCF_RESKEY_CRM_meta_notify_active_* variables were not >>> properly passed to multistate resources with notification enabled. This >>> has been fixed. To help resource agents detect when the fix is >>> available, the CRM feature set has been incremented. (Whenever the >>> feature set changes, mixed-version clusters are supported only during >>> rolling upgrades -- nodes with an older version will not be allowed to >>> rejoin once they shut down.) >> >> Where can I find a description of current "CRM feature sets"? > > It's originally internal-only versioning for cluster to know which > node has the oldest software and hence is predestined to be DC > (in rolling update scenario). > > Ken recently mapped these versions (together with LRMD protocol > versions relevant in the context of pacemaker remote communication) > to proper release versions: http://clusterlabs.org/wiki/ReleaseCalendar There is also a description of how the cluster uses the CRM feature set in the new upgrade documentation: http://clusterlabs.org/doc/en-US/Pacemaker/1.1-pcs/html-single/Pacemaker_Explained/index.html#_rolling_node_by_node ___ Users mailing list: Users@clusterlabs.org http://clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [ClusterLabs] Antw: Pacemaker 1.1.16 - Release Candidate 1
On 04/11/16 08:29 +0100, Ulrich Windl wrote: > Ken Gaillotschrieb am 03.11.2016 um 17:08 in > Nachricht <8af2ff98-05fd-a2c7-f670-58d0ff68e...@redhat.com>: >> ClusterLabs is happy to announce the first release candidate for >> Pacemaker version 1.1.16. Source code is available at: >> >> https://github.com/ClusterLabs/pacemaker/releases/tag/Pacemaker-1.1.16-rc1 >> >> The most significant enhancements in this release are: >> >> [...] >> >> * Previously, the OCF_RESKEY_CRM_meta_notify_active_* variables were not >> properly passed to multistate resources with notification enabled. This >> has been fixed. To help resource agents detect when the fix is >> available, the CRM feature set has been incremented. (Whenever the >> feature set changes, mixed-version clusters are supported only during >> rolling upgrades -- nodes with an older version will not be allowed to >> rejoin once they shut down.) > > Where can I find a description of current "CRM feature sets"? It's originally internal-only versioning for cluster to know which node has the oldest software and hence is predestined to be DC (in rolling update scenario). Ken recently mapped these versions (together with LRMD protocol versions relevant in the context of pacemaker remote communication) to proper release versions: http://clusterlabs.org/wiki/ReleaseCalendar -- Jan (Poki) pgpzDT7kPIwPr.pgp Description: PGP signature ___ Users mailing list: Users@clusterlabs.org http://clusterlabs.org/mailman/listinfo/users Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org