Re: [Pacemaker] racing crm commands... last write wins?
On 12/04/2013, at 11:35 PM, Brian J. Murrell wrote: > On 13-04-10 07:02 PM, Andrew Beekhof wrote: >> >> On 11/04/2013, at 6:33 AM, Brian J. Murrell >> wrote: >>> >>> Does crm_resource suffer from this problem >> >> no > > Excellent. > > I was unable to find any comprehensive documentation on just how to > implement a pacemaker configuration solely with crm_resource and the > manpage for it doesn't seem to indicate any way to create resources, for > example. Right, creation (and any other modifications of the config) is via cibadmin. However that involves dealing with XML which most people have an aversion to, hence the common use of pcs and crmsh. > > Is it typical that when you don't want to use "crm" (or "pcs") and want > to rely on the crm_* group of commands, that you do so in conjunction > with cibadmin for things like creating resources, etc.? Yes. > It seems so, > but I just want to make sure there is not something I have not uncovered > yet. > > Cheers, > b. > > > ___ > Pacemaker mailing list: Pacemaker@oss.clusterlabs.org > http://oss.clusterlabs.org/mailman/listinfo/pacemaker > > Project Home: http://www.clusterlabs.org > Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf > Bugs: http://bugs.clusterlabs.org ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [Pacemaker] racing crm commands... last write wins?
On 13-04-10 07:02 PM, Andrew Beekhof wrote: > > On 11/04/2013, at 6:33 AM, Brian J. Murrell > wrote: >> >> Does crm_resource suffer from this problem > > no Excellent. I was unable to find any comprehensive documentation on just how to implement a pacemaker configuration solely with crm_resource and the manpage for it doesn't seem to indicate any way to create resources, for example. Is it typical that when you don't want to use "crm" (or "pcs") and want to rely on the crm_* group of commands, that you do so in conjunction with cibadmin for things like creating resources, etc.? It seems so, but I just want to make sure there is not something I have not uncovered yet. Cheers, b. signature.asc Description: OpenPGP digital signature ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [Pacemaker] racing crm commands... last write wins?
On 13-04-11 06:00 PM, Andrew Beekhof wrote: > > Actually, I think the semantics of -C are first-write-wins and any subsequent > attempts fail with -EEXSIST Indeed, you are correct. I think my point though was that it didn't matter in my case which writer wins since they should all be trying to write the same thing. But it's good to make the semantics clear for anyone who comes across this thread only to find that mine were in fact inaccurate. Cheers b. signature.asc Description: OpenPGP digital signature ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [Pacemaker] racing crm commands... last write wins?
On 11/04/2013, at 10:46 PM, Rasto Levrinc wrote: > On Thu, Apr 11, 2013 at 2:04 PM, Brian J. Murrell > wrote: >> On 13-04-11 07:37 AM, Brian J. Murrell wrote: >>> >>> In exploring all options, how about pcs? Does pcs' "resource create >>> ..." for example have the same read+modify+replace problem as crm >>> configure or does pcs resource create also only send proper fragments to >>> update just the part of the CIB it's operating on? >> >> Having just cracked pcs open, it doesn't seem to. It seems to create an >> XML string which it then applies to the CIB with: >> >> cibadmin -o resources -C -X $xml_resource_string > > It seems though that it replaces the whole CIB in some cases like m/s > resources. I didn't think it did ever. Chris? > Only LCMC never does. But it's probably probably not that > scriptable, (yet). :) > > Rasto > > ___ > Pacemaker mailing list: Pacemaker@oss.clusterlabs.org > http://oss.clusterlabs.org/mailman/listinfo/pacemaker > > Project Home: http://www.clusterlabs.org > Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf > Bugs: http://bugs.clusterlabs.org ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [Pacemaker] racing crm commands... last write wins?
On 11/04/2013, at 10:04 PM, Brian J. Murrell wrote: > On 13-04-11 07:37 AM, Brian J. Murrell wrote: >> >> In exploring all options, how about pcs? Does pcs' "resource create >> ..." for example have the same read+modify+replace problem as crm >> configure or does pcs resource create also only send proper fragments to >> update just the part of the CIB it's operating on? > > Having just cracked pcs open, it doesn't seem to. It seems to create an > XML string which it then applies to the CIB with: > > cibadmin -o resources -C -X $xml_resource_string > > IIUC, that is compatible with my use-case of multiple nodes in the > cluster creating resources concurrently and not having last-write-wins > problems, correct? Correct. > > Usually the multiple nodes are creating separate resources and in any > case where multiple nodes are creating the same resource (or adjusting > some global parameter), it's configuration is the same from all writers > so last-write-wins there doesn't matter, yes? Actually, I think the semantics of -C are first-write-wins and any subsequent attempts fail with -EEXSIST > > Cheers, > b. > > > ___ > Pacemaker mailing list: Pacemaker@oss.clusterlabs.org > http://oss.clusterlabs.org/mailman/listinfo/pacemaker > > Project Home: http://www.clusterlabs.org > Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf > Bugs: http://bugs.clusterlabs.org ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [Pacemaker] racing crm commands... last write wins?
On Thu, Apr 11, 2013 at 2:04 PM, Brian J. Murrell wrote: > On 13-04-11 07:37 AM, Brian J. Murrell wrote: >> >> In exploring all options, how about pcs? Does pcs' "resource create >> ..." for example have the same read+modify+replace problem as crm >> configure or does pcs resource create also only send proper fragments to >> update just the part of the CIB it's operating on? > > Having just cracked pcs open, it doesn't seem to. It seems to create an > XML string which it then applies to the CIB with: > > cibadmin -o resources -C -X $xml_resource_string It seems though that it replaces the whole CIB in some cases like m/s resources. Only LCMC never does. But it's probably probably not that scriptable, (yet). :) Rasto ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [Pacemaker] racing crm commands... last write wins?
On 13-04-11 07:37 AM, Brian J. Murrell wrote: > > In exploring all options, how about pcs? Does pcs' "resource create > ..." for example have the same read+modify+replace problem as crm > configure or does pcs resource create also only send proper fragments to > update just the part of the CIB it's operating on? Having just cracked pcs open, it doesn't seem to. It seems to create an XML string which it then applies to the CIB with: cibadmin -o resources -C -X $xml_resource_string IIUC, that is compatible with my use-case of multiple nodes in the cluster creating resources concurrently and not having last-write-wins problems, correct? Usually the multiple nodes are creating separate resources and in any case where multiple nodes are creating the same resource (or adjusting some global parameter), it's configuration is the same from all writers so last-write-wins there doesn't matter, yes? Cheers, b. signature.asc Description: OpenPGP digital signature ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [Pacemaker] racing crm commands... last write wins?
On 13-04-10 04:33 PM, Brian J. Murrell wrote: > > Does crm_resource suffer from this problem or does it properly only send > exactly the update to the CIB for the operation it's trying to achieve? In exploring all options, how about pcs? Does pcs' "resource create ..." for example have the same read+modify+replace problem as crm configure or does pcs resource create also only send proper fragments to update just the part of the CIB it's operating on? Cheers, b. signature.asc Description: OpenPGP digital signature ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [Pacemaker] racing crm commands... last write wins?
On 11/04/2013, at 6:33 AM, Brian J. Murrell wrote: > On 13-02-21 07:48 PM, Andrew Beekhof wrote: >> On Fri, Feb 22, 2013 at 5:18 AM, Brian J. Murrell >> wrote: >>> I wonder what happens in the case of two racing "crm" commands that want >>> to update the CIB (with non-overlapping/conflicting data). Is there any >>> locking to ensure that one crm cannot overwrite the other's change? >>> (i.e. second one to get there has to read in the new CIB before being >>> able to apply his change and send it back) Or if there is a situation >>> where one write stomps another's, >> >> If my information is up-to-date, yes. >> >> crmsh uses a read+modify+replace cycle, if B reads after A has read >> but before the replace has happened, data will be lost. > > Does crm_resource suffer from this problem no > or does it properly only send > exactly the update to the CIB for the operation it's trying to achieve? correct > > Cheers, > b. > > > ___ > Pacemaker mailing list: Pacemaker@oss.clusterlabs.org > http://oss.clusterlabs.org/mailman/listinfo/pacemaker > > Project Home: http://www.clusterlabs.org > Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf > Bugs: http://bugs.clusterlabs.org ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [Pacemaker] racing crm commands... last write wins?
On 2013-04-10T16:33:48, "Brian J. Murrell" wrote: > > crmsh uses a read+modify+replace cycle, if B reads after A has read > > but before the replace has happened, data will be lost. > Does crm_resource suffer from this problem or does it properly only send > exactly the update to the CIB for the operation it's trying to achieve? crm_resource doesn't suffer from this problem; by extension, the "crm resource" family of commands doesn't either, it's "only" the crm configure approach. Regards, Lars -- Architect Storage/HA SUSE LINUX Products GmbH, GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 21284 (AG Nürnberg) "Experience is the name everyone gives to their mistakes." -- Oscar Wilde ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [Pacemaker] racing crm commands... last write wins?
On 13-02-21 07:48 PM, Andrew Beekhof wrote: > On Fri, Feb 22, 2013 at 5:18 AM, Brian J. Murrell > wrote: >> I wonder what happens in the case of two racing "crm" commands that want >> to update the CIB (with non-overlapping/conflicting data). Is there any >> locking to ensure that one crm cannot overwrite the other's change? >> (i.e. second one to get there has to read in the new CIB before being >> able to apply his change and send it back) Or if there is a situation >> where one write stomps another's, > > If my information is up-to-date, yes. > > crmsh uses a read+modify+replace cycle, if B reads after A has read > but before the replace has happened, data will be lost. Does crm_resource suffer from this problem or does it properly only send exactly the update to the CIB for the operation it's trying to achieve? Cheers, b. signature.asc Description: OpenPGP digital signature ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [Pacemaker] racing crm commands... last write wins?
On Wed, Mar 20, 2013 at 10:40:10AM -0700, Bob Haxo wrote: > Regarding the replace triggering a DC election ... which is causing > issues with scripted installs ... how do I determine which crm commands > will NOT trigger this election? It seems like every "configure commit" could possible result in new election. But I'm not sure what does it depend on. > I need a way of avoiding this election > while installing. > > I'm finding that when repeating the scripted install with the same > commands, sometimes the DC election gets triggered and sometimes it does > not. With the same configuration updates? > With the DC election, these messages get logged, followed by the > whole xml version of the configuration. > > Call cib_replace failed (-62): Timer expired This is a problem connecting to the cib process, i.e. it's not related to a configuration update (as it cannot proceed anyway). > ERROR: 55: could not replace cib > INFO: 55: offending xml: > > Any suggestions for avoiding replacing rather than incrementally > modifying the configuration? Not right now. The configuration update process in crmsh needs to be modified. Thanks, Dejan > Thanks, > Bob Haxo > SGI > > > > On Mon, 2013-03-04 at 17:25 +0100, Lars Marowsky-Bree wrote: > > On 2013-03-04T17:14:28, Dejan Muhamedagic wrote: > > > > > > Thought so at the time, yes. And I do think it cleaned up a few things, > > > > we just need to improve it. The full CIB replace also seems to trigger > > > > an election ... > > > I think that used to happen in Heartbeat clusters but got fixed > > > in the meantime, the details are a bit foggy. > > > > No, if you look at the current logs on the DC, you'll also see this > > happening. I think it's the replace of the node section that triggers > > it. > > > > > > > > Then most of the logic in crmsh would remain unchanged (i.e., it'd still > > > > operate on whole CIBs only), but the way how it passes it on to > > > > Pacemaker would improve. I hope. > > > crmsh currently doesn't keep the original copy of the CIB. > > > > Right, but that should be a simple thing to add and prototype quickly. > > (Says he who isn't going to be the one doing it ;-) > > > > > Anyway, this approach is worth investigating. > > > > Thanks, let me know how it goes! > > > > > > Regards, > > Lars > > > > > ___ > Pacemaker mailing list: Pacemaker@oss.clusterlabs.org > http://oss.clusterlabs.org/mailman/listinfo/pacemaker > > Project Home: http://www.clusterlabs.org > Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf > Bugs: http://bugs.clusterlabs.org ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [Pacemaker] racing crm commands... last write wins?
Regarding the replace triggering a DC election ... which is causing issues with scripted installs ... how do I determine which crm commands will NOT trigger this election? I need a way of avoiding this election while installing. I'm finding that when repeating the scripted install with the same commands, sometimes the DC election gets triggered and sometimes it does not. With the DC election, these messages get logged, followed by the whole xml version of the configuration. Call cib_replace failed (-62): Timer expired ERROR: 55: could not replace cib INFO: 55: offending xml: Any suggestions for avoiding replacing rather than incrementally modifying the configuration? Thanks, Bob Haxo SGI On Mon, 2013-03-04 at 17:25 +0100, Lars Marowsky-Bree wrote: > On 2013-03-04T17:14:28, Dejan Muhamedagic wrote: > > > > Thought so at the time, yes. And I do think it cleaned up a few things, > > > we just need to improve it. The full CIB replace also seems to trigger > > > an election ... > > I think that used to happen in Heartbeat clusters but got fixed > > in the meantime, the details are a bit foggy. > > No, if you look at the current logs on the DC, you'll also see this > happening. I think it's the replace of the node section that triggers > it. > > > > > Then most of the logic in crmsh would remain unchanged (i.e., it'd still > > > operate on whole CIBs only), but the way how it passes it on to > > > Pacemaker would improve. I hope. > > crmsh currently doesn't keep the original copy of the CIB. > > Right, but that should be a simple thing to add and prototype quickly. > (Says he who isn't going to be the one doing it ;-) > > > Anyway, this approach is worth investigating. > > Thanks, let me know how it goes! > > > Regards, > Lars > ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [Pacemaker] racing crm commands... last write wins?
On 2013-03-04T17:14:28, Dejan Muhamedagic wrote: > > Thought so at the time, yes. And I do think it cleaned up a few things, > > we just need to improve it. The full CIB replace also seems to trigger > > an election ... > I think that used to happen in Heartbeat clusters but got fixed > in the meantime, the details are a bit foggy. No, if you look at the current logs on the DC, you'll also see this happening. I think it's the replace of the node section that triggers it. > > Then most of the logic in crmsh would remain unchanged (i.e., it'd still > > operate on whole CIBs only), but the way how it passes it on to > > Pacemaker would improve. I hope. > crmsh currently doesn't keep the original copy of the CIB. Right, but that should be a simple thing to add and prototype quickly. (Says he who isn't going to be the one doing it ;-) > Anyway, this approach is worth investigating. Thanks, let me know how it goes! Regards, Lars -- Architect Storage/HA SUSE LINUX Products GmbH, GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 21284 (AG Nürnberg) "Experience is the name everyone gives to their mistakes." -- Oscar Wilde ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [Pacemaker] racing crm commands... last write wins?
On Thu, Feb 28, 2013 at 07:34:41PM +0100, Lars Marowsky-Bree wrote: > On 2013-02-28T17:51:03, Dejan Muhamedagic wrote: > > > crmsh used to make modifications in chunks, which was a bit > > complex due to element dependencies, but it worked if I can > > recall correctly. Then it got replaced by a full CIB replace > > (everybody claimed that it was the right thing to do). > > Thought so at the time, yes. And I do think it cleaned up a few things, > we just need to improve it. The full CIB replace also seems to trigger > an election ... I think that used to happen in Heartbeat clusters but got fixed in the meantime, the details are a bit foggy. > > Of course, cmrsh know which elements got modified, which are new, and > > which should be deleted. But I'm not sure cibadmin can accept a diff > > of such a kind. Perhaps doing a shadow apply instead of cibadmin -R > > would help? > > cibadmin -P. > > If crm shell kept the XML it started out from around somewhere (easy for > an uncommitted shadow CIB, in any case), it could use crm_diff --filter > new to generate what has been changed in the current session and apply > that. > > I *think* the CIB client side is supposed to do that already when > cibadmin -R is called; but that doesn't allow merging of updates that > have happened since and has this race. > > Then most of the logic in crmsh would remain unchanged (i.e., it'd still > operate on whole CIBs only), but the way how it passes it on to > Pacemaker would improve. I hope. crmsh currently doesn't keep the original copy of the CIB. Anyway, this approach is worth investigating. Thanks, Dejan > Regards, > Lars > > -- > Architect Storage/HA > SUSE LINUX Products GmbH, GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, > HRB 21284 (AG Nürnberg) > "Experience is the name everyone gives to their mistakes." -- Oscar Wilde > > > ___ > Pacemaker mailing list: Pacemaker@oss.clusterlabs.org > http://oss.clusterlabs.org/mailman/listinfo/pacemaker > > Project Home: http://www.clusterlabs.org > Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf > Bugs: http://bugs.clusterlabs.org ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [Pacemaker] racing crm commands... last write wins?
On 2013-02-28T17:51:03, Dejan Muhamedagic wrote: > crmsh used to make modifications in chunks, which was a bit > complex due to element dependencies, but it worked if I can > recall correctly. Then it got replaced by a full CIB replace > (everybody claimed that it was the right thing to do). Thought so at the time, yes. And I do think it cleaned up a few things, we just need to improve it. The full CIB replace also seems to trigger an election ... > Of course, cmrsh know which elements got modified, which are new, and > which should be deleted. But I'm not sure cibadmin can accept a diff > of such a kind. Perhaps doing a shadow apply instead of cibadmin -R > would help? cibadmin -P. If crm shell kept the XML it started out from around somewhere (easy for an uncommitted shadow CIB, in any case), it could use crm_diff --filter new to generate what has been changed in the current session and apply that. I *think* the CIB client side is supposed to do that already when cibadmin -R is called; but that doesn't allow merging of updates that have happened since and has this race. Then most of the logic in crmsh would remain unchanged (i.e., it'd still operate on whole CIBs only), but the way how it passes it on to Pacemaker would improve. I hope. Regards, Lars -- Architect Storage/HA SUSE LINUX Products GmbH, GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 21284 (AG Nürnberg) "Experience is the name everyone gives to their mistakes." -- Oscar Wilde ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [Pacemaker] racing crm commands... last write wins?
On Thu, Feb 28, 2013 at 11:03:02AM +0100, Lars Marowsky-Bree wrote: > On 2013-02-25T12:20:13, "Brian J. Murrell" wrote: > > > > Perhaps there's a way to improve this. > > > > Well, the CIB is shared resource. Shared resources need to be locked > > against these sort of racy updates. Is there no locking of any kind at > > any level of CIB modifying operations? > > No. There's no CIB lock. > > However, the CIB on the DC serializes all updates. > > > i.e. does even cibadmin suffer from these last-write wins races with no > > option or opportunity to lock the CIB? > > Yes, if you use replace. If you instead apply a diff (incremental > modification etc), those will be merged. (If they don't conflict at the > XML level, obviously. And it doesn't protect against admins making > contradictory changes to the configuration, but that's a different > issue.) > > I'm not sure if crm shell could be modified to instead submit a diff for > the changes it made, or if that solved the problem. crmsh used to make modifications in chunks, which was a bit complex due to element dependencies, but it worked if I can recall correctly. Then it got replaced by a full CIB replace (everybody claimed that it was the right thing to do). Of course, cmrsh know which elements got modified, which are new, and which should be deleted. But I'm not sure cibadmin can accept a diff of such a kind. Perhaps doing a shadow apply instead of cibadmin -R would help? Thanks, Dejan ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [Pacemaker] racing crm commands... last write wins?
On 2013-02-25T12:20:13, "Brian J. Murrell" wrote: > > Perhaps there's a way to improve this. > > Well, the CIB is shared resource. Shared resources need to be locked > against these sort of racy updates. Is there no locking of any kind at > any level of CIB modifying operations? No. There's no CIB lock. However, the CIB on the DC serializes all updates. > i.e. does even cibadmin suffer from these last-write wins races with no > option or opportunity to lock the CIB? Yes, if you use replace. If you instead apply a diff (incremental modification etc), those will be merged. (If they don't conflict at the XML level, obviously. And it doesn't protect against admins making contradictory changes to the configuration, but that's a different issue.) I'm not sure if crm shell could be modified to instead submit a diff for the changes it made, or if that solved the problem. Regards, Lars -- Architect Storage/HA SUSE LINUX Products GmbH, GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 21284 (AG Nürnberg) "Experience is the name everyone gives to their mistakes." -- Oscar Wilde ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [Pacemaker] racing crm commands... last write wins?
On Mon, Feb 25, 2013 at 12:20:13PM -0500, Brian J. Murrell wrote: > On 13-02-25 10:30 AM, Dejan Muhamedagic wrote: > > > > Before doing replace, crmsh queries the CIB and checks if the > > epoch was modified in the meantime. > > But doesn't take out a lock of any sort to prevent an update in the > meanwhile, right? No. > > Those operations are not > > atomic, though. > > Indeed. > > > Perhaps there's a way to improve this. > > Well, the CIB is shared resource. Shared resources need to be locked > against these sort of racy updates. Is there no locking of any kind at > any level of CIB modifying operations? No. > i.e. does even cibadmin suffer from these last-write wins races with no > option or opportunity to lock the CIB? > > > crm node/crm resource invoke crm_attribute or other crm_ tools. > > Yes, that would be my expectation. But somewhere, something has to > implement locking of the shared resource, yes? > > > You should file a bugzilla then. > > Indeed. If there is no locking available anywhere, a ticket most > definitely needs filing. > > Cheers, > b. Thanks, Dejan > ___ > Pacemaker mailing list: Pacemaker@oss.clusterlabs.org > http://oss.clusterlabs.org/mailman/listinfo/pacemaker > > Project Home: http://www.clusterlabs.org > Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf > Bugs: http://bugs.clusterlabs.org ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [Pacemaker] racing crm commands... last write wins?
On 13-02-25 10:30 AM, Dejan Muhamedagic wrote: > > Before doing replace, crmsh queries the CIB and checks if the > epoch was modified in the meantime. But doesn't take out a lock of any sort to prevent an update in the meanwhile, right? > Those operations are not > atomic, though. Indeed. > Perhaps there's a way to improve this. Well, the CIB is shared resource. Shared resources need to be locked against these sort of racy updates. Is there no locking of any kind at any level of CIB modifying operations? i.e. does even cibadmin suffer from these last-write wins races with no option or opportunity to lock the CIB? > crm node/crm resource invoke crm_attribute or other crm_ tools. Yes, that would be my expectation. But somewhere, something has to implement locking of the shared resource, yes? > You should file a bugzilla then. Indeed. If there is no locking available anywhere, a ticket most definitely needs filing. Cheers, b. signature.asc Description: OpenPGP digital signature ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [Pacemaker] racing crm commands... last write wins?
On Fri, Feb 22, 2013 at 11:48:58AM +1100, Andrew Beekhof wrote: > On Fri, Feb 22, 2013 at 5:18 AM, Brian J. Murrell > wrote: > > I wonder what happens in the case of two racing "crm" commands that want > > to update the CIB (with non-overlapping/conflicting data). Is there any > > locking to ensure that one crm cannot overwrite the other's change? > > (i.e. second one to get there has to read in the new CIB before being > > able to apply his change and send it back) Or if there is a situation > > where one write stomps another's, > > If my information is up-to-date, yes. > > crmsh uses a read+modify+replace cycle, if B reads after A has read > but before the replace has happened, data will be lost. Before doing replace, crmsh queries the CIB and checks if the epoch was modified in the meantime. Those operations are not atomic, though. Perhaps there's a way to improve this. However ... > > is there at least some kind of > > notification? > > i dont think so > > > Ultimately, it would be bad for two nodes for example to issue: > > > > # crm node attribute $(uname -n) set name value > > > > at the same time and have one of those updates lost. crm node/crm resource invoke crm_attribute or other crm_ tools. > > But that's what I think I am seeing. You should file a bugzilla then. Thanks, Dejan > > Cheers, > > b. > > > > > > ___ > > Pacemaker mailing list: Pacemaker@oss.clusterlabs.org > > http://oss.clusterlabs.org/mailman/listinfo/pacemaker > > > > Project Home: http://www.clusterlabs.org > > Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf > > Bugs: http://bugs.clusterlabs.org > > > > ___ > Pacemaker mailing list: Pacemaker@oss.clusterlabs.org > http://oss.clusterlabs.org/mailman/listinfo/pacemaker > > Project Home: http://www.clusterlabs.org > Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf > Bugs: http://bugs.clusterlabs.org ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
Re: [Pacemaker] racing crm commands... last write wins?
On Fri, Feb 22, 2013 at 5:18 AM, Brian J. Murrell wrote: > I wonder what happens in the case of two racing "crm" commands that want > to update the CIB (with non-overlapping/conflicting data). Is there any > locking to ensure that one crm cannot overwrite the other's change? > (i.e. second one to get there has to read in the new CIB before being > able to apply his change and send it back) Or if there is a situation > where one write stomps another's, If my information is up-to-date, yes. crmsh uses a read+modify+replace cycle, if B reads after A has read but before the replace has happened, data will be lost. > is there at least some kind of > notification? i dont think so > Ultimately, it would be bad for two nodes for example to issue: > > # crm node attribute $(uname -n) set name value > > at the same time and have one of those updates lost. > > But that's what I think I am seeing. > > Cheers, > b. > > > ___ > Pacemaker mailing list: Pacemaker@oss.clusterlabs.org > http://oss.clusterlabs.org/mailman/listinfo/pacemaker > > Project Home: http://www.clusterlabs.org > Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf > Bugs: http://bugs.clusterlabs.org > ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
[Pacemaker] racing crm commands... last write wins?
I wonder what happens in the case of two racing "crm" commands that want to update the CIB (with non-overlapping/conflicting data). Is there any locking to ensure that one crm cannot overwrite the other's change? (i.e. second one to get there has to read in the new CIB before being able to apply his change and send it back) Or if there is a situation where one write stomps another's, is there at least some kind of notification? Ultimately, it would be bad for two nodes for example to issue: # crm node attribute $(uname -n) set name value at the same time and have one of those updates lost. But that's what I think I am seeing. Cheers, b. signature.asc Description: OpenPGP digital signature ___ Pacemaker mailing list: Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker Project Home: http://www.clusterlabs.org Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf Bugs: http://bugs.clusterlabs.org
