Re: [Pacemaker] Load Balancing, Node Scores and Stickiness
On Fri, Oct 23, 2009 at 2:23 PM, Andrew Beekhof wrote: > On Fri, Oct 23, 2009 at 10:13 AM, Colin wrote: >> On Thu, Oct 22, 2009 at 3:51 PM, Johan Verrept wrote: >>> On Thu, 2009-10-22 at 15:10 +0200, Florian Haas wrote: On 10/22/2009 02:37 PM, Andrew Beekhof wrote: >> I wondered, does it happen dynamically? If one resource starts using a >> lot of resources, are the other migrated to other nodes? > > Not yet. > Such a feature is planned though. > > At the moment pacemaker purely goes on the number of services it has > allocated to the node. > Total/Available RAM, CPU, HDD, none of these things are yet taken into > account. Are there any plans on how this feature would look like in more detail? A daemon monitoring various performance indicators and updating node attributes accordingly? Couldn't that be done today, as a cloneable resource agent? >>> >>> I can see a few problems with such a feature if you wish to implement it >>> today. >>> First of all, you cannot really move services to less loaded nodes if >>> you cannot determine which resource causes which load. If you pick a >>> resource at random, you might move a "too heavy" resource to another >>> less loaded node and cause even more load on that node resulting in >>> something (else?) being moved back. It will create a pretty unstable >>> cluster under load. >>> I am also unsure if it would be wise to mix this directly into the >>> current node scoring. Load numbers will vary wildly and unless the >>> resulting attribute values are in some way stabilised over longer >>> periods, it will also cause unstability. (RRDTool?) >>> It might be possible, but it will be one hell of a complex RA :). A >>> daemon might be better, but both will require a LOT of configuration >>> just to differentiate the load of the different resources. >>> Or are you referring to missing features actually evaluating such information, as in, rather than saying "run this resource on a node with at load average of X or less", being able to say "run this resource on the node with the currently lowest load average"? >>> >>> How will that translate into repeatable node states? At this moment, if >>> you use a timed evaluation of the cluster state, resources should always >>> be assigned to the same nodes (at least, I've never seen it change >>> unless it was under direction of a time contraint). >>> >>> "run this resource on the node with the currently lowest load average" >>> is something that is very unlikely to ever return the same answer twice. >>> >>> Complex indeed! Someone is going to have a considerable amount of fun >>> with this :D >> >> Perhaps static load balancing could be implemented first, before >> trying to go dynamic. >> >> Suppose you could configure an arbitrary set of "measures" (I'd call >> them resources, but that word is already taken in this context), like >> an arbitrary set of keywords. For every such "measure", you can then >> configure (a) how much of it each node has, and (b) how much of it >> each cluster resource/service requires. The cluster can then use some >> heuristics to find a good distribution of resources (perfect could be >> too hard, this is squarely in NP-complete land; PostgreSQL uses a >> genetic algorithm for query optimisation...). > > Thats basically what we're going for this time around. > Maybe with enough experience we'll attempt the dynamic version below. > >> This is not as good as dynamic balancing, but still better than >> nothing, for example this could make sure that a resource with a >> tendency to do I/O runs on the same node with a resource that >> generally uses much CPU... The algorithms should be the same, the difference is whether you run them once on statically configured resource usage, or continuously on dynamically gathered resource consumption (or a weighted average of the two). Probably a good idea to test with static input first... Colin ___ Pacemaker mailing list Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker
Re: [Pacemaker] Load Balancing, Node Scores and Stickiness
On Fri, Oct 23, 2009 at 10:13 AM, Colin wrote: > On Thu, Oct 22, 2009 at 3:51 PM, Johan Verrept wrote: >> On Thu, 2009-10-22 at 15:10 +0200, Florian Haas wrote: >>> On 10/22/2009 02:37 PM, Andrew Beekhof wrote: >>> >> I wondered, does it happen dynamically? If one resource starts using a >>> >> lot of resources, are the other migrated to other nodes? >>> > >>> > Not yet. >>> > Such a feature is planned though. >>> > >>> > At the moment pacemaker purely goes on the number of services it has >>> > allocated to the node. >>> > Total/Available RAM, CPU, HDD, none of these things are yet taken into >>> > account. >>> >>> Are there any plans on how this feature would look like in more detail? >>> A daemon monitoring various performance indicators and updating node >>> attributes accordingly? Couldn't that be done today, as a cloneable >>> resource agent? >> >> I can see a few problems with such a feature if you wish to implement it >> today. >> First of all, you cannot really move services to less loaded nodes if >> you cannot determine which resource causes which load. If you pick a >> resource at random, you might move a "too heavy" resource to another >> less loaded node and cause even more load on that node resulting in >> something (else?) being moved back. It will create a pretty unstable >> cluster under load. >> I am also unsure if it would be wise to mix this directly into the >> current node scoring. Load numbers will vary wildly and unless the >> resulting attribute values are in some way stabilised over longer >> periods, it will also cause unstability. (RRDTool?) >> It might be possible, but it will be one hell of a complex RA :). A >> daemon might be better, but both will require a LOT of configuration >> just to differentiate the load of the different resources. >> >>> Or are you referring to missing features actually evaluating such >>> information, as in, rather than saying "run this resource on a node with >>> at load average of X or less", being able to say "run this resource on >>> the node with the currently lowest load average"? >> >> How will that translate into repeatable node states? At this moment, if >> you use a timed evaluation of the cluster state, resources should always >> be assigned to the same nodes (at least, I've never seen it change >> unless it was under direction of a time contraint). >> >> "run this resource on the node with the currently lowest load average" >> is something that is very unlikely to ever return the same answer twice. >> >> Complex indeed! Someone is going to have a considerable amount of fun >> with this :D > > Perhaps static load balancing could be implemented first, before > trying to go dynamic. > > Suppose you could configure an arbitrary set of "measures" (I'd call > them resources, but that word is already taken in this context), like > an arbitrary set of keywords. For every such "measure", you can then > configure (a) how much of it each node has, and (b) how much of it > each cluster resource/service requires. The cluster can then use some > heuristics to find a good distribution of resources (perfect could be > too hard, this is squarely in NP-complete land; PostgreSQL uses a > genetic algorithm for query optimisation...). Thats basically what we're going for this time around. Maybe with enough experience we'll attempt the dynamic version below. > This is not as good as dynamic balancing, but still better than > nothing, for example this could make sure that a resource with a > tendency to do I/O runs on the same node with a resource that > generally uses much CPU... > > Regards, Colin > > ___ > Pacemaker mailing list > Pacemaker@oss.clusterlabs.org > http://oss.clusterlabs.org/mailman/listinfo/pacemaker > ___ Pacemaker mailing list Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker
Re: [Pacemaker] Load Balancing, Node Scores and Stickiness
On Thu, Oct 22, 2009 at 3:10 PM, Florian Haas wrote: > On 10/22/2009 02:37 PM, Andrew Beekhof wrote: >>> I wondered, does it happen dynamically? If one resource starts using a >>> lot of resources, are the other migrated to other nodes? >> >> Not yet. >> Such a feature is planned though. >> >> At the moment pacemaker purely goes on the number of services it has >> allocated to the node. >> Total/Available RAM, CPU, HDD, none of these things are yet taken into >> account. > > Are there any plans on how this feature would look like in more detail? > A daemon monitoring various performance indicators and updating node > attributes accordingly? Couldn't that be done today, as a cloneable > resource agent? That would be one part, but you still need the logic in the PE to make use of the attributes. > Or are you referring to missing features actually evaluating such > information, as in, rather than saying "run this resource on a node with > at load average of X or less", being able to say "run this resource on > the node with the currently lowest load average"? right, that and "on a machine with 8Gb RAM and two VMs using 6Gb, don't try to put another one that needs 4Gb" ___ Pacemaker mailing list Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker
Re: [Pacemaker] Load Balancing, Node Scores and Stickiness
On Thu, Oct 22, 2009 at 3:51 PM, Johan Verrept wrote: > On Thu, 2009-10-22 at 15:10 +0200, Florian Haas wrote: >> On 10/22/2009 02:37 PM, Andrew Beekhof wrote: >> >> I wondered, does it happen dynamically? If one resource starts using a >> >> lot of resources, are the other migrated to other nodes? >> > >> > Not yet. >> > Such a feature is planned though. >> > >> > At the moment pacemaker purely goes on the number of services it has >> > allocated to the node. >> > Total/Available RAM, CPU, HDD, none of these things are yet taken into >> > account. >> >> Are there any plans on how this feature would look like in more detail? >> A daemon monitoring various performance indicators and updating node >> attributes accordingly? Couldn't that be done today, as a cloneable >> resource agent? > > I can see a few problems with such a feature if you wish to implement it > today. > First of all, you cannot really move services to less loaded nodes if > you cannot determine which resource causes which load. If you pick a > resource at random, you might move a "too heavy" resource to another > less loaded node and cause even more load on that node resulting in > something (else?) being moved back. It will create a pretty unstable > cluster under load. > I am also unsure if it would be wise to mix this directly into the > current node scoring. Load numbers will vary wildly and unless the > resulting attribute values are in some way stabilised over longer > periods, it will also cause unstability. (RRDTool?) > It might be possible, but it will be one hell of a complex RA :). A > daemon might be better, but both will require a LOT of configuration > just to differentiate the load of the different resources. > >> Or are you referring to missing features actually evaluating such >> information, as in, rather than saying "run this resource on a node with >> at load average of X or less", being able to say "run this resource on >> the node with the currently lowest load average"? > > How will that translate into repeatable node states? At this moment, if > you use a timed evaluation of the cluster state, resources should always > be assigned to the same nodes (at least, I've never seen it change > unless it was under direction of a time contraint). > > "run this resource on the node with the currently lowest load average" > is something that is very unlikely to ever return the same answer twice. > > Complex indeed! Someone is going to have a considerable amount of fun > with this :D Perhaps static load balancing could be implemented first, before trying to go dynamic. Suppose you could configure an arbitrary set of "measures" (I'd call them resources, but that word is already taken in this context), like an arbitrary set of keywords. For every such "measure", you can then configure (a) how much of it each node has, and (b) how much of it each cluster resource/service requires. The cluster can then use some heuristics to find a good distribution of resources (perfect could be too hard, this is squarely in NP-complete land; PostgreSQL uses a genetic algorithm for query optimisation...). This is not as good as dynamic balancing, but still better than nothing, for example this could make sure that a resource with a tendency to do I/O runs on the same node with a resource that generally uses much CPU... Regards, Colin ___ Pacemaker mailing list Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker
Re: [Pacemaker] Load Balancing, Node Scores and Stickiness
On Thu, 2009-10-22 at 15:10 +0200, Florian Haas wrote: > On 10/22/2009 02:37 PM, Andrew Beekhof wrote: > >> I wondered, does it happen dynamically? If one resource starts using a > >> lot of resources, are the other migrated to other nodes? > > > > Not yet. > > Such a feature is planned though. > > > > At the moment pacemaker purely goes on the number of services it has > > allocated to the node. > > Total/Available RAM, CPU, HDD, none of these things are yet taken into > > account. > > Are there any plans on how this feature would look like in more detail? > A daemon monitoring various performance indicators and updating node > attributes accordingly? Couldn't that be done today, as a cloneable > resource agent? I can see a few problems with such a feature if you wish to implement it today. First of all, you cannot really move services to less loaded nodes if you cannot determine which resource causes which load. If you pick a resource at random, you might move a "too heavy" resource to another less loaded node and cause even more load on that node resulting in something (else?) being moved back. It will create a pretty unstable cluster under load. I am also unsure if it would be wise to mix this directly into the current node scoring. Load numbers will vary wildly and unless the resulting attribute values are in some way stabilised over longer periods, it will also cause unstability. (RRDTool?) It might be possible, but it will be one hell of a complex RA :). A daemon might be better, but both will require a LOT of configuration just to differentiate the load of the different resources. > Or are you referring to missing features actually evaluating such > information, as in, rather than saying "run this resource on a node with > at load average of X or less", being able to say "run this resource on > the node with the currently lowest load average"? How will that translate into repeatable node states? At this moment, if you use a timed evaluation of the cluster state, resources should always be assigned to the same nodes (at least, I've never seen it change unless it was under direction of a time contraint). "run this resource on the node with the currently lowest load average" is something that is very unlikely to ever return the same answer twice. Complex indeed! Someone is going to have a considerable amount of fun with this :D J. ___ Pacemaker mailing list Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker
Re: [Pacemaker] Load Balancing, Node Scores and Stickiness
On 10/22/2009 02:37 PM, Andrew Beekhof wrote: >> I wondered, does it happen dynamically? If one resource starts using a >> lot of resources, are the other migrated to other nodes? > > Not yet. > Such a feature is planned though. > > At the moment pacemaker purely goes on the number of services it has > allocated to the node. > Total/Available RAM, CPU, HDD, none of these things are yet taken into > account. Are there any plans on how this feature would look like in more detail? A daemon monitoring various performance indicators and updating node attributes accordingly? Couldn't that be done today, as a cloneable resource agent? Or are you referring to missing features actually evaluating such information, as in, rather than saying "run this resource on a node with at load average of X or less", being able to say "run this resource on the node with the currently lowest load average"? Cheers, Florian signature.asc Description: OpenPGP digital signature ___ Pacemaker mailing list Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker
Re: [Pacemaker] Load Balancing, Node Scores and Stickiness
On Thu, Oct 22, 2009 at 1:58 PM, Johan Verrept wrote: > On Thu, 2009-10-22 at 12:37 +0200, Andrew Beekhof wrote: >> > That's easy enough to understand ... and I can't do any fine-tuning, >> > i.e. suppose that 4 nodes of my 10 node cluster fail, and then come up >> > again. If all resources have equal score on all nodes (without >> > counting stickiness), then (a) if stickiness is greater than 0 all >> > resources will stay put, and (b) if stickiness is 0 then the cluster >> > will move around resources to distribute them evenly? >> >> yep > > I wondered, does it happen dynamically? If one resource starts using a > lot of resources, are the other migrated to other nodes? Not yet. Such a feature is planned though. At the moment pacemaker purely goes on the number of services it has allocated to the node. Total/Available RAM, CPU, HDD, none of these things are yet taken into account. ___ Pacemaker mailing list Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker
Re: [Pacemaker] Load Balancing, Node Scores and Stickiness
On Thu, 2009-10-22 at 12:37 +0200, Andrew Beekhof wrote: > > That's easy enough to understand ... and I can't do any fine-tuning, > > i.e. suppose that 4 nodes of my 10 node cluster fail, and then come up > > again. If all resources have equal score on all nodes (without > > counting stickiness), then (a) if stickiness is greater than 0 all > > resources will stay put, and (b) if stickiness is 0 then the cluster > > will move around resources to distribute them evenly? > > yep I wondered, does it happen dynamically? If one resource starts using a lot of resources, are the other migrated to other nodes? J. ___ Pacemaker mailing list Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker
Re: [Pacemaker] Load Balancing, Node Scores and Stickiness
On Thu, Oct 22, 2009 at 11:55 AM, Colin wrote: > On Thu, Oct 22, 2009 at 11:40 AM, Andrew Beekhof wrote: >> On Wed, Oct 21, 2009 at 2:06 PM, Colin wrote: >>> >>> it seems from the documentation that Pacemaker has some inherent >>> tendency to to load-balancing, in the sense of, given equal choice, >>> not starting all resources on a single node... >>> >>> Now, I would like to be able to choose freely on a scale between >>> "always move everything to achieve good load balancing" and "don't >>> gratuitously migrate resources", and would therefore like to >>> understand the algorithms in Pacemaker better. >>> >>> Given a bunch of nodes and resources with a simple setup, i.e. no >>> resource colocation constraints, no groups etc., I understand that a >>> global score is calculated for each resource and each node, where >>> >>> score( resource, node ) = sum of all rsc_location constraints for that >>> resource and node + if the resource is already running on this node, >>> the stickiness (the stickiness of the resource or the global default >>> stickiness) >>> >>> How does the assignment of nodes proceed? My guess is something like: >>> >>> for every resource in order of resource priority >>> choose node with highest score for that resource >> >> if multiple nodes exist with the same score, pick one with the >> least allocated resources > > That's easy enough to understand ... and I can't do any fine-tuning, > i.e. suppose that 4 nodes of my 10 node cluster fail, and then come up > again. If all resources have equal score on all nodes (without > counting stickiness), then (a) if stickiness is greater than 0 all > resources will stay put, and (b) if stickiness is 0 then the cluster > will move around resources to distribute them evenly? yep and if you have any colocation constraints the stickiness is additive. so three colocated resources would prefer their existing host 3 times as much as a single, uncolocated one. > > (Come to think of it, any kind of fine-tuning taking into account > multiple resource weights and more complicated migration resistance > scores would probably be algorithmically really really horrible and > complex (in the complexity theoretic sense, too)...) > exactly :-) ___ Pacemaker mailing list Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker
Re: [Pacemaker] Load Balancing, Node Scores and Stickiness
On Thu, Oct 22, 2009 at 11:40 AM, Andrew Beekhof wrote: > On Wed, Oct 21, 2009 at 2:06 PM, Colin wrote: >> >> it seems from the documentation that Pacemaker has some inherent >> tendency to to load-balancing, in the sense of, given equal choice, >> not starting all resources on a single node... >> >> Now, I would like to be able to choose freely on a scale between >> "always move everything to achieve good load balancing" and "don't >> gratuitously migrate resources", and would therefore like to >> understand the algorithms in Pacemaker better. >> >> Given a bunch of nodes and resources with a simple setup, i.e. no >> resource colocation constraints, no groups etc., I understand that a >> global score is calculated for each resource and each node, where >> >> score( resource, node ) = sum of all rsc_location constraints for that >> resource and node + if the resource is already running on this node, >> the stickiness (the stickiness of the resource or the global default >> stickiness) >> >> How does the assignment of nodes proceed? My guess is something like: >> >> for every resource in order of resource priority >> choose node with highest score for that resource > > if multiple nodes exist with the same score, pick one with the > least allocated resources That's easy enough to understand ... and I can't do any fine-tuning, i.e. suppose that 4 nodes of my 10 node cluster fail, and then come up again. If all resources have equal score on all nodes (without counting stickiness), then (a) if stickiness is greater than 0 all resources will stay put, and (b) if stickiness is 0 then the cluster will move around resources to distribute them evenly? (Come to think of it, any kind of fine-tuning taking into account multiple resource weights and more complicated migration resistance scores would probably be algorithmically really really horrible and complex (in the complexity theoretic sense, too)...) Thanks, Colin ___ Pacemaker mailing list Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker
Re: [Pacemaker] Load Balancing, Node Scores and Stickiness
On Wed, Oct 21, 2009 at 2:06 PM, Colin wrote: > Hi All, > > it seems from the documentation that Pacemaker has some inherent > tendency to to load-balancing, in the sense of, given equal choice, > not starting all resources on a single node... > > Now, I would like to be able to choose freely on a scale between > "always move everything to achieve good load balancing" and "don't > gratuitously migrate resources", and would therefore like to > understand the algorithms in Pacemaker better. > > Given a bunch of nodes and resources with a simple setup, i.e. no > resource colocation constraints, no groups etc., I understand that a > global score is calculated for each resource and each node, where > > score( resource, node ) = sum of all rsc_location constraints for that > resource and node + if the resource is already running on this node, > the stickiness (the stickiness of the resource or the global default > stickiness) > > How does the assignment of nodes proceed? My guess is something like: > > for every resource in order of resource priority > choose node with highest score for that resource if multiple nodes exist with the same score, pick one with the least allocated resources > ??? somehow modify scores to prevent all resources on one node > > Are the details documented anywhere [except for the source]? Just the source so far ___ Pacemaker mailing list Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker
[Pacemaker] Load Balancing, Node Scores and Stickiness
Hi All, it seems from the documentation that Pacemaker has some inherent tendency to to load-balancing, in the sense of, given equal choice, not starting all resources on a single node... Now, I would like to be able to choose freely on a scale between "always move everything to achieve good load balancing" and "don't gratuitously migrate resources", and would therefore like to understand the algorithms in Pacemaker better. Given a bunch of nodes and resources with a simple setup, i.e. no resource colocation constraints, no groups etc., I understand that a global score is calculated for each resource and each node, where score( resource, node ) = sum of all rsc_location constraints for that resource and node + if the resource is already running on this node, the stickiness (the stickiness of the resource or the global default stickiness) How does the assignment of nodes proceed? My guess is something like: for every resource in order of resource priority choose node with highest score for that resource ??? somehow modify scores to prevent all resources on one node Are the details documented anywhere [except for the source]? Regards, Colin ___ Pacemaker mailing list Pacemaker@oss.clusterlabs.org http://oss.clusterlabs.org/mailman/listinfo/pacemaker
