Re: Provision multiple instances of slaves with NodeProvisioner

[Nikolay Borisenko]

Stephen, thank you for detailed answer! Your values works perfect, as I 
expected.

-- 
You received this message because you are subscribed to the Google Groups 
"Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jenkinsci-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/jenkinsci-dev/84d81edb-724d-48e9-9705-0db3e7e22667%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Provision multiple instances of slaves with NodeProvisioner

[Stephen Connolly]

You can get really aggressive provisioning with a fast clock decay rate and
a high margin and margin0

*THE FOLLOWING ARE AT YOUR OWN RISK*

-Dhudson.model.LoadStatistics.decay=0.2
-Dhudson.slaves.NodeProvisioner.MARGIN=50
-Dhudson.slaves.NodeProvisioner.MARGIN0=0.85

*BE WARNED* the above values are *VERY AGRESSIVE* and you will see
over-provisioning.

If you use these with a cloud provider where you pay for provisioned nodes
- e.g. EC2 - then *BE PREPARED FOR A BIG CREDIT CARD BILL*

If you want something less aggressive, perhaps start with

-Dhudson.model.LoadStatistics.decay=0.7
-Dhudson.slaves.NodeProvisioner.MARGIN=30
-Dhudson.slaves.NodeProvisioner.MARGIN0=0.6

Which may still result in over-provisioning, but should be less likely.

Over-provisioning is when you get nodes provisioned and then they sit there
idle because by the time they are provisioned the job they were provisioned
for has completed already.

With Jenkins versions prior to 1.607 you can even see many multiples of
nodes provisioned for the same job.

This issue is further compounded by *almost every single cloud
implementation* doing exactly the wrong thing and adding the node to
Jenkins from the Callable rather than leaving that as the
responsibility of NodeProvisioner (which is what the javadoc says they are
supposed to do) which means that the node is then counted as provisioned
but not available and hence another node needs to be provisioned.

On 4 June 2015 at 20:14, Nikolay Borisenko 
wrote:

> Guys, please advise what values for options I should set to be closer with
> the next feature: "Immediately provide new node if all nodes are busy".
>
> Thank you in advance.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Jenkins Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to jenkinsci-dev+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/jenkinsci-dev/18291b2a-3a79-46e5-b262-b9f957d8ad44%40googlegroups.com
> 
> .
>
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jenkinsci-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/jenkinsci-dev/CA%2BnPnMxNPELP4%2BSbRBwqnRvZ0XJgzG9VxOZdA4kC4RnOgBYjEQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Provision multiple instances of slaves with NodeProvisioner

[Nikolay Borisenko]

Guys, please advise what values for options I should set to be closer with 
the next feature: "Immediately provide new node if all nodes are busy".

Thank you in advance.

-- 
You received this message because you are subscribed to the Google Groups 
"Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jenkinsci-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/jenkinsci-dev/18291b2a-3a79-46e5-b262-b9f957d8ad44%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Provision multiple instances of slaves with NodeProvisioner

[Ivan Kalinin]

Thumbs up for pluggable strategies!

Also, I volunteer to help if reasonable guidance is provided.

On Wednesday, July 16, 2014 6:24:31 PM UTC+4, Stephen Connolly wrote:
>
> idle is based on a long term average.
>
> There are two schools of thought when ramping up build resources.
>
> You can optimize for maximum resource utilization
>
> You can optimize for shortest waiting time
>
> You cannot do both at the same time.
>
> Since it is not instantaneous to start up a build node you can sometimes 
> get a faster build if you just wait for one of the existing builds to 
> complete... as a result firing up a build node for every job in the queue 
> will end up wasting resources.
>
> Similarly the decommissioning strategy is likely to leave build nodes 
> hanging around for a minute or two it would be good if we have some work 
> they could do rather than just throw them away.
>
> The default strategy in Jenkins is to try to maintain the build queue 
> length at approx the number of build nodes. (Yes this may shock you) In 
> other words Jenkins wants there always to be a job ready for each node when 
> it is finished its current work.
>
> That strategy will give the maximum utilization of resources with a 
> secondary minimum waiting time.
>
> I intend in the near future (once I complete getting the scalability 
> framework open sourced) to make the strategy plugable, thus allowing people 
> to, e.g.
>
> Provision nodes while there is a demand for them - which will minimize 
> waiting time at the expense of wasting resources
>
> At present the trick is to play with MARGIN and MARGIN0 and the decay rate 
> so that you get nodes provisioned faster in response to the exponentially 
> weighted average number of idle nodes changing... but be warned, adjusting 
> these values will cause idle nodes to get provisioned depending on the 
> build load.
>
>
> On 16 July 2014 15:00, > wrote:
>
>> Hello,
>>
>> I'm writing a Jenkins Cloud plug-in that should work similar to EC2 
>> plug-in.
>> The sequence of events is as follows:
>> 1. Jenkins decides that there are not enough slaves to run jobs
>> 2. Jenkins access a private resource manager system to reserve and launch 
>> test set-up. The resource manager returns an IP address of the test set-up
>> 3. Jenkins launches slave on the returned IP.
>> 4. The job is executed on the slave
>> 5. Once the job is completed Jenkins kills the slave and updates the 
>> resouce manager that the set-up is free.
>>
>> I have extended Cloud, Slave, ComputerLauncher etc and I have a plugin 
>> that does all of the above.
>> However... Jenkins never provisions more than one slave from the resource 
>> manager cloud.
>>
>> Looking into the code of NodeProvisioner.update, it seems that the 
>> condition
>>
>> *if (idle < MARGIN || needSomeWhenNoneAtAll) {*Is never met once there 
>> is one slave in the pool.
>>
>> When there are no slaves, *needSomeWhenNoneAtAll == true* and the *if* 
>> clause is executed.
>> When there is one slave, *needSomeWhenNoneAtAll == false* and i*dle > 
>> MARGIN* so the *if* clause is not executed.
>>
>> I've found out that MARGIN is set to 0.1 by default with the following 
>> code:
>>
>> *private static final float MARGIN = 
>> Integer.getInteger(NodeProvisioner.class.getName() + ".MARGIN", 10) / 100f;*
>> I've changed this line to 
>> *private static final float MARGIN = 
>> Integer.getInteger(NodeProvisioner.class.getName() + ".MARGIN", 100) / 
>> 100f;*
>> And forced *idle < MARGIN* so the *if* clause always gets executed and 
>> Jenkins launches multiple slaves.
>>
>>
>> *My questions are:*1. Why, by default, *MARGIN* is set to 0.1 so *idle > 
>> MARGIN* and Jenkins does not launch more than one slave?
>> 2. How can I change this default behaviour so Jenkins does not wait at 
>> all? So as soon as it finds out there is awaiting job it will ask to 
>> provision new slave?
>>
>>
>>
>> *Thanks,Yoram *
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "Jenkins Developers" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to jenkinsci-de...@googlegroups.com .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jenkinsci-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Provision multiple instances of slaves with NodeProvisioner

[Stephen Connolly]

Just set the system properties on the JVM before you start Jenkins:

https://github.com/jenkinsci/jenkins/blob/8c4288ca59568e9c8d94022de7871e472f019a2f/core/src/main/java/hudson/slaves/NodeProvisioner.java#L373,
e.g.

-Dhudson.slaves.NodeProvisioner.MARGIN=...



On 19 July 2014 09:03,  wrote:

> Thanks for the detailed response.
>
> I will definitely wait for the updated mechanism, but meanwhile, I would
> like to have multiple instances even if sometimes they are wasted.
>
> So what would be the best way to manipulate MARGIN and MARGIN0 without
> compiling Jenkins (of-course)?
>
> Thanks,
> Yoram
>
>
> On Wednesday, July 16, 2014 5:24:31 PM UTC+3, Stephen Connolly wrote:
>
>> idle is based on a long term average.
>>
>> There are two schools of thought when ramping up build resources.
>>
>> You can optimize for maximum resource utilization
>>
>> You can optimize for shortest waiting time
>>
>> You cannot do both at the same time.
>>
>> Since it is not instantaneous to start up a build node you can sometimes
>> get a faster build if you just wait for one of the existing builds to
>> complete... as a result firing up a build node for every job in the queue
>> will end up wasting resources.
>>
>> Similarly the decommissioning strategy is likely to leave build nodes
>> hanging around for a minute or two it would be good if we have some work
>> they could do rather than just throw them away.
>>
>> The default strategy in Jenkins is to try to maintain the build queue
>> length at approx the number of build nodes. (Yes this may shock you) In
>> other words Jenkins wants there always to be a job ready for each node when
>> it is finished its current work.
>>
>> That strategy will give the maximum utilization of resources with a
>> secondary minimum waiting time.
>>
>> I intend in the near future (once I complete getting the scalability
>> framework open sourced) to make the strategy plugable, thus allowing people
>> to, e.g.
>>
>> Provision nodes while there is a demand for them - which will minimize
>> waiting time at the expense of wasting resources
>>
>> At present the trick is to play with MARGIN and MARGIN0 and the decay
>> rate so that you get nodes provisioned faster in response to the
>> exponentially weighted average number of idle nodes changing... but be
>> warned, adjusting these values will cause idle nodes to get provisioned
>> depending on the build load.
>>
>>
>> On 16 July 2014 15:00,  wrote:
>>
>>> Hello,
>>>
>>> I'm writing a Jenkins Cloud plug-in that should work similar to EC2
>>> plug-in.
>>> The sequence of events is as follows:
>>> 1. Jenkins decides that there are not enough slaves to run jobs
>>> 2. Jenkins access a private resource manager system to reserve and
>>> launch test set-up. The resource manager returns an IP address of the test
>>> set-up
>>> 3. Jenkins launches slave on the returned IP.
>>> 4. The job is executed on the slave
>>> 5. Once the job is completed Jenkins kills the slave and updates the
>>> resouce manager that the set-up is free.
>>>
>>> I have extended Cloud, Slave, ComputerLauncher etc and I have a plugin
>>> that does all of the above.
>>> However... Jenkins never provisions more than one slave from the
>>> resource manager cloud.
>>>
>>> Looking into the code of NodeProvisioner.update, it seems that the
>>> condition
>>>
>>> *if (idle < MARGIN || needSomeWhenNoneAtAll) {*Is never met once there
>>> is one slave in the pool.
>>>
>>> When there are no slaves, *needSomeWhenNoneAtAll == true* and the *if*
>>> clause is executed.
>>> When there is one slave, *needSomeWhenNoneAtAll == false* and i*dle >
>>> MARGIN* so the *if* clause is not executed.
>>>
>>> I've found out that MARGIN is set to 0.1 by default with the following
>>> code:
>>>
>>> *private static final float MARGIN =
>>> Integer.getInteger(NodeProvisioner.class.getName() + ".MARGIN", 10) / 100f;*
>>> I've changed this line to
>>> *private static final float MARGIN =
>>> Integer.getInteger(NodeProvisioner.class.getName() + ".MARGIN", 100) /
>>> 100f;*
>>> And forced *idle < MARGIN* so the *if* clause always gets executed and
>>> Jenkins launches multiple slaves.
>>>
>>>
>>> *My questions are:*1. Why, by default, *MARGIN* is set to 0.1 so *idle
>>> > MARGIN* and Jenkins does not launch more than one slave?
>>> 2. How can I change this default behaviour so Jenkins does not wait at
>>> all? So as soon as it finds out there is awaiting job it will ask to
>>> provision new slave?
>>>
>>>
>>>
>>> *Thanks,Yoram *
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Jenkins Developers" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to jenkinsci-de...@googlegroups.com.
>>>
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>>  --
> You received this message because you are subscribed to the Google Groups
> "Jenkins Developers" group.
> To unsubscribe from this group and stop receiving emails from it, se

Re: Provision multiple instances of slaves with NodeProvisioner

[yoram]

Thanks for the detailed response.

I will definitely wait for the updated mechanism, but meanwhile, I would 
like to have multiple instances even if sometimes they are wasted.

So what would be the best way to manipulate MARGIN and MARGIN0 without 
compiling Jenkins (of-course)?

Thanks,
Yoram


On Wednesday, July 16, 2014 5:24:31 PM UTC+3, Stephen Connolly wrote:
>
> idle is based on a long term average.
>
> There are two schools of thought when ramping up build resources.
>
> You can optimize for maximum resource utilization
>
> You can optimize for shortest waiting time
>
> You cannot do both at the same time.
>
> Since it is not instantaneous to start up a build node you can sometimes 
> get a faster build if you just wait for one of the existing builds to 
> complete... as a result firing up a build node for every job in the queue 
> will end up wasting resources.
>
> Similarly the decommissioning strategy is likely to leave build nodes 
> hanging around for a minute or two it would be good if we have some work 
> they could do rather than just throw them away.
>
> The default strategy in Jenkins is to try to maintain the build queue 
> length at approx the number of build nodes. (Yes this may shock you) In 
> other words Jenkins wants there always to be a job ready for each node when 
> it is finished its current work.
>
> That strategy will give the maximum utilization of resources with a 
> secondary minimum waiting time.
>
> I intend in the near future (once I complete getting the scalability 
> framework open sourced) to make the strategy plugable, thus allowing people 
> to, e.g.
>
> Provision nodes while there is a demand for them - which will minimize 
> waiting time at the expense of wasting resources
>
> At present the trick is to play with MARGIN and MARGIN0 and the decay rate 
> so that you get nodes provisioned faster in response to the exponentially 
> weighted average number of idle nodes changing... but be warned, adjusting 
> these values will cause idle nodes to get provisioned depending on the 
> build load.
>
>
> On 16 July 2014 15:00, > wrote:
>
>> Hello,
>>
>> I'm writing a Jenkins Cloud plug-in that should work similar to EC2 
>> plug-in.
>> The sequence of events is as follows:
>> 1. Jenkins decides that there are not enough slaves to run jobs
>> 2. Jenkins access a private resource manager system to reserve and launch 
>> test set-up. The resource manager returns an IP address of the test set-up
>> 3. Jenkins launches slave on the returned IP.
>> 4. The job is executed on the slave
>> 5. Once the job is completed Jenkins kills the slave and updates the 
>> resouce manager that the set-up is free.
>>
>> I have extended Cloud, Slave, ComputerLauncher etc and I have a plugin 
>> that does all of the above.
>> However... Jenkins never provisions more than one slave from the resource 
>> manager cloud.
>>
>> Looking into the code of NodeProvisioner.update, it seems that the 
>> condition
>>
>> *if (idle < MARGIN || needSomeWhenNoneAtAll) {*Is never met once there 
>> is one slave in the pool.
>>
>> When there are no slaves, *needSomeWhenNoneAtAll == true* and the *if* 
>> clause is executed.
>> When there is one slave, *needSomeWhenNoneAtAll == false* and i*dle > 
>> MARGIN* so the *if* clause is not executed.
>>
>> I've found out that MARGIN is set to 0.1 by default with the following 
>> code:
>>
>> *private static final float MARGIN = 
>> Integer.getInteger(NodeProvisioner.class.getName() + ".MARGIN", 10) / 100f;*
>> I've changed this line to 
>> *private static final float MARGIN = 
>> Integer.getInteger(NodeProvisioner.class.getName() + ".MARGIN", 100) / 
>> 100f;*
>> And forced *idle < MARGIN* so the *if* clause always gets executed and 
>> Jenkins launches multiple slaves.
>>
>>
>> *My questions are:*1. Why, by default, *MARGIN* is set to 0.1 so *idle > 
>> MARGIN* and Jenkins does not launch more than one slave?
>> 2. How can I change this default behaviour so Jenkins does not wait at 
>> all? So as soon as it finds out there is awaiting job it will ask to 
>> provision new slave?
>>
>>
>>
>> *Thanks,Yoram *
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "Jenkins Developers" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to jenkinsci-de...@googlegroups.com .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jenkinsci-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Provision multiple instances of slaves with NodeProvisioner

[Stephen Connolly]

idle is based on a long term average.

There are two schools of thought when ramping up build resources.

You can optimize for maximum resource utilization

You can optimize for shortest waiting time

You cannot do both at the same time.

Since it is not instantaneous to start up a build node you can sometimes
get a faster build if you just wait for one of the existing builds to
complete... as a result firing up a build node for every job in the queue
will end up wasting resources.

Similarly the decommissioning strategy is likely to leave build nodes
hanging around for a minute or two it would be good if we have some work
they could do rather than just throw them away.

The default strategy in Jenkins is to try to maintain the build queue
length at approx the number of build nodes. (Yes this may shock you) In
other words Jenkins wants there always to be a job ready for each node when
it is finished its current work.

That strategy will give the maximum utilization of resources with a
secondary minimum waiting time.

I intend in the near future (once I complete getting the scalability
framework open sourced) to make the strategy plugable, thus allowing people
to, e.g.

Provision nodes while there is a demand for them - which will minimize
waiting time at the expense of wasting resources

At present the trick is to play with MARGIN and MARGIN0 and the decay rate
so that you get nodes provisioned faster in response to the exponentially
weighted average number of idle nodes changing... but be warned, adjusting
these values will cause idle nodes to get provisioned depending on the
build load.


On 16 July 2014 15:00,  wrote:

> Hello,
>
> I'm writing a Jenkins Cloud plug-in that should work similar to EC2
> plug-in.
> The sequence of events is as follows:
> 1. Jenkins decides that there are not enough slaves to run jobs
> 2. Jenkins access a private resource manager system to reserve and launch
> test set-up. The resource manager returns an IP address of the test set-up
> 3. Jenkins launches slave on the returned IP.
> 4. The job is executed on the slave
> 5. Once the job is completed Jenkins kills the slave and updates the
> resouce manager that the set-up is free.
>
> I have extended Cloud, Slave, ComputerLauncher etc and I have a plugin
> that does all of the above.
> However... Jenkins never provisions more than one slave from the resource
> manager cloud.
>
> Looking into the code of NodeProvisioner.update, it seems that the
> condition
>
> *if (idle < MARGIN || needSomeWhenNoneAtAll) {*Is never met once there is
> one slave in the pool.
>
> When there are no slaves, *needSomeWhenNoneAtAll == true* and the *if*
> clause is executed.
> When there is one slave, *needSomeWhenNoneAtAll == false* and i*dle >
> MARGIN* so the *if* clause is not executed.
>
> I've found out that MARGIN is set to 0.1 by default with the following
> code:
>
> *private static final float MARGIN =
> Integer.getInteger(NodeProvisioner.class.getName() + ".MARGIN", 10) / 100f;*
> I've changed this line to
> *private static final float MARGIN =
> Integer.getInteger(NodeProvisioner.class.getName() + ".MARGIN", 100) /
> 100f;*
> And forced *idle < MARGIN* so the *if* clause always gets executed and
> Jenkins launches multiple slaves.
>
>
> *My questions are:*1. Why, by default, *MARGIN* is set to 0.1 so *idle >
> MARGIN* and Jenkins does not launch more than one slave?
> 2. How can I change this default behaviour so Jenkins does not wait at
> all? So as soon as it finds out there is awaiting job it will ask to
> provision new slave?
>
>
>
> *Thanks,Yoram*
>
> --
> You received this message because you are subscribed to the Google Groups
> "Jenkins Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to jenkinsci-dev+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jenkinsci-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.