Re: [Build flow plugin] Cannot load balance multiple workflows

[Ty Satrang]

I think you must be right, my solution was to stop caring where the flow 
runs, and focus on an alternative method of retrieving the report/log data. 
I used the Build Flow Toolbox which lets you copy files with the flow:
job = build('downstream job')
toolbox.copyFiles(job.workspace, build.workspace)
//Unrelated but also useful:
out.println job.build.log



On Sunday, March 8, 2015 at 9:03:30 AM UTC-7, stefan thomasson wrote:
>
> Hi,
> If i understood your description, i am wondering if this Isn't the 
> expected behaviour for buildflow as it is a "fly weight" job and therefore 
> do not consume any executors?
>
> 5 mar 2015 kl. 23:39 skrev Ty Satrang >:
>
> I am not sure how you want to receive that, but here is a gist of the 
> config that will reproduce the issue. 
> Flow: https://gist.github.com/tsatrangtw/73a391aa75b45cb38235
> Downsteam: https://gist.github.com/tsatrangtw/4c20c9a85ce63081c160
>
>
> On Thursday, March 5, 2015 at 2:15:47 PM UTC-8, rginga wrote:
>>
>>  Can you provide a simple example of your main buildflow job?
>>
>>  
>>
>> *From:* jenkins...@googlegroups.com [mailto:jenkins...@googlegroups.com] *On 
>> Behalf Of *Ty Satrang
>> *Sent:* Thursday, March 05, 2015 5:12 PM
>> *To:* jenkins...@googlegroups.com
>> *Subject:* Re: [Build flow plugin] Cannot load balance multiple workflows
>>
>>  
>>  
>> Thank you, I did see this copy pasta in another thread, and I did try 
>> changing the scheduling strategy with a plug-in. It seems that this 
>> particular problem is somewhat exclusive to the Build Flow plugin. It 
>> doesn't just queue jobs up, as you can see in the image link, it runs the 
>> flow jobs all at once on a single node.
>>  
>>
>> On Thursday, March 5, 2015 at 11:46:37 AM UTC-8, rginga wrote:
>>  
>> Maybe I misunderstand your problem. Let’s keep it simple.
>>
>>  
>>
>> The reason the multiple runs of your build flow job run on the same node 
>> is because, when selecting a node, it tries to run it one the same node it 
>> previously ran on (#2 in the Strategy below). If your nodes had just 1 
>> executor defined, then any consecutive runs would run on different nodes. 
>> But with 8 executors, it just runs it there.
>>
>>  
>>
>> *Scheduling strategy*
>>
>> Some slaves are faster, while others are slow. Some slaves are closer 
>> (network wise) to a master, others are far away. So doing a good build 
>> distribution is a challenge. Currently, Jenkins employs the following 
>> strategy:
>>
>>1. If a project is configured to stick to one computer, that's always 
>>honored.
>>2. Jenkins tries to build a project on the same computer that it was 
>>previously built.
>>3. Jenkins tries to move long builds to slaves, because the amount of 
>>network interaction between a master and a slave tends to be logarithmic 
>> to 
>>the duration of a build (IOW, even if project A takes twice as long to 
>>build as project B, it won't require double network transfer.) So this 
>>strategy reduces the network overhead.
>>
>>  If you have interesting ideas (or better yet, implementations), please 
>> let me know.
>>
>>  
>>
>> *From:* jenkins...@googlegroups.com [mailto:jenkins...@googlegroups.com] *On 
>> Behalf Of *Ty Satrang
>> *Sent:* Thursday, March 05, 2015 1:52 PM
>> *To:* jenkins...@googlegroups.com
>> *Subject:* Re: [Build flow plugin] Cannot load balance multiple workflows
>>
>>  
>>  
>> "if you have a label assigned to many slaves, each with one executor, it 
>> will do what you want"
>> My slaves do all share the same label and I tried to "Restrict where 
>> this project can be run" and I get the same behavior.
>>
>> "One way to do what you want is to have each “archive” whatever files 
>> you want which puts them onto the Jenkins server and then use “Copy 
>> artifacts from another project”"
>>  
>> I suppose this could work, but does it copy the files to the workspace or 
>> does it just archive them? I am using a plugin to publish reports from the 
>> workspace.
>>  
>>  
>>  
>>
>> On Thursday, March 5, 2015 at 8:12:59 AM UTC-8, rginga wrote:
>>  
>> Its strategy is to try to use the same node it last built on. If you have 
>> available executors, it will use one. if you have a label assigned to many 
>> slaves, each with one executor, it will do what you want.
>>
>>  
>>
>> One way to do what y

Re: [Build flow plugin] Cannot load balance multiple workflows

[Stefan Thomasson]

Hi,
If i understood your description, i am wondering if this Isn't the expected 
behaviour for buildflow as it is a "fly weight" job and therefore do not 
consume any executors?

> 5 mar 2015 kl. 23:39 skrev Ty Satrang :
> 
> I am not sure how you want to receive that, but here is a gist of the config 
> that will reproduce the issue. 
> Flow: https://gist.github.com/tsatrangtw/73a391aa75b45cb38235
> Downsteam: https://gist.github.com/tsatrangtw/4c20c9a85ce63081c160
> 
> 
>> On Thursday, March 5, 2015 at 2:15:47 PM UTC-8, rginga wrote:
>> Can you provide a simple example of your main buildflow job?
>> 
>>  
>> 
>> From: jenkins...@googlegroups.com [mailto:jenkins...@googlegroups.com] On 
>> Behalf Of Ty Satrang
>> Sent: Thursday, March 05, 2015 5:12 PM
>> To: jenkins...@googlegroups.com
>> Subject: Re: [Build flow plugin] Cannot load balance multiple workflows
>> 
>>  
>> 
>> Thank you, I did see this copy pasta in another thread, and I did try 
>> changing the scheduling strategy with a plug-in. It seems that this 
>> particular problem is somewhat exclusive to the Build Flow plugin. It 
>> doesn't just queue jobs up, as you can see in the image link, it runs the 
>> flow jobs all at once on a single node.
>> 
>> 
>> On Thursday, March 5, 2015 at 11:46:37 AM UTC-8, rginga wrote:
>> 
>> Maybe I misunderstand your problem. Let’s keep it simple.
>> 
>>  
>> 
>> The reason the multiple runs of your build flow job run on the same node is 
>> because, when selecting a node, it tries to run it one the same node it 
>> previously ran on (#2 in the Strategy below). If your nodes had just 1 
>> executor defined, then any consecutive runs would run on different nodes. 
>> But with 8 executors, it just runs it there.
>> 
>>  
>> 
>> Scheduling strategy
>> 
>> Some slaves are faster, while others are slow. Some slaves are closer 
>> (network wise) to a master, others are far away. So doing a good build 
>> distribution is a challenge. Currently, Jenkins employs the following 
>> strategy:
>> 
>> If a project is configured to stick to one computer, that's always honored.
>> Jenkins tries to build a project on the same computer that it was previously 
>> built.
>> Jenkins tries to move long builds to slaves, because the amount of network 
>> interaction between a master and a slave tends to be logarithmic to the 
>> duration of a build (IOW, even if project A takes twice as long to build as 
>> project B, it won't require double network transfer.) So this strategy 
>> reduces the network overhead.
>> If you have interesting ideas (or better yet, implementations), please let 
>> me know.
>> 
>>  
>> 
>> From: jenkins...@googlegroups.com [mailto:jenkins...@googlegroups.com] On 
>> Behalf Of Ty Satrang
>> Sent: Thursday, March 05, 2015 1:52 PM
>> To: jenkins...@googlegroups.com
>> Subject: Re: [Build flow plugin] Cannot load balance multiple workflows
>> 
>>  
>> 
>> "if you have a label assigned to many slaves, each with one executor, it 
>> will do what you want"
>> My slaves do all share the same label and I tried to "Restrict where this 
>> project can be run" and I get the same behavior.
>> 
>> "One way to do what you want is to have each “archive” whatever files you 
>> want which puts them onto the Jenkins server and then use “Copy artifacts 
>> from another project”"
>> 
>> I suppose this could work, but does it copy the files to the workspace or 
>> does it just archive them? I am using a plugin to publish reports from the 
>> workspace.
>> 
>>  
>> 
>> 
>> On Thursday, March 5, 2015 at 8:12:59 AM UTC-8, rginga wrote:
>> 
>> Its strategy is to try to use the same node it last built on. If you have 
>> available executors, it will use one. if you have a label assigned to many 
>> slaves, each with one executor, it will do what you want.
>> 
>>  
>> 
>> One way to do what you want is to have each “archive” whatever files you 
>> want which puts them onto the Jenkins server and then use “Copy artifacts 
>> from another project”
>> 
>>  
>> 
>> From: jenkins...@googlegroups.com [mailto:jenkins...@googlegroups.com] On 
>> Behalf Of Ty Satrang
>> Sent: Thursday, March 05, 2015 3:50 AM
>> To: jenkins...@googlegroups.com
>> Subject: [Build flow plugin] Cannot load balance multiple workflows
>> 
>>  
>> 
>> http://i.imgur.com/VEZ1Erc.png
>

Re: [Build flow plugin] Cannot load balance multiple workflows

[Ty Satrang]

I am using Jenkins to run tests for several teams, and I would like to be 
able to run tests concurrently rather than serially. I have a master job 
that takes parameters (team, tests, code to pull) that I pass using a build 
flow. I use Jenkins so that I can have this happen on demand or on a 
schedule. It appears I may be going about this the wrong way. Do you have 
any suggestions?

On Thursday, March 5, 2015 at 2:54:13 PM UTC-8, rginga wrote:
>
>  Ok, so your upper job Sample Flow does 
>
>  
>
> Build(Sample downstream job, node: name-name)
>
>  
>
> And Sample downstream job does a find grep command
>
>  
>
> And your image represents starting Sample flow job 9 times. Since there 
> are 9 available executors on dvm-qa5, all 9 Sample flow jobs start. I 
> expect that 8 of them pend waiting on Sample downstream job to finish 
> (unless you checked “Run concurrent”)
>
>  
>
> And I think you wanted those 9 jobs to be distributed among other nodes in 
> your group like dvm-qa2 and 1? 
>
>  
>
> I think that I don’t know how to get it to do what you want but I would 
> not call it a “problem”. This strategy “Jenkins tries to build a project on 
> the same computer that it was previously built.” Is called “affinity” and 
> it helps a great deal when jobs are pulling source code and/or building. It 
> can greatly speed up the process.
>
>  
>
> *From:* jenkins...@googlegroups.com  [mailto:
> jenkins...@googlegroups.com ] *On Behalf Of *Ty Satrang
> *Sent:* Thursday, March 05, 2015 5:39 PM
> *To:* jenkins...@googlegroups.com 
> *Subject:* Re: [Build flow plugin] Cannot load balance multiple workflows
>
>  
>  
> I am not sure how you want to receive that, but here is a gist of the 
> config that will reproduce the issue. 
>  
> Flow: https://gist.github.com/tsatrangtw/73a391aa75b45cb38235
> Downsteam: https://gist.github.com/tsatrangtw/4c20c9a85ce63081c160
>  
>  
>  
>
> On Thursday, March 5, 2015 at 2:15:47 PM UTC-8, rginga wrote:
>  
> Can you provide a simple example of your main buildflow job?
>
>  
>
> *From:* jenkins...@googlegroups.com [mailto:jenkins...@googlegroups.com] *On 
> Behalf Of *Ty Satrang
> *Sent:* Thursday, March 05, 2015 5:12 PM
> *To:* jenkins...@googlegroups.com
> *Subject:* Re: [Build flow plugin] Cannot load balance multiple workflows
>
>  
>  
> Thank you, I did see this copy pasta in another thread, and I did try 
> changing the scheduling strategy with a plug-in. It seems that this 
> particular problem is somewhat exclusive to the Build Flow plugin. It 
> doesn't just queue jobs up, as you can see in the image link, it runs the 
> flow jobs all at once on a single node.
>  
>
> On Thursday, March 5, 2015 at 11:46:37 AM UTC-8, rginga wrote:
>  
> Maybe I misunderstand your problem. Let’s keep it simple.
>
>  
>
> The reason the multiple runs of your build flow job run on the same node 
> is because, when selecting a node, it tries to run it one the same node it 
> previously ran on (#2 in the Strategy below). If your nodes had just 1 
> executor defined, then any consecutive runs would run on different nodes. 
> But with 8 executors, it just runs it there.
>
>  
>
> *Scheduling strategy*
>
> Some slaves are faster, while others are slow. Some slaves are closer 
> (network wise) to a master, others are far away. So doing a good build 
> distribution is a challenge. Currently, Jenkins employs the following 
> strategy:
>
>1. If a project is configured to stick to one computer, that's always 
>honored.
>2. Jenkins tries to build a project on the same computer that it was 
>previously built.
>3. Jenkins tries to move long builds to slaves, because the amount of 
>network interaction between a master and a slave tends to be logarithmic 
> to 
>the duration of a build (IOW, even if project A takes twice as long to 
>build as project B, it won't require double network transfer.) So this 
>strategy reduces the network overhead.
>
>  If you have interesting ideas (or better yet, implementations), please 
> let me know.
>
>  
>
> *From:* jenkins...@googlegroups.com [mailto:jenkins...@googlegroups.com] *On 
> Behalf Of *Ty Satrang
> *Sent:* Thursday, March 05, 2015 1:52 PM
> *To:* jenkins...@googlegroups.com
> *Subject:* Re: [Build flow plugin] Cannot load balance multiple workflows
>
>  
>  
> "if you have a label assigned to many slaves, each with one executor, it 
> will do what you want"
> My slaves do all share the same label and I tried to "Restrict where this 
> project can be run" and I get the same behavior.
>
> "One way to do what you want i

RE: [Build flow plugin] Cannot load balance multiple workflows

[Ginga, Dick]

Ok, so your upper job Sample Flow does

Build(Sample downstream job, node: name-name)

And Sample downstream job does a find grep command

And your image represents starting Sample flow job 9 times. Since there are 9 
available executors on dvm-qa5, all 9 Sample flow jobs start. I expect that 8 
of them pend waiting on Sample downstream job to finish (unless you checked 
“Run concurrent”)

And I think you wanted those 9 jobs to be distributed among other nodes in your 
group like dvm-qa2 and 1?

I think that I don’t know how to get it to do what you want but I would not 
call it a “problem”. This strategy “Jenkins tries to build a project on the 
same computer that it was previously built.” Is called “affinity” and it helps 
a great deal when jobs are pulling source code and/or building. It can greatly 
speed up the process.

From: jenkinsci-users@googlegroups.com 
[mailto:jenkinsci-users@googlegroups.com] On Behalf Of Ty Satrang
Sent: Thursday, March 05, 2015 5:39 PM
To: jenkinsci-users@googlegroups.com
Subject: Re: [Build flow plugin] Cannot load balance multiple workflows

I am not sure how you want to receive that, but here is a gist of the config 
that will reproduce the issue.
Flow: https://gist.github.com/tsatrangtw/73a391aa75b45cb38235
Downsteam: https://gist.github.com/tsatrangtw/4c20c9a85ce63081c160


On Thursday, March 5, 2015 at 2:15:47 PM UTC-8, rginga wrote:
Can you provide a simple example of your main buildflow job?

From: jenkins...@googlegroups.com 
[mailto:jenkins...@googlegroups.com] On Behalf Of Ty Satrang
Sent: Thursday, March 05, 2015 5:12 PM
To: jenkins...@googlegroups.com
Subject: Re: [Build flow plugin] Cannot load balance multiple workflows

Thank you, I did see this copy pasta in another thread, and I did try changing 
the scheduling strategy with a plug-in. It seems that this particular problem 
is somewhat exclusive to the Build Flow plugin. It doesn't just queue jobs up, 
as you can see in the image link, it runs the flow jobs all at once on a single 
node.

On Thursday, March 5, 2015 at 11:46:37 AM UTC-8, rginga wrote:
Maybe I misunderstand your problem. Let’s keep it simple.

The reason the multiple runs of your build flow job run on the same node is 
because, when selecting a node, it tries to run it one the same node it 
previously ran on (#2 in the Strategy below). If your nodes had just 1 executor 
defined, then any consecutive runs would run on different nodes. But with 8 
executors, it just runs it there.

Scheduling strategy
Some slaves are faster, while others are slow. Some slaves are closer (network 
wise) to a master, others are far away. So doing a good build distribution is a 
challenge. Currently, Jenkins employs the following strategy:

  1.  If a project is configured to stick to one computer, that's always 
honored.
  2.  Jenkins tries to build a project on the same computer that it was 
previously built.
  3.  Jenkins tries to move long builds to slaves, because the amount of 
network interaction between a master and a slave tends to be logarithmic to the 
duration of a build (IOW, even if project A takes twice as long to build as 
project B, it won't require double network transfer.) So this strategy reduces 
the network overhead.
If you have interesting ideas (or better yet, implementations), please let me 
know.

From: jenkins...@googlegroups.com<mailto:jenkins...@googlegroups.com> 
[mailto:jenkins...@googlegroups.com] On Behalf Of Ty Satrang
Sent: Thursday, March 05, 2015 1:52 PM
To: jenkins...@googlegroups.com<mailto:jenkins...@googlegroups.com>
Subject: Re: [Build flow plugin] Cannot load balance multiple workflows

"if you have a label assigned to many slaves, each with one executor, it will 
do what you want"
My slaves do all share the same label and I tried to "Restrict where this 
project can be run" and I get the same behavior.

"One way to do what you want is to have each “archive” whatever files you want 
which puts them onto the Jenkins server and then use “Copy artifacts from 
another project”"
I suppose this could work, but does it copy the files to the workspace or does 
it just archive them? I am using a plugin to publish reports from the workspace.


On Thursday, March 5, 2015 at 8:12:59 AM UTC-8, rginga wrote:
Its strategy is to try to use the same node it last built on. If you have 
available executors, it will use one. if you have a label assigned to many 
slaves, each with one executor, it will do what you want.

One way to do what you want is to have each “archive” whatever files you want 
which puts them onto the Jenkins server and then use “Copy artifacts from 
another project”

From: jenkins...@googlegroups.com<mailto:jenkins...@googlegroups.com> 
[mailto:jenkins...@googlegroups.com] On Behalf Of Ty Satrang
Sent: Thursday, March 05, 2015 3:50 AM
To: jenkins...@googlegroups.com<mailto:jenkins...@googlegroups.com>
Subject: [Build flow plugin] Cannot load balanc

Re: [Build flow plugin] Cannot load balance multiple workflows

[Ty Satrang]

I am not sure how you want to receive that, but here is a gist of the 
config that will reproduce the issue. 
Flow: https://gist.github.com/tsatrangtw/73a391aa75b45cb38235
Downsteam: https://gist.github.com/tsatrangtw/4c20c9a85ce63081c160


On Thursday, March 5, 2015 at 2:15:47 PM UTC-8, rginga wrote:
>
>  Can you provide a simple example of your main buildflow job?
>
>  
>
> *From:* jenkins...@googlegroups.com  [mailto:
> jenkins...@googlegroups.com ] *On Behalf Of *Ty Satrang
> *Sent:* Thursday, March 05, 2015 5:12 PM
> *To:* jenkins...@googlegroups.com 
> *Subject:* Re: [Build flow plugin] Cannot load balance multiple workflows
>
>  
>  
> Thank you, I did see this copy pasta in another thread, and I did try 
> changing the scheduling strategy with a plug-in. It seems that this 
> particular problem is somewhat exclusive to the Build Flow plugin. It 
> doesn't just queue jobs up, as you can see in the image link, it runs the 
> flow jobs all at once on a single node.
>  
>
> On Thursday, March 5, 2015 at 11:46:37 AM UTC-8, rginga wrote:
>  
> Maybe I misunderstand your problem. Let’s keep it simple.
>
>  
>
> The reason the multiple runs of your build flow job run on the same node 
> is because, when selecting a node, it tries to run it one the same node it 
> previously ran on (#2 in the Strategy below). If your nodes had just 1 
> executor defined, then any consecutive runs would run on different nodes. 
> But with 8 executors, it just runs it there.
>
>  
>
> *Scheduling strategy*
>
> Some slaves are faster, while others are slow. Some slaves are closer 
> (network wise) to a master, others are far away. So doing a good build 
> distribution is a challenge. Currently, Jenkins employs the following 
> strategy:
>
>1. If a project is configured to stick to one computer, that's always 
>honored.
>2. Jenkins tries to build a project on the same computer that it was 
>previously built.
>3. Jenkins tries to move long builds to slaves, because the amount of 
>network interaction between a master and a slave tends to be logarithmic 
> to 
>the duration of a build (IOW, even if project A takes twice as long to 
>build as project B, it won't require double network transfer.) So this 
>strategy reduces the network overhead.
>
>  If you have interesting ideas (or better yet, implementations), please 
> let me know.
>
>  
>
> *From:* jenkins...@googlegroups.com [mailto:jenkins...@googlegroups.com] *On 
> Behalf Of *Ty Satrang
> *Sent:* Thursday, March 05, 2015 1:52 PM
> *To:* jenkins...@googlegroups.com
> *Subject:* Re: [Build flow plugin] Cannot load balance multiple workflows
>
>  
>  
> "if you have a label assigned to many slaves, each with one executor, it 
> will do what you want"
> My slaves do all share the same label and I tried to "Restrict where this 
> project can be run" and I get the same behavior.
>
> "One way to do what you want is to have each “archive” whatever files you 
> want which puts them onto the Jenkins server and then use “Copy artifacts 
> from another project”"
>  
> I suppose this could work, but does it copy the files to the workspace or 
> does it just archive them? I am using a plugin to publish reports from the 
> workspace.
>  
>  
>  
>
> On Thursday, March 5, 2015 at 8:12:59 AM UTC-8, rginga wrote:
>  
> Its strategy is to try to use the same node it last built on. If you have 
> available executors, it will use one. if you have a label assigned to many 
> slaves, each with one executor, it will do what you want.
>
>  
>
> One way to do what you want is to have each “archive” whatever files you 
> want which puts them onto the Jenkins server and then use “Copy artifacts 
> from another project”
>
>  
>
> *From:* jenkins...@googlegroups.com [mailto:jenkins...@googlegroups.com] *On 
> Behalf Of *Ty Satrang
> *Sent:* Thursday, March 05, 2015 3:50 AM
> *To:* jenkins...@googlegroups.com
> *Subject:* [Build flow plugin] Cannot load balance multiple workflows
>
>  
>  
> http://i.imgur.com/VEZ1Erc.png
>  
>  
>  
> For some reason my flow jobs like to all run on a single node instead of 
> choosing an idle node.
>  
>  
>  
> This is a problem for me because I am using the Node and Label parameter 
> plugin 
> <https://wiki.jenkins-ci.org/display/JENKINS/NodeLabel+Parameter+Plugin> to 
> run the jobs on the same VM as the flow job so I can copy the reports and 
> logs to the parent workspace for publishing.
>  
>  
>  
> I tried using the Least Load plugin 
> <http://wiki.jenkins-ci.org/display/JENKINS/Least+Load+Plugin>, but I get

RE: [Build flow plugin] Cannot load balance multiple workflows

[Ginga, Dick]

Can you provide a simple example of your main buildflow job?

From: jenkinsci-users@googlegroups.com 
[mailto:jenkinsci-users@googlegroups.com] On Behalf Of Ty Satrang
Sent: Thursday, March 05, 2015 5:12 PM
To: jenkinsci-users@googlegroups.com
Subject: Re: [Build flow plugin] Cannot load balance multiple workflows

Thank you, I did see this copy pasta in another thread, and I did try changing 
the scheduling strategy with a plug-in. It seems that this particular problem 
is somewhat exclusive to the Build Flow plugin. It doesn't just queue jobs up, 
as you can see in the image link, it runs the flow jobs all at once on a single 
node.

On Thursday, March 5, 2015 at 11:46:37 AM UTC-8, rginga wrote:
Maybe I misunderstand your problem. Let’s keep it simple.

The reason the multiple runs of your build flow job run on the same node is 
because, when selecting a node, it tries to run it one the same node it 
previously ran on (#2 in the Strategy below). If your nodes had just 1 executor 
defined, then any consecutive runs would run on different nodes. But with 8 
executors, it just runs it there.

Scheduling strategy
Some slaves are faster, while others are slow. Some slaves are closer (network 
wise) to a master, others are far away. So doing a good build distribution is a 
challenge. Currently, Jenkins employs the following strategy:

  1.  If a project is configured to stick to one computer, that's always 
honored.
  2.  Jenkins tries to build a project on the same computer that it was 
previously built.
  3.  Jenkins tries to move long builds to slaves, because the amount of 
network interaction between a master and a slave tends to be logarithmic to the 
duration of a build (IOW, even if project A takes twice as long to build as 
project B, it won't require double network transfer.) So this strategy reduces 
the network overhead.
If you have interesting ideas (or better yet, implementations), please let me 
know.

From: jenkins...@googlegroups.com 
[mailto:jenkins...@googlegroups.com] On Behalf Of Ty Satrang
Sent: Thursday, March 05, 2015 1:52 PM
To: jenkins...@googlegroups.com
Subject: Re: [Build flow plugin] Cannot load balance multiple workflows

"if you have a label assigned to many slaves, each with one executor, it will 
do what you want"
My slaves do all share the same label and I tried to "Restrict where this 
project can be run" and I get the same behavior.

"One way to do what you want is to have each “archive” whatever files you want 
which puts them onto the Jenkins server and then use “Copy artifacts from 
another project”"
I suppose this could work, but does it copy the files to the workspace or does 
it just archive them? I am using a plugin to publish reports from the workspace.


On Thursday, March 5, 2015 at 8:12:59 AM UTC-8, rginga wrote:
Its strategy is to try to use the same node it last built on. If you have 
available executors, it will use one. if you have a label assigned to many 
slaves, each with one executor, it will do what you want.

One way to do what you want is to have each “archive” whatever files you want 
which puts them onto the Jenkins server and then use “Copy artifacts from 
another project”

From: jenkins...@googlegroups.com<mailto:jenkins...@googlegroups.com> 
[mailto:jenkins...@googlegroups.com] On Behalf Of Ty Satrang
Sent: Thursday, March 05, 2015 3:50 AM
To: jenkins...@googlegroups.com<mailto:jenkins...@googlegroups.com>
Subject: [Build flow plugin] Cannot load balance multiple workflows

http://i.imgur.com/VEZ1Erc.png

For some reason my flow jobs like to all run on a single node instead of 
choosing an idle node.

This is a problem for me because I am using the Node and Label parameter 
plugin<https://wiki.jenkins-ci.org/display/JENKINS/NodeLabel+Parameter+Plugin> 
to run the jobs on the same VM as the flow job so I can copy the reports and 
logs to the parent workspace for publishing.

I tried using the Least Load 
plugin<http://wiki.jenkins-ci.org/display/JENKINS/Least+Load+Plugin>, but I get 
the same behavior.

I also tried the Throttle Concurrent Builds 
Plug-in<http://wiki.jenkins-ci.org/display/JENKINS/Throttle+Concurrent+Builds+Plugin>
 to restrict the jobs per node to 1, to no effect.

I am just realizing this now but, maybe it would be easier to get the files to 
the parent workspace another way and let the flow jobs all run on whatever 
single node they want. (maybe? 
https://github.com/dnozay/build-flow-toolbox-plugin )


--
You received this message because you are subscribed to the Google Groups 
"Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jenkinsci-use...@googlegroups.com<mailto:jenkinsci-use...@googlegroups.com>.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/jenkinsci-users/3d1d0150-3125-4b54-a846-b0a6687022cf%40googlegroups.com<https://groups.google.com/d/

Re: [Build flow plugin] Cannot load balance multiple workflows

[Ty Satrang]

Thank you, I did see this copy pasta in another thread, and I did try 
changing the scheduling strategy with a plug-in. It seems that this 
particular problem is somewhat exclusive to the Build Flow plugin. It 
doesn't just queue jobs up, as you can see in the image link, it runs the 
flow jobs all at once on a single node.

On Thursday, March 5, 2015 at 11:46:37 AM UTC-8, rginga wrote:
>
>  Maybe I misunderstand your problem. Let’s keep it simple.
>
>  
>
> The reason the multiple runs of your build flow job run on the same node 
> is because, when selecting a node, it tries to run it one the same node it 
> previously ran on (#2 in the Strategy below). If your nodes had just 1 
> executor defined, then any consecutive runs would run on different nodes. 
> But with 8 executors, it just runs it there.
>
>  
>
> *Scheduling strategy*
>
> Some slaves are faster, while others are slow. Some slaves are closer 
> (network wise) to a master, others are far away. So doing a good build 
> distribution is a challenge. Currently, Jenkins employs the following 
> strategy:
>
>1. If a project is configured to stick to one computer, that's always 
>honored.
>2. Jenkins tries to build a project on the same computer that it was 
>previously built.
>3. Jenkins tries to move long builds to slaves, because the amount of 
>network interaction between a master and a slave tends to be logarithmic 
> to 
>the duration of a build (IOW, even if project A takes twice as long to 
>build as project B, it won't require double network transfer.) So this 
>strategy reduces the network overhead.
>
>  If you have interesting ideas (or better yet, implementations), please 
> let me know.
>
>  
>
> *From:* jenkins...@googlegroups.com  [mailto:
> jenkins...@googlegroups.com ] *On Behalf Of *Ty Satrang
> *Sent:* Thursday, March 05, 2015 1:52 PM
> *To:* jenkins...@googlegroups.com 
> *Subject:* Re: [Build flow plugin] Cannot load balance multiple workflows
>
>  
>  
> "if you have a label assigned to many slaves, each with one executor, it 
> will do what you want"
> My slaves do all share the same label and I tried to "Restrict where this 
> project can be run" and I get the same behavior.
>
> "One way to do what you want is to have each “archive” whatever files you 
> want which puts them onto the Jenkins server and then use “Copy artifacts 
> from another project”"
>  
> I suppose this could work, but does it copy the files to the workspace or 
> does it just archive them? I am using a plugin to publish reports from the 
> workspace.
>  
>  
>  
>
> On Thursday, March 5, 2015 at 8:12:59 AM UTC-8, rginga wrote:
>  
> Its strategy is to try to use the same node it last built on. If you have 
> available executors, it will use one. if you have a label assigned to many 
> slaves, each with one executor, it will do what you want.
>
>  
>
> One way to do what you want is to have each “archive” whatever files you 
> want which puts them onto the Jenkins server and then use “Copy artifacts 
> from another project”
>
>  
>
> *From:* jenkins...@googlegroups.com [mailto:jenkins...@googlegroups.com] *On 
> Behalf Of *Ty Satrang
> *Sent:* Thursday, March 05, 2015 3:50 AM
> *To:* jenkins...@googlegroups.com
> *Subject:* [Build flow plugin] Cannot load balance multiple workflows
>
>  
>  
> http://i.imgur.com/VEZ1Erc.png
>  
>  
>  
> For some reason my flow jobs like to all run on a single node instead of 
> choosing an idle node.
>  
>  
>  
> This is a problem for me because I am using the Node and Label parameter 
> plugin 
> <https://wiki.jenkins-ci.org/display/JENKINS/NodeLabel+Parameter+Plugin> to 
> run the jobs on the same VM as the flow job so I can copy the reports and 
> logs to the parent workspace for publishing.
>  
>  
>  
> I tried using the Least Load plugin 
> <http://wiki.jenkins-ci.org/display/JENKINS/Least+Load+Plugin>, but I get 
> the same behavior.
>   
>  
>  
> I also tried the Throttle Concurrent Builds Plug-in 
> <http://wiki.jenkins-ci.org/display/JENKINS/Throttle+Concurrent+Builds+Plugin>
>  to 
> restrict the jobs per node to 1, to no effect.
>  
>  
>  
> I am just realizing this now but, maybe it would be easier to get the 
> files to the parent workspace another way and let the flow jobs all run on 
> whatever single node they want. (maybe? 
> https://github.com/dnozay/build-flow-toolbox-plugin )
>  
>  
>   
>  
>  
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Jenkins Users" group.
> To unsubs

RE: [Build flow plugin] Cannot load balance multiple workflows

[Ginga, Dick]

Maybe I misunderstand your problem. Let’s keep it simple.

The reason the multiple runs of your build flow job run on the same node is 
because, when selecting a node, it tries to run it one the same node it 
previously ran on (#2 in the Strategy below). If your nodes had just 1 executor 
defined, then any consecutive runs would run on different nodes. But with 8 
executors, it just runs it there.

Scheduling strategy
Some slaves are faster, while others are slow. Some slaves are closer (network 
wise) to a master, others are far away. So doing a good build distribution is a 
challenge. Currently, Jenkins employs the following strategy:

  1.  If a project is configured to stick to one computer, that's always 
honored.
  2.  Jenkins tries to build a project on the same computer that it was 
previously built.
  3.  Jenkins tries to move long builds to slaves, because the amount of 
network interaction between a master and a slave tends to be logarithmic to the 
duration of a build (IOW, even if project A takes twice as long to build as 
project B, it won't require double network transfer.) So this strategy reduces 
the network overhead.
If you have interesting ideas (or better yet, implementations), please let me 
know.

From: jenkinsci-users@googlegroups.com 
[mailto:jenkinsci-users@googlegroups.com] On Behalf Of Ty Satrang
Sent: Thursday, March 05, 2015 1:52 PM
To: jenkinsci-users@googlegroups.com
Subject: Re: [Build flow plugin] Cannot load balance multiple workflows

"if you have a label assigned to many slaves, each with one executor, it will 
do what you want"
My slaves do all share the same label and I tried to "Restrict where this 
project can be run" and I get the same behavior.

"One way to do what you want is to have each “archive” whatever files you want 
which puts them onto the Jenkins server and then use “Copy artifacts from 
another project”"
I suppose this could work, but does it copy the files to the workspace or does 
it just archive them? I am using a plugin to publish reports from the workspace.


On Thursday, March 5, 2015 at 8:12:59 AM UTC-8, rginga wrote:
Its strategy is to try to use the same node it last built on. If you have 
available executors, it will use one. if you have a label assigned to many 
slaves, each with one executor, it will do what you want.

One way to do what you want is to have each “archive” whatever files you want 
which puts them onto the Jenkins server and then use “Copy artifacts from 
another project”

From: jenkins...@googlegroups.com 
[mailto:jenkins...@googlegroups.com] On Behalf Of Ty Satrang
Sent: Thursday, March 05, 2015 3:50 AM
To: jenkins...@googlegroups.com
Subject: [Build flow plugin] Cannot load balance multiple workflows

http://i.imgur.com/VEZ1Erc.png

For some reason my flow jobs like to all run on a single node instead of 
choosing an idle node.

This is a problem for me because I am using the Node and Label parameter 
plugin<https://wiki.jenkins-ci.org/display/JENKINS/NodeLabel+Parameter+Plugin> 
to run the jobs on the same VM as the flow job so I can copy the reports and 
logs to the parent workspace for publishing.

I tried using the Least Load 
plugin<http://wiki.jenkins-ci.org/display/JENKINS/Least+Load+Plugin>, but I get 
the same behavior.

I also tried the Throttle Concurrent Builds 
Plug-in<http://wiki.jenkins-ci.org/display/JENKINS/Throttle+Concurrent+Builds+Plugin>
 to restrict the jobs per node to 1, to no effect.

I am just realizing this now but, maybe it would be easier to get the files to 
the parent workspace another way and let the flow jobs all run on whatever 
single node they want. (maybe? 
https://github.com/dnozay/build-flow-toolbox-plugin )


--
You received this message because you are subscribed to the Google Groups 
"Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jenkinsci-use...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/jenkinsci-users/3d1d0150-3125-4b54-a846-b0a6687022cf%40googlegroups.com<https://groups.google.com/d/msgid/jenkinsci-users/3d1d0150-3125-4b54-a846-b0a6687022cf%40googlegroups.com?utm_medium=email&utm_source=footer>.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups 
"Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to 
jenkinsci-users+unsubscr...@googlegroups.com<mailto:jenkinsci-users+unsubscr...@googlegroups.com>.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/jenkinsci-users/0a9cb8a5-d3ef-4f02-8506-0e4b0a986493%40googlegroups.com<https://groups.google.com/d/msgid/jenkinsci-users/0a9cb8a5-d3ef-4f02-8506-0e4b0a986493%40googlegroups.com?utm_medium=email&utm_source=footer>.
For more options, visit https://groups.googl

Re: [Build flow plugin] Cannot load balance multiple workflows

[Ty Satrang]

"if you have a label assigned to many slaves, each with one executor, it 
will do what you want"
My slaves do all share the same label and I tried to "Restrict where this 
project can be run" and I get the same behavior.

"One way to do what you want is to have each “archive” whatever files you 
want which puts them onto the Jenkins server and then use “Copy artifacts 
from another project”"
I suppose this could work, but does it copy the files to the workspace or 
does it just archive them? I am using a plugin to publish reports from the 
workspace.


On Thursday, March 5, 2015 at 8:12:59 AM UTC-8, rginga wrote:
>
>  Its strategy is to try to use the same node it last built on. If you 
> have available executors, it will use one. if you have a label assigned to 
> many slaves, each with one executor, it will do what you want.
>
>  
>
> One way to do what you want is to have each “archive” whatever files you 
> want which puts them onto the Jenkins server and then use “Copy artifacts 
> from another project”
>
>  
>
> *From:* jenkins...@googlegroups.com  [mailto:
> jenkins...@googlegroups.com ] *On Behalf Of *Ty Satrang
> *Sent:* Thursday, March 05, 2015 3:50 AM
> *To:* jenkins...@googlegroups.com 
> *Subject:* [Build flow plugin] Cannot load balance multiple workflows
>
>  
>  
> http://i.imgur.com/VEZ1Erc.png
>  
>  
>  
> For some reason my flow jobs like to all run on a single node instead of 
> choosing an idle node.
>  
>  
>  
> This is a problem for me because I am using the Node and Label parameter 
> plugin 
> <https://wiki.jenkins-ci.org/display/JENKINS/NodeLabel+Parameter+Plugin> to 
> run the jobs on the same VM as the flow job so I can copy the reports and 
> logs to the parent workspace for publishing.
>  
>  
>  
> I tried using the Least Load plugin 
> <http://wiki.jenkins-ci.org/display/JENKINS/Least+Load+Plugin>, but I get 
> the same behavior.
>   
>  
>  
> I also tried the Throttle Concurrent Builds Plug-in 
> <http://wiki.jenkins-ci.org/display/JENKINS/Throttle+Concurrent+Builds+Plugin>
>  to 
> restrict the jobs per node to 1, to no effect.
>  
>  
>  
> I am just realizing this now but, maybe it would be easier to get the 
> files to the parent workspace another way and let the flow jobs all run on 
> whatever single node they want. (maybe? 
> https://github.com/dnozay/build-flow-toolbox-plugin )
>  
>  
>   
>  
>  
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Jenkins Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to jenkinsci-use...@googlegroups.com .
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/jenkinsci-users/3d1d0150-3125-4b54-a846-b0a6687022cf%40googlegroups.com
>  
> <https://groups.google.com/d/msgid/jenkinsci-users/3d1d0150-3125-4b54-a846-b0a6687022cf%40googlegroups.com?utm_medium=email&utm_source=footer>
> .
> For more options, visit https://groups.google.com/d/optout.
>  

-- 
You received this message because you are subscribed to the Google Groups 
"Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jenkinsci-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/jenkinsci-users/0a9cb8a5-d3ef-4f02-8506-0e4b0a986493%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

RE: [Build flow plugin] Cannot load balance multiple workflows

[Ginga, Dick]

Its strategy is to try to use the same node it last built on. If you have 
available executors, it will use one. if you have a label assigned to many 
slaves, each with one executor, it will do what you want.

One way to do what you want is to have each “archive” whatever files you want 
which puts them onto the Jenkins server and then use “Copy artifacts from 
another project”

From: jenkinsci-users@googlegroups.com 
[mailto:jenkinsci-users@googlegroups.com] On Behalf Of Ty Satrang
Sent: Thursday, March 05, 2015 3:50 AM
To: jenkinsci-users@googlegroups.com
Subject: [Build flow plugin] Cannot load balance multiple workflows

http://i.imgur.com/VEZ1Erc.png

For some reason my flow jobs like to all run on a single node instead of 
choosing an idle node.

This is a problem for me because I am using the Node and Label parameter 
plugin<https://wiki.jenkins-ci.org/display/JENKINS/NodeLabel+Parameter+Plugin> 
to run the jobs on the same VM as the flow job so I can copy the reports and 
logs to the parent workspace for publishing.

I tried using the Least Load 
plugin<http://wiki.jenkins-ci.org/display/JENKINS/Least+Load+Plugin>, but I get 
the same behavior.

I also tried the Throttle Concurrent Builds 
Plug-in<http://wiki.jenkins-ci.org/display/JENKINS/Throttle+Concurrent+Builds+Plugin>
 to restrict the jobs per node to 1, to no effect.

I am just realizing this now but, maybe it would be easier to get the files to 
the parent workspace another way and let the flow jobs all run on whatever 
single node they want. (maybe? 
https://github.com/dnozay/build-flow-toolbox-plugin )


--
You received this message because you are subscribed to the Google Groups 
"Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to 
jenkinsci-users+unsubscr...@googlegroups.com<mailto:jenkinsci-users+unsubscr...@googlegroups.com>.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/jenkinsci-users/3d1d0150-3125-4b54-a846-b0a6687022cf%40googlegroups.com<https://groups.google.com/d/msgid/jenkinsci-users/3d1d0150-3125-4b54-a846-b0a6687022cf%40googlegroups.com?utm_medium=email&utm_source=footer>.
For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jenkinsci-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/jenkinsci-users/6C6EE445A6F6CE4E8A0FFB51B071A4E26E507091%40AMERMBX02.PERKINELMER.NET.
For more options, visit https://groups.google.com/d/optout.

[Build flow plugin] Cannot load balance multiple workflows

[Ty Satrang]

http://i.imgur.com/VEZ1Erc.png

For some reason my flow jobs like to all run on a single node instead of 
choosing an idle node.

This is a problem for me because I am using the Node and Label parameter 
plugin 
 to 
run the jobs on the same VM as the flow job so I can copy the reports and 
logs to the parent workspace for publishing.

I tried using the Least Load plugin 
, but I get 
the same behavior.

I also tried the Throttle Concurrent Builds Plug-in 
 
to 
restrict the jobs per node to 1, to no effect.

I am just realizing this now but, maybe it would be easier to get the files 
to the parent workspace another way and let the flow jobs all run on 
whatever single node they want. (maybe? 
https://github.com/dnozay/build-flow-toolbox-plugin )


-- 
You received this message because you are subscribed to the Google Groups 
"Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jenkinsci-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/jenkinsci-users/3d1d0150-3125-4b54-a846-b0a6687022cf%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.