Re: Custom pipeline jobs
Jenn, We do not test with a CI strategy. We have a limited number things that we do with some parallelization. 1. clone or merge 2. in parallel a. build product for (window (32, 64), linux (32, 64)) b. run many different types of tests c. generate report on tests 3. generate consolidated reports Currently I have a different pipeline for many of the preferred configurations we want our developers to use. I want to be able to allow my developers to select from all of the available options. I could put all of these options in 1 pipeline and have if conditions at every decision point. This maybe challenging to get perfect with the diff parallels it would have. I thought it would be powerful to pass a generated script based on available choices to a generic pipeline job. I could then handle all of the complex logic outside of the groovy script. This logic could be unit tested and in revision control. -- You received this message because you are subscribed to the Google Groups "Jenkins Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/73f6c726-82aa-4d25-8e8b-c0cbd768385a%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: Custom pipeline jobs
Anyone have any ideas how I can create dynamic pipeline scripts -- You received this message because you are subscribed to the Google Groups "Jenkins Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/cb9b11ba-7295-4852-86f3-2a54733b0a70%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Custom pipeline jobs
Is it possible to pass groovy scripts to a pipeline job
I want to be able to configure custom groovy scripts based on options
selected by my users.
I could have a massive script with many "if" clauses.
It would be nicer to create a groovy script based on selected options.
this api would then be testable internally.
I have tried a few tests run but it does not appear that the groovy being
passed is being executed.
groovy_script = "stage('testing') {\nbuild job: 'echo'\n}\n"
parameters: [string(name: 'groovy_script', value: groovy_script)
jenkins pipeline job
"""
#!groovy
import hudson.model.*
groovy_script
"""
I was hoping this would evaluate to
"""
#!groovy
import hudson.model.*
stage('testing') {
build job: 'echo'
}
"""
any ideas?
--
You received this message because you are subscribed to the Google Groups
"Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to jenkinsci-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/jenkinsci-users/455449e1-2e5a-4f3a-9930-86472916b27c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Pipeline calling pipeline calling freestyle job
I have: booleanParam(name: 'force_build', value: force_build.toBoolean()) This works Pipeline with parameter (force_build=True) -> freestsyle job with parameter (force_build=False) The value of force_build in the freestyle job is now True This does not work. Pipeline job with a parameter (force_build=True) -> Pipeline with parameter (force_build=False) -> freestsyle job with parameter (force_build=False) The value of force_build in second Pipeline is still False The value of force_build in the freestyle job is still False My expectation is that 1st pipeline force_build=True will override the other parameters down the call stack. Is this not supported? -- You received this message because you are subscribed to the Google Groups "Jenkins Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/d76ff653-1ee0-4ae2-953c-5d6bbb0dde75%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
many parallel stages
Hello,
I was looking for help with a many parallel pipeline.
Here is what I am looking to do
*parallel*(
'linux': {
stage('build_linux') {
build job: 'linux_build', parameters:[]
build job: 'linux_archive', parameters:[]
*parallel*(
stage('test1_linux') {
build job: 'test1_setup1', parameters:[]
build job: 'test1_setup2', parameters:[]
*parallel*(
many_builds...
)
build job: 'test1_finish', parameters:[]
}
stage('test2_linux') {
build job: 'test2_setup1', parameters:[]
build job: 'test2_setup2', parameters:[]
*parallel*(
many_builds...
)
build job: 'test2_finish', parameters:[]
}
)
}
},
'windows': {
stage('build_windows') {
build job: 'windows_build', parameters:[]
build job: 'windows_archive', parameters:[]
*parallel*(
stage('test1_windows') {
build job: 'test1_setup1', parameters:[]
build job: 'test1_setup2', parameters:[]
*parallel*(
many_builds...
)
build job: 'test1_finish', parameters:[]
}
stage('test2_windows') {
build job: 'test2_setup1', parameters:[]
build job: 'test2_setup2', parameters:[]
*parallel*(
many_builds...
)
build job: 'test2_finish', parameters:[]
}
)
}
}
)
As you see most pieces are the same so I tried something like
*parallel*(
'linux': {
generate_build('linux')
*parallel*(
generate_tests('test1')
generate_tests('test2')
)
},
'windows': {
generate_build('windows')
*parallel*(
generate_tests('test1')
generate_tests('test2')
generate_tests('test3')
generate_tests('test4')
)
},
)
def generate_build(system) {
build job: system + '_build', parameters:[]
build job: system + '_archive', parameters:[]
}
def generate_tests(tests) {
build job: tests + '_setup1', parameters:[]
build job: tests + '_setup2', parameters:[]
*parallel*(
many_builds...
)
build job: tests + '_finish', parameters:[]
}
the generate_build ran in parallel , but the generate_tests did not run in
parallel
should I be using a map of maps?
--
You received this message because you are subscribed to the Google Groups
"Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to jenkinsci-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/jenkinsci-users/32f034ba-70d4-4812-b63b-213827169081%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Re: parallel remote trigger builds in a pipeline
I assume you have "Block until the remote triggered projects finish their builds." = False? - Does it not just set your 'done' variable to true as soon as the remote build starts, since it does not block or wait. - Also with this set to False it does not get the log of the remote build. -- You received this message because you are subscribed to the Google Groups "Jenkins Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/3870cfdc-f429-4e82-9b1f-fbf24fc0dc24%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
How to properly use parallels in pipelines
*parallel* (
'windows': {
stage('windows tests') {
*parallel* (
*stage*('some_tests_1') {
*parallel* (
some_tests_1, # is a chunk of tests
)
}
*stage*('some_tests_2') {
*parallel* (
some_tests_2, # is a chunk of tests
)
}
*stage*('some_tests_3') {
*parallel* (
some_tests_3 # is a chunk of tests
)
}
)
}
'linux': ...same as above
Currently we call flows calling flows and the above displays how it would
look in a pipeline.
from:
https://github.com/jenkinsci/pipeline-examples/blob/master/docs/BEST_PRACTICES.md
Nested parallel blocks can lead to *swamping your available executors*, as
each execution of the first parallel block calls multiple executions of the
second parallel block, and so on. In general, think carefully about your
parallelism and your available executors when using parallel.
With this in mind we could
*parallel* (
'windows': {
stage('windows tests') {
*parallel* (
some_tests_1, # is a chunk of tests
some_tests_2, # is a chunk of tests
some_tests_3 # is a chunk of tests
)
}
}
'linux': ...same as above
Also from
https://github.com/jenkinsci/pipeline-examples/blob/master/docs/BEST_PRACTICES.md
*Don’t put stages in parallel **blocks* - that just goes *weird*, breaking
a lot of logic in the Stage View and elsewhere. Save yourself the pain -
don't do it!
So how do you display this work flow properly if we should not use stages
in parallel?
--
You received this message because you are subscribed to the Google Groups
"Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to jenkinsci-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/jenkinsci-users/d33e1ba8-5d60-4ecd-a509-5ac1234a97a5%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
parallel remote trigger builds in a pipeline
#!groovy
import hudson.model.*
stage('Testing') {
parallel (
['windows_remote': {build job: 'remote_trigger_30', parameters:
[string(name: 'branch_name', value: branch_name)]},
'linux_remote': {build job: 'remote_trigger_60', parameters:
[string(name: 'branch_name', value: branch_name)]}
]
)
}
If I set "Block until the remote triggered projects finish their builds."
True
'remote_trigger_30' - blocks until complete
'remote_trigger_60' - blocks until complete
This is not parallel.
If I set "Block until the remote triggered projects finish their builds."
False
'remote_trigger_30' - starts remote build then returns complete
'remote_trigger_60' - starts remote build then returns complete
Both remote builds are still running but Pipeline returns complete
Is there a way to not block but wait?
--
You received this message because you are subscribed to the Google Groups
"Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to jenkinsci-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/jenkinsci-users/d029d9be-d81e-4b4c-9210-396a4add6d46%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Re: Is there a maximum number of parallel builds a pipeline job can run?
So I was not running in a node block I will do more testing thank you.
Here is a sample of my code:
*#!groovy*
*import hudson.model.**
*stage('many_jobs') {*
*parallel (get_many_jobs(6000))*
*}*
*def get_many_jobs(num_builds) {*
*// The map we'll store the parallel steps in before executing them.*
*def stepsForParallel = [:]*
*for (int i = 0; i < num_builds; i++) {*
*def sub_run_config = 'job_' + i*
*stepsForParallel[sub_run_config] = {*
*build job: 'sleep_30', parameters: [string(name:
'job_name', value: sub_run_config)]*
* }*
*}*
*return stepsForParallel*
*}*
--
You received this message because you are subscribed to the Google Groups
"Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to jenkinsci-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/jenkinsci-users/7a53b09c-d8e9-4f9e-93b3-4333e88c25c9%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Re: Is there a maximum number of parallel builds a pipeline job can run?
So I have not run in an ode block I will do more testing thank you.
Here is a sample of my code:
*#!groovy*
*import hudson.model.**
*stage('many_jobs') {*
*parallel (get_many_jobs(6000))*
*}*
*def get_many_jobs(num_builds) {*
*// The map we'll store the parallel steps in before executing them.*
*def stepsForParallel = [:]*
*for (int i = 0; i < num_builds; i++) {*
*def sub_run_config = 'job_' + i*
*stepsForParallel[sub_run_config] = {*
*build job: 'sleep_30', parameters: [string(name:
'job_name', value: sub_run_config)]*
* }*
*}*
*return stepsForParallel*
*}*
--
You received this message because you are subscribed to the Google Groups
"Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to jenkinsci-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/jenkinsci-users/556d4421-e495-43af-9f76-d7ad1c39b07e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Re: Is there a maximum number of parallel builds a pipeline job can run?
So not sure if this is a Jenkins limitation or pipeline. The jobs never exceeded available resources. The garbage collection was stable. So why did it lock-up? It did say I used - 2950 million active threads - 350 threads -- You received this message because you are subscribed to the Google Groups "Jenkins Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/857ec8a7-42ad-411b-8cfe-ad5033bb4774%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: Is there a maximum number of parallel builds a pipeline job can run?
I need to retry it may be that the jobs never all started Branch: job_0 - (1ms in block) Branch: job_9830 - (3min 16sec in block) Branch: job_9831 - (No timing in block) -- You received this message because you are subscribed to the Google Groups "Jenkins Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/40a3d56b-2edb-4079-b30a-44f72dd7ff63%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: Is there a maximum number of parallel builds a pipeline job can run?
So 3 hr 30 min later the pipeline is locked up -- You received this message because you are subscribed to the Google Groups "Jenkins Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/e65fb62e-03cf-49f9-9a45-31297b1fd8f2%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Is there a maximum number of parallel builds a pipeline job can run?
I am doing testing of our Jenkins setup. I have a build job that sleeps for 30 sec. I ran a pipeline that runs 12,000 of these in parallel on 100 agents. Best possible time is 1 hr. The builds seem to finish in a reasonable amount of time. So after build queue is emptied and the pipeline continues to run for a lengthy amount of time. It has been running for 1 hr 30 min and is processing FINISHED job 8880. So 3120 more FINISHED jobs to process. It is unclear what the pipeline is doing? Is there documentation on the basic operations the pipeline is conducting after all the builds finish? Are there setting that affect this behavior? Could my Groovy script be causing this long delay? Thanks, Chris -- You received this message because you are subscribed to the Google Groups "Jenkins Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/07741116-4fc9-45c4-851f-484da065e098%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: How can I add builds to the queue a little at a time?
Thank you for your insights. This reinforces my thoughts after reading https://jenkins.io/blog/2016/11/21/gc-tuning/ -- You received this message because you are subscribed to the Google Groups "Jenkins Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/fd7b3d25-7206-4750-925a-0efefca9716d%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: pipline slow vs flow
I am not trying to say anything I just found it curious. We are doing testing to determine the optimal setup for Jenkins. We currently use flows calling flows. We believe we have too many builds in the build queue (3500). In the process of testing we saw this behavior and I was hoping for insight. We have seen the more builds we do the less efficient pipeline became. going from 1500 builds to 2000 builds the pipeline took twice as long. At 3500 builds - pipeline - 23 mins - flow - 8 min So how does this become negligible if the jobs are doing work? Does this overhead disappear? -- You received this message because you are subscribed to the Google Groups "Jenkins Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/d3dfdfd4-e5df-48e4-9aec-a11552c7040f%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
pipline slow vs flow
So I am doing testing on our current system.
We use flows now.
So I have a flow and a pipeline doing the same task:
- 100 executors
- 2000 builds
- build - echo 'Hello World'
flow time = 2 min 33 sec
pipeline time = 5 min 11 sec
After all the echos are complete the pipeline seems to hang before
completing
pipeline script
#!groovy import hudson.model.* stage('many_jobs') { parallel
(get_many_jobs(2000)) } def get_many_jobs(num_builds) { // The map we'll
store the parallel steps in before executing them. def stepsForParallel =
[:] for (int i = 0; i < num_builds; i++) { def sub_run_config = 'job_' + i
stepsForParallel[sub_run_config] = { build job: 'sq_echo', parameters:
[string(name: 'job_name', value: sub_run_config)] } } return
stepsForParallel }
Thanks
--
You received this message because you are subscribed to the Google Groups
"Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to jenkinsci-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/jenkinsci-users/1d862aad-b1fa-4b29-b09d-212cd528d810%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
How can I add builds to the queue a little at a time?
I am beginning the process of changing to pipelines. We currently use flows to run thousands of jobs(builds). Say we have 6000 builds to run on 100 Windows virtual machines and 6000 builds to run on 100 Linux virtual machines. To reduce available virtual machines with no jobs to run we try to keep the queue as full as possible. Each job takes between 1 - 20 min to execute. I need a way to not over load Jenkins. With our current workflow we have to restart Jenkins every couple of days as it becomes sluggish (we just purchased and are using a new server). We have not determined if Flows with Flows is a contributing factor but are beginning testing with Pipelines next week. We believe the number of jobs in the queue is the cause of our problems, but we have not found a better solution than having that many jobs. My thought was to try and trickle jobs to the queue. I would like to add a new job to the queue only if it drops below say 1000. I am open to other suggestions. -- You received this message because you are subscribed to the Google Groups "Jenkins Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/ca93b1a3-bf6c-4a0d-b41e-60ebf1d2d36d%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
