[JIRA] (JENKINS-59158) Support running Job DSL scripts in parallel in pipeline with DISABLE or DELETE action
Title: Message Title
Daniel Carrington created an issue
Jenkins / JENKINS-59158
Support running Job DSL scripts in parallel in pipeline with DISABLE or DELETE action
Issue Type:
New Feature
Assignee:
Daniel Spilker
Components:
job-dsl-plugin
Created:
2019-08-30 21:40
Environment:
Jenkins 1.176.2 job-dsl-plugin 1.75
Labels:
pipeline performance
Priority:
Minor
Reporter:
Daniel Carrington
One use case for running multiple Job DSL build steps would be to accumulate jobs defined in multiple repositories. Consider a pipeline that does:
stage ("repo a") {
checkout "our/repo-a"
jobDsl "jobs/**/*.groovy"
}
stage ("repo b") {
checkout "our/repo-b"
jobDsl "jobs/**/*.groovy"
}
I run multiple Job DSL build steps in order to speed up the Job DSL configuration seed job. I have about 520 active jobs on my Jenkins instance, and they are all managed in a single repository. They are, thankfully, somewhat organized by project codename. My Job DSL pipeline reads something like:
List folders = ['project-a', 'project-b', 'project-c', ...] // roughly 10-20
[JIRA] (JENKINS-44142) Step jobDsl can be used at most once in pipeline with DELETE
Title: Message Title
Daniel Carrington commented on JENKINS-44142
Re: Step jobDsl can be used at most once in pipeline with DELETE
One use case for running multiple Job DSL build steps would be to accumulate jobs defined in multiple repositories. Consider a pipeline that does:
stage ("repo a") {
checkout "our/repo-a"
jobDsl "jobs/**/*.groovy"
}
stage ("repo b") {
checkout "our/repo-b"
jobDsl "jobs/**/*.groovy"
}
I run multiple Job DSL build steps in order to speed up the Job DSL configuration seed job. I have about 520 active jobs on my Jenkins instance, and they are all managed in a single repository. They are, thankfully, somewhat organized by project codename. My Job DSL pipeline reads something like:
List folders = ['project-a', 'project-b', 'project-c', ...] // roughly 10-20 projects
def startupTasks = [:]
startupTasks["Folder Setup"] = {
node("job_dsl") { stage("Folder Setup) {
checkout scm
sh "generate_some_files_read_by_job_dsl_code"
jobDsl targets: "jobs/folders.groovy"
} }
}
startupTasks["Self-Check"] = {
// Checks that all .groovy files in jobs/** are consumed by exactly one jobDsl build step
}
parallel startupTasks
def allProjectJobs = [:]
for (int i = 0; i < folders.length; i++) {
String projname = folders[i]
allProjectJobs["Project $projname"] = {
node("job_dsl") { stage("Project $projname") {
checkout scm
// Other versions of this have used stash, see commentary below
sh "generate_some_files_read_by_job_dsl_code"
// Classes in src/ are used to implement templates that set up each project similarly.
jobDsl targets: "jobs/$projname/*.groovy", additionalClasspath: "src/"
} }
}
}
parallel allProjectJobs
I use parallel Job DSL execution because I see slow Job DSL performance on running a lot of Job DSL code to generate this many jobs. I have a total of 81 files in my jobs/ folder, and some representative runtimes that I see for each group of files, in the Job DSL build step only, is:
N files in project
total filesize (bytes)
runtime
16
54863
2m 55s
[JIRA] (JENKINS-44142) Step jobDsl can be used at most once in pipeline with DELETE
Title: Message Title Daniel Carrington commented on JENKINS-44142 Re: Step jobDsl can be used at most once in pipeline with DELETE I have used the Job DSL plugin in production for over a year with the DISABLE action in place and I hit a similar issue when using multiple job dsl build steps in a single pipeline. The fix for this issue changed the behavior of the job dsl plugin when using multiple job dsl build steps in parallel. Is that (parallel job dsl build steps) a use case which is interesting to anyone else? (The new behavior of the Job DSL plugin ended up disabling many of my jobs. I spent a day with a mostly broken environment because I had to track this down.) Based on the discussion here, it seems like the correct thing to do is to use the IGNORE behavior on all uses of the job dsl build step except the very last one, which should not run in parallel with any other job dsl build step. Looking at the available hook points in hudson.model.Run and hudson.model.Job, I think that there isn't a great way to solve this. We could register a new hudson.model.listeners.RunListener that disables or deletes jobs after a successful job completes (and runs Job DSL build steps that DISABLE or DELETE jobs). Has that idea already been considered? Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.181769.1494322919000.3123.1565823900600%40Atlassian.JIRA.
[JIRA] (JENKINS-56689) Much-used String.join() method not in default whitelist
Title: Message Title Daniel Carrington created an issue Jenkins / JENKINS-56689 Much-used String.join() method not in default whitelist Issue Type: Improvement Assignee: Daniel Carrington Components: script-security-plugin Created: 2019-03-22 18:51 Priority: Minor Reporter: Daniel Carrington The String.join() method, as referenced in JENKINS-37527, is not actually whitelisted by default. String.split() is whitelisted by default. Shouldn't these be whitelisted? Add Comment This message was sent
[JIRA] (JENKINS-56688) pipeline-syntax Global Variables params.get doesn't work
Title: Message Title Daniel Carrington started work on JENKINS-56688 Change By: Daniel Carrington Status: Open In Progress Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-56688) pipeline-syntax Global Variables params.get doesn't work
Title: Message Title Daniel Carrington commented on JENKINS-56688 Re: pipeline-syntax Global Variables params.get doesn't work https://github.com/jenkinsci/workflow-cps-plugin/pull/275 Minor documentation update fixing the words in the help.jelly to recommend Map.getOrDefault instead. Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-56688) pipeline-syntax Global Variables params.get doesn't work
Title: Message Title Daniel Carrington updated an issue Jenkins / JENKINS-56688 pipeline-syntax Global Variables params.get doesn't work Change By: Daniel Carrington Component/s: workflow-cps-plugin Component/s: workflow-job-plugin Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-56688) pipeline-syntax Global Variables params.get doesn't work
Title: Message Title Daniel Carrington created an issue Jenkins / JENKINS-56688 pipeline-syntax Global Variables params.get doesn't work Issue Type: Bug Assignee: Jesse Glick Components: workflow-job-plugin Created: 2019-03-22 17:36 Environment: jenkins 1.50.2, all workflow plugin versions Priority: Minor Reporter: Daniel Carrington In Pipeline, doing params.get as recommended by Jesse Glick in: https://github.com/jenkinsci/workflow-cps-plugin/commit/7c29a4d24f575e61d80a7aed7b48d6ba5750a5a6 doesn't work because Groovy Map.get(Object, Object) actually tries to modify the map; whereas the params Map is an UnmodifiableMap. When I tried this code, I got the exception: java.lang.UnsupportedOperationException at java.util.Collections$UnmodifiableMap.put(Collections.java:1457) at org.codehaus.groovy.runtime.DefaultGroovyMethods.get(DefaultGroovyMethods.java:7080) I recommend instead Map.getOrDefault, which does not try to modify the Map. Pull request incoming for an update to documentation.
