[JIRA] (JENKINS-37718) Block Pipeline job while upstream or downstream projects are building
Title: Message Title Steve Berube commented on JENKINS-37718 Re: Block Pipeline job while upstream or downstream projects are building This would be very useful for us as well. Add Comment This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-37718) Block Pipeline job while upstream or downstream projects are building
Title: Message Title Dario Simonetti commented on JENKINS-37718 Re: Block Pipeline job while upstream or downstream projects are building What about this situation? A / | \ B | C \ | / D A depends on B, C and D, while B and C depend on D. When build D is successful, it'll trigger a build of A, B and C. But you don't want A to build straight away as it'll get triggered again as soon as B and C are built. So A should block until both B and C are built. Freestyle allow doing this and it works very well, I don't understand why this is not possible for pipeline projects? Add Comment This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-37718) Block Pipeline job while upstream or downstream projects are building
Title: Message Title Jesse Glick commented on JENKINS-37718 Re: Block Pipeline job while upstream or downstream projects are building bq,. Every job has a Jenkinsfile that is about a hundred lines long. Combining it into one job makes it unreadable. Libraries, loadFile Also the parallel step mixes the output of all nodes which renders it close to unreadable. Fix in progress. Also being able to start the intermediate jobs individually isn't possible with one single pipeline job. Actually it is possible, though we do not yet provide a convenient framework for it; possibly Declarative will in the future. Add Comment This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-37718) Block Pipeline job while upstream or downstream projects are building
Title: Message Title
Jesse Glick edited a comment on JENKINS-37718
Re: Block Pipeline job while upstream or downstream projects are building
bq , . Every job has a Jenkinsfile that is about a hundred lines long. Combining it into one job makes it unreadable.Libraries, {{loadFile}}bq. Also the parallel step mixes the output of all nodes which renders it close to unreadable.Fix in progress.bq. Also being able to start the intermediate jobs individually isn't possible with one single pipeline job.Actually it is possible, though we do not yet provide a convenient framework for it; possibly Declarative will in the future.
Add Comment
This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c)
--
You received this message because you are subscribed to the Google Groups "Jenkins Issues" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-37718) Block Pipeline job while upstream or downstream projects are building
Title: Message Title Volker Gimple commented on JENKINS-37718 Re: Block Pipeline job while upstream or downstream projects are building Sorry if that's a dumb question, but I cannot yet see how these orchestration pipelines help address the issue at hand. I do get that they help define the build sequence nicely and without the need for the two options that spawned this discussion thread. However, as far as I understood it, I would either have to run the entire pipeline whenever one of the orchestrated jobs' source code changes or... still need dependency triggers on the individual projects The first scenario is not desirable as it tends to become a significant waste of resources once we are looking at more than just some five to ten jobs. And with the second scenario we're back at the point where having these blocking check boxes become a bit of a necessity. Of course one could start breaking things down into several smaller job groups so that the orchestration jobs make more sense, but then again this is an extra configuration effort that simply wasn't necessary when only using freestyle and matrix jobs and therefore feels like a bit of an overhead. Add Comment This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to
[JIRA] (JENKINS-37718) Block Pipeline job while upstream or downstream projects are building
Title: Message Title
Joshua Hoblitt edited a comment on JENKINS-37718
Re: Block Pipeline job while upstream or downstream projects are building
I believe the advice was to use a pipeline job to orchestrate other jobs. E.g.{code:java}stage('foo') { retry(retries) {build job: 'bar' }}{code}
Add Comment
This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c)
--
You received this message because you are subscribed to the Google Groups "Jenkins Issues" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-37718) Block Pipeline job while upstream or downstream projects are building
Title: Message Title
Joshua Hoblitt commented on JENKINS-37718
Re: Block Pipeline job while upstream or downstream projects are building
I believe the advice was to use a pipeline job orchestrate other jobs. E.g.
stage('foo') {
retry(retries) {
build job: 'bar'
}
}
Add Comment
This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c)
--
You received this message because you are subscribed to the Google Groups "Jenkins Issues" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-37718) Block Pipeline job while upstream or downstream projects are building
Title: Message Title Thorsten Meinl edited a comment on JENKINS-37718 Re: Block Pipeline job while upstream or downstream projects are building No, there are good reason to split this into multiple jobs. One being that the support for multiple SCM checkouts from Git in Jenkins completely sucks. And maintainability. Every job has a Jenkinsfile that is about a hundred lines long. Combining it into one job makes it unreadable. Also the parallel step mixes the output of all nodes which renders it close to unreadable. Also being able to start the intermediate jobs individually isn't possible with one single pipeline job. Shall I continue ;-) Add Comment This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-37718) Block Pipeline job while upstream or downstream projects are building
Title: Message Title Thorsten Meinl commented on JENKINS-37718 Re: Block Pipeline job while upstream or downstream projects are building No, there are good reason to split this into multiple jobs. One being that the support for multiple SCM checkouts from Git in Jenkins completely sucks. And maintainability. Every job has a Jenkinsfile that is about a hundred lines long. Combining it into one job makes it unreadable. Also the parallel step mixes the output of all nodes which renders it close to unreadable. Shall I continue Add Comment This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-37718) Block Pipeline job while upstream or downstream projects are building
Title: Message Title Jesse Glick commented on JENKINS-37718 Re: Block Pipeline job while upstream or downstream projects are building G combines the results from D and F Generally better to make this all one Pipeline job. Add Comment This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-37718) Block Pipeline job while upstream or downstream projects are building
Title: Message Title Volker Gimple commented on JENKINS-37718 Re: Block Pipeline job while upstream or downstream projects are building Hi Thorsten! Thanks for chiming in What I ended up doing was this: For every pipeline job I needed I created a freestyle job as the "trigger buddy" for the pipeline job. The pipeline job does not get any build triggers at all, while the "trigger buddy" has a combination of upstream dependencies and scm triggers that determine when the pipeline should run. As it is a freestyle job, you have a possibility to block it while its up/downstream dependencies are building. The only build step the trigger buddy has is to invoke the pipeline project and wait for its completion. It's a bit of an overhead to manage, but as long as pipelines do not have the blocking options I don't see an easier alternative. Most of the time this approach works fine, but when the load on the Jenkins server became very high we've seen synchronization problems (Pipeline steps hanging or the trigger buddy waiting for the pipeline to finish while it already has), but it's probably unreasonable to expect Jenkins to cope with extreme situations and we managed to throttle the builds to the point where these problems disappeared. If you are concerned about wasting resources because Jenkins tends to pick sub optimum paths through your dependency tree you might want to have a look at the Dependency Queue Plugin. The last maintained version seems incompatible with Jenkins 2.x, but if you go ahead and build the latest sources from github you'll get it working. It also helps to reduce the dependency tree as much as possible - I initially had ours reflect the source code dependencies one by one, but if you reduce it to only absolutely the necessary paths (e.g. you have 3 projects A, B and C; A triggers B and C and B triggers C then it's better to not list A in the upstream dependencies of C because A will implicitly trigger C anyway) then Jenkins will run through your build a lot smoother. Add Comment This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c)
[JIRA] (JENKINS-37718) Block Pipeline job while upstream or downstream projects are building
Title: Message Title Thorsten Meinl commented on JENKINS-37718 Re: Block Pipeline job while upstream or downstream projects are building I also don't see how Lockable Resources may replace this very useful functionality of FreeStyle jobs. Is there any reason, why you don't want to implement this for Pipeline jobs? Our use case is a chain of jobs such as B-C-D / \ A - E - F - G G combines the results from D and F and it doesn't make sense if either D or F (or any of their upstream jobs) are currently executing, because the current results will be replaced quite soon by new results. It doesn't hurt but it's a huge waste of resources. Add Comment This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-37718) Block Pipeline job while upstream or downstream projects are building
Title: Message Title Jesse Glick commented on JENKINS-37718 Re: Block Pipeline job while upstream or downstream projects are building Did not understand the last comment at all. Add Comment This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-37718) Block Pipeline job while upstream or downstream projects are building
Title: Message Title Andras Gaal commented on JENKINS-37718 Re: Block Pipeline job while upstream or downstream projects are building I am affraid lock doe not help. Lets have following jobs: A, B, C, D. A, B, C should block D, but D must not lock A, B, or C. A, B, C are different jobs and can run paralell. Add Comment This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-37718) Block Pipeline job while upstream or downstream projects are building
Title: Message Title Andras Gaal edited a comment on JENKINS-37718 Re: Block Pipeline job while upstream or downstream projects are building I am affraid lock doe does not help.Lets have following jobs: A, B, C, D.A, B, C should block D, but D must not lock A, B, or C. A, B, C are different jobs and can run paralell. Add Comment This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-37718) Block Pipeline job while upstream or downstream projects are building
Title: Message Title Jesse Glick commented on JENKINS-37718 Re: Block Pipeline job while upstream or downstream projects are building hard to maintain lock can take a computed lock name, if that helps. Add Comment This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-37718) Block Pipeline job while upstream or downstream projects are building
Title: Message Title Volker Gimple commented on JENKINS-37718 Re: Block Pipeline job while upstream or downstream projects are building The Lockable Resources plugin is a fine tool. However in the scenario described in the preceding long paragraphs it's not much use because one would either have to use a very broad lock (bad for build efficiency) or define a lock for every identifiable file group (hard to maintain if there are many). Add Comment This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-37718) Block Pipeline job while upstream or downstream projects are building
Title: Message Title Jesse Glick commented on JENKINS-37718 Re: Block Pipeline job while upstream or downstream projects are building A synchronization primitive would be really useful. Did not read any of the preceding long paragraph, but to this: install the Lockable Resources plugin and you will get the lock step. Add Comment This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-37718) Block Pipeline job while upstream or downstream projects are building
Title: Message Title Joshua Hoblitt commented on JENKINS-37718 Re: Block Pipeline job while upstream or downstream projects are building I echo all of Volker Gimple's comments. A synchronization primitive would be really useful. Add Comment This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-37718) Block Pipeline job while upstream or downstream projects are building
Title: Message Title Volker Gimple commented on JENKINS-37718 Re: Block Pipeline job while upstream or downstream projects are building Hi Jesse, First of all: Thanks for looking into this! What I can't get my head around is how to handle the problem of potential file overwrites without the ability to define mutually exclusive execution of different projects. Consider the simplified scenario where project A builds a shared object file and after a successful build stores the results (shared object file plus header file(s) to go with it) on a file server (simple file copy). Project B builds an executable that uses the shared object and header file(s) built and copied by project A. If I cannot make sure that project A and project B never build simultaneously it may (and therefore sooner or later will) happen that project A updates the files on the server while project B is accessing them. If that happens on a Linux node it'll lead to build errors in project B, while on a Windows node the deployment step of project A will fail. The only real workaround I have found so far (except for merging project A and project B into one project - an approach that becomes less of an option if you have not just one but numerous projects of type A) is to split the definition of either project into one project project that holds the pipeline definition (but does not trigger on SCM changes or upstream builds) and define the build trigger(s), dependenc(ies) and synchronization responsibilities in a freestyle project that just triggers the pipeline project and blocks until its execution finishes. This works for me and the overhead caused by the additional freestyle project is manageable, it just feels like an unnecessary detour given the fact that in principle the blocking functionality is available on all the other project types. Add Comment This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c)
[JIRA] (JENKINS-37718) Block Pipeline job while upstream or downstream projects are building
Title: Message Title Jesse Glick commented on JENKINS-37718 Re: Block Pipeline job while upstream or downstream projects are building Unlikely to implement anything like this. Add Comment This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-37718) Block Pipeline job while upstream or downstream projects are building
Title: Message Title Jesse Glick updated an issue Jenkins / JENKINS-37718 Block Pipeline job while upstream or downstream projects are building Change By: Jesse Glick Labels: blocking downstream pipeline upstream Add Comment This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-37718) Block Pipeline job while upstream or downstream projects are building
Title: Message Title Jesse Glick updated an issue Jenkins / JENKINS-37718 Block Pipeline job while upstream or downstream projects are building Change By: Jesse Glick Component/s: workflow-job-plugin Component/s: pipeline Component/s: workflow-multibranch-plugin Add Comment This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-37718) Block Pipeline job while upstream or downstream projects are building
Title: Message Title Volker Gimple updated an issue Jenkins / JENKINS-37718 Block Pipeline job while upstream or downstream projects are building Change By: Volker Gimple Priority: Minor Major Add Comment This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-37718) Block Pipeline job while upstream or downstream projects are building
Title: Message Title Volker Gimple created an issue Jenkins / JENKINS-37718 Block Pipeline job while upstream or downstream projects are building Issue Type: Improvement Assignee: Jesse Glick Components: workflow-multibranch-plugin, workflow-plugin Created: 2016/Aug/26 2:04 PM Environment: Jenkins 2.7.2, Pipeline-Job 2.5 Labels: workflow blocking upstream downstream pipeline Priority: Minor Reporter: Volker Gimple Maybe it's just me, but I think having a possibility on Pipeline and Multibranch Pipeline projects to block execution while up- or downstream-projects are building would be beneficial, especially when migrating existing installations (see my posts in stackoverflow and in Jenkins CI). With Freestyle and Matrix projects there was the option of going to the "Advanced Project Options" and specify that the build should be blocked when upstream/downstream project is building. For Pipeline and Multibranch this option is unavailable, and I did not find a means of synchronizing a Pipeline/Multibranch project to existing freestyle/matrix projects without orchestrating this in another pipeline script. However, having to do that with an installation of about 400 inter-dependent projects is a huge block of work that has to be sorted out in one go. Having those "block when upstream/downstream project is building" options would allow for a gradual and smoother migration.
