[JIRA] (JENKINS-60434) "Prepare for shutdown" should continue executing already running pipelines to completion
Title: Message Title Reinhold Füreder commented on JENKINS-60434 Re: "Prepare for shutdown" should continue executing already running pipelines to completion Thomas de Grenier de Latour Thanks for the insight! And wow, that is impressive and I am not sure you got that right in the first attempt Add Comment This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.203506.1576045591000.5331.1576068900522%40Atlassian.JIRA.
[JIRA] (JENKINS-60434) "Prepare for shutdown" should continue executing already running pipelines to completion
Title: Message Title
Thomas de Grenier de Latour commented on JENKINS-60434
Re: "Prepare for shutdown" should continue executing already running pipelines to completion
In case it can be useful to anyone, here is the "planned upgrade" process we have for Jenkins in my company. It relies on a custom quiet-mode implementation we've implemented in an internal plugin, which basically allows already running builds to terminate (including Pipelines), but forbids starting execution of new builds (expect if they are necessary for termination of the already running builds). The overall process is automated (we have many Jenkins instances), and it goes like this:
activate the custom quiet-down mode (forbid starting new builds)
poll Jenkins until it's idle, for up to X minutes, and then do the upgrade (including an actual restart)
on time-out of this polling, cancel the planned upgrade (cancel the custom quiet-mode), and retry it all later (sometimes we have to find arrangements with users, so that they don't launch their freaking 18 hours tests suite on the day we are planning to do an upgrade)
We don't have plans/time to publish and maintain this as a community plugin, but if someone wants to do something similar, I will dump the code below, feel free to reuse what you want. Note that we would probably never had written this code if we had not been bitten many times by JENKINS-34256. A few years ago, we were simply using the standard Jenkins quiet-mode, but then stuck Pipelines (when the upgrade was cancelled) really became an issue... Now that JENKINS-34256 is fixed, I don't know, we might consider going back to this standard solution. But I think our users prefer having their Pipelines finished before the upgrade, rather than paused/resumed (mainly because the "resume" part is not always smooth: some plugins upgrades might break compatibility of the serialized data, etc.). Anyway, this is the "interesting" part of the code, the QuietDownQueueTaskDispatcher, which filters which new Queue.Item can actually be started when in (custom) quiet-mode.
@Extension
public class QuietDownQueueTaskDispatcher extends QueueTaskDispatcher {
@Inject
QuietDownStateManager quietDownStateManager;
// key: upstreamProject+upstreamBuild from an UpstreamCause
// value: true if children builds should be allowed to run
private ConcurrentHashMap knownUpstreamCauses = new ConcurrentHashMap<>();
// used to decide when cache should be flushed
private AtomicLong quietDownTimestamp = new AtomicLong(0l);
@Override
public @CheckForNull CauseOfBlockage canRun(Queue.Item item) {
QuietDownState currentState = quietDownStateManager.getState();
if (!currentState.isDown()) {
return null;
}
// flush cache if quietDown state has changed
if (quietDownTimestamp.getAndSet(currentState.since()) != currentState.since()) {
knownUpstreamCauses.clear();
}
Queue.Task task = item.task;
// always allow some kind of tasks
if (task instanceof NonBlockingTask || task instanceof ContinuedTask) {
return null;
}
// allow build task because of its upstream cause
if (hasAllowingCause(item.getCauses())) {
return null;
}
// not allowed, let's expl
[JIRA] (JENKINS-60434) "Prepare for shutdown" should continue executing already running pipelines to completion
Title: Message Title Ulrich Köhler commented on JENKINS-60434 Re: "Prepare for shutdown" should continue executing already running pipelines to completion Another use case: The ThinBackup plugin sets Jenkins to shutdown and waits for all jobs to finish. But Pipeline Jobs never finishes: dead lock. Add Comment This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.203506.1576045591000.5040.1576058402783%40Atlassian.JIRA.
[JIRA] (JENKINS-60434) "Prepare for shutdown" should continue executing already running pipelines to completion
Title: Message Title Reinhold Füreder created an issue Jenkins / JENKINS-60434 "Prepare for shutdown" should continue executing already running pipelines to completion Issue Type: Improvement Assignee: Unassigned Components: workflow-cps-plugin Created: 2019-12-11 06:26 Priority: Major Reporter: Reinhold Füreder Based on Devin Nusbaum's comment from JENKINS-34256: A fix for this issue was just released in Pipeline: Groovy Plugin version 2.78. I think there is/was some confusion as to the expected behavior (myself included!), so let me try to clarify: When Jenkins prepares for shutdown, all running Pipelines are paused, and this is the intended behavior. The unintended behavior was that if you canceled shutdown, Pipelines remained paused. This has been fixed in 2.78; Pipelines will now resume execution if shutdown is canceled. Before 2.78, you had to manually pause and unpause each Pipeline to get it to resume execution, or restart Jenkins. Additionally, preparing Jenkins for shutdown and canceling shutdown now each cause a message to be printed to Pipeline build logs indicating that the Pipeline is being paused or resumed due to shutdown so that it is easier to understand what is happening. Based on comments here and elsewhere, I think some users would prefer a variant of "Prepare for shutdown" in which Pipelines continue executing to completion, the same as other types of jobs like Freestyle. If that is something you want, please open a new ticket, describing your use case and the desired behavior. [...] If there is some other aspect of this issue that you would like to see addressed, or a different behavior you would prefer, please open a new ticket describing your particu
