[JIRA] (JENKINS-59152) Jenkins fails to properly abort "bat" step
Title: Message Title Daniel Beck updated an issue Jenkins / JENKINS-59152 Jenkins fails to properly abort "bat" step Change By: Daniel Beck Labels: 2.190.2-rejected 2.190.3-rejected lts-candidate Add Comment This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.201589.1567181048000.5458.1574421300639%40Atlassian.JIRA.
[JIRA] (JENKINS-59152) Jenkins fails to properly abort "bat" step
Title: Message Title Oliver Gondža updated an issue Jenkins / JENKINS-59152 Jenkins fails to properly abort "bat" step Change By: Oliver Gondža Labels: 2.190.2-rejected 2.190.3-rejected lts-candidate Add Comment This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.201589.1567181048000.7797.1572791820625%40Atlassian.JIRA.
[JIRA] (JENKINS-59152) Jenkins fails to properly abort "bat" step
Title: Message Title Oliver Gondža commented on JENKINS-59152 Re: Jenkins fails to properly abort "bat" step Given the fix is disputed (and far from trivial, IMO), I am postponing the backport to 2.190.3 at least. Add Comment This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.201589.1567181048000.3334.1570538700154%40Atlassian.JIRA.
[JIRA] (JENKINS-59152) Jenkins fails to properly abort "bat" step
Title: Message Title Oliver Gondža updated an issue Jenkins / JENKINS-59152 Jenkins fails to properly abort "bat" step Change By: Oliver Gondža Labels: 2.190.2-rejected lts-candidate Add Comment This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.201589.1567181048000.3330.1570538641430%40Atlassian.JIRA.
[JIRA] (JENKINS-59152) Jenkins fails to properly abort "bat" step
Title: Message Title Marat Radchenko commented on JENKINS-59152 Re: Jenkins fails to properly abort "bat" step I do not agree that PR#4225 fully fixed this issue. Race conditions between multiple timers are still there. Shortening of softkill timeout makes issue less often but still possible. Add Comment This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.201589.1567181048000.3133.1570515180229%40Atlassian.JIRA.
[JIRA] (JENKINS-59152) Jenkins fails to properly abort "bat" step
Title: Message Title Oleg Nenashev updated an issue Jenkins / JENKINS-59152 Jenkins fails to properly abort "bat" step Change By: Oleg Nenashev Labels: lts-candidate Add Comment This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.201589.1567181048000.2999.1570493760216%40Atlassian.JIRA.
[JIRA] (JENKINS-59152) Jenkins fails to properly abort "bat" step
Title: Message Title Oleg Nenashev updated JENKINS-59152 Jenkins / JENKINS-59152 Jenkins fails to properly abort "bat" step Change By: Oleg Nenashev Status: Fixed but Unreleased Resolved Add Comment This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.201589.1567181048000.2995.1570493700468%40Atlassian.JIRA.
[JIRA] (JENKINS-59152) Jenkins fails to properly abort "bat" step
Title: Message Title Oleg Nenashev updated an issue Jenkins / JENKINS-59152 Jenkins fails to properly abort "bat" step Change By: Oleg Nenashev Component/s: core Add Comment This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.201589.1567181048000.2997.1570493700496%40Atlassian.JIRA.
[JIRA] (JENKINS-59152) Jenkins fails to properly abort "bat" step
Title: Message Title Oleg Nenashev updated JENKINS-59152 The fix was released in Jenkins 2.199 Jenkins / JENKINS-59152 Jenkins fails to properly abort "bat" step Change By: Oleg Nenashev Status: Open Fixed but Unreleased Resolution: Fixed Released As: Jenkins 2.199 Add Comment This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f) -- You received this message because you are
[JIRA] (JENKINS-59152) Jenkins fails to properly abort "bat" step
Title: Message Title Marat Radchenko commented on JENKINS-59152 Re: Jenkins fails to properly abort "bat" step See comments to PR#4216 for additional technical analysis. Add Comment This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.201589.1567181048000.4232.1569335220051%40Atlassian.JIRA.
[JIRA] (JENKINS-59152) Jenkins fails to properly abort "bat" step
Title: Message Title
Marat Radchenko updated an issue
Jenkins / JENKINS-59152
Jenkins fails to properly abort "bat" step
Change By:
Marat Radchenko
# Windows# Jenkins 2.176.1# Create pipeline:{code}node() { bat "ping 127.0.0.1 -n 10"}{code}# Run pipeline# Abort pipeline# View build logExpected: pipeline aborts fast and without any issuesActual ( 66% reproducibility is less than 100% ):# It takes pipeline 20s to abort# Build log contains "Click here to forcibly terminate running steps" and "After 20s process did not stop", indicating that Jenkins has issues with stopping the pipeline# "Click here to forcibly terminate running steps" link is still visible even after the build has finished # Sometimes ping processes are NOT terminated even when build has aborted. Issue analysis:# There is a race condition between 2 minute timer in {{hudson.util.ProcessTree.WindowsOSProcess#killSoftly}} introduced for JENKINS-17116 by [PR#3414|https://github.com/jenkinsci/jenkins/pull/3414] and 20s timer in {{org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep.Execution#stop}}. It is possible for {{DurableTaskStep}} to pretend that step was cancelled while it fact process is still running. Because of this race condition, it is possible to trick Jenkins into thinking that build has finished while if fact there are still processes running in workspace and potentially locking files there (this happens to us in practice).# {{org.jvnet.winp.WinProcess#sendCtrlC}} that is used in {{hudson.util.ProcessTree.WindowsOSProcess#killSoftly}} is NOT a proper way to terminate processes. Many apps do not interpret CTRL+C as a shutdown signal. ({{cmd.exe}} being the most important one here, because running {{bat}} in pipeline involved TWO {{cmd.exe}} - one running {{jenkins-wrapper.bat}} and second running {{jenkins-main.bat}}. Why you're not using [TerminateProcess function|https://docs.microsoft.com/en-us/windows/win32/api/processthreadsapi/nf-processthreadsapi-terminateprocess] from WinAPI?# There's a race condition between gathering of process list in {{hudson.util.ProcessTree.Windows#Windows}} constructor and killing of the processes, during which build can produce new processes that will not be attempted to be killed.# Usage of {{JENKINS_NODE_COOKIE}} to find what processes to kill is unreliable because 1) processes are free to alter their environment 2) [CreateProcessA|https://docs.microsoft.com/en-us/windows/win32/api/processthreadsapi/nf-processthreadsapi-createprocessa] allows to pass custom environment variables 3) It has unpredictable order 4) It doesn't match Jenkins behavior on Linux
[JIRA] (JENKINS-59152) Jenkins fails to properly abort "bat" step
Title: Message Title Marat Radchenko updated an issue Jenkins / JENKINS-59152 Jenkins fails to properly abort "bat" step Change By: Marat Radchenko Component/s: workflow-durable-task-step-plugin Component/s: core Component/s: durable-task-plugin Add Comment This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.201589.1567181048000.3244.1569245700063%40Atlassian.JIRA.
[JIRA] (JENKINS-59152) Jenkins fails to properly abort "bat" step
Title: Message Title Marat Radchenko commented on JENKINS-59152 Re: Jenkins fails to properly abort "bat" step All that I described in this issue can be reproduced by running org.jenkinsci.plugins.workflow.steps.durable_task.ShellStepTest#abort test on Windows. Sometimes it quickly passes. Sometimes it idles with 20s timeout. Sometimes it fails to kill ping process. Add Comment This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.201589.1567181048000.3018.1569242400043%40Atlassian.JIRA.
[JIRA] (JENKINS-59152) Jenkins fails to properly abort "bat" step
Title: Message Title Marat Radchenko updated an issue Jenkins / JENKINS-59152 Jenkins fails to properly abort "bat" step Change By: Marat Radchenko Component/s: durable-task-plugin Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.201589.1567181048000.5897.1567523040074%40Atlassian.JIRA.
[JIRA] (JENKINS-59152) Jenkins fails to properly abort "bat" step
Title: Message Title
Marat Radchenko updated an issue
Jenkins / JENKINS-59152
Jenkins fails to properly abort "bat" step
Change By:
Marat Radchenko
# Windows# Jenkins 2.176.1# Create pipeline:{code}node() { bat "ping 127.0.0.1 -n 10"}{code}# Run pipeline# Abort pipeline# View build logExpected: pipeline aborts fast and without any issuesActual (66% reproducibility):# It takes pipeline 20s to abort# Build log contains "Click here to forcibly terminate running steps" and "After 20s process did not stop", indicating that Jenkins has issues with stopping the pipeline# "Click here to forcibly terminate running steps" link is still visible even after the build has finishedIssue analysis:# There is a race condition between 2 minute timer in {{hudson.util.ProcessTree.WindowsOSProcess#killSoftly}} introduced for JENKINS-17116 by [PR#3414|https://github.com/jenkinsci/jenkins/pull/3414] and 20s timer in {{org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep.Execution#stop}}. It is possible for {{DurableTaskStep}} to pretend that step was cancelled while it fact process is still running. Because of this race condition, it is possible to trick Jenkins into thinking that build has finished while if fact there are still processes running in workspace and potentially locking files there (this happens to us in practice).# {{org.jvnet.winp.WinProcess#sendCtrlC}} that is used in {{hudson.util.ProcessTree.WindowsOSProcess#killSoftly}} is NOT a proper way to terminate processes. Many apps do not interpret CTRL+C as a shutdown signal. ({{cmd.exe}} being the most important one here, because running {{bat}} in pipeline involved TWO {{cmd.exe}} - one running {{jenkins-wrapper.bat}} and second running {{jenkins-main.bat}}. Why you're not using [TerminateProcess function|https://docs.microsoft.com/en-us/windows/win32/api/processthreadsapi/nf-processthreadsapi-terminateprocess] from WinAPI?# There's a race condition between gathering of process list in {{hudson.util.ProcessTree.Windows#Windows}} constructor and killing of the processes, during which build can produce new processes that will not be attempted to be killed.# Usage of {{JENKINS_NODE_COOKIE}} to find what processes to kill is unreliable because 1) processes are free to alter their environment 2) [CreateProcessA|https://docs.microsoft.com/en-us/windows/win32/api/processthreadsapi/nf-processthreadsapi-createprocessa] allows to pass custom environment variables 3) It has unpredictable order 4) It doesn't match Jenkins behavior on Linux
[JIRA] (JENKINS-59152) Jenkins fails to properly abort "bat" step
Title: Message Title
Marat Radchenko created an issue
Jenkins / JENKINS-59152
Jenkins fails to properly abort "bat" step
Issue Type:
Bug
Assignee:
Unassigned
Components:
core
Created:
2019-08-30 16:04
Environment:
Windows Jenkins 2.176.1
Priority:
Major
Reporter:
Marat Radchenko
Windows
Jenkins 2.176.1
Create pipeline:
node() {
bat "ping 127.0.0.1 -n 10"
}
Run pipeline
Abort pipeline
View build log
Expected: pipeline aborts fast and without any issues Actual (66% reproducibility):
It takes pipeline 20s to abort
Build log contains "Click here to forcibly terminate running steps" and "After 20s process did not stop", indicating that Jenkins has issues with stopping the pipeline
"Click here to forcibly terminate
