[JIRA] (JENKINS-49710) Pipelines run under heavy load sometimes hang running Docker

[abdulla.haw...@saucelabs.com (JIRA)]

Title: Message Title

Abdulla Hawara edited a comment on JENKINS-49710   Re: Pipelines run under heavy load sometimes hang running Docker My *theory* about the issue: Jenkins has `*Text file busy*` error coming from `*durable*` which is used by `*pipeline nodes and processes*` plugin. This issue is easily reproducible by running any command e.g. `*sh "echo 'Hello'"*` in any jenkins job many times in parallel. This hang is caused when running a docker container using the *docker plugin* (the container will stay alive) , but when we run `*docker*` manually using *sh 'docker ...'* it never hangs, but that `*Text file busy*` error still appears because we run a lot of jobs at the same time. The container in this case will stay running forever     {color:#ff0000}To reproduce {color} {code:java} Text file busy{code} error, please follow these steps:   1. Run Jenkins locally `docker run -p 8080:8080 -p 50000:50000 jenkins/jenkins:lts` ver. 2.89.4 2. install just the recommended plugins on it 3. create new pipeline projects called `*hello*` and `*hello2*` 4. put this code in ‘hello’ : {code:java} node{     sh "echo 'hello'" } {code} and this in `*hello2*`: {code:java} node{     sh "echo 'hello'"     sleep(2) } {code} 5. Create a new pipeline project called `*runner*` and put this inside: {code:java} COUNTER = 0 node{     def jobs = [:]          // add 24 instances of the same test to run them later in parallel     24.times {         jobs[('runner' + COUNTER++)] = {triggerProject('hello')()}     }          // add 24 instances of the same test to run them later in parallel     24.times {         jobs[('runner' + COUNTER++)] = {triggerProject('hello2')()}     }          // run them all 20 times in parallel     20.times {         parallel jobs     } } def triggerProject(jobName) {     return {         try{             build job: jobName, parameters: [string(name: 'VALUE', value: String.valueOf(COUNTER++))]         } catch (ex){             println ex         }     } } {code} 6. Go to your jenkins configurations and change the executors to `*50*` 7. try to run the runner once and if you got some sandbox exception, go to the in-procces Script Approval in the Manage Jenkins page and click approve for all commands 8. run runner again and you will see that some of `*hello*` and `*hello2*` has `*text file busy*` error The logs you get afterwards: {code:java} Running in Durability level: MAX_SURVIVABILITY [Pipeline] node Running on Jenkins in /var/jenkins_home/workspace/hello@6 [Pipeline] { [Pipeline] sh [hello@6] Running shell script sh: 1: /var/jenkins_home/workspace/hello@6@tmp/durable-a771d7dd/script.sh: Text file busy [Pipeline] } [Pipeline] // node [Pipeline] End of Pipeline ERROR: script returned exit code 2 Finished: FAILURE {code} Add Comment   This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.