Re: Pipeline: Resource Locking Plugin hangs
tirsdag 13. mars 2018 15.52.32 UTC+1 skrev Steven Foster følgende: > > On Tuesday, March 13, 2018 at 1:02:02 PM UTC, Sverre Moe wrote: >> >> Looks like there is a fix for this in the works. >> https://github.com/jenkinsci/lockable-resources-plugin/pull/70 >> > > I *think* this fix is just for manual reservation/unreservation of a > resource by a user (there's a button in the list of resources) > That's too bad. I had my hopes up that fix would solve my problem. I have asked that question on the GitHub issue. For the time being we are locking around the parallel step with one resource on the Build stage and the same resource on Publish stage. This adds a severe bottleneck to the build time. Would be much better if we could use one resource lock for each parallel stage. -- You received this message because you are subscribed to the Google Groups "Jenkins Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/de2cda60-7ad7-4a99-83b8-542dd23dec32%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: Pipeline: Resource Locking Plugin hangs
On Tuesday, March 13, 2018 at 1:02:02 PM UTC, Sverre Moe wrote: > > Looks like there is a fix for this in the works. > https://github.com/jenkinsci/lockable-resources-plugin/pull/70 > I *think* this fix is just for manual reservation/unreservation of a resource by a user (there's a button in the list of resources) -- You received this message because you are subscribed to the Google Groups "Jenkins Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/cee55f32-131d-4480-a2ee-5e9a5e0fa97d%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: Pipeline: Resource Locking Plugin hangs
Looks like there is a fix for this in the works. https://github.com/jenkinsci/lockable-resources-plugin/pull/70 -- You received this message because you are subscribed to the Google Groups "Jenkins Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/567044aa-7731-49cf-ab6a-82f555658dd9%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: Pipeline: Resource Locking Plugin hangs
Running two builds of pipeline-test2 works fine. The second build gets all
three resources when the first build has released them.
So seems the problem is when locking on label. The build will only get the
first release lock.
tirsdag 13. mars 2018 08.44.21 UTC+1 skrev Sverre Moe følgende:
>
> I created two seperate pipeline jobs to test this with.
> Got the same behaviour.
>
> *pipeline-test1*
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> *def stepsForParallel = [:]stepsForParallel["nodeName4"] =
> transformIntoStep("nodeName4")parallel stepsForParalleldef
> transformIntoStep(nodeName) {return {node("master") {
> lock(label: "testing-repository") {println "Testing lock
> label"sleep 60}}}}*
> OUTPUT
>
> *[Pipeline] parallel[Pipeline] [nodeName4] { (Branch: nodeName4)[Pipeline]
> [nodeName4] node[nodeName4] Running on Jenkins in
> /var/lib/jenkins/workspace/pipeline-test[Pipeline] [nodeName4] {[Pipeline]
> [nodeName4] lock[nodeName4] Trying to acquire lock on [Label:
> testing-repository][nodeName4] Lock acquired on [Label:
> testing-repository][Pipeline] [nodeName4] {[Pipeline] [nodeName4]
> echo[nodeName4] Testing lock label[Pipeline] [nodeName4] sleep[nodeName4]
> Sleeping for 1 min 0 sec[Pipeline] [nodeName4] }[nodeName4] Lock released
> on resource [Label: testing-repository][Pipeline] [nodeName4] //
> lock[Pipeline] [nodeName4] }[Pipeline] [nodeName4] // node[Pipeline]
> [nodeName4] }[Pipeline] // parallel[Pipeline] End of PipelineFinished:
> SUCCESS*
>
> *pipeline-test2*
> def stepsForParallel = [:]
> stepsForParallel["nodeName1"] = transformIntoStep("nodeName1")
> stepsForParallel["nodeName2"] = transformIntoStep("nodeName2")
> stepsForParallel["nodeName3"] = transformIntoStep("nodeName3")
>
> parallel stepsForParallel
>
> def transformIntoStep(nodeName) {
> return {
> node("master") {
> if (nodeName.equals("nodeName1")) {
> lock(resource: 'testing1-repository') {
> println "Testing lock resource"
> sleep 30
> }
> }
>
> if (nodeName.equals("nodeName2")) {
> lock(resource: 'testing2-repository') {
> println "Testing lock resource"
> sleep 30
> }
> }
>
> if (nodeName.equals("nodeName3")) {
> lock(resource: 'testing3-repository') {
> println "Testing lock resource"
> sleep 30
> }
> }
> }
> }
> }
>
> *OUTPUT*
>
> *[Pipeline] parallel[Pipeline] [nodeName1] { (Branch: nodeName1)[Pipeline]
> [nodeName2] { (Branch: nodeName2)[Pipeline] [nodeName3] { (Branch:
> nodeName3)[Pipeline] [nodeName1] node[nodeName1] Running on Jenkins in
> /var/lib/jenkins/workspace/pipeline-test3[Pipeline] [nodeName2]
> node[Pipeline] [nodeName3] node[nodeName2] Running on Jenkins in
> /var/lib/jenkins/workspace/pipeline-test3@2[nodeName3] Running on Jenkins
> in /var/lib/jenkins/workspace/pipeline-test3@3[Pipeline] [nodeName1]
> {[Pipeline] [nodeName2] {[Pipeline] [nodeName3] {[Pipeline] [nodeName1]
> lock[nodeName1] Trying to acquire lock on [testing1-repository][nodeName1]
> Found 0 available resource(s). Waiting for correct amount: 1.[nodeName1]
> [testing1-repository] is locked, waiting...[Pipeline] [nodeName2]
> lock[nodeName2] Trying to acquire lock on [testing2-repository][nodeName2]
> Found 0 available resource(s). Waiting for correct amount: 1.[nodeName2]
> [testing2-repository] is locked, waiting...[Pipeline] [nodeName3]
> lock[nodeName3] Trying to acquire lock on [testing3-repository][nodeName3]
> Found 0 available resource(s). Waiting for correct amount: 1.[nodeName3]
> [testing3-repository] is locked, waiting...[nodeName1] Lock acquired on
> [testing1-repository][Pipeline] [nodeName1] {[Pipeline] [nodeName1]
> echo[nodeName1] Testing lock resource[Pipeline] [nodeName1]
> sleep[nodeName1] Sleeping for 30 sec[Pipeline] [nodeName1] }[nodeName1]
> Lock released on resource [testing1-repository][Pipeline] [nodeName1] //
> lock[Pipeline] [nodeName1] }[Pipeline] [nodeName1] // node[Pipeline]
> [nodeName1] }Aborted by Sverre Moe[Pipeline] [nodeName3] // lock[Pipeline]
> [nodeName2] // lock[Pipeline] [nodeName2] }[Pipeline] [nodeName3]
> }[Pipeline] [nodeName2] // node[Pipeline] [nodeName3] // node[Pipeline]
> [nodeName2] }[nodeName2] Failed in branch nodeName2[Pipeline] [nodeName3]
> }[nodeName3] Failed in branch nodeName3[Pipeline] // parallel[Pipeline] End
> of PipelineException:
> nullorg.jenkinsci.plugins.workflow.steps.FlowInterruptedException at
> org.jenkinsci.plugins.workflow.cps.CpsBodyExecution.cancel(CpsBodyExecution.java:245)
>
> at
> org.jenkinsci.plugins.workflow.steps.BodyExecution.cancel(BodyExecution.java:76)
>
> at
> org.jenkinsci.plugins.work
Re: Pipeline: Resource Locking Plugin hangs
I created two seperate pipeline jobs to test this with.
Got the same behaviour.
*pipeline-test1*
*def stepsForParallel = [:]stepsForParallel["nodeName4"] =
transformIntoStep("nodeName4")parallel stepsForParalleldef
transformIntoStep(nodeName) {return {node("master") {
lock(label: "testing-repository") {println "Testing lock
label"sleep 60}}}}*
OUTPUT
*[Pipeline] parallel[Pipeline] [nodeName4] { (Branch: nodeName4)[Pipeline]
[nodeName4] node[nodeName4] Running on Jenkins in
/var/lib/jenkins/workspace/pipeline-test[Pipeline] [nodeName4] {[Pipeline]
[nodeName4] lock[nodeName4] Trying to acquire lock on [Label:
testing-repository][nodeName4] Lock acquired on [Label:
testing-repository][Pipeline] [nodeName4] {[Pipeline] [nodeName4]
echo[nodeName4] Testing lock label[Pipeline] [nodeName4] sleep[nodeName4]
Sleeping for 1 min 0 sec[Pipeline] [nodeName4] }[nodeName4] Lock released
on resource [Label: testing-repository][Pipeline] [nodeName4] //
lock[Pipeline] [nodeName4] }[Pipeline] [nodeName4] // node[Pipeline]
[nodeName4] }[Pipeline] // parallel[Pipeline] End of PipelineFinished:
SUCCESS*
*pipeline-test2*
def stepsForParallel = [:]
stepsForParallel["nodeName1"] = transformIntoStep("nodeName1")
stepsForParallel["nodeName2"] = transformIntoStep("nodeName2")
stepsForParallel["nodeName3"] = transformIntoStep("nodeName3")
parallel stepsForParallel
def transformIntoStep(nodeName) {
return {
node("master") {
if (nodeName.equals("nodeName1")) {
lock(resource: 'testing1-repository') {
println "Testing lock resource"
sleep 30
}
}
if (nodeName.equals("nodeName2")) {
lock(resource: 'testing2-repository') {
println "Testing lock resource"
sleep 30
}
}
if (nodeName.equals("nodeName3")) {
lock(resource: 'testing3-repository') {
println "Testing lock resource"
sleep 30
}
}
}
}
}
*OUTPUT*
*[Pipeline] parallel[Pipeline] [nodeName1] { (Branch: nodeName1)[Pipeline]
[nodeName2] { (Branch: nodeName2)[Pipeline] [nodeName3] { (Branch:
nodeName3)[Pipeline] [nodeName1] node[nodeName1] Running on Jenkins in
/var/lib/jenkins/workspace/pipeline-test3[Pipeline] [nodeName2]
node[Pipeline] [nodeName3] node[nodeName2] Running on Jenkins in
/var/lib/jenkins/workspace/pipeline-test3@2[nodeName3] Running on Jenkins
in /var/lib/jenkins/workspace/pipeline-test3@3[Pipeline] [nodeName1]
{[Pipeline] [nodeName2] {[Pipeline] [nodeName3] {[Pipeline] [nodeName1]
lock[nodeName1] Trying to acquire lock on [testing1-repository][nodeName1]
Found 0 available resource(s). Waiting for correct amount: 1.[nodeName1]
[testing1-repository] is locked, waiting...[Pipeline] [nodeName2]
lock[nodeName2] Trying to acquire lock on [testing2-repository][nodeName2]
Found 0 available resource(s). Waiting for correct amount: 1.[nodeName2]
[testing2-repository] is locked, waiting...[Pipeline] [nodeName3]
lock[nodeName3] Trying to acquire lock on [testing3-repository][nodeName3]
Found 0 available resource(s). Waiting for correct amount: 1.[nodeName3]
[testing3-repository] is locked, waiting...[nodeName1] Lock acquired on
[testing1-repository][Pipeline] [nodeName1] {[Pipeline] [nodeName1]
echo[nodeName1] Testing lock resource[Pipeline] [nodeName1]
sleep[nodeName1] Sleeping for 30 sec[Pipeline] [nodeName1] }[nodeName1]
Lock released on resource [testing1-repository][Pipeline] [nodeName1] //
lock[Pipeline] [nodeName1] }[Pipeline] [nodeName1] // node[Pipeline]
[nodeName1] }Aborted by Sverre Moe[Pipeline] [nodeName3] // lock[Pipeline]
[nodeName2] // lock[Pipeline] [nodeName2] }[Pipeline] [nodeName3]
}[Pipeline] [nodeName2] // node[Pipeline] [nodeName3] // node[Pipeline]
[nodeName2] }[nodeName2] Failed in branch nodeName2[Pipeline] [nodeName3]
}[nodeName3] Failed in branch nodeName3[Pipeline] // parallel[Pipeline] End
of PipelineException:
nullorg.jenkinsci.plugins.workflow.steps.FlowInterruptedException at
org.jenkinsci.plugins.workflow.cps.CpsBodyExecution.cancel(CpsBodyExecution.java:245)
at
org.jenkinsci.plugins.workflow.steps.BodyExecution.cancel(BodyExecution.java:76)
at
org.jenkinsci.plugins.workflow.cps.steps.ParallelStepExecution.stop(ParallelStepExecution.java:67)
at org.jenkinsci.plugins.workflow.cps.CpsThread.stop(CpsThread.java:296) at
org.jenkinsci.plugins.workflow.cps.CpsFlowExecution$6.onSuccess(CpsFlowExecution.java:1083)
at
org.jenkinsci.plugins.workflow.cps.CpsFlowExecution$6.onSuccess(CpsFlowExecution.java:1072)
at
org.jenkinsci.plugins.workflow.cps.CpsFlowExecution$4$1.run(CpsFlowExecution.java:861)
at
org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$
Re: Pipeline: Resource Locking Plugin hangs
torsdag 8. mars 2018 17.06.12 UTC+1 skrev Sverre Moe følgende:
>
>
>
> torsdag 8. mars 2018 12.37.47 UTC+1 skrev Reinhold Füreder følgende:
>>
>> Very naïve thought(s):
>>
>>
>>
>> While “*Do not know why nodeName1 acquired the lock before nodeName
>> released.*“ might be just a matter of output flushing/buffering,
>>
> I thought so also, probably because of parallell execution.
>
>
> “*The entire build hangs and goes no further.*” sounds really frightening
>> to me.
>>
>>
>>
>> Maybe it is the usage of lock inside parallel step and/or some CPS
>> transformation reason, or some little resource misconfiguration issue!?
>>
>>
>>
>> Ignoring CPS transformations (due to parallel step) I assume/hope the
>> lock plugin does unit testing the “normal” parallelization usage of locking
>> resources?
>>
>
> Perhaps a bug in Lockable Resource Plugin.
> There is no output of CPS transformation exception, nor any other error
> output. It just hangs after the first acquired resource lock is finished.
>
>
We are using the Lockable Resource Plugin, but currently are locking around
the parallell step.
This causes a severe bottleneck when building. If several projects are
building simultaneous it increases the build time by a lot.
stage("build") {
def stepsForParallel = [:]
stepsForParallel[buildNode1] = transformIntoStep(buildNode1)
stepsForParallel[buildNode2] = transformIntoStep(buildNode2)
lock(resource: 'master-repository') {
parallel stepsForParallel
}
}
stage("publish") {
lock(resource: 'master-repository') {
publishYumArtifacts()
}
}
def transformIntoStep() {
return {
node(nodeName) {
preInstall()
compileAndBuild()
postInstall()
}
}
}
This is what I would like to do instead with the lock:
stage("build") {
def stepsForParallel = [:]
stepsForParallel[buildNode1] = transformIntoStep(buildNode1)
stepsForParallel[buildNode2] = transformIntoStep(buildNode2)
parallel stepsForParallel
}
stage("publish") {
lock(label: 'master-repository') {
publishYumArtifacts()
}
}
def transformIntoStep(nodeName) {
return {
node(nodeName) {
lock(resource: nodeName) {
preInstall()
}
compileAndBuild()
lock(resource: nodeName) {
postInstall()
}
}
}
}
When publishing yum artifacts we need to lock all the resources with label.
When building we lock only the build node resource, and if any other build
is going to publish it waits until all resources are free.
--
You received this message because you are subscribed to the Google Groups
"Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to jenkinsci-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/jenkinsci-users/fb1b0b1e-f672-4b6d-809f-c0f6a9931477%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Re: Pipeline: Resource Locking Plugin hangs
torsdag 8. mars 2018 12.37.47 UTC+1 skrev Reinhold Füreder følgende: > > Very naïve thought(s): > > > > While “*Do not know why nodeName1 acquired the lock before nodeName > released.*“ might be just a matter of output flushing/buffering, > I thought so also, probably because of parallell execution. “*The entire build hangs and goes no further.*” sounds really frightening > to me. > > > > Maybe it is the usage of lock inside parallel step and/or some CPS > transformation reason, or some little resource misconfiguration issue!? > > > > Ignoring CPS transformations (due to parallel step) I assume/hope the lock > plugin does unit testing the “normal” parallelization usage of locking > resources? > Perhaps a bug in Lockable Resource Plugin. There is no output of CPS transformation exception, nor any other error output. It just hangs after the first acquired resource lock is finished. -- You received this message because you are subscribed to the Google Groups "Jenkins Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/b7baa7e7-4042-4792-873b-6a1f47f8cf83%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
RE: Pipeline: Resource Locking Plugin hangs
Very naïve thought(s): While “Do not know why nodeName1 acquired the lock before nodeName released.“ might be just a matter of output flushing/buffering, “The entire build hangs and goes no further.” sounds really frightening to me. Maybe it is the usage of lock inside parallel step and/or some CPS transformation reason, or some little resource misconfiguration issue!? Ignoring CPS transformations (due to parallel step) I assume/hope the lock plugin does unit testing the “normal” parallelization usage of locking resources? -- You received this message because you are subscribed to the Google Groups "Jenkins Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/VI1P190MB025507E38826DF5579877B8CF7DF0%40VI1P190MB0255.EURP190.PROD.OUTLOOK.COM. For more options, visit https://groups.google.com/d/optout.
Pipeline: Resource Locking Plugin hangs
Lockable Resource Plugin
Lockable Resources Manager
Lockable Resources
resource: testing1-repository, labels:testing-repository
resource: testing2-repository, labels:testing-repository
resource: testing3-repository, labels:testing-repository
Configured these 3 resources in Jenkins. Different resource name and same
label on all three resources.
Pipeline script:
def stepsForParallel = [:]
stepsForParallel["nodeName1"] = transformIntoStep("nodeName1")
stepsForParallel["nodeName2"] = transformIntoStep("nodeName2")
stepsForParallel["nodeName3"] = transformIntoStep("nodeName3")
stepsForParallel["nodeName4"] = transformIntoStep("nodeName4")
parallel stepsForParallel
def transformIntoStep(nodeName) {
return {
node("master") {
if (nodeName.equals("nodeName4")) {
lock(label: "testing-repository") {
println "Testing lock label"
sleep 20
}
} else {
sleep 5 /* to make sure nodeName4 aquires its lock first. */
if (nodeName.equals("nodeName1")) {
lock(resource: 'testing1-repository') {
println "Testing lock resource"
sleep 10
}
}
if (nodeName.equals("nodeName2")) {
lock(resource: 'testing2-repository') {
println "Testing lock resource"
sleep 10
}
}
if (nodeName.equals("nodeName3")) {
lock(resource: 'testing3-repository') {
println "Testing lock resource"
sleep 10
}
}
}
}
}
}
[Pipeline] parallel
[Pipeline] [nodeName1] { (Branch: nodeName1)
[Pipeline] [nodeName2] { (Branch: nodeName2)
[Pipeline] [nodeName3] { (Branch: nodeName3)
[Pipeline] [nodeName4] { (Branch: nodeName4)
[Pipeline] [nodeName1] node
[nodeName1] Running on Jenkins in /var/lib/jenkins/workspace/pipeline-test
[Pipeline] [nodeName2] node
[nodeName2] Running on Jenkins in /var/lib/jenkins/workspace/pipeline-test@2
[Pipeline] [nodeName3] node
[nodeName3] Running on Jenkins in /var/lib/jenkins/workspace/pipeline-test@3
[Pipeline] [nodeName4] node
[Pipeline] [nodeName1] {
[Pipeline] [nodeName2] {
[Pipeline] [nodeName3] {
[nodeName4] Running on Jenkins in /var/lib/jenkins/workspace/pipeline-test@4
[Pipeline] [nodeName1] sleep
[nodeName1] Sleeping for 5 sec
[Pipeline] [nodeName2] sleep
[nodeName2] Sleeping for 5 sec
[Pipeline] [nodeName3] sleep
[nodeName3] Sleeping for 5 sec
[Pipeline] [nodeName4] {
[Pipeline] [nodeName4] lock
[nodeName4] Trying to acquire lock on [Label: testing-repository]
[nodeName4] Lock acquired on [Label: testing-repository]
[Pipeline] [nodeName4] {
[Pipeline] [nodeName4] echo
[nodeName4] Testing lock label
[Pipeline] [nodeName4] sleep
[nodeName4] Sleeping for 20 sec
[Pipeline] [nodeName1] lock
[nodeName1] Trying to acquire lock on [testing1-repository]
[nodeName1] Found 0 available resource(s). Waiting for correct amount: 1.
[nodeName1] [testing1-repository] is locked, waiting...
[Pipeline] [nodeName2] lock
[nodeName2] Trying to acquire lock on [testing2-repository]
[nodeName2] Found 0 available resource(s). Waiting for correct amount: 1.
[nodeName2] [testing2-repository] is locked, waiting...
[Pipeline] [nodeName3] lock
[nodeName3] Trying to acquire lock on [testing3-repository]
[nodeName3] Found 0 available resource(s). Waiting for correct amount: 1.
[nodeName3] [testing3-repository] is locked, waiting...
[nodeName1] Lock acquired on [testing1-repository]
[Pipeline] [nodeName4] }
[nodeName4] Lock released on resource [Label: testing-repository]
[Pipeline] [nodeName1] {
[Pipeline] [nodeName4] // lock
[Pipeline] [nodeName4] }
[Pipeline] [nodeName1] echo
[nodeName1] Testing lock resource
[Pipeline] [nodeName1] sleep
[nodeName1] Sleeping for 10 sec
[Pipeline] [nodeName4] // node
[Pipeline] [nodeName4] }
[Pipeline] [nodeName1] }
[nodeName1] Lock released on resource [testing1-repository]
[Pipeline] [nodeName1] // lock
[Pipeline] [nodeName1] }
[Pipeline] [nodeName1] // node
[Pipeline] [nodeName1] }
Does not work as I expected it to.
When nodeName4 releases the testing-repository which contains 3 resources,
then the other can acquire the lock on their resources. However only one of
them does so, the others do not. The entire build hangs and goes no
further.
[nodeName1] Lock acquired on [testing1-repository]
[nodeName4] Lock released on resource [Label: testing-repository]
[nodeName1] Lock released on resource [testing1-repository]
Do not know why nodeName1 acquired the lock before nodeName released.
This is the output I expected:
[nodeName4] Trying to acquire lock on [Label: testing-repository]
[nodeName4] Lock acquired on [Label: testing-repository]
[nodeName4] Lock released on resource [Label: testing-repository]
[nodeName1] L
