date:20180318

absolute path customWorkspace with docker-plugin and declarative pipeline

2018-03-18 Thread Alex Suttmiller

Is it possible to use absolute paths for the workspace at the time the
docker container is being created/run with declarative pipeline? I have
done this before with mesos docker cloud, but I am hoping to replicate that
ability locally, without having a mesos cluster on my localhost.

An example pipeline:
pipeline {
agent none
stages {
stage('Foo') {
agent {
docker {
label 'dind'
image 'alpine'
*customWorkspace '/foo'*
}
}

But when I try this, I end up with the results:

sh: can't create /foo@tmp/durable-5a40a590/jenkins-log.txt: nonexistent
directory
sh: can't create /foo@tmp/durable-5a40a590/jenkins-result.txt.tmp: nonexistent
directory
mv: can't rename '/foo@tmp/durable-5a40a590/jenkins-result.txt.tmp': No such
file or directory

This post leads me to believe it may not be possible:
https://github.com/jenkinsci/docker-plugin/issues/540
This post leads me to believe it may be possible:
https://groups.google.com/d/msg/jenkinsci-users/_YNnkdYRXoE/4LifSIOoAAAJ,
but that I haven't figured out the right combination of volumes and
containers.

I am using docker-compose to create my environment. Can recreate locally
from this branch:
https://github.com/NeverOddOrEven/dind-jenkins/commits/custom-ws-dind-agent-not-working.

Any help would be truly appreciated! Thanks.

--
You received this message because you are subscribed to the Google Groups
"Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to jenkinsci-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/jenkinsci-users/01044517-cbf6-4dbb-9b45-17516a903b9d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Windows agent can't delete workspace

2018-03-18 Thread Mark Waite

On Sun, Mar 18, 2018 at 5:40 PM Sean Talts  wrote:

> Thank you both! I particularly like the idea of just deleting before the
> job instead of after. I managed to get something else working too, though I
> think it could be masking the problem. I now run attrib -r -s /s /d before
> deleting (before the run and after, just in case). Seems to be pretty quick?
>
> A run with this:
> http://d1m1s1b1.stat.columbia.edu:8080/blue/organizations/jenkins/Stan/detail/develop/82/pipeline/55
> full Jenkinsfile:
> https://github.com/stan-dev/stan/blob/50be575d95718d077fda81a05d98289392f9acc3/Jenkinsfile
>
> Thanks again!
>
>
If you're willing to use an "attrib" command, then you might also consider
"robocopy /mir" from an empty directory to the workspace directory.  I had
problems deleting directories when they contained a directory structure
that was too long.  Using "robocopy /mir" from an empty directory to the
destination directory was able to delete those too long files when the
Windows "rmdir" command would not.

I preferred that solution in my case because Robocopy is a standard part of
Windows 7 and Windows 10.

Mark Waite



> On Sunday, March 18, 2018 at 2:43:56 AM UTC-4, benjamin.a.lau wrote:
>>
>> We've used scripting around the sysinternals handle tool[1] to make
>> sure that everything which is touching files in the workspace is
>> actually terminated before trying to delete the workspace. We were
>> having more issues with this when trying to delete workspaces at the
>> ends of jobs so in many cases we just let it be and only delete the
>> workspace on the next run.
>>
>> Cheers,
>> Ben
>>
>> [1] https://docs.microsoft.com/en-us/sysinternals/downloads/handle
>>
>> On Sat, Mar 17, 2018 at 3:15 PM, Christian Gagneraud 
>> wrote:
>> > On 18 March 2018 at 03:21, Sean Talts  wrote:
>> >> Hey all,
>> >>
>> >> I'm having this sudden crazy problem where my Windows agent can't
>> delete
>> >> some files in its workspace anymore, failing all builds. It's getting
>> a
>> >> java.nio.file.AccessDeniedException on some files that it created. You
>> can
>> >> see the full exceptions at the end of the log here:
>> >> http://d1m1s1b1.stat.columbia.edu:8080/job/Stan/job/develop/80/console
>> also
>> >> copied below.
>> >>
>> >> I've tried so many things, including switching JVMs on master and
>> agent,
>> >> upgrading and switching to LTS, uninstalling and reinstalling a new
>> agent
>> >> service in a new working directory… Any tips or advice would be
>> extremely
>> >> appreciated; been trying to solve this all day and night since it
>> started
>> >> Thursday when the master rebooted during a job on the Windows agent.
>> >
>> > We have thins kind of errors on Windows when there are still running
>> > processes that have a file descriptor opened.
>> > You could log on to the machine and check if there are any "zombie"
>> processes.
>> >
>> > Chris
>> >
>> >>
>> >> Thanks in advance,
>> >> Sean
>> >>
>> >> Full error:
>> >>
>> >> java.nio.file.AccessDeniedException:
>> >>
>> C:\Jenkins2\workspace\Stan_develop-3CVGV42HAM7J3BLI3M44PCR6M52FN6ZQ65IEZ6TOSHJURSBO2PPA\src\test\test-models\bad\read_only
>>
>> >> at sun.nio.fs.WindowsException.translateToIOException(Unknown Source)
>> >> at sun.nio.fs.WindowsException.rethrowAsIOException(Unknown Source)
>> >> at sun.nio.fs.WindowsException.rethrowAsIOException(Unknown Source)
>> >> at sun.nio.fs.WindowsFileSystemProvider.implDelete(Unknown Source)
>> >> at sun.nio.fs.AbstractFileSystemProvider.deleteIfExists(Unknown
>> Source)
>> >> at java.nio.file.Files.deleteIfExists(Unknown Source)
>> >> at hudson.Util.tryOnceDeleteFile(Util.java:297)
>> >> at hudson.Util.deleteFile(Util.java:253)
>> >> Also:   hudson.remoting.Channel$CallSiteStackTrace: Remote call to
>> >> JNLP4-connect connection from
>> >> gelman-group-win.stat.columbia.edu/128.59.76.64:50229
>> >> at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1737)
>> >> at hudson.remoting.UserResponse.retrieve(UserRequest.java:313)
>> >> at hudson.remoting.Channel.call(Channel.java:952)
>> >> at hudson.FilePath.act(FilePath.java:998)
>> >> at hudson.FilePath.act(FilePath.java:987)
>> >> at hudson.FilePath.deleteRecursive(FilePath.java:1192)
>> >> at
>> >>
>> org.jenkinsci.plugins.workflow.steps.DeleteDirStep$Execution.run(DeleteDirStep.java:77)
>>
>> >> at
>> >>
>> org.jenkinsci.plugins.workflow.steps.DeleteDirStep$Execution.run(DeleteDirStep.java:69)
>>
>> >> at
>> >>
>> org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution$1$1.call(SynchronousNonBlockingStepExecution.java:49)
>>
>> >> at hudson.security.ACL.impersonate(ACL.java:290)
>> >> at
>> >>
>> org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution$1.run(SynchronousNonBlockingStepExecution.java:46)
>>
>> >> at
>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>> >> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>> >> at

Re: Windows agent can't delete workspace

2018-03-18 Thread Sean Talts

Thank you both! I particularly like the idea of just deleting before the 
job instead of after. I managed to get something else working too, though I 
think it could be masking the problem. I now run attrib -r -s /s /d before 
deleting (before the run and after, just in case). Seems to be pretty quick?

A run with this: 
http://d1m1s1b1.stat.columbia.edu:8080/blue/organizations/jenkins/Stan/detail/develop/82/pipeline/55
full Jenkinsfile: 
https://github.com/stan-dev/stan/blob/50be575d95718d077fda81a05d98289392f9acc3/Jenkinsfile

Thanks again!

On Sunday, March 18, 2018 at 2:43:56 AM UTC-4, benjamin.a.lau wrote:
>
> We've used scripting around the sysinternals handle tool[1] to make 
> sure that everything which is touching files in the workspace is 
> actually terminated before trying to delete the workspace. We were 
> having more issues with this when trying to delete workspaces at the 
> ends of jobs so in many cases we just let it be and only delete the 
> workspace on the next run. 
>
> Cheers, 
> Ben 
>
> [1] https://docs.microsoft.com/en-us/sysinternals/downloads/handle 
>
> On Sat, Mar 17, 2018 at 3:15 PM, Christian Gagneraud  > wrote: 
> > On 18 March 2018 at 03:21, Sean Talts  > wrote: 
> >> Hey all, 
> >> 
> >> I'm having this sudden crazy problem where my Windows agent can't 
> delete 
> >> some files in its workspace anymore, failing all builds. It's getting a 
> >> java.nio.file.AccessDeniedException on some files that it created. You 
> can 
> >> see the full exceptions at the end of the log here: 
> >> http://d1m1s1b1.stat.columbia.edu:8080/job/Stan/job/develop/80/console 
> also 
> >> copied below. 
> >> 
> >> I've tried so many things, including switching JVMs on master and 
> agent, 
> >> upgrading and switching to LTS, uninstalling and reinstalling a new 
> agent 
> >> service in a new working directory… Any tips or advice would be 
> extremely 
> >> appreciated; been trying to solve this all day and night since it 
> started 
> >> Thursday when the master rebooted during a job on the Windows agent. 
> > 
> > We have thins kind of errors on Windows when there are still running 
> > processes that have a file descriptor opened. 
> > You could log on to the machine and check if there are any "zombie" 
> processes. 
> > 
> > Chris 
> > 
> >> 
> >> Thanks in advance, 
> >> Sean 
> >> 
> >> Full error: 
> >> 
> >> java.nio.file.AccessDeniedException: 
> >> 
> C:\Jenkins2\workspace\Stan_develop-3CVGV42HAM7J3BLI3M44PCR6M52FN6ZQ65IEZ6TOSHJURSBO2PPA\src\test\test-models\bad\read_only
>  
>
> >> at sun.nio.fs.WindowsException.translateToIOException(Unknown Source) 
> >> at sun.nio.fs.WindowsException.rethrowAsIOException(Unknown Source) 
> >> at sun.nio.fs.WindowsException.rethrowAsIOException(Unknown Source) 
> >> at sun.nio.fs.WindowsFileSystemProvider.implDelete(Unknown Source) 
> >> at sun.nio.fs.AbstractFileSystemProvider.deleteIfExists(Unknown Source) 
> >> at java.nio.file.Files.deleteIfExists(Unknown Source) 
> >> at hudson.Util.tryOnceDeleteFile(Util.java:297) 
> >> at hudson.Util.deleteFile(Util.java:253) 
> >> Also:   hudson.remoting.Channel$CallSiteStackTrace: Remote call to 
> >> JNLP4-connect connection from 
> >> gelman-group-win.stat.columbia.edu/128.59.76.64:50229 
> >> at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1737) 
> >> at hudson.remoting.UserResponse.retrieve(UserRequest.java:313) 
> >> at hudson.remoting.Channel.call(Channel.java:952) 
> >> at hudson.FilePath.act(FilePath.java:998) 
> >> at hudson.FilePath.act(FilePath.java:987) 
> >> at hudson.FilePath.deleteRecursive(FilePath.java:1192) 
> >> at 
> >> 
> org.jenkinsci.plugins.workflow.steps.DeleteDirStep$Execution.run(DeleteDirStep.java:77)
>  
>
> >> at 
> >> 
> org.jenkinsci.plugins.workflow.steps.DeleteDirStep$Execution.run(DeleteDirStep.java:69)
>  
>
> >> at 
> >> 
> org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution$1$1.call(SynchronousNonBlockingStepExecution.java:49)
>  
>
> >> at hudson.security.ACL.impersonate(ACL.java:290) 
> >> at 
> >> 
> org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution$1.run(SynchronousNonBlockingStepExecution.java:46)
>  
>
> >> at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
> >> at java.util.concurrent.FutureTask.run(FutureTask.java:266) 
> >> at 
> >> 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>  
>
> >> at 
> >> 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>  
>
> >> at java.lang.Thread.run(Thread.java:748) 
> >> Caused: java.io.IOException: Unable to delete 
> >> 
> 'C:\Jenkins2\workspace\Stan_develop-3CVGV42HAM7J3BLI3M44PCR6M52FN6ZQ65IEZ6TOSHJURSBO2PPA\src\test\test-models\bad\read_only'.
>  
>
> >> Tried 3 times (of a maximum of 3) waiting 0.1 sec between attempts. 
> >> at hudson.Util.deleteFile(Util.java:258) 
> >> at

Is it possible to detect TIMEOUT event and act upon it?

2018-03-18 Thread Idan Adar

In a declarative pipeline's Jenkinsfile it is possible to add a timeout 
option both global as well as override it per stage. E.g.: 

stage ("SonarQube analysis") {
  options {
timeout(time: 5, unit: 'MINUTES')
  }


  steps {
script {
  STAGE_NAME = "SonarQube analysis"


  withSonarQubeEnv('SonarQube') {
sh "../../../sonar-scanner/bin/sonar-scanner"
  }


  // Check whether coverage threshold is met, otherwise fail the job.
  def qualitygate = waitForQualityGate()
  if (qualitygate.status != "OK") {
slackSend (
  color: '#F01717',
  message: "*$JOB_NAME*: <$BUILD_URL|Build #$BUILD_NUMBER>, 
'$STAGE_NAME' stage failed."
)
error "Pipeline aborted due to quality gate coverage failure: 
${qualitygate.status}"
  }
}
  }
}


Is it possible to make some action in case the timeout was reached? to 
detect it somehow?
In case of a timeout I would like to send a Slack notification.

Right now what happens is that the scanning reached 5 minutes and thus 
never entered the if statement; in which case there is "silent failure".

-- 
You received this message because you are subscribed to the Google Groups 
"Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jenkinsci-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/jenkinsci-users/20346cd1-b31b-46e8-8c74-c290e0bb4cd3%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Windows agent can't delete workspace

2018-03-18 Thread Benjamin Lau

We've used scripting around the sysinternals handle tool[1] to make
sure that everything which is touching files in the workspace is
actually terminated before trying to delete the workspace. We were
having more issues with this when trying to delete workspaces at the
ends of jobs so in many cases we just let it be and only delete the
workspace on the next run.

Cheers,
Ben

[1] https://docs.microsoft.com/en-us/sysinternals/downloads/handle

On Sat, Mar 17, 2018 at 3:15 PM, Christian Gagneraud  wrote:
> On 18 March 2018 at 03:21, Sean Talts  wrote:
>> Hey all,
>>
>> I'm having this sudden crazy problem where my Windows agent can't delete
>> some files in its workspace anymore, failing all builds. It's getting a
>> java.nio.file.AccessDeniedException on some files that it created. You can
>> see the full exceptions at the end of the log here:
>> http://d1m1s1b1.stat.columbia.edu:8080/job/Stan/job/develop/80/console also
>> copied below.
>>
>> I've tried so many things, including switching JVMs on master and agent,
>> upgrading and switching to LTS, uninstalling and reinstalling a new agent
>> service in a new working directory… Any tips or advice would be extremely
>> appreciated; been trying to solve this all day and night since it started
>> Thursday when the master rebooted during a job on the Windows agent.
>
> We have thins kind of errors on Windows when there are still running
> processes that have a file descriptor opened.
> You could log on to the machine and check if there are any "zombie" processes.
>
> Chris
>
>>
>> Thanks in advance,
>> Sean
>>
>> Full error:
>>
>> java.nio.file.AccessDeniedException:
>> C:\Jenkins2\workspace\Stan_develop-3CVGV42HAM7J3BLI3M44PCR6M52FN6ZQ65IEZ6TOSHJURSBO2PPA\src\test\test-models\bad\read_only
>> at sun.nio.fs.WindowsException.translateToIOException(Unknown Source)
>> at sun.nio.fs.WindowsException.rethrowAsIOException(Unknown Source)
>> at sun.nio.fs.WindowsException.rethrowAsIOException(Unknown Source)
>> at sun.nio.fs.WindowsFileSystemProvider.implDelete(Unknown Source)
>> at sun.nio.fs.AbstractFileSystemProvider.deleteIfExists(Unknown Source)
>> at java.nio.file.Files.deleteIfExists(Unknown Source)
>> at hudson.Util.tryOnceDeleteFile(Util.java:297)
>> at hudson.Util.deleteFile(Util.java:253)
>> Also:   hudson.remoting.Channel$CallSiteStackTrace: Remote call to
>> JNLP4-connect connection from
>> gelman-group-win.stat.columbia.edu/128.59.76.64:50229
>> at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1737)
>> at hudson.remoting.UserResponse.retrieve(UserRequest.java:313)
>> at hudson.remoting.Channel.call(Channel.java:952)
>> at hudson.FilePath.act(FilePath.java:998)
>> at hudson.FilePath.act(FilePath.java:987)
>> at hudson.FilePath.deleteRecursive(FilePath.java:1192)
>> at
>> org.jenkinsci.plugins.workflow.steps.DeleteDirStep$Execution.run(DeleteDirStep.java:77)
>> at
>> org.jenkinsci.plugins.workflow.steps.DeleteDirStep$Execution.run(DeleteDirStep.java:69)
>> at
>> org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution$1$1.call(SynchronousNonBlockingStepExecution.java:49)
>> at hudson.security.ACL.impersonate(ACL.java:290)
>> at
>> org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution$1.run(SynchronousNonBlockingStepExecution.java:46)
>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>> at java.lang.Thread.run(Thread.java:748)
>> Caused: java.io.IOException: Unable to delete
>> 'C:\Jenkins2\workspace\Stan_develop-3CVGV42HAM7J3BLI3M44PCR6M52FN6ZQ65IEZ6TOSHJURSBO2PPA\src\test\test-models\bad\read_only'.
>> Tried 3 times (of a maximum of 3) waiting 0.1 sec between attempts.
>> at hudson.Util.deleteFile(Util.java:258)
>> at hudson.FilePath.deleteRecursive(FilePath.java:1225)
>> at hudson.FilePath.deleteContentsRecursive(FilePath.java:1234)
>> at hudson.FilePath.deleteRecursive(FilePath.java:1216)
>> at hudson.FilePath.deleteContentsRecursive(FilePath.java:1234)
>> at hudson.FilePath.deleteRecursive(FilePath.java:1216)
>> at hudson.FilePath.deleteContentsRecursive(FilePath.java:1234)
>> at hudson.FilePath.deleteRecursive(FilePath.java:1216)
>> at hudson.FilePath.deleteContentsRecursive(FilePath.java:1234)
>> at hudson.FilePath.deleteRecursive(FilePath.java:1216)
>> at hudson.FilePath.deleteContentsRecursive(FilePath.java:1234)
>> at hudson.FilePath.deleteRecursive(FilePath.java:1216)
>> at hudson.FilePath.access$1100(FilePath.java:208)
>> at hudson.FilePath$13.invoke(FilePath.java:1195)
>> at hudson.FilePath$13.invoke(FilePath.java:1192)
>> at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2816)
>> at hudson.remoting.UserRequest.perform(UserRequest.java:210)
>> at

absolute path customWorkspace with docker-plugin and declarative pipeline

Re: Windows agent can't delete workspace

Re: Windows agent can't delete workspace

Is it possible to detect TIMEOUT event and act upon it?

Re: Windows agent can't delete workspace

5 matches

Site Navigation

Mail list logo

Footer information