Re: Jenkins, Subversion plugin+ SVNKIT+ JVM garbage collection delay, leads to svn update failure.

[Mark Waite]

Java 7 has a try with resource syntax to reduce file handle leaks, but many
Jenkins plugins support Java versions prior to Java 7.  Jenkins on Linux
has a file leak detection plugin that can help with the debugging.
Unfortunately, that plugin does not work on Windows.

Mark Waite

On Tue, Mar 1, 2016 at 7:14 AM Warren Postma 
wrote:

> We are having a problem on Jenkins server on Windows, that has Subversion
> plugin, which uses a native Java SVNKIT instead of the "real" subversion
> binaries. This has always struck me as problematic, as there have always
> (for the last 3 years) been weird things during checkouts and updates that
> only occurred in svnkit, that were not reproducible for me with main svn
> client. I am experiencing one of those now, it happens once every 24 to 48
> hours, about one in 20 times that our main unit-test job runs.  When I use
> some tools to inspect the situation, it is the java.exe (jvm) that hosts
> the Jenkins task itself, which is holding onto a file-handle of a .tmp file
> inside the .svn folder of my workspace/working-copy, which causes it to
> fail.  It detects a lock after the first chance failure, then the fresh
> working copy fails to check out as well.
>
> Java.exe will not release its lock on
> c:\workspace\smoketest-delta2\.\.svn\svn.e130882e-5301-0010-8f8c-a3d25f9ae1e6.tmp
> until I basically shut down the Jenkins service.
>
> here's the log output:
>
> Started by an SCM change
> Building in workspace c:\workspace\smoketest-delta2
> Updating *http://svn.myorg.biz/core/trunk/powerserver*
>  at revision
> '2016-02-29T14:36:39.012 -0500'
> Workspace appear to be locked, so getting a fresh workspace
> Cleaning local Directory .
> java.nio.file.FileSystemException:
> c:\workspace\smoketest-delta2\.\.svn\svn.e130882e-5301-0010-8f8c-a3d25f9ae1e6.tmp:
> The process cannot access the file because it is being used by another
> process.
>
> at sun.nio.fs.WindowsException.translateToIOException(Unknown Source)
> at sun.nio.fs.WindowsException.rethrowAsIOException(Unknown Source)
> at sun.nio.fs.WindowsException.rethrowAsIOException(Unknown Source)
> at sun.nio.fs.WindowsFileSystemProvider.implDelete(Unknown Source)
> at sun.nio.fs.AbstractFileSystemProvider.delete(Unknown Source)
> at java.nio.file.Files.delete(Unknown Source)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
> at java.lang.reflect.Method.invoke(Unknown Source)
> at hudson.Util.deleteFile(Util.java:255)
> at hudson.Util.deleteRecursive(Util.java:318)
> at hudson.Util.deleteContentsRecursive(Util.java:220)
> at hudson.Util.deleteRecursive(Util.java:309)
> at hudson.Util.deleteContentsRecursive(Util.java:220)
> at hudson.scm.subversion.CheckoutUpdater$1.perform(CheckoutUpdater.java:81)
> at
> hudson.scm.subversion.WorkspaceUpdater$UpdateTask.delegateTo(WorkspaceUpdater.java:162)
> at
> hudson.scm.subversion.WorkspaceUpdater$UpdateTask.delegateTo(WorkspaceUpdater.java:170)
> at
> hudson.scm.subversion.UpdateUpdater$TaskImpl.perform(UpdateUpdater.java:183)
> at
> hudson.scm.subversion.WorkspaceUpdater$UpdateTask.delegateTo(WorkspaceUpdater.java:162)
> at hudson.scm.SubversionSCM$CheckOutTask.perform(SubversionSCM.java:988)
> at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:969)
> at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:945)
> at hudson.FilePath.act(FilePath.java:990)
> at hudson.FilePath.act(FilePath.java:968)
> at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:894)
> at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:830)
> at hudson.scm.SCM.checkout(SCM.java:485)
> at hudson.model.AbstractProject.checkout(AbstractProject.java:1269)
> at
> hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:607)
> at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
> at
> hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:529)
> at hudson.model.Run.execute(Run.java:1738)
> at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
> at hudson.model.ResourceController.execute(ResourceController.java:98)
> at hudson.model.Executor.run(Executor.java:410)
> [description-setter] Description set:
> Sending e-mails to: dev...@com
> Finished: FAILURE
>
> Has anyone got any ideas? I'm so frustrated that I am thinking of writing
> a new Subversion plugin that shells out to a REAL Subversion binary and
> does the SVN updates using that, because I am floored that the design of
> SVNkit makes it even POSSIBLE to not let go of file handles.  Doesn't java
> have a try-with-resource syntax just for this purpose?  How is it possible
> that file handles are not immediately released?  If anyone has ever
> debugged SVNKIT, and has any advice, it would be welcomed. I'm a C#, C++,
> Pascal 

Jenkins, Subversion plugin+ SVNKIT+ JVM garbage collection delay, leads to svn update failure.

[Warren Postma]

We are having a problem on Jenkins server on Windows, that has Subversion 
plugin, which uses a native Java SVNKIT instead of the "real" subversion 
binaries. This has always struck me as problematic, as there have always 
(for the last 3 years) been weird things during checkouts and updates that 
only occurred in svnkit, that were not reproducible for me with main svn 
client. I am experiencing one of those now, it happens once every 24 to 48 
hours, about one in 20 times that our main unit-test job runs.  When I use 
some tools to inspect the situation, it is the java.exe (jvm) that hosts 
the Jenkins task itself, which is holding onto a file-handle of a .tmp file 
inside the .svn folder of my workspace/working-copy, which causes it to 
fail.  It detects a lock after the first chance failure, then the fresh 
working copy fails to check out as well.  

Java.exe will not release its lock on 
c:\workspace\smoketest-delta2\.\.svn\svn.e130882e-5301-0010-8f8c-a3d25f9ae1e6.tmp
 
until I basically shut down the Jenkins service.

here's the log output:

Started by an SCM change
Building in workspace c:\workspace\smoketest-delta2
Updating *http://svn.myorg.biz/core/trunk/powerserver* 
 at revision 
'2016-02-29T14:36:39.012 -0500'
Workspace appear to be locked, so getting a fresh workspace
Cleaning local Directory .
java.nio.file.FileSystemException: 
c:\workspace\smoketest-delta2\.\.svn\svn.e130882e-5301-0010-8f8c-a3d25f9ae1e6.tmp:
 
The process cannot access the file because it is being used by another 
process.

at sun.nio.fs.WindowsException.translateToIOException(Unknown Source)
at sun.nio.fs.WindowsException.rethrowAsIOException(Unknown Source)
at sun.nio.fs.WindowsException.rethrowAsIOException(Unknown Source)
at sun.nio.fs.WindowsFileSystemProvider.implDelete(Unknown Source)
at sun.nio.fs.AbstractFileSystemProvider.delete(Unknown Source)
at java.nio.file.Files.delete(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at hudson.Util.deleteFile(Util.java:255)
at hudson.Util.deleteRecursive(Util.java:318)
at hudson.Util.deleteContentsRecursive(Util.java:220)
at hudson.Util.deleteRecursive(Util.java:309)
at hudson.Util.deleteContentsRecursive(Util.java:220)
at hudson.scm.subversion.CheckoutUpdater$1.perform(CheckoutUpdater.java:81)
at 
hudson.scm.subversion.WorkspaceUpdater$UpdateTask.delegateTo(WorkspaceUpdater.java:162)
at 
hudson.scm.subversion.WorkspaceUpdater$UpdateTask.delegateTo(WorkspaceUpdater.java:170)
at 
hudson.scm.subversion.UpdateUpdater$TaskImpl.perform(UpdateUpdater.java:183)
at 
hudson.scm.subversion.WorkspaceUpdater$UpdateTask.delegateTo(WorkspaceUpdater.java:162)
at hudson.scm.SubversionSCM$CheckOutTask.perform(SubversionSCM.java:988)
at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:969)
at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:945)
at hudson.FilePath.act(FilePath.java:990)
at hudson.FilePath.act(FilePath.java:968)
at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:894)
at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:830)
at hudson.scm.SCM.checkout(SCM.java:485)
at hudson.model.AbstractProject.checkout(AbstractProject.java:1269)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:607)
at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
at 
hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:529)
at hudson.model.Run.execute(Run.java:1738)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
at hudson.model.ResourceController.execute(ResourceController.java:98)
at hudson.model.Executor.run(Executor.java:410)
[description-setter] Description set: 
Sending e-mails to: dev...@com
Finished: FAILURE

Has anyone got any ideas? I'm so frustrated that I am thinking of writing a 
new Subversion plugin that shells out to a REAL Subversion binary and does 
the SVN updates using that, because I am floored that the design of SVNkit 
makes it even POSSIBLE to not let go of file handles.  Doesn't java have a 
try-with-resource syntax just for this purpose?  How is it possible that 
file handles are not immediately released?  If anyone has ever debugged 
SVNKIT, and has any advice, it would be welcomed. I'm a C#, C++, Pascal 
developer with only minimal Java knowledge, but I'm quite annoyed at SVNKIT 
for doing this, and perhaps, since I love Jenkins, this is a way I can help 
the Jenkins community out.

The Server is running Windows Server 2012 R2. Windows Search feature is NOT 
installed.  Windows Endpoint Protection (Defender) realtime protection is 
disabled, and exclusions are also defined for good measure.

Jenkins version is 1.647,  Subversion plugin version is 2.5.7.

I am not sure how to configure