[JIRA] [subversion] (JENKINS-13835) E175002 in org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request
centic commented on JENKINS-13835 E175002 in org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request So either the Garbage Collector that you run is not ideal for your type of load or you have enough memory, but just barely enough and because of this the GC needs to do lots of work to make some memory available. In the first case a look at some garbage collection tuning will help, in the second a bit more memory might do the trick. Looking at the Java VM with jvisualvm can give you an idea how memory of the application looks like and which solution might give better results. Maybe you can provide some facts, e.g. actual memory settings and what the memory graphs look like. This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators. For more information on JIRA, see: http://www.atlassian.com/software/jira -- You received this message because you are subscribed to the Google Groups Jenkins Issues group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
[JIRA] [subversion] (JENKINS-13835) E175002 in org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request
Greg Hansen commented on JENKINS-13835 E175002 in org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request Just to close the loop, what I have found is that this issue is related to garbage collection in the underlying JVM. I installed the monitoring plugin that uses JavaMelody (see https://wiki.jenkins-ci.org/display/JENKINS/Monitoring), and if I trigger two successive garbage collections before the build of the builds run, then I don't get this error any more. (The clue came from this post: http://jenkins-ci.361315.n4.nabble.com/Hudson-garbage-collection-on-schedule-td3050696.html). From looking at the graphs generated by the plugin, what seems to happen is that if there is too much VM available, then the amount of time that garbage collection takes when it finally is triggered exceeds the amount of time required to maintain the connection to SVN. I have not characterized the times involved, just noted empirically that starting my daily builds runs with the smallest amount of VM possible seems to help a lot. Maybe there's such a thing as having too much VM? Perhaps mention of this could be made in the "setting up Jenkins" articles? This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators. For more information on JIRA, see: http://www.atlassian.com/software/jira -- You received this message because you are subscribed to the Google Groups Jenkins Issues group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
[JIRA] [subversion] (JENKINS-13835) E175002 in org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request
Greg Hansen edited a comment on JENKINS-13835 E175002 in org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request Just to close the loop, what I have found is that this issue is related to garbage collection in the underlying JVM. I installed the monitoring plugin that uses JavaMelody (see https://wiki.jenkins-ci.org/display/JENKINS/Monitoring), and if I trigger two successive garbage collections before the bulk of the builds run, then I don't get this error any more. (The clue came from this post: http://jenkins-ci.361315.n4.nabble.com/Hudson-garbage-collection-on-schedule-td3050696.html). From looking at the graphs generated by the plugin, what seems to happen is that if there is too much VM available, then the amount of time that garbage collection takes when it finally is triggered exceeds the amount of time required to maintain the connection to SVN. I have not characterized the times involved, just noted empirically that starting my daily builds runs with the smallest amount of VM possible seems to help a lot. Maybe there's such a thing as having too much VM? Perhaps mention of this could be made in the "setting up Jenkins" articles? This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators. For more information on JIRA, see: http://www.atlassian.com/software/jira -- You received this message because you are subscribed to the Google Groups Jenkins Issues group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
[JIRA] [subversion] (JENKINS-13835) E175002 in org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request
Greg Hansen commented on JENKINS-13835 E175002 in org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request I'm on a VM, I'm hoping that's not the issue. The VM is configured with 4 virtual cpus (each Intel E5-2650 at 2.0 GHz), 8 GB RAM, and 10K RPM SAS drives. We just bought the servers a month or two ago, and from examining the performance logs of the VMware vSphere Center, I'm barely using 8% of the cpu available, max. The SVN server is hardware that's a couple of years old, but I'm getting those on newer hardware too (the 1.8 installation I mentioned). We're currently running SVN 1.6 in our repos, so the 1.7 issue isn't it. Good thought, though! I haven't looked at the Jenkins GC options. I'll research that. It would make sense if it jumped in and hogged the server for long periods of time, which is likely given that I'm running 27 projects/day. I did bump the timeout to 10 minutes on both sides (both the SVN machine and the Jenkins slave that's doing the checkout); all that did is make it wait longer to time out. If it helps any, here's my latest call stack: ERROR: Failed to check out https://gigasrc.gigamon.com/repos/gv_g/mainline/gv216 org.tmatesoft.svn.core.SVNException: svn: E175002: REPORT /repos/!svn/vcc/default failed at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:379) at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:364) at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:352) at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.performHttpRequest(DAVConnection.java:708) at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.doReport(DAVConnection.java:335) at org.tmatesoft.svn.core.internal.io.dav.DAVRepository.runReport(DAVRepository.java:1289) at org.tmatesoft.svn.core.internal.io.dav.DAVRepository.update(DAVRepository.java:837) at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgAbstractUpdate.updateInternal(SvnNgAbstractUpdate.java:192) at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgAbstractUpdate.update(SvnNgAbstractUpdate.java:76) at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgAbstractUpdate.checkout(SvnNgAbstractUpdate.java:752) at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgCheckout.run(SvnNgCheckout.java:14) at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgCheckout.run(SvnNgCheckout.java:9) at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgOperationRunner.run(SvnNgOperationRunner.java:20) at org.tmatesoft.svn.core.internal.wc2.SvnOperationRunner.run(SvnOperationRunner.java:20) at org.tmatesoft.svn.core.wc2.SvnOperationFactory.run(SvnOperationFactory.java:1235) at org.tmatesoft.svn.core.wc2.SvnOperation.run(SvnOperation.java:291) at org.tmatesoft.svn.core.wc.SVNUpdateClient.doCheckout(SVNUpdateClient.java:777) at hudson.scm.subversion.CheckoutUpdater$1.perform(CheckoutUpdater.java:99) at hudson.scm.subversion.WorkspaceUpdater$UpdateTask.delegateTo(WorkspaceUpdater.java:153) at hudson.scm.SubversionSCM$CheckOutTask.perform(SubversionSCM.java:903) at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:884) at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:867) at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2388) at hudson.remoting.UserRequest.perform(UserRequest.java:118) at hudson.remoting.UserRequest.perform(UserRequest.java:48) at hudson.remoting.Request$2.run(Request.java:326) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72) at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source) at java.util.concurrent.FutureTask.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Caused by: svn: E175002: REPORT /repos/!svn/vcc/default failed at org.tmatesoft.svn.core.SVNErrorMessage.create(SVNErrorMessage.java:208) at org.tmatesoft.svn.core.SVNErrorMessage.create(SVNErrorMessage.java:154) at org.tmatesoft.svn.core.SVNErrorMessage.create(SVNErrorMessage.java:97) ... 32 more Caused by: org.tmatesoft.svn.core.SVNException: svn: E175002: REPORT request failed on '/repos/!svn/vcc/default' svn: E175002: timed out waiting for server at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:64) at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:51)
[JIRA] [subversion] (JENKINS-13835) E175002 in org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request
pancake
commented on JENKINS-13835
E175002 in org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request
Greg, the problem is indeed on the Jenkins side. Not necessarily on Jenkins server itself. Basically speaking this glitch occurs when svn client (Jenkins server or slave) stops communicating to svn server for unexpectedly long time (KeepAlive). In our Jenkins deployment we saw that svn checkout on slave consumed 100% of one CPU core, i.e. checkout speed was limited by per-core performance of Jenkins node. So, to workaround the alleviate the situation you can:
See if CPU/disk performance is a bottleneck for svn checkout and upgrade the hardware of relevant Jenkins node.
Tune Jenkins GC options. We didn't check, but it's very likely that delays are due to GC.
Increase KeepAlive on svn server. You'll probably want to setup a dedicated svn mirror for Jenkins and set KeepAlive on that mirror only. That's not good to have insanely big KeepAlive on a publicly available servers.
Note that we're using 1.6 WC format on Jenkins ("Manage Jenkins" "Configure System" "Subversion"). I don't quite remember what exactly bug made us downgrade 1.7 - 1.6 and not even sure if it's still there. But you can try it if the above doesn't help.
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators.
For more information on JIRA, see: http://www.atlassian.com/software/jira
--
You received this message because you are subscribed to the Google Groups Jenkins Issues group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
[JIRA] [subversion] (JENKINS-13835) E175002 in org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request
Greg Hansen commented on JENKINS-13835 E175002 in org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request Just tried this with a new installation of the latest version of Subversion server (1.8) on a freshly-installed machine running CentOS 6.3, and get the same error. We tried increasing the "KeepAlive" timeout from 15 seconds to 60 seconds, but still observed the issue. This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators. For more information on JIRA, see: http://www.atlassian.com/software/jira -- You received this message because you are subscribed to the Google Groups Jenkins Issues group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
[JIRA] [subversion] (JENKINS-13835) E175002 in org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request
Greg Hansen edited a comment on JENKINS-13835 E175002 in org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request Just tried this with a new installation of the latest version of Subversion server (1.8) on a freshly-installed machine running CentOS 6.3, and get the same error. We tried increasing the "KeepAlive" timeout from 15 seconds to 60 seconds (restarted both httpd and jenkins), but still observed the issue. This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators. For more information on JIRA, see: http://www.atlassian.com/software/jira -- You received this message because you are subscribed to the Google Groups Jenkins Issues group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
[JIRA] [subversion] (JENKINS-13835) E175002 in org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request
pancake commented on JENKINS-13835 E175002 in org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request Greg, it should be minutes or even tens of minutes (depends on WC size and hardware of the box that performs the checkout). Jenkins' KeepAlive has nothing to do with the problem. This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators. For more information on JIRA, see: http://www.atlassian.com/software/jira -- You received this message because you are subscribed to the Google Groups Jenkins Issues group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
[JIRA] [subversion] (JENKINS-13835) E175002 in org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request
Greg Hansen commented on JENKINS-13835 E175002 in org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request I'm grasping at anything that might alleviate the situation. Configuring httpd to maintain an established connection for a while to see if there was any more activity that could be sent across it before dismantling sounded like a reasonable thing to try. From the experiment where I ran the same check-outs against both a revision 1.6 repo and a newly-installed 1.8 repo (Dell R620, two six-core processors, 32 GB RAM, 10K RPM SAS drives, CentOS 6.3), the fact that the problem occurs on both repos suggests that the problem is not on the Subversion side it's on the Jenkins server side. Anyone with suggestions about Jenkins or CentOS configuration that might help this, please speak up! It's killing one or two out of 23 builds every day. This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators. For more information on JIRA, see: http://www.atlassian.com/software/jira -- You received this message because you are subscribed to the Google Groups Jenkins Issues group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
[JIRA] [subversion] (JENKINS-13835) E175002 in org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request
Greg Hansen commented on JENKINS-13835 E175002 in org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request I bumped the timeout to five minutes, and now ten. We'll see if that helps. This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators. For more information on JIRA, see: http://www.atlassian.com/software/jira -- You received this message because you are subscribed to the Google Groups Jenkins Issues group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
[JIRA] [subversion] (JENKINS-13835) E175002 in org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request
Greg Hansen commented on JENKINS-13835 E175002 in org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request Jenkins version 1.519 Jenkins Subversion plugin version 1.50 CentOS 6.3 x86_64 ERROR: Failed to check out https://gigasrc.gigamon.com/repos/gv_g/branches/gv_8300/gs org.tmatesoft.svn.core.SVNException: svn: E175002: REPORT /repos/!svn/vcc/default failed at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:379) at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:364) at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:352) at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.performHttpRequest(DAVConnection.java:708) at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.doReport(DAVConnection.java:335) at org.tmatesoft.svn.core.internal.io.dav.DAVRepository.runReport(DAVRepository.java:1289) at org.tmatesoft.svn.core.internal.io.dav.DAVRepository.update(DAVRepository.java:837) at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgAbstractUpdate.updateInternal(SvnNgAbstractUpdate.java:192) at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgAbstractUpdate.update(SvnNgAbstractUpdate.java:76) at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgAbstractUpdate.checkout(SvnNgAbstractUpdate.java:752) at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgCheckout.run(SvnNgCheckout.java:14) at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgCheckout.run(SvnNgCheckout.java:9) at org.tmatesoft.svn.core.internal.wc2.ng.SvnNgOperationRunner.run(SvnNgOperationRunner.java:20) at org.tmatesoft.svn.core.internal.wc2.SvnOperationRunner.run(SvnOperationRunner.java:20) at org.tmatesoft.svn.core.wc2.SvnOperationFactory.run(SvnOperationFactory.java:1235) at org.tmatesoft.svn.core.wc2.SvnOperation.run(SvnOperation.java:291) at org.tmatesoft.svn.core.wc.SVNUpdateClient.doCheckout(SVNUpdateClient.java:777) at hudson.scm.subversion.CheckoutUpdater$1.perform(CheckoutUpdater.java:99) at hudson.scm.subversion.WorkspaceUpdater$UpdateTask.delegateTo(WorkspaceUpdater.java:153) at hudson.scm.SubversionSCM$CheckOutTask.perform(SubversionSCM.java:903) at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:884) at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:867) at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2388) at hudson.remoting.UserRequest.perform(UserRequest.java:118) at hudson.remoting.UserRequest.perform(UserRequest.java:48) at hudson.remoting.Request$2.run(Request.java:326) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72) at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source) at java.util.concurrent.FutureTask.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Caused by: svn: E175002: REPORT /repos/!svn/vcc/default failed at org.tmatesoft.svn.core.SVNErrorMessage.create(SVNErrorMessage.java:208) at org.tmatesoft.svn.core.SVNErrorMessage.create(SVNErrorMessage.java:154) at org.tmatesoft.svn.core.SVNErrorMessage.create(SVNErrorMessage.java:97) ... 32 more Caused by: org.tmatesoft.svn.core.SVNException: svn: E175002: REPORT request failed on '/repos/!svn/vcc/default' svn: E175002: SSL peer shut down incorrectly at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:64) at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:51) at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection._request(HTTPConnection.java:754) at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:373) ... 31 more Caused by: svn: E175002: REPORT request failed on '/repos/!svn/vcc/default' at org.tmatesoft.svn.core.SVNErrorMessage.create(SVNErrorMessage.java:208) at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection._request(HTTPConnection.java:752) ... 32 more Caused by: svn: E175002: SSL peer shut down incorrectly at org.tmatesoft.svn.core.SVNErrorMessage.create(SVNErrorMessage.java:109) at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection._request(HTTPConnection.java:505) ... 32 more Caused by: javax.net.ssl.SSLException: SSL peer shut down incorrectly at com.sun.net.ssl.internal.ssl.InputRecord.readV3Record(Unknown Source) at
