[JIRA] [cli] (JENKINS-23248) CLI: get-job calls are causing file descriptor leaks.
Daniel Beck commented on JENKINS-23248 CLI: get-job calls are causing file descriptor leaks. Damon G Looks like a duplicate. If you could do a "binary search" for the responsible Jenkins release (1.546 good, 1.564 bad is too many possible versions) that would help. This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators. For more information on JIRA, see: http://www.atlassian.com/software/jira -- You received this message because you are subscribed to the Google Groups Jenkins Issues group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] [cli] (JENKINS-23248) CLI: get-job calls are causing file descriptor leaks.
Damon G commented on JENKINS-23248 CLI: get-job calls are causing file descriptor leaks. Hi Daniel, It may take me a few days to get back to you but yes i could run this test against the intermediary versions. I've cloned this vm off into an isolated environment for testing. Isn't there something else I could do at the same time to narrow it down? lsof just shows me that the file descriptors are leaking from java. Any sort of jenkins tracing I might want to enable at the same time that might narrow down where it's coming from? This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators. For more information on JIRA, see: http://www.atlassian.com/software/jira -- You received this message because you are subscribed to the Google Groups Jenkins Issues group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] [cli] (JENKINS-23248) CLI: get-job calls are causing file descriptor leaks.
Damon G edited a comment on JENKINS-23248 CLI: get-job calls are causing file descriptor leaks. Hi Daniel, It may take me a few days to get back to you but yes i could run this test against the intermediary versions. I've cloned this vm off into an isolated environment for testing. Isn't there something else I could do at the same time to narrow it down? lsof just shows me that the file descriptors are leaking from java. Any sort of jenkins tracing I might want to enable at the same time that might narrow down where it's coming from? Actually it looks like we'll be able to try this on Thursday. This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators. For more information on JIRA, see: http://www.atlassian.com/software/jira -- You received this message because you are subscribed to the Google Groups Jenkins Issues group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] [cli] (JENKINS-23248) CLI: get-job calls are causing file descriptor leaks.
Daniel Beck commented on JENKINS-23248 CLI: get-job calls are causing file descriptor leaks. Damon G: If you know how to search a heap dump for these, go for it. Otherwise, it's probably easiest narrowing it down to the first broken release. If you then have Git and Maven (3.0.5+) available, check out https://github.com/jenkinsci/jenkins and try using git bisect to find the responsible commit. mvn -DskipTests=true clean verify java -jar war/target/jenkins.war can be used to compile and run Jenkins from source. This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators. For more information on JIRA, see: http://www.atlassian.com/software/jira -- You received this message because you are subscribed to the Google Groups Jenkins Issues group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] [cli] (JENKINS-23248) CLI: get-job calls are causing file descriptor leaks.
Matthew Reiter commented on JENKINS-23248 CLI: get-job calls are causing file descriptor leaks. From what I can tell, the problem is with the CLI, and specifically, its use of the Jenkins remoting library. By going back to the 1.546 version of the CLI and adjusting the version of the remoting library used, I determined that the leaks occur on version 2.38 but not on 2.37. Those two version are significantly different, so I'm not sure of the exact change that introduced the problem. This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators. For more information on JIRA, see: http://www.atlassian.com/software/jira -- You received this message because you are subscribed to the Google Groups Jenkins Issues group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] [cli] (JENKINS-23248) CLI: get-job calls are causing file descriptor leaks.
Damon G commented on JENKINS-23248 CLI: get-job calls are causing file descriptor leaks. I just logged jenkins-23572 about a week ago and it looks like this exact same issue. Happy to mark my a duplicate of this or run tests to determine the root cause. We are calling the CLI from within our groovy test code inside of our job while it's executing on the slaves. I don't see how I could work around this by closing file descriptors in my wrapper code that's executing on the slave since it's on the master where there are too many open file descriptors. Maybe your job was executing directly on the master? This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators. For more information on JIRA, see: http://www.atlassian.com/software/jira -- You received this message because you are subscribed to the Google Groups Jenkins Issues group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] [cli] (JENKINS-23248) CLI: get-job calls are causing file descriptor leaks.
Craig Phillips commented on JENKINS-23248 CLI: get-job calls are causing file descriptor leaks. I know what the cause was. I found this article: https://wiki.jenkins-ci.org/display/JENKINS/Spawning+processes+from+build I'd say the article needs updating. It assumes that the bug/feature is only prevalent for sub processes that hold pipes open that become detached from the main process. This is not entirely true, since I was able to reproduce the same effect by running the CLI utility as a foreground process from a job spawned on the Jenkins server. The output and input from the CLI was attached directly to the build pipeline. There was no detaching or backgrounding going on, simply execute and return the exit code. However, what I think may be happening (I am a Java noob, so this is complete speculation) is that the same bug/feature found in Java that causes the problem for build processes, was emitted by the Java process spawned from running the CLI utility. I suspect this created some kind of circular file descriptor reference inside the JVM, preventing the EOF from being transmitted by the CLI utility when it exited. I have cured the problem completely, by modifying my wrapper script to explicitly close all descriptors apart from stdin, stdout and stderr on invocation, before running the Java util. Also, after running the util, stdin, stdout and stderr are then explicitly closed before the shell script exits. Since making this change, the file descriptor leak has been stable at 823 total file descriptors open. My guess is that the CLI utility should be doing something similar. This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators. For more information on JIRA, see: http://www.atlassian.com/software/jira -- You received this message because you are subscribed to the Google Groups Jenkins Issues group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] [cli] (JENKINS-23248) CLI: get-job calls are causing file descriptor leaks.
Craig Phillips commented on JENKINS-23248 CLI: get-job calls are causing file descriptor leaks. 1.564 This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators. For more information on JIRA, see: http://www.atlassian.com/software/jira -- You received this message because you are subscribed to the Google Groups Jenkins Issues group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] [cli] (JENKINS-23248) CLI: get-job calls are causing file descriptor leaks.
Craig Phillips commented on JENKINS-23248 CLI: get-job calls are causing file descriptor leaks. Just upgraded to 1.565 and the issue still persists. This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators. For more information on JIRA, see: http://www.atlassian.com/software/jira -- You received this message because you are subscribed to the Google Groups Jenkins Issues group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] [cli] (JENKINS-23248) CLI: get-job calls are causing file descriptor leaks.
Craig Phillips updated JENKINS-23248 CLI: get-job calls are causing file descriptor leaks. Here is a delta with approximately an hour between each snapshot. Change By: Craig Phillips (02/Jun/14 12:52 PM) Attachment: jkfd.txt This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators. For more information on JIRA, see: http://www.atlassian.com/software/jira -- You received this message because you are subscribed to the Google Groups Jenkins Issues group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] [cli] (JENKINS-23248) CLI: get-job calls are causing file descriptor leaks.
Craig Phillips commented on JENKINS-23248 CLI: get-job calls are causing file descriptor leaks. /tmp/jenk_fd.1 2014-06-01 21:03:34.006155887 +0100 +++ /tmp/jenk_fd.2 2014-06-01 21:09:27.053382015 +0100 @@ -52342 +52341,0 @@ -l-wx-- 1 kcc users 64 Jun 1 20:45 57103 - socket:282820885 @@ -55657 +55655,0 @@ lr-x- 1 kcc users 64 Jun 1 20:59 60087 - /user1/jenkins/jobs/scm-poll-jenkins-branch-monitor/builds/2014-06-01_21-01-01/log @@ -55662 +55660 @@ lr-x- 1 kcc users 64 Jun 1 21:01 60091 - pipe:282533977 +lr-x-- 1 kcc users 64 Jun 1 21:01 60091 - socket:283188722 @@ -55747 +55745 @@ lrwx- 1 kcc users 64 Jun 1 21:03 60169 - socket:282813330 +lrwx-- 1 kcc users 64 Jun 1 21:03 60169 - socket:283188811 @@ -55807 +55804,0 @@ lrwx- 1 kcc users 64 Jun 1 21:03 60222 - socket:282819646 @@ -55824,0 +55822 @@ +lrwx-- 1 kcc users 64 Jun 1 21:04 60239 - socket:282821225 @@ -55825,0 +55824,5 @@ +lrwx-- 1 kcc users 64 Jun 1 21:04 60240 - socket:282821552 +lrwx-- 1 kcc users 64 Jun 1 21:05 60241 - socket:282821858 +l-wx-- 1 kcc users 64 Jun 1 21:05 60242 - socket:282947156 +lrwx-- 1 kcc users 64 Jun 1 21:07 60243 - socket:282947065 +lr-x-- 1 kcc users 64 Jun 1 21:05 60244 - socket:283309559 This is just a small sample within several minutes of one another. I can send a larger delta if needs be. The leak is now at 6 files, which you can see from the line numbers in the diff. This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators. For more information on JIRA, see: http://www.atlassian.com/software/jira -- You received this message because you are subscribed to the Google Groups Jenkins Issues group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] [cli] (JENKINS-23248) CLI: get-job calls are causing file descriptor leaks.
Craig Phillips commented on JENKINS-23248 CLI: get-job calls are causing file descriptor leaks. Running garbage collection in the JVM doesn't clear them down either. I think I mentioned this already. This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators. For more information on JIRA, see: http://www.atlassian.com/software/jira -- You received this message because you are subscribed to the Google Groups Jenkins Issues group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] [cli] (JENKINS-23248) CLI: get-job calls are causing file descriptor leaks.
Daniel Beck commented on JENKINS-23248 CLI: get-job calls are causing file descriptor leaks. Jenkins version? This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators. For more information on JIRA, see: http://www.atlassian.com/software/jira -- You received this message because you are subscribed to the Google Groups Jenkins Issues group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] [cli] (JENKINS-23248) CLI: get-job calls are causing file descriptor leaks.
Craig Phillips created JENKINS-23248 CLI: get-job calls are causing file descriptor leaks. Issue Type: Bug Affects Versions: current Assignee: Unassigned Components: cli Created: 30/May/14 11:12 AM Description: I have a job that dynamically creates jobs using the CLI. I have noticed that since installing the job, which verifies the existing of jobs by calling 'get-job', Jenkins is leaking file descriptors. I am currently making around 40 calls per build, which runs on every CVS commit. I have a job setup to monitor the number of FD's in /proc/$jenkins_pid/fd. Calling garbage collection in the JVM doesn't release the FD's and thus the only cure is to restart Jenkins before the number of files reaches the Open file ulimit. I have set my ulimit to 65356 so I don't have to reboot so frequently. I restarted Jenkins at 7:49 this morning and the file descriptor count is currently at 6147 files; it's now 12:10 in the afternoon, so it's been steadily leaking FD's at approximately 1500 per hour. Project: Jenkins Labels: jenkins Priority: Major Reporter: Craig Phillips This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators. For more information on JIRA, see: http://www.atlassian.com/software/jira -- You received this message because you are subscribed to the Google Groups Jenkins Issues group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] [cli] (JENKINS-23248) CLI: get-job calls are causing file descriptor leaks.
Kohsuke Kawaguchi commented on JENKINS-23248 CLI: get-job calls are causing file descriptor leaks. When you look at ls -la /proc/$PID/fd output, can you tell which files are left open? Can you take a diff between two points of time and give us the delta? This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators. For more information on JIRA, see: http://www.atlassian.com/software/jira -- You received this message because you are subscribed to the Google Groups Jenkins Issues group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
