[JIRA] (JENKINS-27514) Jenkins leaks thousands of Computer.threadPoolForRemoting threads leading to eventual server OOM
Title: Message Title David Johansen commented on JENKINS-27514 Re: Jenkins leaks thousands of Computer.threadPoolForRemoting threads leading to eventual server OOM So, we were hitting this issue a while back, threads stacking until we had to restart. I tracked it down to some automation that I had written, using http://javadoc.jenkins-ci.org/hudson/model/Computer.html#connect(boolean). I was using connect(true) which is supposed to spawn a new thread if the agent is "disconnected", and cancel the current thread that was disconnected. It doesn't cancel it right away though. Anyways, it was spawning them much more frequently than Jenkins was killing those old threads, when a host was not in a state where it could be connected. I changed my call to connect(false) to not force reconnect if the agent was unavailable, and we stopped seeing the crazy thread stacking. Kinda sounds like what you are seeing, but either way figured I'd share what we did. Add Comment This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] [core] (JENKINS-17619) Allow directory excludes for REKEY process: jenkins.security.RekeySecretAdminMonitor
David Johansen created JENKINS-17619 Allow directory excludes for REKEY process: jenkins.security.RekeySecretAdminMonitor Issue Type: Improvement Affects Versions: current Assignee: Unassigned Components: core Created: 15/Apr/13 5:30 PM Description: When re-keying we run into an issue where jenkins is scanning all directories, even the hidden NAS directories with backups, ".snapshots" which causes it to scan 30x the files it needs to while failing due to permissions on this directory. It would be great to have an option to exclude directories from this process. I couldn't find any info about this problem yet, so as far as I know this is the first ISSUE created for it... ERROR: Failed to rewrite /var/lib/jenkins/.snapshot/nightly.29/job_history/jobs/JOBNAME REMOVED/2013-01-04_15-09-39/config.xml hudson.util.IOException2: Failed to create a temporary file in /var/lib/jenkins/.snapshot/nightly.29/job_history/jobs/JOBNAME REMOVED/2013-01-04_15-09-39 at hudson.util.AtomicFileWriter.init(AtomicFileWriter.java:67) at hudson.util.SecretRewriter.rewrite(SecretRewriter.java:79) at hudson.util.SecretRewriter.rewriteRecursive(SecretRewriter.java:170) at hudson.util.SecretRewriter.rewriteRecursive(SecretRewriter.java:182) at hudson.util.SecretRewriter.rewriteRecursive(SecretRewriter.java:182) at hudson.util.SecretRewriter.rewriteRecursive(SecretRewriter.java:182) at hudson.util.SecretRewriter.rewriteRecursive(SecretRewriter.java:182) at hudson.util.SecretRewriter.rewriteRecursive(SecretRewriter.java:182) at hudson.util.SecretRewriter.rewriteRecursive(SecretRewriter.java:182) at hudson.util.SecretRewriter.rewriteRecursive(SecretRewriter.java:143) at jenkins.security.RekeySecretAdminMonitor$RekeyThread.run(RekeySecretAdminMonitor.java:182) Caused by: java.io.IOException: Read-only file system at java.io.UnixFileSystem.createFileExclusively(Native Method) at java.io.File.checkAndCreate(File.java:1717) at java.io.File.createTempFile0(File.java:1738) at java.io.File.createTempFile(File.java:1815) at hudson.util.AtomicFileWriter.init(AtomicFileWriter.java:65) ... 10 more Environment: RHEL/CentOS Linux Fix Versions: current Project: Jenkins Labels: jenkins Priority: Minor Reporter: David Johansen This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators. For more information on JIRA, see: http://www.atlassian.com/software/jira -- You received this message because you are subscribed to the Google Groups Jenkins Issues group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
