[JIRA] (JENKINS-12672) Jobs take a long time to complete - get stuck sending success/failure email
[ https://issues.jenkins-ci.org/browse/JENKINS-12672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rob Petti resolved JENKINS-12672. - Resolution: Fixed Resolving as fixed. Feel free to reopen if anything weird happens again. ;) > Jobs take a long time to complete - get stuck sending success/failure email > --- > > Key: JENKINS-12672 > URL: https://issues.jenkins-ci.org/browse/JENKINS-12672 > Project: Jenkins > Issue Type: Bug > Components: perforce >Affects Versions: current > Environment: Jenkins 1.450 > Perforce plugin 1.3.7 > Red Hat Enterprise Linux 4.5 >Reporter: Ed Palazzo >Assignee: Rob Petti > > We've noticed that sometimes our jobs get in a state where they're taking a > long time to complete. Investigation reveals that the maven build has > completing successfully, and it's "stuck" trying to send a success email. > Left alone, the build will eventually complete but a 10 minute build may take > several hours to succeed and blocks other jobs since the executors are busy. > Manual monitoring and intervention is usually required to free up the jobs. > Here's an example thread taken from a Jenkins thread dump that shows the > "stuck" job. It appears to be processing the change list to retrieve the > checkin user(s) that checked in code. Why is this taking so long? > The exact stack changes with each thread dump, but it is consistent up to > PerforceChangeLogSet.parse(PerforceChangeLogSet.java:111). > "Executor #2 for master : executing markview-7.0.x_document-server #103" > prio=10 tid=0x48865400 nid=0x5109 runnable [0x49bfa000] >java.lang.Thread.State: RUNNABLE > at java.util.ArrayList.size(ArrayList.java:177) > at org.dom4j.tree.BackedList.(BackedList.java:36) > at > org.dom4j.tree.AbstractBranch.createResultList(AbstractBranch.java:373) > at org.dom4j.tree.AbstractElement.elements(AbstractElement.java:392) > at > org.dom4j.tree.AbstractElement.elementIterator(AbstractElement.java:428) > at > org.jaxen.dom4j.DocumentNavigator.getChildAxisIterator(DocumentNavigator.java:233) > at > org.jaxen.expr.iter.IterableChildAxis.namedAccessIterator(IterableChildAxis.java:98) > at org.jaxen.expr.DefaultNameStep.evaluate(DefaultNameStep.java:180) > at > org.jaxen.expr.DefaultLocationPath.evaluate(DefaultLocationPath.java:140) > at org.jaxen.expr.DefaultXPathExpr.asList(DefaultXPathExpr.java:102) > at org.jaxen.BaseXPath.selectNodesForContext(BaseXPath.java:674) > at org.jaxen.BaseXPath.selectNodes(BaseXPath.java:213) > at org.jaxen.BaseXPath.selectSingleNode(BaseXPath.java:234) > at > org.dom4j.xpath.DefaultXPath.selectSingleNode(DefaultXPath.java:159) > at org.dom4j.tree.AbstractNode.selectSingleNode(AbstractNode.java:185) > at > hudson.plugins.perforce.PerforceChangeLogSet.parse(PerforceChangeLogSet.java:111) > at > hudson.plugins.perforce.PerforceChangeLogParser.parse(PerforceChangeLogParser.java:18) > at hudson.model.AbstractBuild.calcChangeSet(AbstractBuild.java:808) > at hudson.model.AbstractBuild.getChangeSet(AbstractBuild.java:782) > at hudson.model.AbstractBuild.hasParticipant(AbstractBuild.java:356) > at > hudson.model.AbstractProject.hasParticipant(AbstractProject.java:1366) > at hudson.model.User.getProjects(User.java:402) > at > hudson.plugins.perforce.PerforceMailResolver.findMailAddressFor(PerforceMailResolver.java:38) > at > hudson.tasks.MailAddressResolver.resolve(MailAddressResolver.java:100) > at hudson.tasks.Mailer$UserProperty.getAddress(Mailer.java:514) > at > hudson.plugins.emailext.ExtendedEmailPublisher.createMail(ExtendedEmailPublisher.java:331) > at > hudson.plugins.emailext.ExtendedEmailPublisher.sendMail(ExtendedEmailPublisher.java:251) > at > hudson.plugins.emailext.ExtendedEmailPublisher._perform(ExtendedEmailPublisher.java:243) > at > hudson.plugins.emailext.ExtendedEmailPublisher.perform(ExtendedEmailPublisher.java:203) > at hudson.tasks.BuildStepMonitor$3.perform(BuildStepMonitor.java:36) > at > hudson.model.AbstractBuild$AbstractRunner.perform(AbstractBuild.java:700) > at > hudson.model.AbstractBuild$AbstractRunner.performAllBuildSteps(AbstractBuild.java:675) > at > hudson.maven.MavenModuleSetBuild$RunnerImpl.cleanUp(MavenModuleSetBuild.java:1022) > at hudson.model.Run.run(Run.java:1453) > at hudson.maven.MavenModuleSetBuild.run(MavenModuleSetBuild.java:481) > at hudson.model.ResourceController.execute(ResourceController.java:88) > at hudson.model.Executor.run(Executor.java:238) -- This message is automat
[JIRA] (JENKINS-12672) Jobs take a long time to complete - get stuck sending success/failure email
[ https://issues.jenkins-ci.org/browse/JENKINS-12672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=158998#comment-158998 ] Ed Palazzo commented on JENKINS-12672: -- I've had the fix in place for a week now, and the problem has not recurred. A very good sign! Thanks again for the quick fix. I'll let you know if anything changes here. > Jobs take a long time to complete - get stuck sending success/failure email > --- > > Key: JENKINS-12672 > URL: https://issues.jenkins-ci.org/browse/JENKINS-12672 > Project: Jenkins > Issue Type: Bug > Components: perforce >Affects Versions: current > Environment: Jenkins 1.450 > Perforce plugin 1.3.7 > Red Hat Enterprise Linux 4.5 >Reporter: Ed Palazzo >Assignee: Rob Petti > > We've noticed that sometimes our jobs get in a state where they're taking a > long time to complete. Investigation reveals that the maven build has > completing successfully, and it's "stuck" trying to send a success email. > Left alone, the build will eventually complete but a 10 minute build may take > several hours to succeed and blocks other jobs since the executors are busy. > Manual monitoring and intervention is usually required to free up the jobs. > Here's an example thread taken from a Jenkins thread dump that shows the > "stuck" job. It appears to be processing the change list to retrieve the > checkin user(s) that checked in code. Why is this taking so long? > The exact stack changes with each thread dump, but it is consistent up to > PerforceChangeLogSet.parse(PerforceChangeLogSet.java:111). > "Executor #2 for master : executing markview-7.0.x_document-server #103" > prio=10 tid=0x48865400 nid=0x5109 runnable [0x49bfa000] >java.lang.Thread.State: RUNNABLE > at java.util.ArrayList.size(ArrayList.java:177) > at org.dom4j.tree.BackedList.(BackedList.java:36) > at > org.dom4j.tree.AbstractBranch.createResultList(AbstractBranch.java:373) > at org.dom4j.tree.AbstractElement.elements(AbstractElement.java:392) > at > org.dom4j.tree.AbstractElement.elementIterator(AbstractElement.java:428) > at > org.jaxen.dom4j.DocumentNavigator.getChildAxisIterator(DocumentNavigator.java:233) > at > org.jaxen.expr.iter.IterableChildAxis.namedAccessIterator(IterableChildAxis.java:98) > at org.jaxen.expr.DefaultNameStep.evaluate(DefaultNameStep.java:180) > at > org.jaxen.expr.DefaultLocationPath.evaluate(DefaultLocationPath.java:140) > at org.jaxen.expr.DefaultXPathExpr.asList(DefaultXPathExpr.java:102) > at org.jaxen.BaseXPath.selectNodesForContext(BaseXPath.java:674) > at org.jaxen.BaseXPath.selectNodes(BaseXPath.java:213) > at org.jaxen.BaseXPath.selectSingleNode(BaseXPath.java:234) > at > org.dom4j.xpath.DefaultXPath.selectSingleNode(DefaultXPath.java:159) > at org.dom4j.tree.AbstractNode.selectSingleNode(AbstractNode.java:185) > at > hudson.plugins.perforce.PerforceChangeLogSet.parse(PerforceChangeLogSet.java:111) > at > hudson.plugins.perforce.PerforceChangeLogParser.parse(PerforceChangeLogParser.java:18) > at hudson.model.AbstractBuild.calcChangeSet(AbstractBuild.java:808) > at hudson.model.AbstractBuild.getChangeSet(AbstractBuild.java:782) > at hudson.model.AbstractBuild.hasParticipant(AbstractBuild.java:356) > at > hudson.model.AbstractProject.hasParticipant(AbstractProject.java:1366) > at hudson.model.User.getProjects(User.java:402) > at > hudson.plugins.perforce.PerforceMailResolver.findMailAddressFor(PerforceMailResolver.java:38) > at > hudson.tasks.MailAddressResolver.resolve(MailAddressResolver.java:100) > at hudson.tasks.Mailer$UserProperty.getAddress(Mailer.java:514) > at > hudson.plugins.emailext.ExtendedEmailPublisher.createMail(ExtendedEmailPublisher.java:331) > at > hudson.plugins.emailext.ExtendedEmailPublisher.sendMail(ExtendedEmailPublisher.java:251) > at > hudson.plugins.emailext.ExtendedEmailPublisher._perform(ExtendedEmailPublisher.java:243) > at > hudson.plugins.emailext.ExtendedEmailPublisher.perform(ExtendedEmailPublisher.java:203) > at hudson.tasks.BuildStepMonitor$3.perform(BuildStepMonitor.java:36) > at > hudson.model.AbstractBuild$AbstractRunner.perform(AbstractBuild.java:700) > at > hudson.model.AbstractBuild$AbstractRunner.performAllBuildSteps(AbstractBuild.java:675) > at > hudson.maven.MavenModuleSetBuild$RunnerImpl.cleanUp(MavenModuleSetBuild.java:1022) > at hudson.model.Run.run(Run.java:1453) > at hudson.maven.MavenModuleSetBuild.run(MavenModuleSetBuild.java:481) > at hudson.model.ResourceContro
[JIRA] (JENKINS-12672) Jobs take a long time to complete - get stuck sending success/failure email
[ https://issues.jenkins-ci.org/browse/JENKINS-12672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=158779#comment-158779 ] Ed Palazzo commented on JENKINS-12672: -- Thanks, Rob. I really appreciate your quick response. Yes, we have lots of projects (something like 300 jobs across 6 branches of code) and lots of builds of those projects. I'll give the snapshot a whirl next time I bring the server down. > Jobs take a long time to complete - get stuck sending success/failure email > --- > > Key: JENKINS-12672 > URL: https://issues.jenkins-ci.org/browse/JENKINS-12672 > Project: Jenkins > Issue Type: Bug > Components: perforce >Affects Versions: current > Environment: Jenkins 1.450 > Perforce plugin 1.3.7 > Red Hat Enterprise Linux 4.5 >Reporter: Ed Palazzo >Assignee: Rob Petti > > We've noticed that sometimes our jobs get in a state where they're taking a > long time to complete. Investigation reveals that the maven build has > completing successfully, and it's "stuck" trying to send a success email. > Left alone, the build will eventually complete but a 10 minute build may take > several hours to succeed and blocks other jobs since the executors are busy. > Manual monitoring and intervention is usually required to free up the jobs. > Here's an example thread taken from a Jenkins thread dump that shows the > "stuck" job. It appears to be processing the change list to retrieve the > checkin user(s) that checked in code. Why is this taking so long? > The exact stack changes with each thread dump, but it is consistent up to > PerforceChangeLogSet.parse(PerforceChangeLogSet.java:111). > "Executor #2 for master : executing markview-7.0.x_document-server #103" > prio=10 tid=0x48865400 nid=0x5109 runnable [0x49bfa000] >java.lang.Thread.State: RUNNABLE > at java.util.ArrayList.size(ArrayList.java:177) > at org.dom4j.tree.BackedList.(BackedList.java:36) > at > org.dom4j.tree.AbstractBranch.createResultList(AbstractBranch.java:373) > at org.dom4j.tree.AbstractElement.elements(AbstractElement.java:392) > at > org.dom4j.tree.AbstractElement.elementIterator(AbstractElement.java:428) > at > org.jaxen.dom4j.DocumentNavigator.getChildAxisIterator(DocumentNavigator.java:233) > at > org.jaxen.expr.iter.IterableChildAxis.namedAccessIterator(IterableChildAxis.java:98) > at org.jaxen.expr.DefaultNameStep.evaluate(DefaultNameStep.java:180) > at > org.jaxen.expr.DefaultLocationPath.evaluate(DefaultLocationPath.java:140) > at org.jaxen.expr.DefaultXPathExpr.asList(DefaultXPathExpr.java:102) > at org.jaxen.BaseXPath.selectNodesForContext(BaseXPath.java:674) > at org.jaxen.BaseXPath.selectNodes(BaseXPath.java:213) > at org.jaxen.BaseXPath.selectSingleNode(BaseXPath.java:234) > at > org.dom4j.xpath.DefaultXPath.selectSingleNode(DefaultXPath.java:159) > at org.dom4j.tree.AbstractNode.selectSingleNode(AbstractNode.java:185) > at > hudson.plugins.perforce.PerforceChangeLogSet.parse(PerforceChangeLogSet.java:111) > at > hudson.plugins.perforce.PerforceChangeLogParser.parse(PerforceChangeLogParser.java:18) > at hudson.model.AbstractBuild.calcChangeSet(AbstractBuild.java:808) > at hudson.model.AbstractBuild.getChangeSet(AbstractBuild.java:782) > at hudson.model.AbstractBuild.hasParticipant(AbstractBuild.java:356) > at > hudson.model.AbstractProject.hasParticipant(AbstractProject.java:1366) > at hudson.model.User.getProjects(User.java:402) > at > hudson.plugins.perforce.PerforceMailResolver.findMailAddressFor(PerforceMailResolver.java:38) > at > hudson.tasks.MailAddressResolver.resolve(MailAddressResolver.java:100) > at hudson.tasks.Mailer$UserProperty.getAddress(Mailer.java:514) > at > hudson.plugins.emailext.ExtendedEmailPublisher.createMail(ExtendedEmailPublisher.java:331) > at > hudson.plugins.emailext.ExtendedEmailPublisher.sendMail(ExtendedEmailPublisher.java:251) > at > hudson.plugins.emailext.ExtendedEmailPublisher._perform(ExtendedEmailPublisher.java:243) > at > hudson.plugins.emailext.ExtendedEmailPublisher.perform(ExtendedEmailPublisher.java:203) > at hudson.tasks.BuildStepMonitor$3.perform(BuildStepMonitor.java:36) > at > hudson.model.AbstractBuild$AbstractRunner.perform(AbstractBuild.java:700) > at > hudson.model.AbstractBuild$AbstractRunner.performAllBuildSteps(AbstractBuild.java:675) > at > hudson.maven.MavenModuleSetBuild$RunnerImpl.cleanUp(MavenModuleSetBuild.java:1022) > at hudson.model.Run.run(Run.java:1453) > at hudson.maven.MavenModuleSetBuild.run
[JIRA] (JENKINS-12672) Jobs take a long time to complete - get stuck sending success/failure email
[ https://issues.jenkins-ci.org/browse/JENKINS-12672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=158758#comment-158758 ] Rob Petti commented on JENKINS-12672: - And here's the snapshot with the fix if you'd like to test it before the next release. :) http://ci.jenkins-ci.org/job/plugins_perforce/175/org.jvnet.hudson.plugins$perforce/artifact/org.jvnet.hudson.plugins/perforce/1.3.8-SNAPSHOT/perforce-1.3.8-SNAPSHOT.hpi > Jobs take a long time to complete - get stuck sending success/failure email > --- > > Key: JENKINS-12672 > URL: https://issues.jenkins-ci.org/browse/JENKINS-12672 > Project: Jenkins > Issue Type: Bug > Components: perforce >Affects Versions: current > Environment: Jenkins 1.450 > Perforce plugin 1.3.7 > Red Hat Enterprise Linux 4.5 >Reporter: Ed Palazzo >Assignee: Rob Petti > > We've noticed that sometimes our jobs get in a state where they're taking a > long time to complete. Investigation reveals that the maven build has > completing successfully, and it's "stuck" trying to send a success email. > Left alone, the build will eventually complete but a 10 minute build may take > several hours to succeed and blocks other jobs since the executors are busy. > Manual monitoring and intervention is usually required to free up the jobs. > Here's an example thread taken from a Jenkins thread dump that shows the > "stuck" job. It appears to be processing the change list to retrieve the > checkin user(s) that checked in code. Why is this taking so long? > The exact stack changes with each thread dump, but it is consistent up to > PerforceChangeLogSet.parse(PerforceChangeLogSet.java:111). > "Executor #2 for master : executing markview-7.0.x_document-server #103" > prio=10 tid=0x48865400 nid=0x5109 runnable [0x49bfa000] >java.lang.Thread.State: RUNNABLE > at java.util.ArrayList.size(ArrayList.java:177) > at org.dom4j.tree.BackedList.(BackedList.java:36) > at > org.dom4j.tree.AbstractBranch.createResultList(AbstractBranch.java:373) > at org.dom4j.tree.AbstractElement.elements(AbstractElement.java:392) > at > org.dom4j.tree.AbstractElement.elementIterator(AbstractElement.java:428) > at > org.jaxen.dom4j.DocumentNavigator.getChildAxisIterator(DocumentNavigator.java:233) > at > org.jaxen.expr.iter.IterableChildAxis.namedAccessIterator(IterableChildAxis.java:98) > at org.jaxen.expr.DefaultNameStep.evaluate(DefaultNameStep.java:180) > at > org.jaxen.expr.DefaultLocationPath.evaluate(DefaultLocationPath.java:140) > at org.jaxen.expr.DefaultXPathExpr.asList(DefaultXPathExpr.java:102) > at org.jaxen.BaseXPath.selectNodesForContext(BaseXPath.java:674) > at org.jaxen.BaseXPath.selectNodes(BaseXPath.java:213) > at org.jaxen.BaseXPath.selectSingleNode(BaseXPath.java:234) > at > org.dom4j.xpath.DefaultXPath.selectSingleNode(DefaultXPath.java:159) > at org.dom4j.tree.AbstractNode.selectSingleNode(AbstractNode.java:185) > at > hudson.plugins.perforce.PerforceChangeLogSet.parse(PerforceChangeLogSet.java:111) > at > hudson.plugins.perforce.PerforceChangeLogParser.parse(PerforceChangeLogParser.java:18) > at hudson.model.AbstractBuild.calcChangeSet(AbstractBuild.java:808) > at hudson.model.AbstractBuild.getChangeSet(AbstractBuild.java:782) > at hudson.model.AbstractBuild.hasParticipant(AbstractBuild.java:356) > at > hudson.model.AbstractProject.hasParticipant(AbstractProject.java:1366) > at hudson.model.User.getProjects(User.java:402) > at > hudson.plugins.perforce.PerforceMailResolver.findMailAddressFor(PerforceMailResolver.java:38) > at > hudson.tasks.MailAddressResolver.resolve(MailAddressResolver.java:100) > at hudson.tasks.Mailer$UserProperty.getAddress(Mailer.java:514) > at > hudson.plugins.emailext.ExtendedEmailPublisher.createMail(ExtendedEmailPublisher.java:331) > at > hudson.plugins.emailext.ExtendedEmailPublisher.sendMail(ExtendedEmailPublisher.java:251) > at > hudson.plugins.emailext.ExtendedEmailPublisher._perform(ExtendedEmailPublisher.java:243) > at > hudson.plugins.emailext.ExtendedEmailPublisher.perform(ExtendedEmailPublisher.java:203) > at hudson.tasks.BuildStepMonitor$3.perform(BuildStepMonitor.java:36) > at > hudson.model.AbstractBuild$AbstractRunner.perform(AbstractBuild.java:700) > at > hudson.model.AbstractBuild$AbstractRunner.performAllBuildSteps(AbstractBuild.java:675) > at > hudson.maven.MavenModuleSetBuild$RunnerImpl.cleanUp(MavenModuleSetBuild.java:1022) > at hudson.model.Run.run(Run.java:1453) > at hudson.maven.MavenM
[JIRA] (JENKINS-12672) Jobs take a long time to complete - get stuck sending success/failure email
[ https://issues.jenkins-ci.org/browse/JENKINS-12672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=158757#comment-158757 ] dogfood commented on JENKINS-12672: --- Integrated in !http://ci.jenkins-ci.org/images/16x16/blue.png! [plugins_perforce #175|http://ci.jenkins-ci.org/job/plugins_perforce/175/] [JENKINS-12672] change method for getting list of projects for mail resolver (Revision 414cd9f0a6409b46eef58d1229953e5093daa940) Result = SUCCESS Rob Petti : Files : * src/main/java/hudson/plugins/perforce/PerforceMailResolver.java > Jobs take a long time to complete - get stuck sending success/failure email > --- > > Key: JENKINS-12672 > URL: https://issues.jenkins-ci.org/browse/JENKINS-12672 > Project: Jenkins > Issue Type: Bug > Components: perforce >Affects Versions: current > Environment: Jenkins 1.450 > Perforce plugin 1.3.7 > Red Hat Enterprise Linux 4.5 >Reporter: Ed Palazzo >Assignee: Rob Petti > > We've noticed that sometimes our jobs get in a state where they're taking a > long time to complete. Investigation reveals that the maven build has > completing successfully, and it's "stuck" trying to send a success email. > Left alone, the build will eventually complete but a 10 minute build may take > several hours to succeed and blocks other jobs since the executors are busy. > Manual monitoring and intervention is usually required to free up the jobs. > Here's an example thread taken from a Jenkins thread dump that shows the > "stuck" job. It appears to be processing the change list to retrieve the > checkin user(s) that checked in code. Why is this taking so long? > The exact stack changes with each thread dump, but it is consistent up to > PerforceChangeLogSet.parse(PerforceChangeLogSet.java:111). > "Executor #2 for master : executing markview-7.0.x_document-server #103" > prio=10 tid=0x48865400 nid=0x5109 runnable [0x49bfa000] >java.lang.Thread.State: RUNNABLE > at java.util.ArrayList.size(ArrayList.java:177) > at org.dom4j.tree.BackedList.(BackedList.java:36) > at > org.dom4j.tree.AbstractBranch.createResultList(AbstractBranch.java:373) > at org.dom4j.tree.AbstractElement.elements(AbstractElement.java:392) > at > org.dom4j.tree.AbstractElement.elementIterator(AbstractElement.java:428) > at > org.jaxen.dom4j.DocumentNavigator.getChildAxisIterator(DocumentNavigator.java:233) > at > org.jaxen.expr.iter.IterableChildAxis.namedAccessIterator(IterableChildAxis.java:98) > at org.jaxen.expr.DefaultNameStep.evaluate(DefaultNameStep.java:180) > at > org.jaxen.expr.DefaultLocationPath.evaluate(DefaultLocationPath.java:140) > at org.jaxen.expr.DefaultXPathExpr.asList(DefaultXPathExpr.java:102) > at org.jaxen.BaseXPath.selectNodesForContext(BaseXPath.java:674) > at org.jaxen.BaseXPath.selectNodes(BaseXPath.java:213) > at org.jaxen.BaseXPath.selectSingleNode(BaseXPath.java:234) > at > org.dom4j.xpath.DefaultXPath.selectSingleNode(DefaultXPath.java:159) > at org.dom4j.tree.AbstractNode.selectSingleNode(AbstractNode.java:185) > at > hudson.plugins.perforce.PerforceChangeLogSet.parse(PerforceChangeLogSet.java:111) > at > hudson.plugins.perforce.PerforceChangeLogParser.parse(PerforceChangeLogParser.java:18) > at hudson.model.AbstractBuild.calcChangeSet(AbstractBuild.java:808) > at hudson.model.AbstractBuild.getChangeSet(AbstractBuild.java:782) > at hudson.model.AbstractBuild.hasParticipant(AbstractBuild.java:356) > at > hudson.model.AbstractProject.hasParticipant(AbstractProject.java:1366) > at hudson.model.User.getProjects(User.java:402) > at > hudson.plugins.perforce.PerforceMailResolver.findMailAddressFor(PerforceMailResolver.java:38) > at > hudson.tasks.MailAddressResolver.resolve(MailAddressResolver.java:100) > at hudson.tasks.Mailer$UserProperty.getAddress(Mailer.java:514) > at > hudson.plugins.emailext.ExtendedEmailPublisher.createMail(ExtendedEmailPublisher.java:331) > at > hudson.plugins.emailext.ExtendedEmailPublisher.sendMail(ExtendedEmailPublisher.java:251) > at > hudson.plugins.emailext.ExtendedEmailPublisher._perform(ExtendedEmailPublisher.java:243) > at > hudson.plugins.emailext.ExtendedEmailPublisher.perform(ExtendedEmailPublisher.java:203) > at hudson.tasks.BuildStepMonitor$3.perform(BuildStepMonitor.java:36) > at > hudson.model.AbstractBuild$AbstractRunner.perform(AbstractBuild.java:700) > at > hudson.model.AbstractBuild$AbstractRunner.performAllBuildSteps(AbstractBuild.java:675) > at > hudson.maven.MavenModuleSetBuild$RunnerImpl.
[JIRA] (JENKINS-12672) Jobs take a long time to complete - get stuck sending success/failure email
[ https://issues.jenkins-ci.org/browse/JENKINS-12672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=158756#comment-158756 ] SCM/JIRA link daemon commented on JENKINS-12672: Code changed in jenkins User: Rob Petti Path: src/main/java/hudson/plugins/perforce/PerforceMailResolver.java http://jenkins-ci.org/commit/perforce-plugin/414cd9f0a6409b46eef58d1229953e5093daa940 Log: [JENKINS-12672] change method for getting list of projects for mail resolver user.getProjects() is very slow, since it scans every changelog of every project. Just go through all the projects instead. > Jobs take a long time to complete - get stuck sending success/failure email > --- > > Key: JENKINS-12672 > URL: https://issues.jenkins-ci.org/browse/JENKINS-12672 > Project: Jenkins > Issue Type: Bug > Components: perforce >Affects Versions: current > Environment: Jenkins 1.450 > Perforce plugin 1.3.7 > Red Hat Enterprise Linux 4.5 >Reporter: Ed Palazzo >Assignee: Rob Petti > > We've noticed that sometimes our jobs get in a state where they're taking a > long time to complete. Investigation reveals that the maven build has > completing successfully, and it's "stuck" trying to send a success email. > Left alone, the build will eventually complete but a 10 minute build may take > several hours to succeed and blocks other jobs since the executors are busy. > Manual monitoring and intervention is usually required to free up the jobs. > Here's an example thread taken from a Jenkins thread dump that shows the > "stuck" job. It appears to be processing the change list to retrieve the > checkin user(s) that checked in code. Why is this taking so long? > The exact stack changes with each thread dump, but it is consistent up to > PerforceChangeLogSet.parse(PerforceChangeLogSet.java:111). > "Executor #2 for master : executing markview-7.0.x_document-server #103" > prio=10 tid=0x48865400 nid=0x5109 runnable [0x49bfa000] >java.lang.Thread.State: RUNNABLE > at java.util.ArrayList.size(ArrayList.java:177) > at org.dom4j.tree.BackedList.(BackedList.java:36) > at > org.dom4j.tree.AbstractBranch.createResultList(AbstractBranch.java:373) > at org.dom4j.tree.AbstractElement.elements(AbstractElement.java:392) > at > org.dom4j.tree.AbstractElement.elementIterator(AbstractElement.java:428) > at > org.jaxen.dom4j.DocumentNavigator.getChildAxisIterator(DocumentNavigator.java:233) > at > org.jaxen.expr.iter.IterableChildAxis.namedAccessIterator(IterableChildAxis.java:98) > at org.jaxen.expr.DefaultNameStep.evaluate(DefaultNameStep.java:180) > at > org.jaxen.expr.DefaultLocationPath.evaluate(DefaultLocationPath.java:140) > at org.jaxen.expr.DefaultXPathExpr.asList(DefaultXPathExpr.java:102) > at org.jaxen.BaseXPath.selectNodesForContext(BaseXPath.java:674) > at org.jaxen.BaseXPath.selectNodes(BaseXPath.java:213) > at org.jaxen.BaseXPath.selectSingleNode(BaseXPath.java:234) > at > org.dom4j.xpath.DefaultXPath.selectSingleNode(DefaultXPath.java:159) > at org.dom4j.tree.AbstractNode.selectSingleNode(AbstractNode.java:185) > at > hudson.plugins.perforce.PerforceChangeLogSet.parse(PerforceChangeLogSet.java:111) > at > hudson.plugins.perforce.PerforceChangeLogParser.parse(PerforceChangeLogParser.java:18) > at hudson.model.AbstractBuild.calcChangeSet(AbstractBuild.java:808) > at hudson.model.AbstractBuild.getChangeSet(AbstractBuild.java:782) > at hudson.model.AbstractBuild.hasParticipant(AbstractBuild.java:356) > at > hudson.model.AbstractProject.hasParticipant(AbstractProject.java:1366) > at hudson.model.User.getProjects(User.java:402) > at > hudson.plugins.perforce.PerforceMailResolver.findMailAddressFor(PerforceMailResolver.java:38) > at > hudson.tasks.MailAddressResolver.resolve(MailAddressResolver.java:100) > at hudson.tasks.Mailer$UserProperty.getAddress(Mailer.java:514) > at > hudson.plugins.emailext.ExtendedEmailPublisher.createMail(ExtendedEmailPublisher.java:331) > at > hudson.plugins.emailext.ExtendedEmailPublisher.sendMail(ExtendedEmailPublisher.java:251) > at > hudson.plugins.emailext.ExtendedEmailPublisher._perform(ExtendedEmailPublisher.java:243) > at > hudson.plugins.emailext.ExtendedEmailPublisher.perform(ExtendedEmailPublisher.java:203) > at hudson.tasks.BuildStepMonitor$3.perform(BuildStepMonitor.java:36) > at > hudson.model.AbstractBuild$AbstractRunner.perform(AbstractBuild.java:700) > at > hudson.model.AbstractBuild$AbstractRunner.performAllBuildSteps(AbstractBuild.java:675) >
[JIRA] (JENKINS-12672) Jobs take a long time to complete - get stuck sending success/failure email
[ https://issues.jenkins-ci.org/browse/JENKINS-12672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=158751#comment-158751 ] Rob Petti commented on JENKINS-12672: - Ok, I see what's happening here. Mail resolvers aren't given a hint as to which project to look at when determining the email address, so unfortunately this means that we need to know which projects that user has ever committed changes to. The project is needed so the plugin knows what settings to use to connect to perforce and obtain the email address. It would seem that the problem lies with the User.getProjects() function that we use to get the list of candidates. It will run through every single changelog in every single project. I imagine this is taking a long time because you have a lot of builds. > Jobs take a long time to complete - get stuck sending success/failure email > --- > > Key: JENKINS-12672 > URL: https://issues.jenkins-ci.org/browse/JENKINS-12672 > Project: Jenkins > Issue Type: Bug > Components: perforce >Affects Versions: current > Environment: Jenkins 1.450 > Perforce plugin 1.3.7 > Red Hat Enterprise Linux 4.5 >Reporter: Ed Palazzo > > We've noticed that sometimes our jobs get in a state where they're taking a > long time to complete. Investigation reveals that the maven build has > completing successfully, and it's "stuck" trying to send a success email. > Left alone, the build will eventually complete but a 10 minute build may take > several hours to succeed and blocks other jobs since the executors are busy. > Manual monitoring and intervention is usually required to free up the jobs. > Here's an example thread taken from a Jenkins thread dump that shows the > "stuck" job. It appears to be processing the change list to retrieve the > checkin user(s) that checked in code. Why is this taking so long? > The exact stack changes with each thread dump, but it is consistent up to > PerforceChangeLogSet.parse(PerforceChangeLogSet.java:111). > "Executor #2 for master : executing markview-7.0.x_document-server #103" > prio=10 tid=0x48865400 nid=0x5109 runnable [0x49bfa000] >java.lang.Thread.State: RUNNABLE > at java.util.ArrayList.size(ArrayList.java:177) > at org.dom4j.tree.BackedList.(BackedList.java:36) > at > org.dom4j.tree.AbstractBranch.createResultList(AbstractBranch.java:373) > at org.dom4j.tree.AbstractElement.elements(AbstractElement.java:392) > at > org.dom4j.tree.AbstractElement.elementIterator(AbstractElement.java:428) > at > org.jaxen.dom4j.DocumentNavigator.getChildAxisIterator(DocumentNavigator.java:233) > at > org.jaxen.expr.iter.IterableChildAxis.namedAccessIterator(IterableChildAxis.java:98) > at org.jaxen.expr.DefaultNameStep.evaluate(DefaultNameStep.java:180) > at > org.jaxen.expr.DefaultLocationPath.evaluate(DefaultLocationPath.java:140) > at org.jaxen.expr.DefaultXPathExpr.asList(DefaultXPathExpr.java:102) > at org.jaxen.BaseXPath.selectNodesForContext(BaseXPath.java:674) > at org.jaxen.BaseXPath.selectNodes(BaseXPath.java:213) > at org.jaxen.BaseXPath.selectSingleNode(BaseXPath.java:234) > at > org.dom4j.xpath.DefaultXPath.selectSingleNode(DefaultXPath.java:159) > at org.dom4j.tree.AbstractNode.selectSingleNode(AbstractNode.java:185) > at > hudson.plugins.perforce.PerforceChangeLogSet.parse(PerforceChangeLogSet.java:111) > at > hudson.plugins.perforce.PerforceChangeLogParser.parse(PerforceChangeLogParser.java:18) > at hudson.model.AbstractBuild.calcChangeSet(AbstractBuild.java:808) > at hudson.model.AbstractBuild.getChangeSet(AbstractBuild.java:782) > at hudson.model.AbstractBuild.hasParticipant(AbstractBuild.java:356) > at > hudson.model.AbstractProject.hasParticipant(AbstractProject.java:1366) > at hudson.model.User.getProjects(User.java:402) > at > hudson.plugins.perforce.PerforceMailResolver.findMailAddressFor(PerforceMailResolver.java:38) > at > hudson.tasks.MailAddressResolver.resolve(MailAddressResolver.java:100) > at hudson.tasks.Mailer$UserProperty.getAddress(Mailer.java:514) > at > hudson.plugins.emailext.ExtendedEmailPublisher.createMail(ExtendedEmailPublisher.java:331) > at > hudson.plugins.emailext.ExtendedEmailPublisher.sendMail(ExtendedEmailPublisher.java:251) > at > hudson.plugins.emailext.ExtendedEmailPublisher._perform(ExtendedEmailPublisher.java:243) > at > hudson.plugins.emailext.ExtendedEmailPublisher.perform(ExtendedEmailPublisher.java:203) > at hudson.tasks.BuildStepMonitor$3.perform(BuildStepMonitor.java:36) > at > hudson.model.AbstractBuild$Abstr
[JIRA] (JENKINS-12672) Jobs take a long time to complete - get stuck sending success/failure email
[ https://issues.jenkins-ci.org/browse/JENKINS-12672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rob Petti reassigned JENKINS-12672: --- Assignee: Rob Petti > Jobs take a long time to complete - get stuck sending success/failure email > --- > > Key: JENKINS-12672 > URL: https://issues.jenkins-ci.org/browse/JENKINS-12672 > Project: Jenkins > Issue Type: Bug > Components: perforce >Affects Versions: current > Environment: Jenkins 1.450 > Perforce plugin 1.3.7 > Red Hat Enterprise Linux 4.5 >Reporter: Ed Palazzo >Assignee: Rob Petti > > We've noticed that sometimes our jobs get in a state where they're taking a > long time to complete. Investigation reveals that the maven build has > completing successfully, and it's "stuck" trying to send a success email. > Left alone, the build will eventually complete but a 10 minute build may take > several hours to succeed and blocks other jobs since the executors are busy. > Manual monitoring and intervention is usually required to free up the jobs. > Here's an example thread taken from a Jenkins thread dump that shows the > "stuck" job. It appears to be processing the change list to retrieve the > checkin user(s) that checked in code. Why is this taking so long? > The exact stack changes with each thread dump, but it is consistent up to > PerforceChangeLogSet.parse(PerforceChangeLogSet.java:111). > "Executor #2 for master : executing markview-7.0.x_document-server #103" > prio=10 tid=0x48865400 nid=0x5109 runnable [0x49bfa000] >java.lang.Thread.State: RUNNABLE > at java.util.ArrayList.size(ArrayList.java:177) > at org.dom4j.tree.BackedList.(BackedList.java:36) > at > org.dom4j.tree.AbstractBranch.createResultList(AbstractBranch.java:373) > at org.dom4j.tree.AbstractElement.elements(AbstractElement.java:392) > at > org.dom4j.tree.AbstractElement.elementIterator(AbstractElement.java:428) > at > org.jaxen.dom4j.DocumentNavigator.getChildAxisIterator(DocumentNavigator.java:233) > at > org.jaxen.expr.iter.IterableChildAxis.namedAccessIterator(IterableChildAxis.java:98) > at org.jaxen.expr.DefaultNameStep.evaluate(DefaultNameStep.java:180) > at > org.jaxen.expr.DefaultLocationPath.evaluate(DefaultLocationPath.java:140) > at org.jaxen.expr.DefaultXPathExpr.asList(DefaultXPathExpr.java:102) > at org.jaxen.BaseXPath.selectNodesForContext(BaseXPath.java:674) > at org.jaxen.BaseXPath.selectNodes(BaseXPath.java:213) > at org.jaxen.BaseXPath.selectSingleNode(BaseXPath.java:234) > at > org.dom4j.xpath.DefaultXPath.selectSingleNode(DefaultXPath.java:159) > at org.dom4j.tree.AbstractNode.selectSingleNode(AbstractNode.java:185) > at > hudson.plugins.perforce.PerforceChangeLogSet.parse(PerforceChangeLogSet.java:111) > at > hudson.plugins.perforce.PerforceChangeLogParser.parse(PerforceChangeLogParser.java:18) > at hudson.model.AbstractBuild.calcChangeSet(AbstractBuild.java:808) > at hudson.model.AbstractBuild.getChangeSet(AbstractBuild.java:782) > at hudson.model.AbstractBuild.hasParticipant(AbstractBuild.java:356) > at > hudson.model.AbstractProject.hasParticipant(AbstractProject.java:1366) > at hudson.model.User.getProjects(User.java:402) > at > hudson.plugins.perforce.PerforceMailResolver.findMailAddressFor(PerforceMailResolver.java:38) > at > hudson.tasks.MailAddressResolver.resolve(MailAddressResolver.java:100) > at hudson.tasks.Mailer$UserProperty.getAddress(Mailer.java:514) > at > hudson.plugins.emailext.ExtendedEmailPublisher.createMail(ExtendedEmailPublisher.java:331) > at > hudson.plugins.emailext.ExtendedEmailPublisher.sendMail(ExtendedEmailPublisher.java:251) > at > hudson.plugins.emailext.ExtendedEmailPublisher._perform(ExtendedEmailPublisher.java:243) > at > hudson.plugins.emailext.ExtendedEmailPublisher.perform(ExtendedEmailPublisher.java:203) > at hudson.tasks.BuildStepMonitor$3.perform(BuildStepMonitor.java:36) > at > hudson.model.AbstractBuild$AbstractRunner.perform(AbstractBuild.java:700) > at > hudson.model.AbstractBuild$AbstractRunner.performAllBuildSteps(AbstractBuild.java:675) > at > hudson.maven.MavenModuleSetBuild$RunnerImpl.cleanUp(MavenModuleSetBuild.java:1022) > at hudson.model.Run.run(Run.java:1453) > at hudson.maven.MavenModuleSetBuild.run(MavenModuleSetBuild.java:481) > at hudson.model.ResourceController.execute(ResourceController.java:88) > at hudson.model.Executor.run(Executor.java:238) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your
[JIRA] (JENKINS-12672) Jobs take a long time to complete - get stuck sending success/failure email
Ed Palazzo created JENKINS-12672: Summary: Jobs take a long time to complete - get stuck sending success/failure email Key: JENKINS-12672 URL: https://issues.jenkins-ci.org/browse/JENKINS-12672 Project: Jenkins Issue Type: Bug Components: perforce Affects Versions: current Environment: Jenkins 1.450 Perforce plugin 1.3.7 Red Hat Enterprise Linux 4.5 Reporter: Ed Palazzo We've noticed that sometimes our jobs get in a state where they're taking a long time to complete. Investigation reveals that the maven build has completing successfully, and it's "stuck" trying to send a success email. Left alone, the build will eventually complete but a 10 minute build may take several hours to succeed and blocks other jobs since the executors are busy. Manual monitoring and intervention is usually required to free up the jobs. Here's an example thread taken from a Jenkins thread dump that shows the "stuck" job. It appears to be processing the change list to retrieve the checkin user(s) that checked in code. Why is this taking so long? The exact stack changes with each thread dump, but it is consistent up to PerforceChangeLogSet.parse(PerforceChangeLogSet.java:111). "Executor #2 for master : executing markview-7.0.x_document-server #103" prio=10 tid=0x48865400 nid=0x5109 runnable [0x49bfa000] java.lang.Thread.State: RUNNABLE at java.util.ArrayList.size(ArrayList.java:177) at org.dom4j.tree.BackedList.(BackedList.java:36) at org.dom4j.tree.AbstractBranch.createResultList(AbstractBranch.java:373) at org.dom4j.tree.AbstractElement.elements(AbstractElement.java:392) at org.dom4j.tree.AbstractElement.elementIterator(AbstractElement.java:428) at org.jaxen.dom4j.DocumentNavigator.getChildAxisIterator(DocumentNavigator.java:233) at org.jaxen.expr.iter.IterableChildAxis.namedAccessIterator(IterableChildAxis.java:98) at org.jaxen.expr.DefaultNameStep.evaluate(DefaultNameStep.java:180) at org.jaxen.expr.DefaultLocationPath.evaluate(DefaultLocationPath.java:140) at org.jaxen.expr.DefaultXPathExpr.asList(DefaultXPathExpr.java:102) at org.jaxen.BaseXPath.selectNodesForContext(BaseXPath.java:674) at org.jaxen.BaseXPath.selectNodes(BaseXPath.java:213) at org.jaxen.BaseXPath.selectSingleNode(BaseXPath.java:234) at org.dom4j.xpath.DefaultXPath.selectSingleNode(DefaultXPath.java:159) at org.dom4j.tree.AbstractNode.selectSingleNode(AbstractNode.java:185) at hudson.plugins.perforce.PerforceChangeLogSet.parse(PerforceChangeLogSet.java:111) at hudson.plugins.perforce.PerforceChangeLogParser.parse(PerforceChangeLogParser.java:18) at hudson.model.AbstractBuild.calcChangeSet(AbstractBuild.java:808) at hudson.model.AbstractBuild.getChangeSet(AbstractBuild.java:782) at hudson.model.AbstractBuild.hasParticipant(AbstractBuild.java:356) at hudson.model.AbstractProject.hasParticipant(AbstractProject.java:1366) at hudson.model.User.getProjects(User.java:402) at hudson.plugins.perforce.PerforceMailResolver.findMailAddressFor(PerforceMailResolver.java:38) at hudson.tasks.MailAddressResolver.resolve(MailAddressResolver.java:100) at hudson.tasks.Mailer$UserProperty.getAddress(Mailer.java:514) at hudson.plugins.emailext.ExtendedEmailPublisher.createMail(ExtendedEmailPublisher.java:331) at hudson.plugins.emailext.ExtendedEmailPublisher.sendMail(ExtendedEmailPublisher.java:251) at hudson.plugins.emailext.ExtendedEmailPublisher._perform(ExtendedEmailPublisher.java:243) at hudson.plugins.emailext.ExtendedEmailPublisher.perform(ExtendedEmailPublisher.java:203) at hudson.tasks.BuildStepMonitor$3.perform(BuildStepMonitor.java:36) at hudson.model.AbstractBuild$AbstractRunner.perform(AbstractBuild.java:700) at hudson.model.AbstractBuild$AbstractRunner.performAllBuildSteps(AbstractBuild.java:675) at hudson.maven.MavenModuleSetBuild$RunnerImpl.cleanUp(MavenModuleSetBuild.java:1022) at hudson.model.Run.run(Run.java:1453) at hudson.maven.MavenModuleSetBuild.run(MavenModuleSetBuild.java:481) at hudson.model.ResourceController.execute(ResourceController.java:88) at hudson.model.Executor.run(Executor.java:238) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jenkins-ci.org/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
