[JIRA] [teamconcert-plugin] (JENKINS-24965) Personal builds can encounter deadlock when fetching source
James Clark commented on JENKINS-24965 Personal builds can encounter deadlock when fetching source Hi, Just updated the jazz work item with the full threadDump as reported from Jenkins at the time of another personal build hanging. Thanks, James This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators. For more information on JIRA, see: http://www.atlassian.com/software/jira -- You received this message because you are subscribed to the Google Groups Jenkins Issues group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] [teamconcert-plugin] (JENKINS-24965) Personal builds can encounter deadlock when fetching source
James Clark commented on JENKINS-24965 Personal builds can encounter deadlock when fetching source Hi Heather, The build had failed before that. The load step had passed and it was a legitimate failure during the build. I'll try and grab all thread data next time . Thanks, James This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators. For more information on JIRA, see: http://www.atlassian.com/software/jira -- You received this message because you are subscribed to the Google Groups Jenkins Issues group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] [teamconcert-plugin] (JENKINS-24965) Personal builds can encounter deadlock when fetching source
James Clark commented on JENKINS-24965 Personal builds can encounter deadlock when fetching source Hi Heather, Had another hanging build today. Here is the Jenkins console output: Started by user anonymous Building on master in workspace /ccbdata/build/TRUNKP RTC : checkout... RTC : Build initiated by request from RTC RTC Checkout : Source control setup RTC Checkout : Fetching files to fetch destination "/ccbdata/build/TRUNKP/src" ... It got stuck there and just had the loading symbol. Thanks, James This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators. For more information on JIRA, see: http://www.atlassian.com/software/jira -- You received this message because you are subscribed to the Google Groups Jenkins Issues group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] [teamconcert-plugin] (JENKINS-24965) Personal builds can encounter deadlock when fetching source
James Clark commented on JENKINS-24965 Personal builds can encounter deadlock when fetching source Hi Heather, I've attached a screenshot of the activities tab for the build result as viewed in RTC to the jazz.net work item that you posted. I unfortunately can't tell if all the source code was loaded as subsequent personal builds have been done which have re-used the workspace. The thread dump is no longer available for that build, however I did look at it and it was waiting for a lock at the same place. I can't say for this build in particular but generally, we have 2 executors running and they will both be RTC builds. Therefore it is likely that there was another build running at the same time that would have checked out source from RTC etc. The previous personal build before that had failed for a legitimate reason. If you're talking about a build for any job that was running at the same time as the one that hanged, I'm not sure what build that was as the timing info for the hanging build has been lost after killing the Jenkins process. Thanks, James This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators. For more information on JIRA, see: http://www.atlassian.com/software/jira -- You received this message because you are subscribed to the Google Groups Jenkins Issues group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] [teamconcert] (JENKINS-24965) Personal builds can encounter deadlock when fetching source
James Clark
commented on JENKINS-24965
Personal builds can encounter deadlock when fetching source
Hi Heather, no worries - late replying myself .
I've seen it happen to a handful of different people, including myself. My workspace is public (as we encourage all of our team to do) so visibility shouldn't be an issue. We only have a couple of components that rarely change so I imagine it's unlikely any of the hanging builds had something different, component-wise.
The load directory for the build definition is set to 'src' so that it checks out to {JOB_WORKSPACE}/src.
There are no special actions checked nor any excludes set for the load actions or accept options. I guess these are default values but I can send you exactly what option has what value if you need (assuming some might not be blank by default).
There haven't been many personal builds lately so I unfortunately don't have the output log to hand. Next time I see a hanging build, I will update this ticket.
Not quite sure what you mean by the build result having an activity for fetching files but the build doesn't get to the point of actually building anything. It tries to checkout the src code and never comes back from there.
Trying to abort the build via Jenkins has no effect. The only way to stop it is to kill the master Jenkins process.
Hope that helps, let me know if you need anything else!
Thanks,
James
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators.
For more information on JIRA, see: http://www.atlassian.com/software/jira
--
You received this message because you are subscribed to the Google Groups Jenkins Issues group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
[JIRA] [teamconcert] (JENKINS-24965) Personal builds can encounter deadlock when fetching source
James Clark commented on JENKINS-24965 Personal builds can encounter deadlock when fetching source Hi Heather, thanks for the reply. We're using the 4.0.3 version of the toolkit. There is only ever one instance of a job running at one time. I'm not sure whether the checkout finished or not. I believe the console output in Jenkins keeps saying that's it's fetching the source. I'll have to confirm next time I see it happen. I'm not too sure on the meaning of sandbox here but the when starting a personal build, each user selects their workspace with changes checked in for the build to use. In this respect, each repository workspace is different. All personal builds use the same job workspace as configured in Jenkins, but as there is only one job running at a time, there shouldn't be any overwriting going on. The jobs run on the Jenkins master. Thanks! JC This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators. For more information on JIRA, see: http://www.atlassian.com/software/jira -- You received this message because you are subscribed to the Google Groups Jenkins Issues group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] [teamconcert] (JENKINS-24965) Personal builds can encounter deadlock when fetching source
James Clark created JENKINS-24965 Personal builds can encounter deadlock when fetching source Issue Type: Bug Assignee: Unassigned Components: teamconcert Created: 02/Oct/14 10:44 AM Description: Sometimes, when running personal builds from RTC, the job will hang indefinitely. The console output for the job shows that the job is in the team concert pluging and is fetching source from RTC. Looking at the output from jenkins/ThreadDump, it would seem there is a deadlock situation occurring. Here is some example output: Executor #1 for master : executing Mainline personal build #458 "Executor #1 for master : executing Mainline personal build #458" Id=945 Group=main TIMED_WAITING on java.util.ArrayList@5663b111 at java.lang.Object.wait(Native Method) waiting on java.util.ArrayList@5663b111 at com.ibm.team.filesystem.client.internal.copyfileareas.BatchingLock.acquire(BatchingLock.java:446) at com.ibm.team.filesystem.client.internal.copyfileareas.CopyFileAreaManager.deregister(CopyFileAreaManager.java:350) at com.ibm.team.filesystem.client.internal.SharingManager.deregister(SharingManager.java:1358) at com.ibm.team.filesystem.client.internal.SharingManager.deregister(SharingManager.java:1316) at com.ibm.team.build.internal.hjplugin.rtc.RepositoryConnection.checkout(RepositoryConnection.java:457) at com.ibm.team.build.internal.hjplugin.rtc.RTCFacade.checkout(RTCFacade.java:390) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:622) at com.ibm.team.build.internal.hjplugin.RTCFacadeFactory$RTCFacadeWrapper.invoke(RTCFacadeFactory.java:115) at com.ibm.team.build.internal.hjplugin.RTCCheckoutTask.invoke(RTCCheckoutTask.java:166) at com.ibm.team.build.internal.hjplugin.RTCCheckoutTask.invoke(RTCCheckoutTask.java:32) at hudson.FilePath.act(FilePath.java:920) at hudson.FilePath.act(FilePath.java:893) at com.ibm.team.build.internal.hjplugin.RTCScm.checkout(RTCScm.java:1079) at hudson.model.AbstractProject.checkout(AbstractProject.java:1252) at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:615) at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86) at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:524) at hudson.model.Run.execute(Run.java:1706) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43) at hudson.model.ResourceController.execute(ResourceController.java:88) at hudson.model.Executor.run(Executor.java:232) I spoke with Scott Cowan over Sametime and he suggested that it was trying to access a lock on a file on the master filesystem and there might be some shared state between builds. I've had a quick look over our builds and I can't see anything obvious that would cause the problem but I will continue to investigate. It should be noted that this only seems to occur for personal builds where a particular user's workspace is being used. Our normal builds have not seen this issue. Thanks, James Environment: Jenkins 1.565.1 TeamConcert 1.1.8 Host: Red Hat Enterprise Linux Server 6.5 RTC 4.0.3 Jenkins running with 2 executors Project: Jenkins Labels: hang deadlock personal teamconcert Priority: Major Reporter: James Clark This message
