Barry Pape wrote: > I changed the ant task to use sshexec instead of exec with ssh as the > executable. I still have the seemingly random hung process. I executed > the ant task from outside cruisecontrol and it completed successfully. > The last time I found a hung process I grabbed a thread dump and am > including it here. Thanks for the suggestions and if there are any > other ideas I'd love to hear them. >
Hello Barry, Do I understand well that in your case CruiseControl is running ant within its own process ? If this is the case, there is a way to make CruiseControl start ant as an external process, I think using a script attribute on the ant builder. Try this, it might help. Also, there might be an underlying bug, either in CruiseControl or in Ant. Opening a ticket either in CruiseControl's JIRA or in ant's bugzilla depending on which application you think is misbehaving might help. If CruiseControl is starting ant within the CruiseControl process, I would say it is primarily a CruiseControl bug. Regards, Antoine > Thanks, > Barry > > [cc]Feb-22 11:12:13 ScriptRunner - 2010-02-22 11:12:13 > [cc]Feb-22 11:12:13 ScriptRunner - Full thread dump Java HotSpot(TM) > Server VM (14.2-b01 mixed mode): > [cc]Feb-22 11:12:13 ScriptRunner - > [cc]Feb-22 11:12:13 ScriptRunner - "Thread-3" prio=3 tid=0x08c5b400 > nid=0x12 sleeping[0xb667c000] > [cc]Feb-22 11:12:13 ScriptRunner - java.lang.Thread.State: > TIMED_WAITING (sleeping) > [cc]Feb-22 11:12:13 ScriptRunner - at java.lang.Thread.sleep(Native > Method) > [cc]Feb-22 11:12:13 ScriptRunner - at > org.apache.tools.ant.taskdefs.optional.ssh.SSHExec$1.run(SSHExec.java:15 > 8) > [cc]Feb-22 11:12:13 ScriptRunner - > [cc]Feb-22 11:12:13 ScriptRunner - "Connect thread > testb-pub-web01.test.aus.netspend.net session" prio=3 tid=0x08830800 > nid=0x11 runnable [0xb65da000] > [cc]Feb-22 11:12:13 ScriptRunner - java.lang.Thread.State: RUNNABLE > [cc]Feb-22 11:12:13 ScriptRunner - at > java.net.SocketInputStream.socketRead0(Native Method) > [cc]Feb-22 11:12:13 ScriptRunner - at > java.net.SocketInputStream.read(SocketInputStream.java:129) > [cc]Feb-22 11:12:13 ScriptRunner - at > com.jcraft.jsch.IO.getByte(IO.java:82) > [cc]Feb-22 11:12:13 ScriptRunner - at > com.jcraft.jsch.Session.read(Session.java:811) > [cc]Feb-22 11:12:13 ScriptRunner - at > com.jcraft.jsch.Session.run(Session.java:1195) > [cc]Feb-22 11:12:13 ScriptRunner - at > java.lang.Thread.run(Thread.java:619) > [cc]Feb-22 11:12:13 ScriptRunner - > [cc]Feb-22 11:12:13 ScriptRunner - "Low Memory Detector" daemon prio=3 > tid=0x08140000 nid=0xb runnable [0x00000000] > [cc]Feb-22 11:12:13 ScriptRunner - java.lang.Thread.State: RUNNABLE > [cc]Feb-22 11:12:13 ScriptRunner - > [cc]Feb-22 11:12:13 ScriptRunner - "CompilerThread1" daemon prio=3 > tid=0x0813d000 nid=0xa waiting on condition [0x00000000] > [cc]Feb-22 11:12:13 ScriptRunner - java.lang.Thread.State: RUNNABLE > [cc]Feb-22 11:12:13 ScriptRunner - > [cc]Feb-22 11:12:13 ScriptRunner - "CompilerThread0" daemon prio=3 > tid=0x0813a800 nid=0x9 waiting on condition [0x00000000] > [cc]Feb-22 11:12:13 ScriptRunner - java.lang.Thread.State: RUNNABLE > [cc]Feb-22 11:12:13 ScriptRunner - > [cc]Feb-22 11:12:13 ScriptRunner - "Signal Dispatcher" daemon prio=3 > tid=0x08139000 nid=0x8 waiting on condition [0x00000000][cc]Feb-22 > 11:12:13 ScriptRunner - java.lang.Thread.State: RUNNABLE > [cc]Feb-22 11:12:13 ScriptRunner - > [cc]Feb-22 11:12:13 ScriptRunner - "Finalizer" daemon prio=3 > tid=0x08124c00 nid=0x7 in Object.wait() [0xfb0ab000] > [cc]Feb-22 11:12:13 ScriptRunner - java.lang.Thread.State: WAITING > (on object monitor) > [cc]Feb-22 11:12:13 ScriptRunner - at java.lang.Object.wait(Native > Method) > [cc]Feb-22 11:12:13 ScriptRunner - - waiting on <0xbaf16bc0> (a > java.lang.ref.ReferenceQueue$Lock) > [cc]Feb-22 11:12:13 ScriptRunner - at > java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118) > [cc]Feb-22 11:12:13 ScriptRunner - - locked <0xbaf16bc0> (a > java.lang.ref.ReferenceQueue$Lock) > [cc]Feb-22 11:12:13 ScriptRunner - at > java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134) > [cc]Feb-22 11:12:13 ScriptRunner - at > java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159) > [cc]Feb-22 11:12:13 ScriptRunner - > [cc]Feb-22 11:12:13 ScriptRunner - "Reference Handler" daemon prio=3 > tid=0x08123400 nid=0x6 in Object.wait() [0xfb0fc000] > [cc]Feb-22 11:12:13 ScriptRunner - java.lang.Thread.State: WAITING > (on object monitor) > [cc]Feb-22 11:12:13 ScriptRunner - at java.lang.Object.wait(Native > Method) > [cc]Feb-22 11:12:13 ScriptRunner - - waiting on <0xbb097080> (a > java.lang.ref.Reference$Lock) > [cc]Feb-22 11:12:13 ScriptRunner - at > java.lang.Object.wait(Object.java:485) > [cc]Feb-22 11:12:14 ScriptRunner - at > java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116) > [cc]Feb-22 11:12:14 ScriptRunner - - locked <0xbb097080> (a > java.lang.ref.Reference$Lock) > [cc]Feb-22 11:12:14 ScriptRunner - > [cc]Feb-22 11:12:14 ScriptRunner - "main" prio=3 tid=0x08070800 nid=0x2 > in Object.wait() [0xfe3dd000] > [cc]Feb-22 11:12:14 ScriptRunner - java.lang.Thread.State: WAITING > (on object monitor) > [cc]Feb-22 11:12:14 ScriptRunner - at java.lang.Object.wait(Native > Method) > [cc]Feb-22 11:12:14 ScriptRunner - - waiting on <0xf7f41278> (a > org.apache.tools.ant.taskdefs.optional.ssh.SSHExec$1) > [cc]Feb-22 11:12:14 ScriptRunner - at > java.lang.Thread.join(Thread.java:1143) > [cc]Feb-22 11:12:14 ScriptRunner - - locked <0xf7f41278> (a > org.apache.tools.ant.taskdefs.optional.ssh.SSHExec$1) > [cc]Feb-22 11:12:14 ScriptRunner - at > org.apache.tools.ant.taskdefs.optional.ssh.SSHExec.execute(SSHExec.java: > 167) > [cc]Feb-22 11:12:14 ScriptRunner - at > org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:288) > [cc]Feb-22 11:12:14 ScriptRunner - at > sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source) > [cc]Feb-22 11:12:14 ScriptRunner - at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor > Impl.java:25) > [cc]Feb-22 11:12:14 ScriptRunner - at > java.lang.reflect.Method.invoke(Method.java:597) > [cc]Feb-22 11:12:14 ScriptRunner - at > org.apache.tools.ant.dispatch.DispatchUtils.execute(DispatchUtils.java:1 > 05) > [cc]Feb-22 11:12:14 ScriptRunner - at > org.apache.tools.ant.Task.perform(Task.java:348) > [cc]Feb-22 11:12:14 ScriptRunner - at > org.apache.tools.ant.taskdefs.Sequential.execute(Sequential.java:62) > [cc]Feb-22 11:12:14 ScriptRunner - at > org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:288) > [cc]Feb-22 11:12:14 ScriptRunner - at > sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source) > [cc]Feb-22 11:12:14 ScriptRunner - at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor > Impl.java:25) > [cc]Feb-22 11:12:14 ScriptRunner - at > java.lang.reflect.Method.invoke(Method.java:597) > [cc]Feb-22 11:12:14 ScriptRunner - at > org.apache.tools.ant.dispatch.DispatchUtils.execute(DispatchUtils.java:1 > 05) > [cc]Feb-22 11:12:14 ScriptRunner - at > org.apache.tools.ant.Task.perform(Task.java:348) > [cc]Feb-22 11:12:14 ScriptRunner - at > org.apache.tools.ant.taskdefs.MacroInstance.execute(MacroInstance.java:3 > 91) > [cc]Feb-22 11:12:14 ScriptRunner - at > org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:288) > [cc]Feb-22 11:12:14 ScriptRunner - at > sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source) > [cc]Feb-22 11:12:14 ScriptRunner - at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor > Impl.java:25) > [cc]Feb-22 11:12:14 ScriptRunner - at > java.lang.reflect.Method.invoke(Method.java:597) > [cc]Feb-22 11:12:14 ScriptRunner - at > org.apache.tools.ant.dispatch.DispatchUtils.execute(DispatchUtils.java:1 > 05) > [cc]Feb-22 11:12:14 ScriptRunner - at > org.apache.tools.ant.Task.perform(Task.java:348) > [cc]Feb-22 11:12:14 ScriptRunner - at > org.apache.tools.ant.taskdefs.Sequential.execute(Sequential.java:62) > [cc]Feb-22 11:12:14 ScriptRunner - at > net.sf.antcontrib.logic.IfTask.execute(IfTask.java:217) > [cc]Feb-22 11:12:14 ScriptRunner - at > sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > [cc]Feb-22 11:12:14 ScriptRunner - at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.jav > a:39) > [cc]Feb-22 11:12:14 ScriptRunner - at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor > Impl.java:25) > [cc]Feb-22 11:12:14 ScriptRunner - at > java.lang.reflect.Method.invoke(Method.java:597) > [cc]Feb-22 11:12:14 ScriptRunner - at > org.apache.tools.ant.dispatch.DispatchUtils.execute(DispatchUtils.java:1 > 05) > [cc]Feb-22 11:12:14 ScriptRunner - at > org.apache.tools.ant.TaskAdapter.execute(TaskAdapter.java:134) > [cc]Feb-22 11:12:14 ScriptRunner - at > org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:288) > [cc]Feb-22 11:12:14 ScriptRunner - at > sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > [cc]Feb-22 11:12:14 ScriptRunner - at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.jav > a:39) > [cc]Feb-22 11:12:14 ScriptRunner - at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor > Impl.java:25) > [cc]Feb-22 11:12:14 ScriptRunner - at > java.lang.reflect.Method.invoke(Method.java:597) > [cc]Feb-22 11:12:14 ScriptRunner - at > org.apache.tools.ant.dispatch.DispatchUtils.execute(DispatchUtils.java:1 > 05) > [cc]Feb-22 11:12:14 ScriptRunner - at > org.apache.tools.ant.Task.perform(Task.java:348) > [cc]Feb-22 11:12:14 ScriptRunner - at > org.apache.tools.ant.Target.execute(Target.java:357) > [cc]Feb-22 11:12:14 ScriptRunner - at > org.apache.tools.ant.Target.performTasks(Target.java:385) > [cc]Feb-22 11:12:14 ScriptRunner - at > org.apache.tools.ant.Project.executeSortedTargets(Project.java:1329) > [cc]Feb-22 11:12:14 ScriptRunner - at > org.apache.tools.ant.Project.executeTarget(Project.java:1298) > [cc]Feb-22 11:12:14 ScriptRunner - at > org.apache.tools.ant.helper.DefaultExecutor.executeTargets(DefaultExecut > or.java:41)[cc]Feb-22 11:12:14 ScriptRunner - at > org.apache.tools.ant.Project.executeTargets(Project.java:1181) > [cc]Feb-22 11:12:14 ScriptRunner - at > org.apache.tools.ant.Main.runBuild(Main.java:698) > [cc]Feb-22 11:12:14 ScriptRunner - at > org.apache.tools.ant.Main.startAnt(Main.java:199) > [cc]Feb-22 11:12:14 ScriptRunner - at > org.apache.tools.ant.launch.Launcher.run(Launcher.java:257) > [cc]Feb-22 11:12:14 ScriptRunner - at > org.apache.tools.ant.launch.Launcher.main(Launcher.java:104) > [cc]Feb-22 11:12:14 ScriptRunner - > [cc]Feb-22 11:12:14 ScriptRunner - "VM Thread" prio=3 tid=0x0811f400 > nid=0x5 runnable > [cc]Feb-22 11:12:14 ScriptRunner - > [cc]Feb-22 11:12:14 ScriptRunner - "GC task thread#0 (ParallelGC)" > prio=3 tid=0x08076c00 nid=0x3 runnable > [cc]Feb-22 11:12:14 ScriptRunner - > [cc]Feb-22 11:12:14 ScriptRunner - "GC task thread#1 (ParallelGC)" > prio=3 tid=0x08078400 nid=0x4 runnable > [cc]Feb-22 11:12:14 ScriptRunner - > [cc]Feb-22 11:12:14 ScriptRunner - "VM Periodic Task Thread" prio=3 > tid=0x08141800 nid=0xc waiting on condition > [cc]Feb-22 11:12:14 ScriptRunner - > [cc]Feb-22 11:12:14 ScriptRunner - JNI global references: 1235 > [cc]Feb-22 11:12:14 ScriptRunner - > [cc]Feb-22 11:12:14 ScriptRunner - Heap > [cc]Feb-22 11:12:14 ScriptRunner - PSYoungGen total 67136K, used > 55220K [0xf3e00000, 0xfb000000, 0xfb000000) > [cc]Feb-22 11:12:14 ScriptRunner - eden space 64640K, 81% used > [0xf3e00000,0xf71b52b8,0xf7d20000) > [cc]Feb-22 11:12:14 ScriptRunner - from space 2496K, 91% used > [0xf7d20000,0xf7f58010,0xf7f90000) > [cc]Feb-22 11:12:14 ScriptRunner - to space 2752K, 0% used > [0xfad50000,0xfad50000,0xfb000000) > [cc]Feb-22 11:12:14 ScriptRunner - PSOldGen total 69632K, used > 2971K [0xbae00000, 0xbf200000, 0xf3e00000) > [cc]Feb-22 11:12:14 ScriptRunner - object space 69632K, 4% used > [0xbae00000,0xbb0e6d80,0xbf200000) > [cc]Feb-22 11:12:14 ScriptRunner - PSPermGen total 16384K, used > 7839K [0xb6e00000, 0xb7e00000, 0xbae00000) > [cc]Feb-22 11:12:14 ScriptRunner - object space 16384K, 47% used > [0xb6e00000,0xb75a7d08,0xb7e00000) > [cc]Feb-22 11:12:14 ScriptRunner - > > > -----Original Message----- > From: Antoine Levy Lambert [mailto:[email protected]] > Sent: Saturday, February 13, 2010 4:36 PM > To: Ant Users List > Subject: Re: Exec Connection Not Closing > > Barry Pape wrote: > >> Hi Everyone, >> >> >> >> I seem to have an issue with an exec task not closing the ssh >> > connection > >> it opens. Basically we use CruiseControl and Ant to build and deploy >> our code. In the CruiseControl build.xml I have an ant task that >> > ssh's > >> to a remote server and starts a shell script. Once the script >> > completes > >> CruiseControl continues along its merry way...until recently. Now at >> seemingly random times, one of these ssh connections will not close >> > thus > >> causing CruiseControl to hang. If I kill the process, CruiseControl >> continues like normal. >> >> >> >> > I am using CruiseControl and never saw CruiseControl hanging because a > build is hanging. However if CruiseControl is configured with only > one thread, only one build can run at one time. > >> The ant task is: >> >> <exec executable="ssh" logError="true"> >> >> <arg line="${remote.host} ${sudo.cmd} -u >> ${username} /usr/local/qa/deploy_build.sh ${deploy.param}" /> >> >> </exec> >> >> >> > You could use the <sshexec/> ant task instead which is shipping with > ant, and requires jsch. > Maybe the underlying problem is that sudo is now configured to request a > > password ? or your remote user is expired ? > You should try to run this ssh command by hand to see what is wrong. > >> >> >> This had been working fine for the past year. Now we are having >> > issues. > >> It started when we migrated to a new build server. It has a more >> > recent > >> OS (solaris 2.10 instead of 2.9). >> >> >> >> I tried adding the timeout attribute to the exec task, but all that >> seemed to do was hide the stuck process. >> >> >> >> Any thoughts or suggestions are much appreciated! >> >> >> >> Thanks, >> >> Barry >> >> >> --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
