I changed the ant task to use sshexec instead of exec with ssh as the executable. I still have the seemingly random hung process. I executed the ant task from outside cruisecontrol and it completed successfully. The last time I found a hung process I grabbed a thread dump and am including it here. Thanks for the suggestions and if there are any other ideas I'd love to hear them.
Thanks, Barry [cc]Feb-22 11:12:13 ScriptRunner - 2010-02-22 11:12:13 [cc]Feb-22 11:12:13 ScriptRunner - Full thread dump Java HotSpot(TM) Server VM (14.2-b01 mixed mode): [cc]Feb-22 11:12:13 ScriptRunner - [cc]Feb-22 11:12:13 ScriptRunner - "Thread-3" prio=3 tid=0x08c5b400 nid=0x12 sleeping[0xb667c000] [cc]Feb-22 11:12:13 ScriptRunner - java.lang.Thread.State: TIMED_WAITING (sleeping) [cc]Feb-22 11:12:13 ScriptRunner - at java.lang.Thread.sleep(Native Method) [cc]Feb-22 11:12:13 ScriptRunner - at org.apache.tools.ant.taskdefs.optional.ssh.SSHExec$1.run(SSHExec.java:15 8) [cc]Feb-22 11:12:13 ScriptRunner - [cc]Feb-22 11:12:13 ScriptRunner - "Connect thread testb-pub-web01.test.aus.netspend.net session" prio=3 tid=0x08830800 nid=0x11 runnable [0xb65da000] [cc]Feb-22 11:12:13 ScriptRunner - java.lang.Thread.State: RUNNABLE [cc]Feb-22 11:12:13 ScriptRunner - at java.net.SocketInputStream.socketRead0(Native Method) [cc]Feb-22 11:12:13 ScriptRunner - at java.net.SocketInputStream.read(SocketInputStream.java:129) [cc]Feb-22 11:12:13 ScriptRunner - at com.jcraft.jsch.IO.getByte(IO.java:82) [cc]Feb-22 11:12:13 ScriptRunner - at com.jcraft.jsch.Session.read(Session.java:811) [cc]Feb-22 11:12:13 ScriptRunner - at com.jcraft.jsch.Session.run(Session.java:1195) [cc]Feb-22 11:12:13 ScriptRunner - at java.lang.Thread.run(Thread.java:619) [cc]Feb-22 11:12:13 ScriptRunner - [cc]Feb-22 11:12:13 ScriptRunner - "Low Memory Detector" daemon prio=3 tid=0x08140000 nid=0xb runnable [0x00000000] [cc]Feb-22 11:12:13 ScriptRunner - java.lang.Thread.State: RUNNABLE [cc]Feb-22 11:12:13 ScriptRunner - [cc]Feb-22 11:12:13 ScriptRunner - "CompilerThread1" daemon prio=3 tid=0x0813d000 nid=0xa waiting on condition [0x00000000] [cc]Feb-22 11:12:13 ScriptRunner - java.lang.Thread.State: RUNNABLE [cc]Feb-22 11:12:13 ScriptRunner - [cc]Feb-22 11:12:13 ScriptRunner - "CompilerThread0" daemon prio=3 tid=0x0813a800 nid=0x9 waiting on condition [0x00000000] [cc]Feb-22 11:12:13 ScriptRunner - java.lang.Thread.State: RUNNABLE [cc]Feb-22 11:12:13 ScriptRunner - [cc]Feb-22 11:12:13 ScriptRunner - "Signal Dispatcher" daemon prio=3 tid=0x08139000 nid=0x8 waiting on condition [0x00000000][cc]Feb-22 11:12:13 ScriptRunner - java.lang.Thread.State: RUNNABLE [cc]Feb-22 11:12:13 ScriptRunner - [cc]Feb-22 11:12:13 ScriptRunner - "Finalizer" daemon prio=3 tid=0x08124c00 nid=0x7 in Object.wait() [0xfb0ab000] [cc]Feb-22 11:12:13 ScriptRunner - java.lang.Thread.State: WAITING (on object monitor) [cc]Feb-22 11:12:13 ScriptRunner - at java.lang.Object.wait(Native Method) [cc]Feb-22 11:12:13 ScriptRunner - - waiting on <0xbaf16bc0> (a java.lang.ref.ReferenceQueue$Lock) [cc]Feb-22 11:12:13 ScriptRunner - at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118) [cc]Feb-22 11:12:13 ScriptRunner - - locked <0xbaf16bc0> (a java.lang.ref.ReferenceQueue$Lock) [cc]Feb-22 11:12:13 ScriptRunner - at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134) [cc]Feb-22 11:12:13 ScriptRunner - at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159) [cc]Feb-22 11:12:13 ScriptRunner - [cc]Feb-22 11:12:13 ScriptRunner - "Reference Handler" daemon prio=3 tid=0x08123400 nid=0x6 in Object.wait() [0xfb0fc000] [cc]Feb-22 11:12:13 ScriptRunner - java.lang.Thread.State: WAITING (on object monitor) [cc]Feb-22 11:12:13 ScriptRunner - at java.lang.Object.wait(Native Method) [cc]Feb-22 11:12:13 ScriptRunner - - waiting on <0xbb097080> (a java.lang.ref.Reference$Lock) [cc]Feb-22 11:12:13 ScriptRunner - at java.lang.Object.wait(Object.java:485) [cc]Feb-22 11:12:14 ScriptRunner - at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116) [cc]Feb-22 11:12:14 ScriptRunner - - locked <0xbb097080> (a java.lang.ref.Reference$Lock) [cc]Feb-22 11:12:14 ScriptRunner - [cc]Feb-22 11:12:14 ScriptRunner - "main" prio=3 tid=0x08070800 nid=0x2 in Object.wait() [0xfe3dd000] [cc]Feb-22 11:12:14 ScriptRunner - java.lang.Thread.State: WAITING (on object monitor) [cc]Feb-22 11:12:14 ScriptRunner - at java.lang.Object.wait(Native Method) [cc]Feb-22 11:12:14 ScriptRunner - - waiting on <0xf7f41278> (a org.apache.tools.ant.taskdefs.optional.ssh.SSHExec$1) [cc]Feb-22 11:12:14 ScriptRunner - at java.lang.Thread.join(Thread.java:1143) [cc]Feb-22 11:12:14 ScriptRunner - - locked <0xf7f41278> (a org.apache.tools.ant.taskdefs.optional.ssh.SSHExec$1) [cc]Feb-22 11:12:14 ScriptRunner - at org.apache.tools.ant.taskdefs.optional.ssh.SSHExec.execute(SSHExec.java: 167) [cc]Feb-22 11:12:14 ScriptRunner - at org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:288) [cc]Feb-22 11:12:14 ScriptRunner - at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source) [cc]Feb-22 11:12:14 ScriptRunner - at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor Impl.java:25) [cc]Feb-22 11:12:14 ScriptRunner - at java.lang.reflect.Method.invoke(Method.java:597) [cc]Feb-22 11:12:14 ScriptRunner - at org.apache.tools.ant.dispatch.DispatchUtils.execute(DispatchUtils.java:1 05) [cc]Feb-22 11:12:14 ScriptRunner - at org.apache.tools.ant.Task.perform(Task.java:348) [cc]Feb-22 11:12:14 ScriptRunner - at org.apache.tools.ant.taskdefs.Sequential.execute(Sequential.java:62) [cc]Feb-22 11:12:14 ScriptRunner - at org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:288) [cc]Feb-22 11:12:14 ScriptRunner - at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source) [cc]Feb-22 11:12:14 ScriptRunner - at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor Impl.java:25) [cc]Feb-22 11:12:14 ScriptRunner - at java.lang.reflect.Method.invoke(Method.java:597) [cc]Feb-22 11:12:14 ScriptRunner - at org.apache.tools.ant.dispatch.DispatchUtils.execute(DispatchUtils.java:1 05) [cc]Feb-22 11:12:14 ScriptRunner - at org.apache.tools.ant.Task.perform(Task.java:348) [cc]Feb-22 11:12:14 ScriptRunner - at org.apache.tools.ant.taskdefs.MacroInstance.execute(MacroInstance.java:3 91) [cc]Feb-22 11:12:14 ScriptRunner - at org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:288) [cc]Feb-22 11:12:14 ScriptRunner - at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source) [cc]Feb-22 11:12:14 ScriptRunner - at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor Impl.java:25) [cc]Feb-22 11:12:14 ScriptRunner - at java.lang.reflect.Method.invoke(Method.java:597) [cc]Feb-22 11:12:14 ScriptRunner - at org.apache.tools.ant.dispatch.DispatchUtils.execute(DispatchUtils.java:1 05) [cc]Feb-22 11:12:14 ScriptRunner - at org.apache.tools.ant.Task.perform(Task.java:348) [cc]Feb-22 11:12:14 ScriptRunner - at org.apache.tools.ant.taskdefs.Sequential.execute(Sequential.java:62) [cc]Feb-22 11:12:14 ScriptRunner - at net.sf.antcontrib.logic.IfTask.execute(IfTask.java:217) [cc]Feb-22 11:12:14 ScriptRunner - at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [cc]Feb-22 11:12:14 ScriptRunner - at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.jav a:39) [cc]Feb-22 11:12:14 ScriptRunner - at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor Impl.java:25) [cc]Feb-22 11:12:14 ScriptRunner - at java.lang.reflect.Method.invoke(Method.java:597) [cc]Feb-22 11:12:14 ScriptRunner - at org.apache.tools.ant.dispatch.DispatchUtils.execute(DispatchUtils.java:1 05) [cc]Feb-22 11:12:14 ScriptRunner - at org.apache.tools.ant.TaskAdapter.execute(TaskAdapter.java:134) [cc]Feb-22 11:12:14 ScriptRunner - at org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:288) [cc]Feb-22 11:12:14 ScriptRunner - at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [cc]Feb-22 11:12:14 ScriptRunner - at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.jav a:39) [cc]Feb-22 11:12:14 ScriptRunner - at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor Impl.java:25) [cc]Feb-22 11:12:14 ScriptRunner - at java.lang.reflect.Method.invoke(Method.java:597) [cc]Feb-22 11:12:14 ScriptRunner - at org.apache.tools.ant.dispatch.DispatchUtils.execute(DispatchUtils.java:1 05) [cc]Feb-22 11:12:14 ScriptRunner - at org.apache.tools.ant.Task.perform(Task.java:348) [cc]Feb-22 11:12:14 ScriptRunner - at org.apache.tools.ant.Target.execute(Target.java:357) [cc]Feb-22 11:12:14 ScriptRunner - at org.apache.tools.ant.Target.performTasks(Target.java:385) [cc]Feb-22 11:12:14 ScriptRunner - at org.apache.tools.ant.Project.executeSortedTargets(Project.java:1329) [cc]Feb-22 11:12:14 ScriptRunner - at org.apache.tools.ant.Project.executeTarget(Project.java:1298) [cc]Feb-22 11:12:14 ScriptRunner - at org.apache.tools.ant.helper.DefaultExecutor.executeTargets(DefaultExecut or.java:41)[cc]Feb-22 11:12:14 ScriptRunner - at org.apache.tools.ant.Project.executeTargets(Project.java:1181) [cc]Feb-22 11:12:14 ScriptRunner - at org.apache.tools.ant.Main.runBuild(Main.java:698) [cc]Feb-22 11:12:14 ScriptRunner - at org.apache.tools.ant.Main.startAnt(Main.java:199) [cc]Feb-22 11:12:14 ScriptRunner - at org.apache.tools.ant.launch.Launcher.run(Launcher.java:257) [cc]Feb-22 11:12:14 ScriptRunner - at org.apache.tools.ant.launch.Launcher.main(Launcher.java:104) [cc]Feb-22 11:12:14 ScriptRunner - [cc]Feb-22 11:12:14 ScriptRunner - "VM Thread" prio=3 tid=0x0811f400 nid=0x5 runnable [cc]Feb-22 11:12:14 ScriptRunner - [cc]Feb-22 11:12:14 ScriptRunner - "GC task thread#0 (ParallelGC)" prio=3 tid=0x08076c00 nid=0x3 runnable [cc]Feb-22 11:12:14 ScriptRunner - [cc]Feb-22 11:12:14 ScriptRunner - "GC task thread#1 (ParallelGC)" prio=3 tid=0x08078400 nid=0x4 runnable [cc]Feb-22 11:12:14 ScriptRunner - [cc]Feb-22 11:12:14 ScriptRunner - "VM Periodic Task Thread" prio=3 tid=0x08141800 nid=0xc waiting on condition [cc]Feb-22 11:12:14 ScriptRunner - [cc]Feb-22 11:12:14 ScriptRunner - JNI global references: 1235 [cc]Feb-22 11:12:14 ScriptRunner - [cc]Feb-22 11:12:14 ScriptRunner - Heap [cc]Feb-22 11:12:14 ScriptRunner - PSYoungGen total 67136K, used 55220K [0xf3e00000, 0xfb000000, 0xfb000000) [cc]Feb-22 11:12:14 ScriptRunner - eden space 64640K, 81% used [0xf3e00000,0xf71b52b8,0xf7d20000) [cc]Feb-22 11:12:14 ScriptRunner - from space 2496K, 91% used [0xf7d20000,0xf7f58010,0xf7f90000) [cc]Feb-22 11:12:14 ScriptRunner - to space 2752K, 0% used [0xfad50000,0xfad50000,0xfb000000) [cc]Feb-22 11:12:14 ScriptRunner - PSOldGen total 69632K, used 2971K [0xbae00000, 0xbf200000, 0xf3e00000) [cc]Feb-22 11:12:14 ScriptRunner - object space 69632K, 4% used [0xbae00000,0xbb0e6d80,0xbf200000) [cc]Feb-22 11:12:14 ScriptRunner - PSPermGen total 16384K, used 7839K [0xb6e00000, 0xb7e00000, 0xbae00000) [cc]Feb-22 11:12:14 ScriptRunner - object space 16384K, 47% used [0xb6e00000,0xb75a7d08,0xb7e00000) [cc]Feb-22 11:12:14 ScriptRunner - -----Original Message----- From: Antoine Levy Lambert [mailto:[email protected]] Sent: Saturday, February 13, 2010 4:36 PM To: Ant Users List Subject: Re: Exec Connection Not Closing Barry Pape wrote: > Hi Everyone, > > > > I seem to have an issue with an exec task not closing the ssh connection > it opens. Basically we use CruiseControl and Ant to build and deploy > our code. In the CruiseControl build.xml I have an ant task that ssh's > to a remote server and starts a shell script. Once the script completes > CruiseControl continues along its merry way...until recently. Now at > seemingly random times, one of these ssh connections will not close thus > causing CruiseControl to hang. If I kill the process, CruiseControl > continues like normal. > > > I am using CruiseControl and never saw CruiseControl hanging because a build is hanging. However if CruiseControl is configured with only one thread, only one build can run at one time. > The ant task is: > > <exec executable="ssh" logError="true"> > > <arg line="${remote.host} ${sudo.cmd} -u > ${username} /usr/local/qa/deploy_build.sh ${deploy.param}" /> > > </exec> > > You could use the <sshexec/> ant task instead which is shipping with ant, and requires jsch. Maybe the underlying problem is that sudo is now configured to request a password ? or your remote user is expired ? You should try to run this ssh command by hand to see what is wrong. > > > This had been working fine for the past year. Now we are having issues. > It started when we migrated to a new build server. It has a more recent > OS (solaris 2.10 instead of 2.9). > > > > I tried adding the timeout attribute to the exec task, but all that > seemed to do was hide the stuck process. > > > > Any thoughts or suggestions are much appreciated! > > > > Thanks, > > Barry > > Regards, Antoine --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected] Confidentiality Notice! This electronic transmission and any attached documents or other writings are confidential and are for the sole use of the intended recipient(s) identified above. This message may contain information that is privileged, confidential or otherwise protected from disclosure under applicable law. If the receiver of this information is not the intended recipient, or the employee, or agent responsible for delivering the information to the intended recipient, you are hereby notified that any use, reading, dissemination, distribution, copying or storage of this information is strictly prohibited. If you have received this information in error, please notify the sender by return email and delete the electronic transmission, including all attachments from your system. --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
