I have no console to type that in because all our companies java processes are indirectly started from a process controller. However, I have a point where I log out the stack traces to file when trying to alter that transfer job. Here is where the transfer job hangs indefinitely:
2012-09-14T08:00:41.446Z TRACE [auto.AutoService.handleThreadShutdown] java.net.SocketInputStream.socketRead0(Native Method) 2012-09-14T08:00:41.446Z TRACE [auto.AutoService.handleThreadShutdown] java.net.SocketInputStream.read(SocketInputStream.java:129) 2012-09-14T08:00:41.446Z TRACE [auto.AutoService.handleThreadShutdown] sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:264) 2012-09-14T08:00:41.446Z TRACE [auto.AutoService.handleThreadShutdown] sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:306) 2012-09-14T08:00:41.446Z TRACE [auto.AutoService.handleThreadShutdown] sun.nio.cs.StreamDecoder.read(StreamDecoder.java:158) 2012-09-14T08:00:41.447Z TRACE [auto.AutoService.handleThreadShutdown] java.io.InputStreamReader.read(InputStreamReader.java:167) 2012-09-14T08:00:41.447Z TRACE [auto.AutoService.handleThreadShutdown] java.io.BufferedReader.fill(BufferedReader.java:136) 2012-09-14T08:00:41.447Z TRACE [auto.AutoService.handleThreadShutdown] java.io.BufferedReader.readLine(BufferedReader.java:299) 2012-09-14T08:00:41.447Z TRACE [auto.AutoService.handleThreadShutdown] java.io.BufferedReader.readLine(BufferedReader.java:362) 2012-09-14T08:00:41.447Z TRACE [auto.AutoService.handleThreadShutdown] org.apache.commons.net.ftp.FTPFileEntryParserImpl.readNextEntry(FTPFileEntryParserImpl.java:52) 2012-09-14T08:00:41.447Z TRACE [auto.AutoService.handleThreadShutdown] org.apache.commons.net.ftp.FTPListParseEngine.readStream(FTPListParseEngine.java:133) 2012-09-14T08:00:41.447Z TRACE [auto.AutoService.handleThreadShutdown] org.apache.commons.net.ftp.FTPListParseEngine.readServerList(FTPListParseEngine.java:102) 2012-09-14T08:00:41.447Z TRACE [auto.AutoService.handleThreadShutdown] org.apache.commons.net.ftp.FTPClient.initiateListParsing(FTPClient.java:3103) 2012-09-14T08:00:41.448Z TRACE [auto.AutoService.handleThreadShutdown] org.apache.commons.net.ftp.FTPClient.initiateListParsing(FTPClient.java:3072) 2012-09-14T08:00:41.448Z TRACE [auto.AutoService.handleThreadShutdown] org.apache.commons.net.ftp.FTPClient.listFiles(FTPClient.java:2731) Note that this stacktrace was made with commons-net 3.1. I already looked in the source and it's waiting for a readLine() which never returns. Given the fact that the log line "<-- 226 >> Transfer complete. 117 bytes transferred. 7,14 KB/sec." is missing in the logs (see my initial post) I assume the library waits for this line to come from the server, which for some reason never makes it. What I find interesting though is that no timeout seems to interrupt this endless waiting. Is this a misconfiguration on my part or a bug in the software? I do ftpClient.setSoTimeout(30000); In my software. Which other Timout might I have to set to cause the readLine() to timeout at some point? Kind regards, Johannes Frank From: Gary Gregory <garydgreg...@gmail.com> To: Commons Users List <user@commons.apache.org> Date: 11.09.2012 18:44 Subject: Re: [NET] FTP Connection occasionally hangs on listFiles() If you start the JVM from a command line, you can type control-break to get a thread dump at any time. Gary -------------------------------------------------------------------------------------------------------------------------------------------- Johannes Frank - KISTERS AG - Charlottenburger Allee 5 - 52068 Aachen - Germany Handelsregister Aachen, HRB-Nr. 7838 | Vorstand: Klaus Kisters, Hanns Kisters | Aufsichtsratsvorsitzender: Dr. Thomas Klevers Phone: +49 241 9671 -0 | Fax: +49 241 9671 -555 | E-Mail: johannes.fr...@kisters.de | WWW: -------------------------------------------------------------------------------------------------------------------------------------------- This e-mail may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and destroy this e-mail. Any unauthorised copying, disclosure or distribution of the material in this e-mail is strictly forbidden.On Sep 11, 2012, at 12:06, sebb <seb...@gmail.com> wrote: > On 11 September 2012 16:05, Johannes Frank <johannes.fr...@kisters.de> wrote: >> Hello! >> >> As stated in my initial post, the only way of "error log" is the missing >> of the log line stating that the list files command completed "<-- 226 >> Transfer complete. 117 bytes transferred. 7,14 KB/sec." >> >> I have to check if I have a data Timeout set, I do definitely have a >> socket timeout set. Will give the data timout a look. >> I do not really understand the CommandListener suggestion. >> Regarding the thread dump I will try if I can find one in the logs we had >> on the customer machine. > > Unlikely; a thread dump normally has to be requested manually. > > Next time it happens, if possible please generate the thread dump > before killing the process. > > This can be done with jvisualvm which is part of the JDK. > >> I am relatively sure it's NOT an issue with the server (as we had the >> issue with two different servers on two different operating systems), > > On the other hand, you say that Commons NET works OK in 99% of cases ... > >> which does not mean that the error has to be in NET (could still easily be >> something occult in their network...) > >> Thanks by the way for your effort! >> >> Regards, >> Johannes. >> >> >> >> From: >> sebb <seb...@gmail.com> >> To: >> Commons Users List <user@commons.apache.org> >> Date: >> 11.09.2012 16:52 >> Subject: >> Re: [NET] FTP Connection occasionally hangs on listFiles() >> >> >> >> On 11 September 2012 12:53, Johannes Frank <johannes.fr...@kisters.de> >> wrote: >>> Hello! >>> >>> Thank you for your reply. However, I cannot provide such a program since >>> (as said) I cannot reliably reproduce that problem (on 99% of the >> systems >>> not at all and on the two mentioned systems only sporadically). From >> what >>> I see, simply calling FTPClient.listFiles() on these two systems often >>> enough will suffice to "reproduce" this problem :-( >> >> How are you initialising the FTPClient instance? >> >> You could add a command listener using >> FTPClient#addProtocolCommandListener; if listFiles() returns OK, then >> delete the output. >> This might give some useful info. >> >> Also, try setting a data timeout using FTPClient#setDataTimeout >> >> Is there any error indication in the server logs? >> >> Also, can you create a thread dump when the problem occurs? >> That should show where it is waiting. >> >> Without a way to reproduce the issue, it's going to be tricky to debug >> and fix (if it is a problem with NET rather than a server issue). >> >>> Regards, >>> Johannes Frank >>> >>> >>> >>> >>> From: >>> sebb <seb...@gmail.com> >>> To: >>> Commons Users List <user@commons.apache.org> >>> Date: >>> 11.09.2012 01:00 >>> Subject: >>> Re: [NET] FTP Connection occasionally hangs on listFiles() >>> >>> >>> >>> On 10 September 2012 08:09, Johannes Frank <johannes.fr...@kisters.de> >>> wrote: >>>> Hello together. >>>> >>>> I have a weird problem which unfortunately leads to production halts at >>>> two of our customers so I urge for a pointer in the right direction >>> here. >>>> >>>> My software periodically detects files on a FTP Server and downloads >>> them. >>>> This works fine for some time but quite randomly, out of a suddin this >>>> loop starts to hang indefinitely until manually killed and restarted by >>>> the Administrator. >>>> >>>> I narrowed it down so far that I can say that the FTPClient.listFiles() >>>> method hangs and never returns. Here is what the ftp communication log >>>> says on a normal run: >>>> >>>> 2012-09-04T12:41:01.130Z DEBUG >>> [location.FtpLocation.protocolCommandSent] >>>> Veribox A-Dateien: --> NOOP >>>> 2012-09-04T12:41:01.132Z DEBUG >>>> [location.FtpLocation.protocolReplyReceived] Veribox A-Dateien: <-- 200 >>>> Command okay. >>>> 2012-09-04T12:41:01.132Z DEBUG >>> [location.FtpLocation.protocolCommandSent] >>>> Veribox A-Dateien: --> PASV >>>> 2012-09-04T12:41:01.140Z DEBUG >>>> [location.FtpLocation.protocolReplyReceived] Veribox A-Dateien: <-- 227 >>>> Entering Passive Mode (172,35,1,12,7,70 >>>> ) >>>> 2012-09-04T12:41:01.141Z DEBUG >>> [location.FtpLocation.protocolCommandSent] >>>> Veribox A-Dateien: --> LIST * >>>> 2012-09-04T12:41:01.146Z DEBUG >>>> [location.FtpLocation.protocolReplyReceived] Veribox A-Dateien: <-- 150 >>>> Opening ASCII mode data connection for >>>> /bin/ls. >>>> 2012-09-04T12:41:01.186Z DEBUG >>>> [location.FtpLocation.protocolReplyReceived] Veribox A-Dateien: <-- 226 >>>> Transfer complete. 117 bytes transferre >>>> d. 7,14 KB/sec. >>>> >>>> And here is what it says the moment it hangs up: >>>> >>>> 2012-09-04T12:41:51.196Z DEBUG >>> [location.FtpLocation.protocolCommandSent] >>>> Veribox A-Dateien: --> NOOP >>>> 2012-09-04T12:41:51.197Z DEBUG >>>> [location.FtpLocation.protocolReplyReceived] Veribox A-Dateien: <-- 200 >>>> Command okay. >>>> 2012-09-04T12:41:51.198Z DEBUG >>> [location.FtpLocation.protocolCommandSent] >>>> Veribox A-Dateien: --> PASV >>>> 2012-09-04T12:41:51.204Z DEBUG >>>> [location.FtpLocation.protocolReplyReceived] Veribox A-Dateien: <-- 227 >>>> Entering Passive Mode (172,35,1,12,7,71 >>>> ) >>>> 2012-09-04T12:41:51.205Z DEBUG >>> [location.FtpLocation.protocolCommandSent] >>>> Veribox A-Dateien: --> LIST * >>>> 2012-09-04T12:41:51.211Z DEBUG >>>> [location.FtpLocation.protocolReplyReceived] Veribox A-Dateien: <-- 150 >>>> Opening ASCII mode data connection for >>>> /bin/ls. >>>> >>>> As you can see, the server answers that it's opening an ascii mode data >>>> connection but then nothing seems to happen. >>>> >>>> This is what netstat -np says: >>>> tcp 91 0 172.30.3.21:39009 172.35.1.12:2730 CLOSE_WAIT >>>> 24094/java >>>> tcp 0 0 172.30.3.21:45273 172.35.1.12:1863 >> ESTABLISHED >>>> 24094/java >>>> tcp 0 0 172.30.3.21:56584 172.35.1.12:2299 >> ESTABLISHED >>>> 24094/java >>>> tcp 0 0 172.30.3.21:53582 172.35.1.12:2730 >> ESTABLISHED >>>> 24094/java >>>> tcp 0 0 172.30.3.21:57945 172.35.1.12:2730 >> ESTABLISHED >>>> 24094/java >>>> tcp 95 0 172.30.3.21:39010 172.35.1.12:2730 CLOSE_WAIT >>>> 24094/java >>>> >>>> 2730 is the control port of the ftp server, 2299, 1863 are data >>>> connections. Note the non-empty Send-Q for the CLOSE_WAIT connections. >>>> >>>> The issue I have here seems to be relatively independent of everything >> I >>>> can come up with. >>>> We have this problem in both linux (SuSE 11.4 64bit) and Windows XP, >> The >>>> FTP server being once a Serv-U under Windows 2003 Standard and vsftp >>> under >>>> linux. The java virtual machine is openjdk under linux and sun java 6 >>>> under windows. As far as I can see, no proxies or firewalls are in the >>> way >>>> on either of the customers systems. I tested with both commons-net 2.0 >>> and >>>> 3.1 with no difference in result. >>>> >>>> Did anyone on this list ever encounter an issue like this before and is >>>> there a way to circumvent this stalling of the listFiles() call? >>> >>> If you can provide a sample program that reproduces the problem, then >>> please raise a JIRA issue and attach the code and sample output from >>> the run. >>> Make sure you enable command logging. >>> >>>> Kind regards, >>>> Johannes Frank >>>> >>> >> -------------------------------------------------------------------------------------------------------------------------------------------- >>>> Johannes Frank - KISTERS AG - Charlottenburger Allee 5 - 52068 Aachen - >>> Germany >>>> Handelsregister Aachen, HRB-Nr. 7838 | Vorstand: Klaus Kisters, Hanns >>> Kisters | Aufsichtsratsvorsitzender: Dr. Thomas Klevers >>>> Phone: +49 241 9671 -0 | Fax: +49 241 9671 -555 | E-Mail: >>> johannes.fr...@kisters.de | WWW: >>>> >>> >> -------------------------------------------------------------------------------------------------------------------------------------------- >>>> This e-mail may contain confidential and/or privileged information. If >>> you are not the intended recipient (or have received this e-mail in >> error) >>> please notify the sender immediately and destroy this e-mail. Any >>> unauthorised copying, disclosure or distribution of the material in this >>> e-mail is strictly forbidden. >>> >>> --------------------------------------------------------------------- >>> To unsubscribe, e-mail: user-unsubscr...@commons.apache.org >>> For additional commands, e-mail: user-h...@commons.apache.org >>> >>> >>> >>> >>> >> -------------------------------------------------------------------------------------------------------------------------------------------- >>> Johannes Frank - KISTERS AG - Charlottenburger Allee 5 - 52068 Aachen - >> Germany >>> Handelsregister Aachen, HRB-Nr. 7838 | Vorstand: Klaus Kisters, Hanns >> Kisters | Aufsichtsratsvorsitzender: Dr. Thomas Klevers >>> Phone: +49 241 9671 -0 | Fax: +49 241 9671 -555 | E-Mail: >> johannes.fr...@kisters.de | WWW: >>> >> -------------------------------------------------------------------------------------------------------------------------------------------- >>> This e-mail may contain confidential and/or privileged information. If >> you are not the intended recipient (or have received this e-mail in error) >> please notify the sender immediately and destroy this e-mail. Any >> unauthorised copying, disclosure or distribution of the material in this >> e-mail is strictly forbidden. >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: user-unsubscr...@commons.apache.org >> For additional commands, e-mail: user-h...@commons.apache.org >> >> >> >> >> -------------------------------------------------------------------------------------------------------------------------------------------- >> Johannes Frank - KISTERS AG - Charlottenburger Allee 5 - 52068 Aachen - Germany >> Handelsregister Aachen, HRB-Nr. 7838 | Vorstand: Klaus Kisters, Hanns Kisters | Aufsichtsratsvorsitzender: Dr. Thomas Klevers >> Phone: +49 241 9671 -0 | Fax: +49 241 9671 -555 | E-Mail: johannes.fr...@kisters.de | WWW: >> -------------------------------------------------------------------------------------------------------------------------------------------- >> This e-mail may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and destroy this e-mail. Any unauthorised copying, disclosure or distribution of the material in this e-mail is strictly forbidden. > > --------------------------------------------------------------------- > To unsubscribe, e-mail: user-unsubscr...@commons.apache.org > For additional commands, e-mail: user-h...@commons.apache.org > --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@commons.apache.org For additional commands, e-mail: user-h...@commons.apache.org
