Doing further research on this, we've uncovered the following: The problem we're having is large files being truncated in ASCII mode when lftp receives a timeout from a server where REST is not supported (which it is not in ASCII mode). We originally saw this originating from an AIX 5.3 machine to an IBM mainframe. We have replicated on other systems such as RHEL Linux and Solaris. >From our lftp debug, cut from the end of the transfer (points of interest bolded, our comments in red): ******** >>>>> Put_LL: xbreak5b, len=8291 ***** > res=75 ***** > if(res==len && skip_cr) ***** > put_ll_timer->Reset() ***** > (FINAL) return res=77 So lftp writes 77 bytes (we assume this is from the beginning of the current buffer to the next new line) ******** >>>>> Put_LL: xbreak5b, len=8214 ***** > res=75 ***** > if(res==len && skip_cr) ***** > put_ll_timer->Reset() ***** > (FINAL) return res=77 The next line (this is a structured file, so all lines are 75 bytes long), at this time, the server has transferred the entire file. ---- Put/Get: Got EOF on data connection ---- Closing data socket --- isLargeFile : 0 **** Timeout - reconnecting The server has a 10 minutes timeout on the control channel, and this transfer took little less than an hour. ---- Closing control socket dns cache hit ---- Connecting to host.example.com (x.x.x.x) port 21 <--- 220-FTP161 IBM FTP CS V1R7 at host.example.com, 18:47:08 on 2007-08-13. ******** >>>>> Put_LL: xbreak5b, len=8137 ***** > res=75 ***** > if(res==len && skip_cr) ***** > put_ll_timer->Reset() ***** > (FINAL) return res=77 We see that the "put" process is still writing its existing buffer to the file. ******** >>>>> Put_LL: xbreak5b, len=8060 ***** > res=75 ***** > if(res==len && skip_cr) ***** > put_ll_timer->Reset() ***** > (FINAL) return res=77 <--- 220 Connection will close if idle for more than 10 minutes. ---> AUTH TLS <--- 234 Security environment established - ready for negotiation ---> USER user ******** >>>>> Put_LL: xbreak5b, len=7983 ***** > res=75 ***** > if(res==len && skip_cr) ***** > put_ll_timer->Reset() ***** > (FINAL) return res=77
<...snip certificate information...> <--- 331 Send password please. ---> PASS XXXX ******** >>>>> Put_LL: xbreak5b, len=7906 ***** > res=75 ***** > if(res==len && skip_cr) ***** > put_ll_timer->Reset() ***** > (FINAL) return res=77 <--- 230 user is logged on. Working directory is "user.". ---> FEAT <--- 211- Extensions supported ******** >>>>> Put_LL: xbreak5b, len=7829 ***** > res=75 ***** > if(res==len && skip_cr) ***** > put_ll_timer->Reset() ***** > (FINAL) return res=77 Put is still running.... <--- AUTH TLS <--- PBSZ <--- PROT <--- 211 End ---> PBSZ 0 <--- 200 Protection buffer size accepted ---> PROT P <--- 200 Data connection protection set to private ---> PROT P <--- 200 Data connection protection set to private command = RETR ---> PASV FileCopy::Do: get rolled back to 0, seeking put accordingly At this point, we assume that "get" rolled back to zero because the connection reconnected and REST is not supported (regardless of ASCII mode). "Put" realized this and rolled back. However, above we see that 7752 still remain in "put's" buffer. This buffer is then lost! copy: put confirmed store ***** > STEP 111 ***** > STEP 222 FileCopy(2003d2a8) enters state GET_DONE_WAIT copy: get is finished - all done We would expect that the file transfer would start the beginning but instead it just replies "all done". Note that we don't want it to restart since that will just become an infinite transfer loop. FileCopy(2003d2a8) enters state ALL_DONE ---- Closing data socket ---- Closing control socket 396716651 bytes transferred in 3342 seconds (115.9K/s) We also noticed that in binary mode, FileCopyPeerFDStream will write its entire buffer out on each Put_LL pass. In ASCII mode, this writes out only the part of the buffer up until the next new line. What we propose is that Put in the Buffer class loop until FDStream has written out its buffer completely each pass. However, we wanted to pose the question to Alex and the community to see if there are any concerns to performance, etc... We also wanted to see if anyone else has observed this problem and might have more familiarity with the code to propose a better solution. Thanks! Jason Burns Information Security Technology CAST -> Cryptography Services -> Secure Communications and Data Encryption UnixSecure Lead Engineer _____ From: Chang, Shannon H. Sent: Friday, August 17, 2007 10:27 AM To: [email protected] Subject: Large files truncated As this is my first post on this site, please excuse any gaps in understanding... It seems that data still in the buffer is not getting written out when transferring large files in ASCII mode that encounters a timeout. Currently with ipswitch this behavior is not observed. Here are our steps: 1. Set timeout on proftpd for 10 minutes 2. Set ASCII mode 3. Start transferring large file in ASCII mode 4. Timeout occurs and there is still data in buffer 5. REST in ASCII mode is not supported and data in buffer is not written out and lost. Is there an existing fix out there that changes step 4 so that the data in the buffer is flushed and written out? If not I can attempt to add this. Shannon
