On Jun 6, 11:21 pm, "Hendrik van Rooyen" <[EMAIL PROTECTED]> wrote: > <[EMAIL PROTECTED]> wrote: > > Hi all, > > > I'm using ftplib to transfer large files to remote sites. The process > > seems to work perfectly with small files, but when the file gets to > > large ~20GB I begin getting errors that sometimes seem to be non- > > fatal, and other times the transfer does not complete. I've debugged > > the hell out of the code, and can't figure out what is going on..is it > > the remote site? Packet loss? > > 20 gig is a big thing - what is your link speed, and how long does the thing > run before falling over - Is your ISP maybe giving you or the remote site a > new IP addy periodically? > > Are there any "anti jabber" precautions taken anywhere? > > you may have to break it up into smaller chunks and glue them together > remotely... > > hth - Hendrik
It's different sites (state to state) within the same company, so I'm sure of the IP addresses. Whats odd is that the transfers don't fail most of the time. The transfer usually completes properly but ftplib.FTP.storbinary() throws an IOError at the end of the file instead of returning the status of the transfer. I've resorted to just catching and logging the exception and then comparing remote vs. local md5's to check for integrity, but it seems like if a connection error occurs, storbinary should handle the error and return an error code rather than throwing an exception. The 20gb files take about an hour to transfer, and I've had success with files up to 60gb so far... once I figured out what ftplib was doing and starting cathing the IOError. Why would storbinary throw an exception even when the file transfer was successful? Why would the exception not be thrown until after the file was sent? Shouldn't ftplib return something like (104, "Connection reset by peer") instead of throwing an error? Is my reasoning off somewhere? ~Sean -- http://mail.python.org/mailman/listinfo/python-list