Re: ftplib error- Large file
[EMAIL PROTECTED] wrote: > Ok. I guess that makes sense. But what about the other > questions...mainly: Why would it throw an exception even though the > file was properly transferred? Je, well, I answered the one I knew about, :) Regarding the error... es hard to say. What happens if you transmit the file using a standard ftp program? Or better, could you sniff your network to see what is the TCP dialog? > I guess this could be blamed on the FTP server. Maybe. Maybe not... -- . Facundo . Blog: http://www.taniquetil.com.ar/plog/ PyAr: http://www.python.org/ar/ -- http://mail.python.org/mailman/listinfo/python-list
Re: ftplib error- Large file
On Jun 7, 8:54 am, Facundo Batista <[EMAIL PROTECTED]> wrote: > [EMAIL PROTECTED] wrote: > > Why would storbinary throw an exception even when the file transfer > > was successful? Why would the exception not be thrown until after the > > file was sent? Shouldn't ftplib return something like (104, > > "Connection reset by peer") instead of throwing an error? Is my > > reasoning off somewhere? > > There was an error, the connection was reset by peer. The error code is > 104. As it *was* an error, an exception is correct. > > You always can hide it yourself, with try/except... > > Regards, > > -- > . Facundo > . > Blog:http://www.taniquetil.com.ar/plog/ > PyAr:http://www.python.org/ar/ Ok. I guess that makes sense. But what about the other questions...mainly: Why would it throw an exception even though the file was properly transferred? I guess this could be blamed on the FTP server. ~Sean -- http://mail.python.org/mailman/listinfo/python-list
Re: ftplib error- Large file
[EMAIL PROTECTED] wrote: > Why would storbinary throw an exception even when the file transfer > was successful? Why would the exception not be thrown until after the > file was sent? Shouldn't ftplib return something like (104, > "Connection reset by peer") instead of throwing an error? Is my > reasoning off somewhere? There was an error, the connection was reset by peer. The error code is 104. As it *was* an error, an exception is correct. You always can hide it yourself, with try/except... Regards, -- . Facundo . Blog: http://www.taniquetil.com.ar/plog/ PyAr: http://www.python.org/ar/ -- http://mail.python.org/mailman/listinfo/python-list
Re: ftplib error- Large file
On Jun 6, 11:21 pm, "Hendrik van Rooyen" <[EMAIL PROTECTED]> wrote: > <[EMAIL PROTECTED]> wrote: > > Hi all, > > > I'm using ftplib to transfer large files to remote sites. The process > > seems to work perfectly with small files, but when the file gets to > > large ~20GB I begin getting errors that sometimes seem to be non- > > fatal, and other times the transfer does not complete. I've debugged > > the hell out of the code, and can't figure out what is going on..is it > > the remote site? Packet loss? > > 20 gig is a big thing - what is your link speed, and how long does the thing > run before falling over - Is your ISP maybe giving you or the remote site a > new IP addy periodically? > > Are there any "anti jabber" precautions taken anywhere? > > you may have to break it up into smaller chunks and glue them together > remotely... > > hth - Hendrik It's different sites (state to state) within the same company, so I'm sure of the IP addresses. Whats odd is that the transfers don't fail most of the time. The transfer usually completes properly but ftplib.FTP.storbinary() throws an IOError at the end of the file instead of returning the status of the transfer. I've resorted to just catching and logging the exception and then comparing remote vs. local md5's to check for integrity, but it seems like if a connection error occurs, storbinary should handle the error and return an error code rather than throwing an exception. The 20gb files take about an hour to transfer, and I've had success with files up to 60gb so far... once I figured out what ftplib was doing and starting cathing the IOError. Why would storbinary throw an exception even when the file transfer was successful? Why would the exception not be thrown until after the file was sent? Shouldn't ftplib return something like (104, "Connection reset by peer") instead of throwing an error? Is my reasoning off somewhere? ~Sean -- http://mail.python.org/mailman/listinfo/python-list
Re: ftplib error- Large file
<[EMAIL PROTECTED]> wrote: > Hi all, > > I'm using ftplib to transfer large files to remote sites. The process > seems to work perfectly with small files, but when the file gets to > large ~20GB I begin getting errors that sometimes seem to be non- > fatal, and other times the transfer does not complete. I've debugged > the hell out of the code, and can't figure out what is going on..is it > the remote site? Packet loss? > 20 gig is a big thing - what is your link speed, and how long does the thing run before falling over - Is your ISP maybe giving you or the remote site a new IP addy periodically? Are there any "anti jabber" precautions taken anywhere? you may have to break it up into smaller chunks and glue them together remotely... hth - Hendrik -- http://mail.python.org/mailman/listinfo/python-list
Re: ftplib error- Large file
En Wed, 06 Jun 2007 13:39:57 -0300, <[EMAIL PROTECTED]> escribió: > I'm using ftplib to transfer large files to remote sites. The process > seems to work perfectly with small files, but when the file gets to > large ~20GB I begin getting errors that sometimes seem to be non- > fatal, and other times the transfer does not complete. I've debugged > the hell out of the code, and can't figure out what is going on..is it > the remote site? Packet loss? > > > IOError: [Errno 104] Connection reset by peer You always have to consider a connection failure - it's just more likely to happen on a big file. The REST ftp command is used to restart the transfer from a certain offset. -- Gabriel Genellina -- http://mail.python.org/mailman/listinfo/python-list
ftplib error- Large file
Hi all, I'm using ftplib to transfer large files to remote sites. The process seems to work perfectly with small files, but when the file gets to large ~20GB I begin getting errors that sometimes seem to be non- fatal, and other times the transfer does not complete. I've debugged the hell out of the code, and can't figure out what is going on..is it the remote site? Packet loss? Traceback (most recent call last): File "/usr/local/sw/bin/FTPput.py", line 142, in execute self.ftp.put(file.path) File "/usr/local/sw/lib/FTPconn.py", line 32, in put return self.storbinary("STOR %s" % destfile, open(f.path, 'rb')) File "/usr/lib/python2.2/ftplib.py", line 422, in storbinary return self.voidresp() File "/usr/lib/python2.2/ftplib.py", line 222, in voidresp resp = self.getresp() File "/usr/lib/python2.2/ftplib.py", line 208, in getresp resp = self.getmultiline() File "/usr/lib/python2.2/ftplib.py", line 194, in getmultiline line = self.getline() File "/usr/lib/python2.2/ftplib.py", line 181, in getline line = self.file.readline() IOError: [Errno 104] Connection reset by peer I'd appreciate any input or ideas. ~Sean -- http://mail.python.org/mailman/listinfo/python-list