convert ftp.retrbinary to file object? - Python language lacks expression?
I just tried to convert a (hugh size) ftp.retrbinary run into a pseudo-file object with .read(bytes) method in order to not consume 500MB on a copy operation. First I thought, its easy as usual with python using something like 'yield' or so. Yet I didn't manage to do (without using threads or rewriting 'retrbinary')? Any ideas? I tried a pattern like: def open(self,ftppath,mode='rb'): class FTPFile: #TODO ... def iter_retr() ... def callback(blk): how-to-yield-from-here-to-iter_retr blk??? ftp.retrbinary(RETR %s % relpath,callback) def read(self, bytes=-1): ... self.buf+=self.iter.next() ... -- http://mail.python.org/mailman/listinfo/python-list
Re: convert ftp.retrbinary to file object? - Python language lacks expression?
Robert wrote: I just tried to convert a (hugh size) ftp.retrbinary run into a pseudo-file object with .read(bytes) method in order to not consume 500MB on a copy operation. First I thought, its easy as usual with python using something like 'yield' or so. Yet I didn't manage to do (without using threads or rewriting 'retrbinary')? Any ideas? I tried a pattern like: def open(self,ftppath,mode='rb'): class FTPFile: #TODO ... def iter_retr() ... def callback(blk): how-to-yield-from-here-to-iter_retr blk??? ftp.retrbinary(RETR %s % relpath,callback) def read(self, bytes=-1): ... self.buf+=self.iter.next() ... H this is nearly there I think...: import ftplib class TransferAbort(Exception): pass class FTPFile: def __init__(self, server, filename): self.server = server self.filename = filename self.offset = 0 def callback(self, data): self.offset = self.offset + len(data) self.data = data ## now quit the RETR command? raise TransferAbort(stop right now) def read(self, amount): self.ftp = ftplib.FTP(self.server) self.ftp.login() try: self.ftp.retrbinary(RETR %s %self.filename, self.callback, blocksize=amount, rest=self.offset) except TransferAbort: return self.data f = FTPFile(HOSTNAME, FILENAME) print f.read(24) print f.read(24) I open the ftp connection inside the read method as it caused an error (on the second call to read) when I opened it in __init__ ??? HTH Martin -- http://mail.python.org/mailman/listinfo/python-list
Re: convert ftp.retrbinary to file object? - Python language lacks expression?
Martin Franklin wrote: Martin Franklin wrote: Robert wrote: I just tried to convert a (hugh size) ftp.retrbinary run into a pseudo-file object with .read(bytes) method in order to not consume 500MB on a copy operation. [snip] H this is nearly there I think...: whoops... spoke too soon.. Trigger happy this morning... import ftplib class TransferAbort(Exception): pass class FTPFile: def __init__(self, server, filename): self.server = server self.filename = filename self.offset = 0 def callback(self, data): self.offset = self.offset + len(data) self.data = data ## now quit the RETR command? raise TransferAbort(stop right now) def read(self, amount): self.ftp = ftplib.FTP(self.server) self.ftp.login() I needed to insert a time.sleep(0.1) here as the connections were falling over themselves - I guess testing with a blocksize of 24 is a little silly. try: self.ftp.retrbinary(RETR %s %self.filename, self.callback, blocksize=amount, rest=self.offset) except TransferAbort: also need to close the ftp connection here! self.ftp.close() return self.data f = FTPFile(HOSTNAME, FILENAME) print f.read(24) print f.read(24) ## new test... f = FTPFile(HOSTNAME, FILENAME) while 1: data = f.read(24) if not data: break print data, I open the ftp connection inside the read method as it caused an error (on the second call to read) when I opened it in __init__ ??? HTH Martin -- http://mail.python.org/mailman/listinfo/python-list
Re: convert ftp.retrbinary to file object? - Python language lacks expression?
Robert wrote: That turns into periodic new RETR commands with offset. Think its more an odd trick. I'd even prefer a threaded approach (thread puts the blocks into a stack; a while ... yield generator loop in the main thread serves the .read() function of the pseudo file object, which is my wish). Yet such tricks are all kind of OS-level tricks with a lot of overhead. I wonder really, if the Python language itself can express an elegant flat solution to turn the block delivering callback function into a generator/.read(bytes) solution? I found no way. Don't know whether this would be helpful as a starting point, but a while (hmm, some years ...) ago I wrote an example of how FTP could be used as a file-like object. Look for ftpStream.py on http://www.holdenweb.com/Python/ Of course, in those days files could do a bit less than they can now, so there's no attempt to provide an iterator interface. (Looking over some Ruby stuff, Ruby seems to be able to do so from the language. I am not really familiar to Ruby. I always felt Python to be as complete - but much more clean. I became somewhat jealous ... :-) ) As the solution in my case has to stand many different file systems compatibly ( file.read(bytes) function !) and also other FTPS SFTP classes with different retrbinary functions have to be compatible, I cannot even make a simple FTP subclassed retrbinary without getting really weired. Thus the existing .retrbinary with callback is the official interface in this game. You will note that my code uses delegation to an FTP object rather than inheritance. Maybe you would find that approach more fruitful for your application. regards Steve -- Meet the Python developers and your c.l.py favorites March 23-25 Come to PyCon DC 2005 http://www.pycon.org/ Steve Holden http://www.holdenweb.com/ -- http://mail.python.org/mailman/listinfo/python-list
