On Tue, Aug 17, 2010 at 9:40 AM, Rodrick Brown <rodrick.br...@gmail.com> wrote: > I have a fairly large file 1-2GB in size that I need to process line by line > but I first need to convert the file to text using a 3rd party tool that > prints the records also line by line. > > I've tried using Popen to do this with no luck. I'm trying to simulate > > /bin/foo myfile.dat > > And as the records are being printed do some calculations. > > pipe = Popen(exttool,shell=True,stdout=PIPE).stdout
I'd strongly suggest trying to avoid shell=True. > for data in pipe.readlines(): for data in pipe: > print data, > > This operation blocks forever I'm guessing it's trying to process the entire > file at once. Indeed, that's how readlines() works, so don't use it. http://docs.python.org/tutorial/inputoutput.html#methods-of-file-objects Cheers, Chris -- http://blog.rebertia.com -- http://mail.python.org/mailman/listinfo/python-list
