On Tuesday 17 August 2010, it occurred to Rodrick Brown to exclaim: > I have a fairly large file 1-2GB in size that I need to process line by > line but I first need to convert the file to text using a 3rd party tool > that prints the records also line by line. > > I've tried using Popen to do this with no luck. I'm trying to simulate > > /bin/foo myfile.dat > > And as the records are being printed do some calculations. > > pipe = Popen(exttool,shell=True,stdout=PIPE).stdout > > for data in pipe.readlines(): > print data, > > This operation blocks forever I'm guessing it's trying to process the > entire file at once.
Yes. It is. That's what you're telling it to do: file.readline returns a list of all the lines in the file. What you want to do is iterate over the stream, as in: for line in pipe: process(line) Also, there's probably no need to use shell=True. > > Sent from my iPhone 4. Is that a fact? This is so interesting. Phones these days. Almost as annoyingly obnoxious as gmx and yahoo mail. -- http://mail.python.org/mailman/listinfo/python-list