New submission from Gunnar Aastrand Grimnes:
When reading large files with fileinput, it will work as expected and only
process a line at a time when used normally, but if you add an hook_encoded
openhook it will read the whole file into memory before returning the first
line.
Verify by running this program on a large text file:
import fileinput
for l in fileinput.input(openhook=fileinput.hook_encoded('iso-8859-1')):
raw_input()
and check how much memory it uses. Remove the openhook and memory usage goes
down to nothing.
----------
components: IO
messages: 210130
nosy: gromgull
priority: normal
severity: normal
status: open
title: fileinput module will read whole file into memory when using
fileinput.hook_encoded
type: behavior
versions: Python 2.7
_______________________________________
Python tracker <[email protected]>
<http://bugs.python.org/issue20501>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
