STINNER Victor added the comment: On Fedora 20/x86_64, running test_file2k takes up to 4.8 GB (5114316 kB) of RSS memory (VmPeak in /proc/pid/status).
It looks like readahead_get_line_skip() has an efficient code to handle buffer. It uses recursive calls: --- readahead: allocate 0.0 MB readahead: allocate 0.0 MB readahead: allocate 0.0 MB readahead: allocate 0.0 MB readahead: allocate 0.0 MB readahead: allocate 0.0 MB readahead: allocate 0.0 MB readahead: allocate 0.0 MB readahead: allocate 0.0 MB readahead: allocate 0.1 MB readahead: allocate 0.1 MB readahead: allocate 0.1 MB readahead: allocate 0.1 MB readahead: allocate 0.1 MB readahead: allocate 0.2 MB readahead: allocate 0.2 MB readahead: allocate 0.3 MB readahead: allocate 0.3 MB readahead: allocate 0.4 MB readahead: allocate 0.5 MB readahead: allocate 0.7 MB readahead: allocate 0.8 MB readahead: allocate 1.1 MB readahead: allocate 1.3 MB readahead: allocate 1.7 MB readahead: allocate 2.1 MB readahead: allocate 2.6 MB readahead: allocate 3.2 MB readahead: allocate 4.0 MB readahead: allocate 5.0 MB readahead: allocate 6.3 MB readahead: allocate 7.9 MB readahead: allocate 9.9 MB readahead: allocate 12.3 MB readahead: allocate 15.4 MB readahead: allocate 19.3 MB readahead: allocate 24.1 MB readahead: allocate 30.1 MB readahead: allocate 37.6 MB readahead: allocate 47.0 MB readahead: allocate 58.8 MB readahead: allocate 73.5 MB readahead: allocate 91.8 MB readahead: allocate 114.8 MB readahead: allocate 143.5 MB readahead: allocate 179.4 MB readahead: allocate 224.2 MB readahead: allocate 280.2 MB readahead: allocate 350.3 MB readahead: allocate 437.9 MB readahead: allocate 547.3 MB Breakpoint 2, PyObject_Malloc (nbytes=2147483733) at Objects/obmalloc.c:792 792 if (nbytes > PY_SSIZE_T_MAX) (gdb) where #0 PyObject_Malloc (nbytes=2147483733) at Objects/obmalloc.c:792 #1 0x0000000000464af1 in _PyObject_DebugMallocApi (id=111 'o', nbytes=2147483701) at Objects/obmalloc.c:1474 #2 0x0000000000464a48 in _PyObject_DebugMalloc (nbytes=2147483701) at Objects/obmalloc.c:1441 #3 0x000000000046efdc in PyString_FromStringAndSize (str=0x0, size=2147483648) at Objects/stringobject.c:88 #4 0x0000000000436c30 in readahead_get_line_skip (f=0x7fffeea2bf40, skip=2147483648, bufsize=573933340) at Objects/fileobject.c:2291 #5 0x0000000000436da6 in readahead_get_line_skip (f=0x7fffeea2bf40, skip=1836553986, bufsize=459146672) at Objects/fileobject.c:2311 #6 0x0000000000436da6 in readahead_get_line_skip (f=0x7fffeea2bf40, skip=1469236648, bufsize=367317338) at Objects/fileobject.c:2311 #7 0x0000000000436da6 in readahead_get_line_skip (f=0x7fffeea2bf40, skip=1175382777, bufsize=293853871) at Objects/fileobject.c:2311 #8 0x0000000000436da6 in readahead_get_line_skip (f=0x7fffeea2bf40, skip=940299680, bufsize=235083097) at Objects/fileobject.c:2311 #9 0x0000000000436da6 in readahead_get_line_skip (f=0x7fffeea2bf40, skip=752233202, bufsize=188066478) at Objects/fileobject.c:2311 #10 0x0000000000436da6 in readahead_get_line_skip (f=0x7fffeea2bf40, skip=601780019, bufsize=150453183) at Objects/fileobject.c:2311 #11 0x0000000000436da6 in readahead_get_line_skip (f=0x7fffeea2bf40, skip=481417472, bufsize=120362547) at Objects/fileobject.c:2311 #12 0x0000000000436da6 in readahead_get_line_skip (f=0x7fffeea2bf40, skip=385127434, bufsize=96290038) at Objects/fileobject.c:2311 #13 0x0000000000436da6 in readahead_get_line_skip (f=0x7fffeea2bf40, skip=308095403, bufsize=77032031) at Objects/fileobject.c:2311 #14 0x0000000000436da6 in readahead_get_line_skip (f=0x7fffeea2bf40, skip=246469778, bufsize=61625625) at Objects/fileobject.c:2311 #15 0x0000000000436da6 in readahead_get_line_skip (f=0x7fffeea2bf40, skip=197169278, bufsize=49300500) at Objects/fileobject.c:2311 #16 0x0000000000436da6 in readahead_get_line_skip (f=0x7fffeea2bf40, skip=157728878, bufsize=39440400) at Objects/fileobject.c:2311 #17 0x0000000000436da6 in readahead_get_line_skip (f=0x7fffeea2bf40, skip=126176558, bufsize=31552320) at Objects/fileobject.c:2311 #18 0x0000000000436da6 in readahead_get_line_skip (f=0x7fffeea2bf40, skip=100934702, bufsize=25241856) at Objects/fileobject.c:2311 #19 0x0000000000436da6 in readahead_get_line_skip (f=0x7fffeea2bf40, skip=80741217, bufsize=20193485) at Objects/fileobject.c:2311 #20 0x0000000000436da6 in readahead_get_line_skip (f=0x7fffeea2bf40, skip=64586429, bufsize=16154788) at Objects/fileobject.c:2311 #21 0x0000000000436da6 in readahead_get_line_skip (f=0x7fffeea2bf40, skip=51662598, bufsize=12923831) at Objects/fileobject.c:2311 #22 0x0000000000436da6 in readahead_get_line_skip (f=0x7fffeea2bf40, skip=41323533, bufsize=10339065) at Objects/fileobject.c:2311 #23 0x0000000000436da6 in readahead_get_line_skip (f=0x7fffeea2bf40, skip=33052281, bufsize=8271252) at Objects/fileobject.c:2311 #24 0x0000000000436da6 in readahead_get_line_skip (f=0x7fffeea2bf40, skip=26435279, bufsize=6617002) at Objects/fileobject.c:2311 #25 0x0000000000436da6 in readahead_get_line_skip (f=0x7fffeea2bf40, skip=21141677, bufsize=5293602) at Objects/fileobject.c:2311 #26 0x0000000000436da6 in readahead_get_line_skip (f=0x7fffeea2bf40, skip=16906795, bufsize=4234882) at Objects/fileobject.c:2311 #27 0x0000000000436da6 in readahead_get_line_skip (f=0x7fffeea2bf40, skip=13518889, bufsize=3387906) at Objects/fileobject.c:2311 #28 0x0000000000436da6 in readahead_get_line_skip (f=0x7fffeea2bf40, skip=10808564, bufsize=2710325) at Objects/fileobject.c:2311 #29 0x0000000000436da6 in readahead_get_line_skip (f=0x7fffeea2bf40, skip=8640304, bufsize=2168260) at Objects/fileobject.c:2311 #30 0x0000000000436da6 in readahead_get_line_skip (f=0x7fffeea2bf40, skip=6905696, bufsize=1734608) at Objects/fileobject.c:2311 #31 0x0000000000436da6 in readahead_get_line_skip (f=0x7fffeea2bf40, skip=5518009, bufsize=1387687) at Objects/fileobject.c:2311 #32 0x0000000000436da6 in readahead_get_line_skip (f=0x7fffeea2bf40, skip=4407859, bufsize=1110150) at Objects/fileobject.c:2311 #33 0x0000000000436da6 in readahead_get_line_skip (f=0x7fffeea2bf40, skip=3519739, bufsize=888120) at Objects/fileobject.c:2311 #34 0x0000000000436da6 in readahead_get_line_skip (f=0x7fffeea2bf40, skip=2809243, bufsize=710496) at Objects/fileobject.c:2311 #35 0x0000000000436da6 in readahead_get_line_skip (f=0x7fffeea2bf40, skip=2240846, bufsize=568397) at Objects/fileobject.c:2311 #36 0x0000000000436da6 in readahead_get_line_skip (f=0x7fffeea2bf40, skip=1786128, bufsize=454718) at Objects/fileobject.c:2311 #37 0x0000000000436da6 in readahead_get_line_skip (f=0x7fffeea2bf40, skip=1422353, bufsize=363775) at Objects/fileobject.c:2311 #38 0x0000000000436da6 in readahead_get_line_skip (f=0x7fffeea2bf40, skip=1131333, bufsize=291020) at Objects/fileobject.c:2311 #39 0x0000000000436da6 in readahead_get_line_skip (f=0x7fffeea2bf40, skip=898517, bufsize=232816) at Objects/fileobject.c:2311 #40 0x0000000000436da6 in readahead_get_line_skip (f=0x7fffeea2bf40, skip=712264, bufsize=186253) at Objects/fileobject.c:2311 #41 0x0000000000436da6 in readahead_get_line_skip (f=0x7fffeea2bf40, skip=563261, bufsize=149003) at Objects/fileobject.c:2311 #42 0x0000000000436da6 in readahead_get_line_skip (f=0x7fffeea2bf40, skip=444058, bufsize=119203) at Objects/fileobject.c:2311 #43 0x0000000000436da6 in readahead_get_line_skip (f=0x7fffeea2bf40, skip=348695, bufsize=95363) at Objects/fileobject.c:2311 #44 0x0000000000436da6 in readahead_get_line_skip (f=0x7fffeea2bf40, skip=272404, bufsize=76291) at Objects/fileobject.c:2311 #45 0x0000000000436da6 in readahead_get_line_skip (f=0x7fffeea2bf40, skip=211371, bufsize=61033) at Objects/fileobject.c:2311 #46 0x0000000000436da6 in readahead_get_line_skip (f=0x7fffeea2bf40, skip=162544, bufsize=48827) at Objects/fileobject.c:2311 #47 0x0000000000436da6 in readahead_get_line_skip (f=0x7fffeea2bf40, skip=123482, bufsize=39062) at Objects/fileobject.c:2311 #48 0x0000000000436da6 in readahead_get_line_skip (f=0x7fffeea2bf40, skip=92232, bufsize=31250) at Objects/fileobject.c:2311 #49 0x0000000000436da6 in readahead_get_line_skip (f=0x7fffeea2bf40, skip=67232, bufsize=25000) at Objects/fileobject.c:2311 #50 0x0000000000436da6 in readahead_get_line_skip (f=0x7fffeea2bf40, skip=47232, bufsize=20000) at Objects/fileobject.c:2311 #51 0x0000000000436da6 in readahead_get_line_skip (f=0x7fffeea2bf40, skip=31232, bufsize=16000) at Objects/fileobject.c:2311 #52 0x0000000000436da6 in readahead_get_line_skip (f=0x7fffeea2bf40, skip=18432, bufsize=12800) at Objects/fileobject.c:2311 #53 0x0000000000436da6 in readahead_get_line_skip (f=0x7fffeea2bf40, skip=8192, bufsize=10240) at Objects/fileobject.c:2311 #54 0x0000000000436da6 in readahead_get_line_skip (f=0x7fffeea2bf40, skip=0, bufsize=8192) at Objects/fileobject.c:2311 #55 0x0000000000436e52 in file_iternext (f=0x7fffeea2bf40) at Objects/fileobject.c:2335 --- ---------- _______________________________________ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue22526> _______________________________________ _______________________________________________ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com