[issue4533] 3.0 file.read dreadfully slow

[Antoine Pitrou]

Antoine Pitrou  added the comment:

This has been fixed as part of the big IO update in trunk. I assume
nobody really cares about making a separate patch for 2.6, please
re-open if you are interested!

--
resolution: accepted -> fixed
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue4533] 3.0 file.read dreadfully slow

[Antoine Pitrou]

Antoine Pitrou  added the comment:

Since it is solved for 3.x and only needs to be bacported to 2.x (where
the "io" module isn't the default), downgrading to critical.

--
nosy: +pitrou
priority: release blocker -> critical

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue4533] 3.0 file.read dreadfully slow

[Martin v. Löwis]

Changes by Martin v. Löwis :


--
priority: deferred blocker -> release blocker

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue4533] 3.0 file.read dreadfully slow

[Martin v. Löwis]

Changes by Martin v. Löwis <[EMAIL PROTECTED]>:


--
priority: release blocker -> deferred blocker

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue4533] 3.0 file.read dreadfully slow

[Christian Heimes]

Christian Heimes <[EMAIL PROTECTED]> added the comment:

The updated patch has been committed to 3.0 and 3.1. I'm going to
backport the patch to 2.x later.

--
resolution:  -> accepted
stage: patch review -> commit review
versions: +Python 2.6, Python 2.7 -Python 3.0, Python 3.1

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue4533] 3.0 file.read dreadfully slow

[Gregory P. Smith]

Gregory P. Smith <[EMAIL PROTECTED]> added the comment:

fileio_buffer2.patch looks good other than minor touchups:

Turn the XXX comment into:

/* NOTE: overflow impossible due to limits on BUFSIZ *

Also, 2 << 24 is 32MB yet your error message test says >= 64MB.  I think
you meant 1 << 26.

fix those and commit. :)

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue4533] 3.0 file.read dreadfully slow

[Christian Heimes]

Christian Heimes <[EMAIL PROTECTED]> added the comment:

The preprocessor doesn't handle power. 2 << 24 (64MB) sounds sufficient
for me.

Added file: http://bugs.python.org/file12228/fileio_buffer2.patch

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue4533] 3.0 file.read dreadfully slow

[Gregory P. Smith]

Gregory P. Smith <[EMAIL PROTECTED]> added the comment:

patch looks good to me.

nitpick comments: use += instead of = and + in:

newsize = newsize + newsize
 and
newsize = newsize + BIGCHUNK.

As for the XXX about overflow, so long as BUFSIZ is not defined to be an
insanely large number (it should never be) this will be fine.  add a
preprocessor test for that in.

#if (BUFSIZ >= 2**30)
#error "unreasonable BUFSIZ defined"
#endif

--
nosy: +gregory.p.smith

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue4533] 3.0 file.read dreadfully slow

[Christian Heimes]

Christian Heimes <[EMAIL PROTECTED]> added the comment:

The fileio_buffer.patch implements the same progressive buffer as Python
2.x' Object/fileobject.c.

--
keywords: +patch
stage: test needed -> patch review
Added file: http://bugs.python.org/file12227/fileio_buffer.patch

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue4533] 3.0 file.read dreadfully slow

[Christian Heimes]

Christian Heimes <[EMAIL PROTECTED]> added the comment:

The small buffer size in Modules/_fileio.c is one reason for the slowness.

$ dd if=/dev/zero of=zeros bs=1MB count=50
$ cat testread.py
open("zeros", "rb").read()
$ ./python -m cProfile testread.py
 40 function calls (39 primitive calls) in 4.246 CPU seconds

   Ordered by: standard name

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
10.0160.0164.2464.246 :1() 
10.0000.0000.0000.000 io.py:277(__new__)   
20.0000.0000.0000.000 io.py:355(flush) 
20.0000.0000.0000.000 io.py:364(close) 
20.0000.0000.0000.000 io.py:376(__del__)   
10.0000.0000.0000.000 io.py:413(_checkReadable)
10.0000.0000.0000.000 io.py:614(__init__)  
20.0000.0000.0000.000 io.py:618(close) 
10.0000.0000.0000.000 io.py:708(__init__)  
10.0000.0000.0000.000 io.py:733(flush) 
10.0000.0000.0000.000 io.py:736(close) 
10.0000.0000.0000.000 io.py:755(closed)
10.0000.0000.0000.000 io.py:82(open)   
10.0000.0000.0000.000 io.py:896(__init__)  
20.0000.0000.0000.000 io.py:905(_reset_read_buf)
10.0210.0214.2304.230 io.py:909(read)   
10.0000.0004.2094.209 io.py:920(_read_unlocked) 
10.0000.0000.0000.000 {built-in method
allocate_lock}
  2/10.0000.0004.2464.246 {built-in method exec}   
 
10.0000.0000.0000.000 {built-in method fstat}  
 
20.0000.0000.0000.000 {built-in method
isinstance}   
30.0000.0000.0000.000 {built-in method len}
 
10.0000.0000.0000.000 {method '__enter__' of
'_thread.lock' objects}
10.0000.0000.0000.000 {method 'append' of 'list'
objects}   
10.0000.0000.0000.000 {method 'disable' of
'_lsprof.Profiler' objects}
10.0000.0000.0000.000 {method 'fileno' of
'_FileIO' objects}  
10.0000.0000.0000.000 {method 'isatty' of
'_FileIO' objects}  
10.8250.8250.8250.825 {method 'join' of 'bytes'
objects}  
23.3841.6923.3841.692 {method 'read' of
'_FileIO' objects}
10.0000.0000.0000.000 {method 'readable' of
'_FileIO' objects} 

$ vi Modules/_fileio.c
-#define DEFAULT_BUFFER_SIZE (8*1024)
+#define DEFAULT_BUFFER_SIZE (80*1024)
$ ./python -m cProfile testread.py 
 40 function calls (39 primitive calls) in 1.273 CPU seconds   


   Ordered by: standard name

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
10.0190.0191.2731.273 :1()
10.0000.0000.0000.000 io.py:277(__new__)
20.0000.0000.0000.000 io.py:355(flush)
20.0000.0000.0000.000 io.py:364(close)
20.0000.0000.0000.000 io.py:376(__del__)
10.0000.0000.0000.000 io.py:413(_checkReadable)
10.0000.0000.0000.000 io.py:614(__init__)
20.0000.0000.0000.000 io.py:618(close)
10.0000.0000.0000.000 io.py:708(__init__)
10.0000.0000.0000.000 io.py:733(flush)
10.0000.0000.0000.000 io.py:736(close)
10.0000.0000.0000.000 io.py:755(closed)
10.0000.0000.0000.000 io.py:82(open)
10.0000.0000.0000.000 io.py:896(__init__)
20.0000.0000.0000.000 io.py:905(_reset_read_buf)
10.0160.0161.2541.254 io.py:909(read)
10.0000.0001.2381.238 io.py:920(_read_unlocked)
10.0000.0000.0000.000 {built-in method
allocate_lock}
  2/10.0000.0001.2731.273 {built-in method exec}
10.0000.0000.0000.000 {built-in method fstat}
20.0000.0000.0000.000 {built-in method isinstance}
30.0000.0000.0000.000 {built-in method len}
10.0000.0000.0000.000 {method '__enter__' of
'_thread.lock' objects}
10.0000.0000.0000.000 {method 'append' of 'list'
objects}
10.0000.0000.0000.000 {method 'disable' of
'_lsprof.Profiler' objects}
10.0000.0000.0000.000 {method 'fileno' of
'_FileIO' objects}
10.0000.0000.0000.000 {method 'isatty' of

[issue4533] 3.0 file.read dreadfully slow

[Christian Heimes]

Christian Heimes <[EMAIL PROTECTED]> added the comment:

This needs definitely some testing!

--
components: +Extension Modules
nosy: +christian.heimes
priority:  -> release blocker
stage:  -> test needed
versions: +Python 3.1

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue4533] 3.0 file.read dreadfully slow

[Terry J. Reedy]

New submission from Terry J. Reedy <[EMAIL PROTECTED]>:

C.l.p poster reported that 3.0 file.read is orders of magnitude slower
than with 2.5 (but confused issue with buffer = 0).  Jerry Hill reported

"Here's a quick comparison between 2.5 and
3.0 on a relatively small 17 meg file:

C:\>c:\Python30\python -m timeit -n 1
"open('C:\\work\\temp\\bppd_vsub.csv', 'rb').read()"
1 loops, best of 3: 36.8 sec per loop

C:\>c:\Python25\python -m timeit -n 1
"open('C:\\work\\temp\\bppd_vsub.csv', 'rb').read()"
1 loops, best of 3: 33 msec per loop

That's 3 orders of magnitude slower on python3.0!"

I verified this informally on WinXP by opening and then reading
Doc/Pythonxy.chm (about 4 megs) -- an eye blink versus 3 seconds,
repeated.  Even the open seemed slower but I did not time it.
>>> f=open('Doc/Python30.chm','rb')
>>> d=f.read()

--
components: Interpreter Core, Library (Lib)
messages: 76915
nosy: tjreedy
severity: normal
status: open
title: 3.0 file.read dreadfully slow
type: performance
versions: Python 3.0

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com