[issue4565] io write() performance very slow

[Antoine Pitrou]

Antoine Pitrou  added the comment:

We can't solve this for 3.0.1, downgrading to critical.

--
priority: release blocker -> critical

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue4565] io write() performance very slow

[Martin v. Löwis]

Changes by Martin v. Löwis :


--
priority: deferred blocker -> release blocker

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue4565] io write() performance very slow

[Martin v. Löwis]

Changes by Martin v. Löwis <[EMAIL PROTECTED]>:


--
priority: release blocker -> deferred blocker

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue4565] io write() performance very slow

[Winfried Plappert]

Changes by Winfried Plappert <[EMAIL PROTECTED]>:


--
nosy: +wplappert

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue4565] io write() performance very slow

[Barry A. Warsaw]

Changes by Barry A. Warsaw <[EMAIL PROTECTED]>:


--
priority: high -> release blocker

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue4565] io write() performance very slow

[Amaury Forgeot d'Arc]

Amaury Forgeot d'Arc <[EMAIL PROTECTED]> added the comment:

> I'm a bit surprised by the adopted approach. 
> It seems you are merely translating the Python code into C. 
> I think the proper approach for the buffered IO classes would be 
> to use a fixed-size buffer which never gets reallocated.

You are certainly right, but the code io.py is already difficult to understand 
and 
maintain; the corresponding C code adds one level of complexity;
had I changed the buffering strategy at the same time, it would have been 
impossible 
to ensure a correct implementation.
Now that my C implementation of the Buffered classes seems correct (all tests 
pass, 
except a few about destructors) we could try alternative approaches.

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue4565] io write() performance very slow

[Christian Heimes]

Christian Heimes <[EMAIL PROTECTED]> added the comment:

For more bug reports see #4533 and #4561.

I suggest we close this bug report as duplicate and keep the discussion
in #4561.

--
nosy: +christian.heimes

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue4565] io write() performance very slow

[Antoine Pitrou]

Antoine Pitrou <[EMAIL PROTECTED]> added the comment:

Hi Amaury,

> There is a project to rewrite it in C

Thanks for publicizing this. I'm a bit surprised by the adopted
approach. It seems you are merely translating the Python code into C. I
think the proper approach for the buffered IO classes would be to use a
fixed-size buffer which never gets reallocated.

If you look at bufferedwriter2.patch in #3476, I had rewritten
BufferedWriter using a fixed-size buffer (although not for performance
reasons), I think it would be a good starting point for a C implementation.

--
nosy: +pitrou

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue4565] io write() performance very slow

[Giampaolo Rodola']

Changes by Giampaolo Rodola' <[EMAIL PROTECTED]>:


--
nosy: +giampaolo.rodola

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue4565] io write() performance very slow

[Istvan Albert]

Istvan Albert <[EMAIL PROTECTED]> added the comment:

Well I would strongly dispute that anyone other than the developers
expected this. The release documentation states:

"The net result of the 3.0 generalizations is that Python 3.0 runs the
pystone benchmark around 10% slower than Python 2.5."

There is no indication of an order of magnitudes in read/write slowdown.
I believe that this issue is extremely serious! IO is an essential part
of a program, and today we live in the world of gigabytes of data. I am
reading reports of even more severe io slowdowns than what I saw:

http://bugs.python.org/issue4561

Java has had a hard time getting rid of the "it is very slow" stigma
even after getting a JIT compiler, so there is a danger there for a
lasting negative impression.

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue4565] io write() performance very slow

[Amaury Forgeot d'Arc]

Amaury Forgeot d'Arc <[EMAIL PROTECTED]> added the comment:

This is expected: the I/O stack has been completely rewritten... in 
almost pure-python code.

There is a project to rewrite it in C. It started at
http://svn.python.org/view/sandbox/trunk/io-c/

--
assignee:  -> amaury.forgeotdarc
nosy: +amaury.forgeotdarc
priority:  -> high

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue4565] io write() performance very slow

[Istvan Albert]

New submission from Istvan Albert <[EMAIL PROTECTED]>:

The write performance into text files is substantially slower (5x-8x)
than that of python 2.5. This makes python 3.0 unsuited to any
application that needs to write larger amounts of data.

test code follows ---

import time

lo, hi, step = 10**5, 10**6, 10**5

# writes increasingly more lines to a file
for N in range(lo, hi, step):
fp = open('foodata.txt', 'wt')
start = time.time()
for i in range( N ):
fp.write( '%s\n' % i)
fp.close()
stop = time.time()
print ( "%s\t%s" % (N, stop-start) )

--
components: Interpreter Core
messages: 77132
nosy: ialbert
severity: normal
status: open
title: io write() performance very slow
type: performance
versions: Python 3.0

___
Python tracker <[EMAIL PROTECTED]>

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com