[issue4194] Miserable subprocess.Popen performance

Changes by Brad Hall [EMAIL PROTECTED]: -- nosy: +bgh ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue4194 ___ ___ Python-bugs-list mailing list

[issue4194] Miserable subprocess.Popen performance

Winfried Plappert [EMAIL PROTECTED] added the comment: Hi is the dramatic difference on Solaris-10 / Python2.6: I dtraced the popentest.py and counted syscalls: with os_popen: read =243 with process:Popen read = 589018 That explains a lot! The rest of the system calls are similir

[issue4194] Miserable subprocess.Popen performance

Winfried Plappert [EMAIL PROTECTED] added the comment: The created testfile size is 588890 bytes, which implies that subprocess.Popen reads the file in completely unbuffered mode, one byte at a time. If I modify the popentest.py programme by specifying a bufsize of 1_000_000, the execution time

[issue4194] Miserable subprocess.Popen performance

Skip Montanaro [EMAIL PROTECTED] added the comment: Using a nonzero bufsize parameter makes all the difference in the world: Using the default (bufsize=0 == unbuffered): % python popentest.py time with os.popen : 0.035032 time with subprocess.Popen : 1.496455 Creating the

[issue4194] Miserable subprocess.Popen performance

Skip Montanaro [EMAIL PROTECTED] added the comment: Using a nonzero bufsize parameter makes all the difference in the world: ... In fact, looking at posix_popen in posixmodule.c it appears the default value for bufsize there is -1, implying that I/O is fully buffered. Even if the

[issue4194] Miserable subprocess.Popen performance

Changes by Giampaolo Rodola' [EMAIL PROTECTED]: -- nosy: +giampaolo.rodola ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue4194 ___ ___ Python-bugs-list

[issue4194] Miserable subprocess.Popen performance

Skip Montanaro [EMAIL PROTECTED] added the comment: I've been thinking about it, and I think even though it would be a slight change to the API, I agree with Winfried that the default value for bufsize should be -1, not 0. In my own use of os.popen and friends, almost all the time I use them

[issue4194] Miserable subprocess.Popen performance

Winfried Plappert [EMAIL PROTECTED] added the comment: On the other hand, we will silently break all those applications which are out there relying on the fact that a pipe is an unbuffered device. You might consider it for Python 3.0, but I don't know if it is a good idea for Python 2.x. The

[issue4194] Miserable subprocess.Popen performance

New submission from Skip Montanaro [EMAIL PROTECTED]: I noticed a colleague at work today checked in a change to his code to switch back from subprocess.Popen to os.popen. I asked him about it and he reported that subprocess.Popen was about 10x slower than os.popen. I asked him for a simple

[issue4194] Miserable subprocess.Popen performance

Changes by Sameer [EMAIL PROTECTED]: -- nosy: +sameerd ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue4194 ___ ___ Python-bugs-list mailing list

[issue4194] Miserable subprocess.Popen performance

Winfried Plappert [EMAIL PROTECTED] added the comment: Hi Skip, I find different measurements om Windows/XP: I copied the script and ran it under Python 2.5.2 and Python 2.6 (as downloaded from http://python.org/ftp/python/2.6/python-2.6.msi): [EMAIL PROTECTED] /cygdrive/e/tmp $ python

[issue4194] Miserable subprocess.Popen performance

Sameer [EMAIL PROTECTED] added the comment: The subprocess module does different things depending on whether the systems are Windows or Posix. This could explain the bad performance on Solaris and the Mac and the good performance on Windows. If this is correct, then we should see poor

[issue4194] Miserable subprocess.Popen performance

Skip Montanaro [EMAIL PROTECTED] added the comment: Good suggestion Sameer. I tried it out with Python 2.5 on a Linux host here and saw essentially identical results for the two alternatives (~ 0.08s). S ___ Python tracker [EMAIL PROTECTED]

[issue4194] Miserable subprocess.Popen performance

David W. Lambert [EMAIL PROTECTED] added the comment: cygwin Python 2.5.1 (similar) time with os.popen : 0.43818802 time with subprocess.Popen : 0.36161035 linux python 2.4.2 (similar) time with os.popen : 0.0344939231873 time with subprocess.Popen : 0.0354421138763

[issue4194] Miserable subprocess.Popen performance

Skip Montanaro [EMAIL PROTECTED] added the comment: I don't expect Python3 to be all that great io performance-wise yet. Still, for me on the Mac os.popen beats subprocess.Popen pretty handily: % python3.0 popentest.py time with os.popen : 0.874988 time with subprocess.Popen :

[issue4194] Miserable subprocess.Popen performance

Winfried Plappert [EMAIL PROTECTED] added the comment: Here are my figures from a different processor on Linux (Ubuntu): [EMAIL PROTECTED]:~/python$ python2.5 popentest.py time with os.popen : 0.0478880405426 time with subprocess.Popen : 0.0596849918365 [EMAIL PROTECTED]:~/python$ python2.6

[issue4194] Miserable subprocess.Popen performance

Winfried Plappert [EMAIL PROTECTED] added the comment: Yes, I can confirm that the performance is lousy on Solaris. Solaris-9/Python 2.5.1: time with os.popen : 0.124045133591 time with subprocess.Popen : 1.60335588455 Solaris-9/Python 2.6: time with os.popen : 0.115752220154 time with

[issue4194] Miserable subprocess.Popen performance

Changes by Winfried Plappert [EMAIL PROTECTED]: -- type: - performance versions: +Python 2.4, Python 2.5, Python 2.6 ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue4194 ___