[issue4194] Miserable subprocess.Popen performance

2008-11-02 Thread Brad Hall 

Changes by Brad Hall [EMAIL PROTECTED]:


--
nosy: +bgh

___
Python tracker [EMAIL PROTECTED]
http://bugs.python.org/issue4194
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue4194] Miserable subprocess.Popen performance

2008-10-25 Thread Winfried Plappert 

Winfried Plappert [EMAIL PROTECTED] added the comment:

Hi is the dramatic difference on Solaris-10 / Python2.6:

I dtraced the popentest.py and counted syscalls: 

with os_popen: read =243
with process:Popen read = 589018

That explains a lot! 

The rest of the system calls are similir in count and appearance.
I have not looked into the C-source (yet).

___
Python tracker [EMAIL PROTECTED]
http://bugs.python.org/issue4194
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue4194] Miserable subprocess.Popen performance

2008-10-25 Thread Winfried Plappert 

Winfried Plappert [EMAIL PROTECTED] added the comment:

The created testfile size is 588890 bytes, which implies that
subprocess.Popen reads the file in completely unbuffered mode, one byte
at a time. If I modify the popentest.py programme by specifying a
bufsize of 1_000_000, the execution time drops quite a bit:
time with os.popen 0.069 sec
time with subprocess.Popen 0.118 sec
This Solaris 9/Python 2.6.

I think I have got it: Lib/subprocess.py should have a default bufsize
of -1, not 0! I tested it by modifying the subprocess.Popen call and
here is the result:
time with os.popen 0.086 sec
time with subprocess.Popen 0.080 sec

See also Modules/posixmodule.c.

___
Python tracker [EMAIL PROTECTED]
http://bugs.python.org/issue4194
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue4194] Miserable subprocess.Popen performance

2008-10-25 Thread Skip Montanaro 

Skip Montanaro [EMAIL PROTECTED] added the comment:

Using a nonzero bufsize parameter makes all the difference in the world:

  Using the default (bufsize=0 == unbuffered):

% python popentest.py
time with os.popen :  0.035032
time with subprocess.Popen :  1.496455

  Creating the Popen object with bufsize=8192:

% python popentest.py
time with os.popen :  0.034642
time with subprocess.Popen :  0.041376

  Creating the Popen object with bufsize=1 (== line buffered):

% python popentest.py
time with os.popen :  0.034658
time with subprocess.Popen :  0.04211

Maybe the default for that parameter shouldn't be zero?

Skip

___
Python tracker [EMAIL PROTECTED]
http://bugs.python.org/issue4194
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue4194] Miserable subprocess.Popen performance

2008-10-25 Thread Skip Montanaro 

Skip Montanaro [EMAIL PROTECTED] added the comment:

 Using a nonzero bufsize parameter makes all the difference in the
 world:
...

In fact, looking at posix_popen in posixmodule.c it appears the default
value for bufsize there is -1, implying that I/O is fully buffered.

Even if the default bufsize value for subprocess.Popen is not changed its
documentation should be corrected to reflect the different default values
for bufsize between os.popen and subprocess.Popen:

Replacing os.popen*
---
pipe = os.popen(cmd, mode='r', [bufsize])
==
pipe = Popen(cmd, shell=True, bufsize=bufsize, stdout=PIPE).stdout

If you use the default bufsize with os.popen you must set bufsize to -1
when creating a Popen object.  Otherwise your I/O will be unbuffered and
you will be disappointed at the I/O performance.

--
assignee:  - georg.brandl
components: +Documentation, Library (Lib)
nosy: +georg.brandl
versions: +Python 2.7, Python 3.0

___
Python tracker [EMAIL PROTECTED]
http://bugs.python.org/issue4194
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue4194] Miserable subprocess.Popen performance

2008-10-25 Thread Giampaolo Rodola' 

Changes by Giampaolo Rodola' [EMAIL PROTECTED]:


--
nosy: +giampaolo.rodola

___
Python tracker [EMAIL PROTECTED]
http://bugs.python.org/issue4194
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue4194] Miserable subprocess.Popen performance

2008-10-25 Thread Skip Montanaro 

Skip Montanaro [EMAIL PROTECTED] added the comment:

I've been thinking about it, and I think even though it would be a slight 
change to the API, I agree with Winfried that the default value for 
bufsize should be -1, not 0.  In my own use of os.popen and friends, 
almost all the time I use them to process the output of a traditional Unix 
pipeline, which means the output will be plain text, line buffered.  In 
those cases it makes absolutely no sense to read from the pipe one 
character at a time.

S

___
Python tracker [EMAIL PROTECTED]
http://bugs.python.org/issue4194
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue4194] Miserable subprocess.Popen performance

2008-10-25 Thread Winfried Plappert 

Winfried Plappert [EMAIL PROTECTED] added the comment:

On the other hand, we will silently break all those applications which
are out there relying on the fact that a pipe is an unbuffered device.
You might consider it for Python 3.0, but I don't know if it is a good
idea for Python 2.x. The documentation of subprocess.Popen clear states
that the default is unbuffered, bufsize=0. I quote from
http://docs.python.org/library/subprocess.html:

bufsize, if given, has the same meaning as the corresponding argument to
the built-in open() function: 0 means unbuffered, 1 means line buffered,
any other positive value means use a buffer of (approximately) that
size. A negative bufsize means to use the system default, which usually
means fully buffered. The default value for bufsize is 0 (unbuffered).

___
Python tracker [EMAIL PROTECTED]
http://bugs.python.org/issue4194
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue4194] Miserable subprocess.Popen performance

2008-10-24 Thread Skip Montanaro 

New submission from Skip Montanaro [EMAIL PROTECTED]:

I noticed a colleague at work today checked in a change to his code to
switch back from subprocess.Popen to os.popen.  I asked him about it and he
reported that subprocess.Popen was about 10x slower than os.popen.  I asked
him for a simple test case, which is attached.  Here are my results with
Python 2.4 through 2.7 (aka CVS HEAD):

tmp% python2.4 popentest.py 
time with os.popen :  0.09
time with subprocess.Popen :  2.27
tmp% python2.5 popentest.py 
time with os.popen :  0.03
time with subprocess.Popen :  1.52
tmp% python2.6 popentest.py 
time with os.popen :  0.038824
time with subprocess.Popen :  1.517056
tmp% python2.7 popentest.py 
time with os.popen :  0.033746
time with subprocess.Popen :  1.512331

These times are on my Mac laptop, all writing to the local disk.  It seems
there was a bit of improvement in the 2.5 release but that it is still
miserably slow when compared with os.popen.

His original test used time.clock() as the timer.  I changed to time.time()
but got essentially the same result.

Skip

--
files: popentest.py
messages: 75173
nosy: skip.montanaro
severity: normal
status: open
title: Miserable subprocess.Popen performance
Added file: http://bugs.python.org/file11871/popentest.py

___
Python tracker [EMAIL PROTECTED]
http://bugs.python.org/issue4194
___import os, subprocess, time

os_filename = tmp_os
sub_filename = tmp_subprocess

def write_file(filename): # write out a reasonably big file
   fh = file(filename, 'w')
   for i in xrange(10):
   print fh, i
   fh.close()

write_file(os_filename)
write_file(sub_filename)

start = time.time()
read_handle = os.popen('cat %s' % os_filename)
a = [i for i in read_handle]
print time with os.popen : , time.time() - start

start = time.time()
read_handle = subprocess.Popen(['cat', sub_filename], 
stdout=subprocess.PIPE).stdout
a = [i for i in read_handle]
print time with subprocess.Popen : , time.time() - start

os.unlink(os_filename)
os.unlink(sub_filename)
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue4194] Miserable subprocess.Popen performance

2008-10-24 Thread Sameer 

Changes by Sameer [EMAIL PROTECTED]:


--
nosy: +sameerd

___
Python tracker [EMAIL PROTECTED]
http://bugs.python.org/issue4194
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue4194] Miserable subprocess.Popen performance

2008-10-24 Thread Winfried Plappert 

Winfried Plappert [EMAIL PROTECTED] added the comment:

Hi Skip,

I find different measurements om Windows/XP: I copied the script and ran
it under Python 2.5.2 and Python 2.6 (as downloaded from
http://python.org/ftp/python/2.6/python-2.6.msi):

[EMAIL PROTECTED] /cygdrive/e/tmp $ python ./timing.py # 2.5.2
time with os.popen :  0.29631335
time with subprocess.Popen :  0.20368665

in other words: identical or subprocess.Popen even faster.

and it is getting even better with Python2.6:

[EMAIL PROTECTED] /cygdrive/e/tmp $ python26 timing.py # 2.6
time with os.popen :  0.26632425
time with subprocess.Popen :  0.1873624

So could that be a Mac OS issue?

--
nosy: +wplappert

___
Python tracker [EMAIL PROTECTED]
http://bugs.python.org/issue4194
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue4194] Miserable subprocess.Popen performance

2008-10-24 Thread Sameer 

Sameer [EMAIL PROTECTED] added the comment:

The subprocess module does different things depending on whether the
systems are Windows or Posix. This could explain the bad performance on
Solaris and the Mac and the good performance on Windows. 

If this is correct, then we should see poor performance on Linux also.
Also, I don't have a box to test that out.

___
Python tracker [EMAIL PROTECTED]
http://bugs.python.org/issue4194
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue4194] Miserable subprocess.Popen performance

2008-10-24 Thread Skip Montanaro 

Skip Montanaro [EMAIL PROTECTED] added the comment:

Good suggestion Sameer.  I tried it out with Python 2.5 on a Linux host 
here and saw essentially identical results for the two alternatives (~ 
0.08s).

S

___
Python tracker [EMAIL PROTECTED]
http://bugs.python.org/issue4194
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue4194] Miserable subprocess.Popen performance

2008-10-24 Thread David W. Lambert 

David W. Lambert [EMAIL PROTECTED] added the comment:

cygwin Python 2.5.1  (similar)
time with os.popen :  0.43818802
time with subprocess.Popen :  0.36161035

linux python 2.4.2 (similar)
time with os.popen :  0.0344939231873
time with subprocess.Popen :  0.0354421138763

linux python3K (both awful)
time with os.popen :  0.715471029282
time with subprocess.Popen :  1.58627915382

--
nosy: +LambertDW

___
Python tracker [EMAIL PROTECTED]
http://bugs.python.org/issue4194
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue4194] Miserable subprocess.Popen performance

2008-10-24 Thread Skip Montanaro 

Skip Montanaro [EMAIL PROTECTED] added the comment:

I don't expect Python3 to be all that great io performance-wise yet.  
Still, for me on the Mac os.popen beats subprocess.Popen pretty handily:

  % python3.0 popentest.py
  time with os.popen :  0.874988
  time with subprocess.Popen :  2.112976

___
Python tracker [EMAIL PROTECTED]
http://bugs.python.org/issue4194
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue4194] Miserable subprocess.Popen performance

2008-10-24 Thread Winfried Plappert 

Winfried Plappert [EMAIL PROTECTED] added the comment:

Here are my figures from a different processor on Linux (Ubuntu):

[EMAIL PROTECTED]:~/python$ python2.5 popentest.py 
time with os.popen :  0.0478880405426
time with subprocess.Popen :  0.0596849918365
[EMAIL PROTECTED]:~/python$ python2.6 popentest.py 
time with os.popen :  0.0481159687042
time with subprocess.Popen :  0.0592260360718
uname -a: ... 2.6.24-19-generic #1 SMP Wed Aug 20 22:56:21 UTC 2008 i686
GNU/Linux

___
Python tracker [EMAIL PROTECTED]
http://bugs.python.org/issue4194
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue4194] Miserable subprocess.Popen performance

2008-10-24 Thread Winfried Plappert 

Winfried Plappert [EMAIL PROTECTED] added the comment:

Yes, I can confirm that the performance is lousy on Solaris.

Solaris-9/Python 2.5.1:
time with os.popen :  0.124045133591
time with subprocess.Popen :  1.60335588455

Solaris-9/Python 2.6:
time with os.popen :  0.115752220154
time with subprocess.Popen :  1.61768198013

Solaris-10/Python 2.4.4:
time with os.popen :  0.124130010605
time with subprocess.Popen :  1.45624995232

Solaris-10/Python 2.6:
time with os.popen :  0.113790988922
time with subprocess.Popen :  1.42739701271

All machines are idendital in processor speed: v240.

___
Python tracker [EMAIL PROTECTED]
http://bugs.python.org/issue4194
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue4194] Miserable subprocess.Popen performance

2008-10-24 Thread Winfried Plappert 

Changes by Winfried Plappert [EMAIL PROTECTED]:


--
type:  - performance
versions: +Python 2.4, Python 2.5, Python 2.6

___
Python tracker [EMAIL PROTECTED]
http://bugs.python.org/issue4194
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com