[issue8557] subprocess PATH semantics and portability

[R. David Murray]

R. David Murray  added the comment:

Well, it seems I was mistaken when I thought I knew how this worked :)
Checking the os.exec documentation linked from the subprocess page, I see that 
when an environment is supplied PATH is indeed checked in it.  The 
documentation for CreateProcess, however, indicates that PATH is ignored, and 
any extension must be supplied explicitly.

At the very least the docs should be updated to clarify that execvpe is used 
when an environment is passed on posix, and to link to the CreateProcess docs.  
A discussion of PATH could perhaps be put in a note or footnote (probably 
footnote, there are enough notes already in those docs!)

I'm not sure how one creates a good portability story out of these pieces.  It 
doesn't seem as though there is any way to harmonize the two, since we are 
dealing with the semantics of system calls over which we have no control.

For reference, here is (a?) link to CreateProcess docs that I found via Google:

http://msdn.microsoft.com/en-us/library/ms682425(VS.85).aspx

It doesn't look like the kind of link that one could trust to be stable, 
though, so I'm not sure if we should include it in the docs.

I'm adding Brian Curtin as nosy to see if he knows whether or not there are 
permalink-like links to the relevant Microsoft documentation that we could use.

--
nosy: +brian.curtin
stage:  -> needs patch
title: subprocess PATH semantics -> subprocess PATH semantics and portability
versions: +Python 2.7, Python 3.1, Python 3.2

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8557] subprocess PATH semantics and portability

[Brian Curtin]

Brian Curtin  added the comment:

You could take the "(VS8.5)" part out of the link which will give the latest 
version, which may not always be the relevant version (although I doubt this 
specific API would change).

That's about the best permalink-like feature you'll find, but overall, leaving 
the link as-is is pretty safe in my experience.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8557] subprocess PATH semantics and portability

[Dave Abrahams]

Dave Abrahams  added the comment:

@r.david.murray: did you try running my test?  I think it shows that we are 
pretty darned close to fully portable.  I believe we could fix Popen to make it 
fully portable pretty easily.  In fact, there may be a pure-python fix.  
Documenting the differences would also not be hard.  I would discourage you 
from relying *solely* on a description such as "uses execvpe on POSIX" to 
describe the semantics.  Aside from being a nonportable description, it doesn't 
help anyone who isn't familiar with the POSIX system calls.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8557] subprocess PATH semantics and portability

[R. David Murray]

R. David Murray  added the comment:

I didn't run the script.  I have now, but I'm not clear from its output what 
each test is actually doing, and don't really have the time to figure it out 
from the code right now.

I think it is probably more efficient to just ask you what your suggestion is 
for making things more portable?

As for the docs, the docs link to the os.exec python docs, which explain the 
PATH semantics.  Linking to the Microsoft documentation would equivalently 
explain the Windows semantics.  An explicit footnote discussing the differences 
in PATH behavior in the subprocess context would probably be helpful.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8557] subprocess PATH semantics and portability

[Dave Abrahams]

Dave Abrahams  added the comment:

I've uploaded a new probe.py that contains a win32 Popen wrapper that I think 
acts just like *nix's Popen w.r.t. PATH and environment (pass --fix to 
demonstrate).  I suggest using this or an equivalent wrapper for Win32, and 
documenting the fact that with shell=False, filename extensions need to be 
supplied explicitly on windows.

--
Added file: http://bugs.python.org/file17180/probe.py

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8557] subprocess PATH semantics and portability

[Dave Abrahams]

Changes by Dave Abrahams :


Removed file: http://bugs.python.org/file17142/probe.py

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8557] subprocess PATH semantics and portability

[Dave Abrahams]

Dave Abrahams  added the comment:

Not to appear impatient, but
It's a fairly tidy answer, I think :-)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8557] subprocess PATH semantics and portability

[R. David Murray]

R. David Murray  added the comment:

Sorry for my Windows ignorance, but if CreateProcess ignores the PATH, how does 
updating the PATH fix the problem?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8557] subprocess PATH semantics and portability

[Dave Abrahams]

Dave Abrahams  added the comment:

I'm probably as ignorant as you are of Windows issues. I just know what my 
experiments tell me: if you force the contents of any explicit 'env' argument 
into os.environ before calling Popen, you get the same behavior as on *nix.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8557] subprocess PATH semantics and portability

[R. David Murray]

R. David Murray  added the comment:

Well, it wouldn't be the first time the microsoft docs were wrong.

There are two questions here: (1) is this behavior consistent across all 
microsoft platforms we support?  (2) is this *change* in behavior of Popen 
acceptable?

For (1) we need a unit test added to the subprocess unit tests that can check 
this.

For (2)...well, I think it would be good for the behavior to be as consistent 
as practical, so I'd be in favor.  We need some second opinions, though, to 
make sure we aren't overlooking some negative consequence.  I'm also not sure 
that this qualifies as a bug fix, so it may only be possible to get it into 
3.2, assuming it is acceptable.

Note that I have not tested your program on Windows myself, I'm taking your 
word for it that it works ;)  I'll be more inclined to test things if the tests 
are in the form of unit tests, which should be much easier to understand than 
your test program.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8557] subprocess PATH semantics and portability

[Dave Abrahams]

Dave Abrahams  added the comment:

R. David Murray wrote:
> There are two questions here: (1) is this behavior consistent across all 
> microsoft platforms we support?  

I'll be honest: I don't know.

> (2) is this *change* in behavior of Popen acceptable?

I don't know that either.

> I'll be more inclined to
> test things if the tests are in the form of unit tests, which should
> be much easier to understand than your test program.

I guess no good deed goes unpunished ;-)

I also guess that whether you think unit tests will be easier to
understand depends on what kind of information you expect to glean
from the code.  My script was designed to probe for all
inconsistencies between ‘doze and POSIX behaviors, and it is more
revealing in that respect than a unit test would be.  The unit test
that would prompt the particular code change I'm suggesting would look
more like:

put directory X in the env argument's PATH (but not in os.environ)
attempt to launch X/some_executable as simply “some_executable”
assert that X/some_executable actually ran

I don't know what Popen's unit tests look like, and to be honest,
right now I just don't have any more time to pour into this particular
issue.  Even if it doesn't get fixed in Python I'm going to be using
my wrapper for uniformity.  I hope everything I've done so far is
useful to the community but if not, I still have what I need.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8557] subprocess PATH semantics and portability

[R. David Murray]

R. David Murray  added the comment:

Fair enough.  Thank you for your detective work, and hopefully someone will be 
interested enough to pick this up again later.

--
status: open -> languishing

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8557] subprocess PATH semantics and portability

[Dave Abrahams]

Dave Abrahams  added the comment:

New data point: in some contexts on Windows (not sure of the exact cause but I 
was dealing with multiple drives), even this workaround isn't enough.  I ended 
up having to do something like this (i.e. manually search the path) on win32:


def full_executable_path(invoked, environ):

if os.path.splitext(invoked)[1]:
return invoked

explicit_dir = os.path.dirname(invoked)

if explicit_dir:
path = [ explicit_dir ]
else:
path = environ.get('PATH').split(os.path.pathsep)

extensions = environ.get(
'PATHEXT',
# Use *something* in case the environment variable is
# empty.  These come from my machine's defaults
'.COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.PSC1'
).split(os.path.pathsep)

for dir in path:
for ext in extensions:
full_path = os.path.join(dir, invoked+ext)
if os.path.exists( full_path ):
return full_path

return invoked # Not found; invoking it will likely fail

class Popen(subprocess.Popen):
def __init__(
self, args, bufsize=0, executable=None,
stdin=None, stdout=None, stderr=None,
preexec_fn=None, close_fds=False, shell=False, 
cwd=None, env=None, 
*args_, **kw):

if executable is None and not shell:
executable = full_executable_path(args[0], env or os.environ)

super(Popen,self).__init__(
args, bufsize, executable, stdin, stdout, stderr, 
preexec_fn, close_fds, shell, cwd, env, *args_, **kw)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com