STINNER Victor added the comment:
In Python 3.3, open_noinherit_ctypes() can be written:
def opener_noinherit(filename, flags):
return os.open(filename, flags | os.O_NOINHERIT)
f = open(filename, opener=opener_noinherit)
Example on Linux with O_CLOEXEC:
$ python3
Python 3.3.0 (default,
Bernt Røskar Brenna added the comment:
I just tested on 3.4a0.
Observed the following changes:
- subprocess_devnull now NEVER fails.
- subprocess_redirfile does not fail as often as before, but still fails.
I changed the number of tasks to 20 and increased max_workers to 5 to get
Bernt Røskar Brenna added the comment:
Increasing max_workers to 5 and running 20 tasks highlighs the 3.3-3.4
difference (code in attached file testscript4.py):
Version: 3.4.0a4 (v3.4.0a4:e245b0d7209b, Oct 20 2013, 19:23:45) [MSC v.1600 32 b
it (Intel)]
Platform: Windows-XP-5.1.2600-SP3
Tasks:
Changes by Bernt Røskar Brenna bernt.bre...@gmail.com:
--
versions: +Python 3.4
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue19575
___
___
Changes by Bernt Røskar Brenna bernt.bre...@gmail.com:
--
components: +IO
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue19575
___
___
STINNER Victor added the comment:
Ok, this issue is the corner case of the PEP 446:
http://www.python.org/dev/peps/pep-0446/#only-inherit-some-handles-on-windows
The PEP explicitly does nothing for this case. It can change in the future.
Until the point is fixed, you have to use a lock around
Bernt Røskar Brenna added the comment:
I tested, and locking around the subprocess.Popen call indeed works on Python
3.4.
Thanks!
Do you have any tips on how to accomplish the same thing on Python 3.3 (locking
around Popen did not make any difference on 3.3)?
--
Bernt Røskar Brenna added the comment:
Never mind, I figured it:
On Python 3.3, the combination of locking around Popen and opening the file
that I redirect to using the code below works (code from scons):
def open_noinherit(*args, **kwargs):
fp = open(*args, **kwargs)
Bernt Røskar Brenna added the comment:
And here's a function that does not require pywin32:
def open_noinherit_ctypes(*args, **kwargs):
HANDLE_FLAG_INHERIT = 1
import msvcrt
from ctypes import windll, WinError
fp = open(*args, **kwargs)
if not
Charles-François Natali added the comment:
R. David Murray added the comment:
neologix noted that *when redirection is used* the way that *all* windows
file handles are inherited changes.
That's true (but that was from Richard actually).
--
___
STINNER Victor added the comment:
I think it's simply due to file descriptor inheritance
File handles are now non-inheritable by default in Python 3.4 (PEP 446). Does
it change anything?
--
nosy: +haypo
___
Python tracker rep...@bugs.python.org
Bernt Røskar Brenna added the comment:
Another script, another test case.
Four different tasks are run:
subprocess_redirfile: Popen(stdout=file)
subprocess_devnull: Popen(stdout=DEVNULL)
subprocess_noredirect: Popen()
nosubprocess: No Popen() call
Judging from the output it looks as if it is
Bernt Røskar Brenna added the comment:
Similar result on Windows Server 2008:
Platform: Windows-2008ServerR2-6.1.7600
task_type #threads result
subprocess_redirfile 2 9 errors
subprocess_redirfile 1 OK
subprocess_devnull2 9 errors
R. David Murray added the comment:
Unfortunately I currently lack a windows environment on which to test this.
I've added some people to nosy who might be able to help out with this.
--
nosy: +gps, tim.golden
___
Python tracker
Charles-François Natali added the comment:
I think it's simply due to file descriptor inheritance (files being
inherited by other subprocess instance): since Windows can't remove open
files, kaboom. It doesn't have anything to do with threads.
--
nosy: +neologix
R. David Murray added the comment:
That was my initial thought, too, but the subprocess context manager waits for
the subprocess to end, so those file descriptors should be closed by the time
the deletion attempt happens, shouldn't they?
--
___
Richard Oudkerk added the comment:
Note that on Windows if you redirect the standard streams then *all*
inheritable handles are inherited by the child process.
Presumably the handle for f_w file object (and/or a duplicate of it) created in
one thread is accidentally leaked to the other child
Bernt Røskar Brenna added the comment:
@neologix: How can it not have anything to do with threads?
- It always works with max_workers == 1
- When max_workers == 2, shutil.rmtree sometimes fails
--
___
Python tracker rep...@bugs.python.org
Changes by Bernt Røskar Brenna bernt.bre...@gmail.com:
--
nosy: +astrand
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue19575
___
___
Charles-François Natali added the comment:
Bernt Røskar Brenna added the comment:
@neologix: How can it not have anything to do with threads?
- It always works with max_workers == 1
- When max_workers == 2, shutil.rmtree sometimes fails
It has nothing to do with threads.
You could
Bernt Røskar Brenna added the comment:
@neologix: But how do you explain this:
subprocess_devnull2 9 errors
subprocess_devnull1 OK
subprocess_devnull creates a file, then starts a subprocess (that redirects to
DEVNULL, does not use the file), then tries to
R. David Murray added the comment:
neologix noted that *when redirection is used* the way that *all* windows file
handles are inherited changes. But that's about the end of *my* understanding
of the issue :)
--
___
Python tracker
New submission from Bernt Røskar Brenna:
Running the following task using concurrent.futures.ThreadPoolExecutor works
with max_workers == 1 and fails when max_workers 1 :
def task():
dirname = tempfile.mkdtemp()
f_w = open(os.path.join(dirname, stdout.txt), w)
f_r =
Bernt Røskar Brenna added the comment:
Simpler task method that still reproduces the problem:
def task():
dirname = tempfile.mkdtemp()
f_w = open(os.path.join(dirname, stdout.txt), w)
e_w = open(os.path.join(dirname, stderr.txt), w)
with subprocess.Popen(dir, shell=True,
R. David Murray added the comment:
It doesn't look like you are waiting for the subprocess to complete (and
therefore close the files) before doing the rmtree. What happens if you do
that?
--
nosy: +r.david.murray
___
Python tracker
R. David Murray added the comment:
Nevermind, I forgot that the context manager also does the wait.
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue19575
___
R. David Murray added the comment:
Most likely this is a case of a virus scanner or file indexer having the file
open when rmtree runs. See issue 7443, which was about solving this for our
test suite. My last message there asks about exposing the workaround in
shutil, but no one appears to
Bernt Røskar Brenna added the comment:
No indexing and no virus scanner running.
Note that it never fails if running in a single thread. IMO, this indicates
that external processes interfering is not the problem.
--
___
Python tracker
Bernt Røskar Brenna added the comment:
Here's an improved repro script.
I believe it demonstrates that it is the combination of subprocess.Popen and
threading that causes the problem.
Here's the output from my Windows XP VM:
***
c:\... c:\Python33\python.exe repro_improved.py
29 matches
Mail list logo
