[issue37380] subprocess.Popen._cleanup() "The handle is invalid" error when some old process is gone

Steve Dower added the comment: > import subprocess > subprocess._cleanup = lambda: None -- ___ Python tracker ___ ___

[issue37380] subprocess.Popen._cleanup() "The handle is invalid" error when some old process is gone

Sylvain Corlay added the comment: Yes, I understand that. I was wondering if there was a workaround that we could use to not drop 3.6 support right away! On Fri, May 29, 2020, 19:29 STINNER Victor wrote: > > STINNER Victor added the comment: > > Sadly, 3.6 no longer get bug fixes, only

[issue37380] subprocess.Popen._cleanup() "The handle is invalid" error when some old process is gone

STINNER Victor added the comment: Sadly, 3.6 no longer get bug fixes, only security fixes: https://devguide.python.org/#status-of-python-branches -- ___ Python tracker ___

[issue37380] subprocess.Popen._cleanup() "The handle is invalid" error when some old process is gone

Sylvain Corlay added the comment: Hello, Is there a means to work around that bug for Python 3.6 ? It seems that the fix was only backported to 3.7, and we were not planning on dropping Python 3.6 support quite yet in our project. -- nosy: +sylvain.corlay

[issue37380] subprocess.Popen._cleanup() "The handle is invalid" error when some old process is gone

Eryk Sun added the comment: > When a Popen instance is finalized by the garbage collector, the > internal handle is also finalized and closed despite the instance > being put on the active list. This results in _cleanup throwing > because the handle can no longer be used. Thanks. I hadn't

[issue37380] subprocess.Popen._cleanup() "The handle is invalid" error when some old process is gone

STINNER Victor added the comment: Steve Dower: "I triggered automatic backports to see how they go. Doesn't necessarily mean we'll merge them without looking more closely, but may as well check out what our automated tools suggest." Thanks, I merged the backports. --

[issue37380] subprocess.Popen._cleanup() "The handle is invalid" error when some old process is gone

STINNER Victor added the comment: Thanks Ruslan Kuprieiev for the bug report and the fix. Thanks Eryk Sun (as usual!) for the great analysis of this issue. Ok, the bug should now be fixed in 3.7, 3.8 and master branches. I chose to not fix Python 2.7: see my previous comment. See bpo-37410

[issue37380] subprocess.Popen._cleanup() "The handle is invalid" error when some old process is gone

STINNER Victor added the comment: New changeset 1e2707d7e82aedf73c59772bc7aa228286323c3c by Victor Stinner (Miss Islington (bot)) in branch '3.7': bpo-37380: subprocess: don't use _active on win (GH-14360) (GH-15706)

[issue37380] subprocess.Popen._cleanup() "The handle is invalid" error when some old process is gone

STINNER Victor added the comment: New changeset 4d1abedce9422473af2ac78047e55cde73208208 by Victor Stinner (Miss Islington (bot)) in branch '3.8': bpo-37380: subprocess: don't use _active on win (GH-14360) (GH-15707)

[issue37380] subprocess.Popen._cleanup() "The handle is invalid" error when some old process is gone

STINNER Victor added the comment: > When a Popen instance is finalized by the garbage collector, the internal > handle is also finalized and closed despite the instance being put on the > active list. This results in _cleanup throwing because the handle can no > longer be used. Right,

[issue37380] subprocess.Popen._cleanup() "The handle is invalid" error when some old process is gone

Steve Dower added the comment: I triggered automatic backports to see how they go. Doesn't necessarily mean we'll merge them without looking more closely, but may as well check out what our automated tools suggest. -- ___ Python tracker

[issue37380] subprocess.Popen._cleanup() "The handle is invalid" error when some old process is gone

Change by miss-islington : -- pull_requests: +15361 pull_request: https://github.com/python/cpython/pull/15707 ___ Python tracker ___

[issue37380] subprocess.Popen._cleanup() "The handle is invalid" error when some old process is gone

Change by miss-islington : -- pull_requests: +15360 pull_request: https://github.com/python/cpython/pull/15706 ___ Python tracker ___

[issue37380] subprocess.Popen._cleanup() "The handle is invalid" error when some old process is gone

Change by Steve Dower : -- stage: patch review -> backport needed versions: +Python 3.8, Python 3.9 ___ Python tracker ___ ___

[issue37380] subprocess.Popen._cleanup() "The handle is invalid" error when some old process is gone

Chip Lynch added the comment: I see there is a merged patch for this (https://github.com/python/cpython/pull/14360); is it possible to get it tagged for backport to 3.7? I haven't seen the 3.7.5 tag drop yet so wanted to recommend it's included, it's giving us fits. On our Windows 10 dev

[issue37380] subprocess.Popen._cleanup() "The handle is invalid" error when some old process is gone

Emanuel Zephir added the comment: I believe this to be a bug in the standard library instead of solely being the result of an application error. When a Popen instance is finalized by the garbage collector, the internal handle is also finalized and closed despite the instance being put on

[issue37380] subprocess.Popen._cleanup() "The handle is invalid" error when some old process is gone

STINNER Victor added the comment: > Say a library calls CreateEventW and gets handle 32. It passes this handle to > some other library, which uses the event and closes the handle when it no > longer needs it. But due to a miscommunication in the documentation, the > first library thinks the

[issue37380] subprocess.Popen._cleanup() "The handle is invalid" error when some old process is gone

Eryk Sun added the comment: > Without accessing private attributes, I don't see how someone can > discover the private handle. So for me, it's more a serious bug in an > application, no? Blindly closing random handles doesn't sound like a > good idea to me. Say a library calls CreateEventW

[issue37380] subprocess.Popen._cleanup() "The handle is invalid" error when some old process is gone

STINNER Victor added the comment: > The example is just emulating a problem from someone else's code that's > closing our handle. Typically this situation occurs because the code is > holding onto a handle value for a kernel object (File, Section, Job, Process, > Thread, Event, etc) that

[issue37380] subprocess.Popen._cleanup() "The handle is invalid" error when some old process is gone

Eryk Sun added the comment: >> subprocess._active[0]._handle.Close() > > Why would you do that? You should not access the private _active list, > nor access the private _handle attribute. I understand that it's a way > to trigger such bug, but is it possible to trigger this bug without >

[issue37380] subprocess.Popen._cleanup() "The handle is invalid" error when some old process is gone

STINNER Victor added the comment: > subprocess._active[0]._handle.Close() Why would you do that? You should not access the private _active list, nor access the private _handle attribute. I understand that it's a way to trigger such bug, but is it possible to trigger this bug without

[issue37380] subprocess.Popen._cleanup() "The handle is invalid" error when some old process is gone

Eryk Sun added the comment: >> If one of the processes in that list is gone, then (...) "The handle >> is invalid" (...) will prevent you from creating any new processes >> with Popen after that > > It's the first time that I see such bug report. IIRC it's the second time I've seen that

[issue37380] subprocess.Popen._cleanup() "The handle is invalid" error when some old process is gone

STINNER Victor added the comment: > To me, it doesn't seem dangerous to backport that, but I would love to hear > what Eryk and others think :) In my experience, any change is dangerous :-) The initial bug doesn't contain any reproducer. I'm talking about: > If one of the processes in that

[issue37380] subprocess.Popen._cleanup() "The handle is invalid" error when some old process is gone

Ruslan Kuprieiev added the comment: The `_active` code is pretty old and has not changed since 2.7, so it makes sense to backport it for now(also, should be very trivial to do that). To me, it doesn't seem dangerous to backport that, but I would love to hear what Eryk and others think :)

[issue37380] subprocess.Popen._cleanup() "The handle is invalid" error when some old process is gone

STINNER Victor added the comment: I merged PR 14360 into master. Are Python 2.7, 3.7 and 3.8 impacted by the bug as well? Should we backport the change, or would it be too dangerous? -- ___ Python tracker

[issue37380] subprocess.Popen._cleanup() "The handle is invalid" error when some old process is gone

STINNER Victor added the comment: New changeset 042821ae3cf537e01963c9ec85d1a454d921e826 by Victor Stinner (Ruslan Kuprieiev) in branch 'master': bpo-37380: subprocess: don't use _active on win (GH-14360) https://github.com/python/cpython/commit/042821ae3cf537e01963c9ec85d1a454d921e826

[issue37380] subprocess.Popen._cleanup() "The handle is invalid" error when some old process is gone

STINNER Victor added the comment: Here is a concrete example of ResourceWarning when debugging/stressing multiprocessing code, I interrupt (CTRL+c) test_resource_tracker while it's running: vstinner@apu$ ./python -m test test_multiprocessing_spawn --fail-env-changed -v -m

[issue37380] subprocess.Popen._cleanup() "The handle is invalid" error when some old process is gone

STINNER Victor added the comment: Sorry, I'm lost :-( More and more different issues are discussed here. Would it be possible please to stick this issue to _active list and _cleanup() on Windows? From what I read, remove/disable _active and _cleanup() makes sense on Windows. I already

[issue37380] subprocess.Popen._cleanup() "The handle is invalid" error when some old process is gone

Eryk Sun added the comment: > The process is not polled to be able to emit the warning in more > cases, again, to ease debug. It emits an incorrect warning if the process has already exited: "subprocess %s is still running". This can be rectified. Here's my current understanding: def

[issue37380] subprocess.Popen._cleanup() "The handle is invalid" error when some old process is gone

STINNER Victor added the comment: > In Unix, the zombie remains visible in the task list (marked as in > Linux), but in Windows an exited process is removed from the Process > Manager's active list, so it's no longer visible to users. Also, a Process > object is reaped as soon as the last

[issue37380] subprocess.Popen._cleanup() "The handle is invalid" error when some old process is gone

Eryk Sun added the comment: > One issue on Linux is that the zombie process keeps the pid used until > the parent reads the child exit status, and Linux pids are limited to > 32768 by default. Windows allocates Process and Thread IDs out of a kernel handle table, which can grow to about

[issue37380] subprocess.Popen._cleanup() "The handle is invalid" error when some old process is gone

Steve Dower added the comment: The handle can deliberately live beyond the process, but it does not have to. If it is closed early then the kernel object will be freed when no handles remain, which will be at process exit. So it's a classic __exit__/__del__ case, where both are needed if

[issue37380] subprocess.Popen._cleanup() "The handle is invalid" error when some old process is gone

STINNER Victor added the comment: > See issue 36067 for a related discussion. The _active list and _cleanup > function are not required in Windows. When a process exits, it gets rundown > to free its handle table and virtual memory. Only the kernel object remains, > which is kept alive by

[issue37380] subprocess.Popen._cleanup() "The handle is invalid" error when some old process is gone

Ruslan Kuprieiev added the comment: Thanks for pointing that out! I've submitted a tiny patch to skip ``_collect()`` and ``__del__``. Please find it in the Github PR attached. Looking forward to your feedback! -- ___ Python tracker

[issue37380] subprocess.Popen._cleanup() "The handle is invalid" error when some old process is gone

Change by Ruslan Kuprieiev : -- pull_requests: +14176 pull_request: https://github.com/python/cpython/pull/14360 ___ Python tracker ___

[issue37380] subprocess.Popen._cleanup() "The handle is invalid" error when some old process is gone

STINNER Victor added the comment: > 2) ignore EBADF(OSError: [WinError 6] The handle is invalid) in terminate() > and probably some other methods That sounds like a bad idea. We should avoid using a handle if it can become invalid out of our control. A handle might be recycled, no?

[issue37380] subprocess.Popen._cleanup() "The handle is invalid" error when some old process is gone

Eryk Sun added the comment: > 1) disable _cleanup, _active, and remove _internal_poll() for windows _active only gets appended to in __del__. We can skip the entire body of __del__. Also, calling _cleanup can be skipped in __init__. _internal_poll is required for poll(). 2) ignore

[issue37380] subprocess.Popen._cleanup() "The handle is invalid" error when some old process is gone

Ruslan Kuprieiev added the comment: Hi Eryk! Thanks for a swift reply! So the current plan for fixing it is: 1) disable _cleanup, _active, and remove _internal_poll() for windows 2) ignore EBADF(OSError: [WinError 6] The handle is invalid) in terminate() and probably some other methods

[issue37380] subprocess.Popen._cleanup() "The handle is invalid" error when some old process is gone

Change by Roundup Robot : -- keywords: +patch pull_requests: +14147 stage: -> patch review pull_request: https://github.com/python/cpython/pull/14327 ___ Python tracker ___

[issue37380] subprocess.Popen._cleanup() "The handle is invalid" error when some old process is gone

Eryk Sun added the comment: See issue 36067 for a related discussion. The _active list and _cleanup function are not required in Windows. When a process exits, it gets rundown to free its handle table and virtual memory. Only the kernel object remains, which is kept alive by pointer and

[issue37380] subprocess.Popen._cleanup() "The handle is invalid" error when some old process is gone

Change by Karthikeyan Singaravelan : -- nosy: +gregory.p.smith ___ Python tracker ___ ___ Python-bugs-list mailing list

[issue37380] subprocess.Popen._cleanup() "The handle is invalid" error when some old process is gone

New submission from Ruslan Kuprieiev : subprocess keeps the list of active processes in its `_active` list. Whenever you try to Popen a new process, it is immediately (like right in the first line of Popen.__init__ [1]) trying to go clean up the old ones. If one of the processes in that list