[issue37224] test__xxsubinterpreters fails randomly

Kyle Stanley added the comment: I have a few spare cycles to take another stab at this issue. I can say with some certainty that the failure in test__xxsubinterpreters.DestroyTests does not occur on the latest commit to master (3.9): ``` $ ./python -m test test__xxsubinterpreters --match

[issue37224] test__xxsubinterpreters fails randomly

Kyle Stanley added the comment: > Thanks, Kyle. That helps at least a little. :) No problem. (: I'll certainly spend some additional time investigating the main issue when I have the chance to, but in the meantime that test change should make it slightly easier to determine the point of

[issue37224] test__xxsubinterpreters fails randomly

Eric Snow added the comment: Thanks, Kyle. That helps at least a little. :) -- ___ Python tracker ___ ___ Python-bugs-list

[issue37224] test__xxsubinterpreters fails randomly

miss-islington added the comment: New changeset 9a740b6c7e7a88185d79128b8a1993ac387d5091 by Miss Islington (bot) in branch '3.8': bpo-37224: Improve test__xxsubinterpreters.DestroyTests (GH-18058) https://github.com/python/cpython/commit/9a740b6c7e7a88185d79128b8a1993ac387d5091 --

[issue37224] test__xxsubinterpreters fails randomly

Change by miss-islington : -- pull_requests: +17693 pull_request: https://github.com/python/cpython/pull/18318 ___ Python tracker ___

[issue37224] test__xxsubinterpreters fails randomly

miss-islington added the comment: New changeset f03a8f8d5001963ad5b5b28dbd95497e9cc15596 by Kyle Stanley in branch 'master': bpo-37224: Improve test__xxsubinterpreters.DestroyTests (GH-18058) https://github.com/python/cpython/commit/f03a8f8d5001963ad5b5b28dbd95497e9cc15596 -- nosy:

[issue37224] test__xxsubinterpreters fails randomly

Change by Kyle Stanley : -- pull_requests: +17452 pull_request: https://github.com/python/cpython/pull/18058 ___ Python tracker ___

[issue37224] test__xxsubinterpreters fails randomly

Eric Snow added the comment: On Wed, Jan 15, 2020 at 12:20 AM Kyle Stanley wrote: > As can be seen from the results above, the interpreter is not even running in > the first place before > it's destroyed, so of course destroy() won't raise an RuntimeError. I think > this proves that >

[issue37224] test__xxsubinterpreters fails randomly

Change by STINNER Victor : -- nosy: -vstinner ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue37224] test__xxsubinterpreters fails randomly

Kyle Stanley added the comment: I just made a rather interesting discovery. Instead of specifically focusing my efforts on the logic with interp_destroy(), I decided to take a closer look at the failing unit test itself. The main test within DestroyTests that's failing is the following:

[issue37224] test__xxsubinterpreters fails randomly

Kyle Stanley added the comment: > I also just realized that I can run > "test.test__xxsubinterpreters.DestroyTests" by itself with: > ./python -m test test__xxsubinterpreters.DestroyTests -j200 -F -v Oops, the correct syntax is: ./python -m test test__xxsubinterpreters --match DestroyTests

[issue37224] test__xxsubinterpreters fails randomly

Kyle Stanley added the comment: I also just realized that I can run "test.test__xxsubinterpreters.DestroyTests" by itself with: ./python -m test test__xxsubinterpreters.DestroyTests -j200 -F -v For some reason, I hadn't thought of running that class of tests by itself to isolate the

[issue37224] test__xxsubinterpreters fails randomly

Kyle Stanley added the comment: Update: I have a bit of good news and not so great news. The good news is that I had some time to work on this again, specifically with isolating the failure in test__xxsubinterpreters.DestroyTests. Locally, I added a few temporary "@unittest.skip()"

[issue37224] test__xxsubinterpreters fails randomly

Kyle Stanley added the comment: > For a struct-specific getter we usually end the prefix with an underscore: _PyInterpreter_IsFinalizing. Otherwise, that's the same name I would have used. :) Good to know, thanks! > No worries (or hurries). Just request a review from me when you're ready.

[issue37224] test__xxsubinterpreters fails randomly

Eric Snow added the comment: On Fri, Dec 13, 2019 at 8:08 PM Kyle Stanley wrote: > Yeah, I named it "_PyInterpreterIsFinalizing" and it's within > Include/cpython. Definitely open > to suggestions on the name though, it's basically just a private getter for > interp->finalizing. For a

[issue37224] test__xxsubinterpreters fails randomly

Kyle Stanley added the comment: > Yep, it has to use the public C-API just like any other module. The > function has a "_Py" prefix and be defined in Include/cpython, right? Yeah, I named it "_PyInterpreterIsFinalizing" and it's within Include/cpython. Definitely open to suggestions on the

[issue37224] test__xxsubinterpreters fails randomly

Eric Snow added the comment: On Sat, Nov 30, 2019 at 9:23 PM Kyle Stanley wrote: > I have a few ideas that I'd like to test out for fixing this failure, and if > any of them produce positive results I'll report back. Sounds good. > Since the failures are still consistently occurring, I

[issue37224] test__xxsubinterpreters fails randomly

Eric Snow added the comment: On Sat, Nov 30, 2019 at 9:23 PM Kyle Stanley wrote: > Based on the above hint, I was able to make some progress on a potential > solution. Thanks Eric. That's great! > Instead of only checking "frame->f_executing", I changed "_is_running()" to > also check the

[issue37224] test__xxsubinterpreters fails randomly

Kyle Stanley added the comment: > so that operations (such as running scripts, destroying the interpreter, etc) > can't occur during finalization Clarification: by "destroying the interpreter" I am specifically referring to calling `interp_destroy()` after finalization has already started.

[issue37224] test__xxsubinterpreters fails randomly

Kyle Stanley added the comment: > Regarding "is_running()", notice that it relies almost entirely on > "frame->f_executing". That might not be enough (or maybe the behavior there > changed). That would be worth checking out. Based on the above hint, I was able to make some progress on a

[issue37224] test__xxsubinterpreters fails randomly

Kyle Stanley added the comment: > So, I was finally able to replicate a failure in test_still_running locally, > it required using a rather ridiculous number of parallel workers I forgot to mention that I was able to replicate the above failure on the latest commit to the 3.8 branch. I was

[issue37224] test__xxsubinterpreters fails randomly

Kyle Stanley added the comment: > I was able to consistently reproduce the above failure using 200 parallel > workers, even without `-f`. Oops, I didn't mean without passing `-F`, as this would result in only a single test being ran. I meant without letting it repeat multiple times, as in

[issue37224] test__xxsubinterpreters fails randomly

Kyle Stanley added the comment: So, I was finally able to replicate a failure in test_still_running locally, it required using a rather ridiculous number of parallel workers: $ ./python -m test test__xxsubinterpreters -j200 -F ... Exception in thread Thread-7: Traceback (most recent call

[issue37224] test__xxsubinterpreters fails randomly

Kyle Stanley added the comment: > Sorry I haven't gotten back to you sooner, Kyle. Thanks for working on this. > I'm looking at your PR right now. > BTW, Kyle, your problem-solving approach on this is on-track. Don't get > discouraged. This stuff is tricky. :) No problem at all, and

[issue37224] test__xxsubinterpreters fails randomly

Eric Snow added the comment: Thus far these are the failures we've seen: * not running when we expect it to be running: * interpreters.is_running(interp) * interpreters.run_string(interp, ...) * interpreters.destroy(interp) * can't find the interpreter even though we expect it to

[issue37224] test__xxsubinterpreters fails randomly

Eric Snow added the comment: BTW, Kyle, your problem-solving approach on this is on-track. Don't get discouraged. This stuff is tricky. :) -- ___ Python tracker ___

[issue37224] test__xxsubinterpreters fails randomly

Eric Snow added the comment: Sorry I haven't gotten back to you sooner, Kyle. Thanks for working on this. I'm looking at your PR right now. -- ___ Python tracker ___

[issue37224] test__xxsubinterpreters fails randomly

Change by Eric Snow : -- keywords: +patch pull_requests: +16843 stage: -> patch review pull_request: https://github.com/python/cpython/pull/16293 ___ Python tracker ___

[issue37224] test__xxsubinterpreters fails randomly

STINNER Victor added the comment: AMD64 Windows8.1 Refleaks 3.8: https://buildbot.python.org/all/#/builders/224/builds/151 0:54:23 load avg: 5.62 [306/423/3] test__xxsubinterpreters failed -- running: test_asyncio (4 min 26 sec), test_zipfile (6 min 6 sec), test_compileall (6 min 21 sec)

[issue37224] test__xxsubinterpreters fails randomly

STINNER Victor added the comment: See also bpo-33868: test__xxsubinterpreters fails randomly since at least 2018-06-15. -- ___ Python tracker ___

[issue37224] test__xxsubinterpreters fails randomly

STINNER Victor added the comment: AMD64 FreeBSD Shared 3.x: https://buildbot.python.org/all/#/builders/371/builds/7 test_subinterpreter (test.test__xxsubinterpreters.IsRunningTests) ... Exception in thread Thread-8: Traceback (most recent call last): File

[issue37224] test__xxsubinterpreters fails randomly

STINNER Victor added the comment: The race condition can be reproduced on Linux using a lot of parallel test worker processes. For example, I reproduce it on my laptop (8 CPUs) using: $ ./python -m test test__xxsubinterpreters -F -j30 ... 0:00:26 load avg: 17.86 [ 24] test__xxsubinterpreters

[issue37224] test__xxsubinterpreters fails randomly

Kyle Stanley added the comment: > Kyle Stanley proposed a fix: PR 16293. Note that I'm not confident about the fix I proposed in GH-16293, it was more of an idea to fix the intermittent failures more than anything. It definitely needs review from someone knowledgeable about sub-interpreters

[issue37224] test__xxsubinterpreters fails randomly

STINNER Victor added the comment: AMD64 FreeBSD CURRENT Shared 3.8: https://buildbot.python.org/all/#/builders/212/builds/310 FAIL: test_still_running (test.test__xxsubinterpreters.DestroyTests) FAIL: test_run_coroutine_threadsafe_with_timeout

[issue37224] test__xxsubinterpreters fails randomly

STINNER Victor added the comment: AMD64 FreeBSD CURRENT Shared 3.x: https://buildbot.python.org/all/#/builders/168/builds/1630 FAIL: test_still_running (test.test__xxsubinterpreters.DestroyTests) 1:07:09 load avg: 8.18 Re-running failed tests in verbose mode 1:07:09 load avg: 8.18 Re-running

[issue37224] test__xxsubinterpreters fails randomly

Change by STINNER Victor : -- title: test__xxsubinterpreters failed on AMD64 Windows8.1 Refleaks 3.8 -> test__xxsubinterpreters fails randomly ___ Python tracker ___