[issue36867] Make semaphore_tracker track other system resources

STINNER Victor added the comment: The new test is not reliable, see: bpo-37244 "test_multiprocessing_forkserver: test_resource_tracker() failed on x86 Gentoo Refleaks 3.8". -- ___ Python tracker

[issue36867] Make semaphore_tracker track other system resources

STINNER Victor added the comment: It seems like all known bugs are fixed, I close again the issue. Thanks! -- resolution: -> fixed stage: patch review -> resolved status: open -> closed ___ Python tracker

[issue36867] Make semaphore_tracker track other system resources

STINNER Victor added the comment: New changeset cbe72d842646ded2454784679231e3d1e6252e72 by Victor Stinner (Pierre Glaser) in branch 'master': bpo-36867: _test_multiprocessing: avoid weak sync primitive (GH-13292)

[issue36867] Make semaphore_tracker track other system resources

Antoine Pitrou added the comment: New changeset b1dfcad6f0d3a52c9ac31fb9763fc7962a84b27c by Antoine Pitrou (Pierre Glaser) in branch 'master': bpo-36867: Create the resource_tracker before launching SharedMemoryManagers (GH-13276)

[issue36867] Make semaphore_tracker track other system resources

Change by Pierre Glaser : -- pull_requests: +13203 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue36867] Make semaphore_tracker track other system resources

Antoine Pitrou added the comment: New changeset 50466c66509de556a8579172f82d1abb1560d8e4 by Antoine Pitrou (Pierre Glaser) in branch 'master': bpo-36867: DOC update multiprocessing.rst (GH-13289) https://github.com/python/cpython/commit/50466c66509de556a8579172f82d1abb1560d8e4 --

[issue36867] Make semaphore_tracker track other system resources

Change by Antoine Pitrou : -- pull_requests: +13200 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue36867] Make semaphore_tracker track other system resources

STINNER Victor added the comment: I like Olivier's pattern. Maybe we can slowly increase the sleep to stop shortly if the resource goes away shortly. deadline = time.monotonic() + 60.0 sleep = 0.010 while ...: if : break if time.monotonic() > deadline: ... assert error ... sleep

[issue36867] Make semaphore_tracker track other system resources

Pierre Glaser added the comment: We can do that, or maybe we can try to wait on the `resource_tracker's` pid? -- ___ Python tracker ___

[issue36867] Make semaphore_tracker track other system resources

Olivier Grisel added the comment: As Victor said, the `time.sleep(1.0)` might lead to Heisen failures. I am not sure how to write proper strong synchronization in this case but we could instead go for something intermediate such as the following pattern: ... p.terminate()

[issue36867] Make semaphore_tracker track other system resources

Change by Pierre Glaser : -- pull_requests: +13182 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue36867] Make semaphore_tracker track other system resources

STINNER Victor added the comment: test_shared_memory_cleaned_after_process_termination() uses time as a weak synchronization primitive: # killing abruptly processes holding reference to a shared memory # segment should not leak the given memory segment. p.terminate()

[issue36867] Make semaphore_tracker track other system resources

Pierre Glaser added the comment: Actually, I was properly unlinking the shared_memory segments. The warning messages are due to bad interactions between the ResourceTracker and the SharedMemoryManager object. In this particular case, it's easy to change a little bit the problematic test to

[issue36867] Make semaphore_tracker track other system resources

Pierre Glaser added the comment: Yes, one test I wrote in an unrelated commit does not unlink a memory segment. Now the ResourceTracker complains. Fixing it now. -- ___ Python tracker

[issue36867] Make semaphore_tracker track other system resources

Karthikeyan Singaravelan added the comment: Since the commit there is a warning in CI and locally while running tests. Travis CI : * https://travis-ci.org/python/cpython/jobs/531304357#L2099 (test_multiprocessing_forkserver) * https://travis-ci.org/python/cpython/jobs/531304357#L2296

[issue36867] Make semaphore_tracker track other system resources

Pierre Glaser added the comment: Shared memory segments are now tracked by the brand new resource_tracker! Thanks Antoine for the review. Does anyone have an opinion on introducing a public API for users to make the resource_tracker track resources of their choice? What We have in mind is: -

[issue36867] Make semaphore_tracker track other system resources

Antoine Pitrou added the comment: New changeset f22cc69b012f52882d434a5c44a004bc3aa5c33c by Antoine Pitrou (Pierre Glaser) in branch 'master': bpo-36867: Make semaphore_tracker track other system resources (GH-13222)

[issue36867] Make semaphore_tracker track other system resources

Change by Pierre Glaser : -- keywords: +patch pull_requests: +13132 stage: -> patch review ___ Python tracker ___ ___

[issue36867] Make semaphore_tracker track other system resources

New submission from Pierre Glaser : Hi all, Olivier Grisel, Thomas Moreau and myself are currently working on increasing the range of action of the semaphore_tracker in Python. multiprocessing.semaphore_tracker is a little known module, that launches a server process used to track the life