Pierre Glaser <[email protected]> added the comment:
Shared memory segments are now tracked by the brand new resource_tracker!
Thanks Antoine for the review.
Does anyone have an opinion on introducing a public API for users to make the
resource_tracker track resources of their choice?
What We have in mind is:
- making public the existing resource_tracker.register/unregister
- adding a new function, resource_tracker.make_trackable(resource_type,
cleanup_func), where:
* resource_type is a string (an identifier that will be used each time a
resource of the type resource_type needs tracking, via the call
resource_tracker.register(resource_name, resource_type)
* cleanup_func must be a callable taking a single string as argument (the
name of the resource that needs tracking). This function will be called
after the end of a process for reach resource of type resource_type the
process did not clean properly
Under the hood, make_trackable simply populates resource_tracker._CLEANUP_FUNCS
with new items.
Here is a simple example:
import os
import resource_tracker
import shutil
from multiprocessing import util
class ClassCreatingAFolder:
"""Class where each instance creates a temporary folder.
Each temporary folder is supposed to exist for the duration of the
instance
that created it.
"""
def __init__(self, folder_name):
self.folder_name = folder_name
os.mkdir(folder_name)
# any instance normally garbage-collected should remove its
folder, and
# notice the resource_tracker that its folder was correctly
removed.
util.Finalize(self, ClassCreatingAFolder.cleanup,
args=(folder_name,))
# If this session quits abruptly, the finalizer will not be
called for
# the instances of ClassCreatingAFolder that were still alive
# before the shutdown. The resource_tracker comes into play,
and removes
# the folders associated to each of these resources.
resource_tracker.register(
folder_name, # argument to shutil.rmtree
"ClassCreatingAFolder")
@staticmethod
def cleanup(folder_name):
resource_tracker.unregister(folder_name, "ClassCreatingAFolder")
shutil.rmtree(folder_name)
# Tell the resource_tracker how to cleanup resources created by
# ClassCreatingAFolder instances
resource_tracker.make_trackable("ClassCreatingAFolder", shutil.rmtree)
Typical resources that can be made trackable include memmaped objects,
temporary folders. Our use-case is joblib that has its own mmap type that we
would like to track using the semaphore_tracker.
Any thoughts?
----------
_______________________________________
Python tracker <[email protected]>
<https://bugs.python.org/issue36867>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
