New submission from STINNER Victor: Thanks to the PEP 445, it becomes possible to trace easily memory allocations. I propose to add a new tracemalloc module computing the memory usage per file and per line number.
I implemented the module as a third party module for Python 2.5-3.4, but it requires to patch and recompile Python: https://pypi.python.org/pypi/pytracemalloc My proposed implementation is different: * use a simple C implementation of an hash table called "cfuhash" (coming from the libcfu project) instead of depending on the glib library ; I simplified and adapted the implementation for my usage * no enable() / disable() function: tracemalloc can only be enabled before startup by setting PYTHONTRACEMALLOC=1 environment variable * traces (size of the memory block, Python filename, Python line number) are stored directly in the memory block, not in a separated hash table I chose PYTHONTRACEMALLOC env var instead of enable()/disable() functions to be able to really trace *all* memory allocated by Python, especially memory allocated at startup, during Python initialization. TODO: * The (high-level) API should be reviewed and discussed * The documention should be improved * PyMem_Raw API is not hooked yet because the code is not thread-safe (it relies on the Python GIL) * Filenames should not be encoded: the hash table should directly use Unicode strings (PyObject*), encoding a filename allocates memory and may call the garbage collector * Memory is not released at exit For the documentation, see the following page: https://pypi.python.org/pypi/pytracemalloc See also the https://bitbucket.org/haypo/pyfailmalloc project and the issue #18408. ---------- hgrepos: 206 messages: 196436 nosy: haypo priority: normal severity: normal status: open title: Add a new tracemalloc module to trace memory allocations type: enhancement versions: Python 3.4 _______________________________________ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue18874> _______________________________________ _______________________________________________ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
