STINNER Victor added the comment:
py_setallocators-filename.patch: Here is a try to define an API providing the
filename and line number of the C code. The Py_SetAllocators() API is unchanged:
PyAPI_FUNC(int) Py_SetAllocators(
char api,
void* (*malloc) (size_t size, void *user_data),
void* (*realloc) (void *ptr, size_t size, void *user_data),
void (*free) (void *ptr, void *user_data),
void *user_data
);
If Python is compiled with -DPYMEM_TRACE_MALLOC, user_data is not the last
parameter passed to Py_SetAllocators() but a pointer to a _PyMem_Trace
structure:
typedef struct {
void *data;
/* NULL and -1 when unknown */
const char *filename;
int lineno;
} _PyMem_Trace;
The problem is that the module using Py_SetAllocators() must be compiled
differently depending on PYMEM_TRACE_MALLOC. Example from pytracemalloc,
modified for this patch:
---
_PyMem_Trace *ctrace;
trace_api_t *api;
void *call_data;
void *ptr;
#ifdef PYMEM_TRACE_MALLOC
ctrace = (_PyMem_Trace *)data;
api = (trace_api_t *)ctrace->data;
ctrace->data = api->data;
call_data = data;
#else
ctrace = NULL;
api = (trace_api_t *)data;
call_data = api->data;
#endif
ptr = api->malloc(size, call_data);
...
---
I didn't like the "ctrace->data = api->data;" instruction: pytracemalloc
modifies the input _PyMem_Trace structure.
pytracemalloc code is a little bit more complex, but "it works". pytracemalloc
can reuse the filename and line number of the C module, or of the Python
module. It can be configured at runtime. Example of output for the C module:
---
2013-06-11 00:36:30: Top 15 allocations per file and line (compared to
2013-06-11 00:36:25)
#1: Objects/dictobject.c:352: size=6 MiB (+4324 KiB), count=9818 (+7773),
average=663 B
#2: Objects/unicodeobject.c:1085: size=6 MiB (+2987 KiB), count=61788 (+26197),
average=111 B
#3: Objects/tupleobject.c:104: size=4054 KiB (+2176 KiB), count=44569 (+24316),
average=93 B
#4: Objects/typeobject.c:770: size=2440 KiB (+1626 KiB), count=13906 (+10360),
average=179 B
#5: Objects/bytesobject.c:107: size=2395 KiB (+1114 KiB), count=24846 (+11462),
average=98 B
#6: Objects/funcobject.c:12: size=1709 KiB (+1103 KiB), count=11516 (+7431),
average=152 B
#7: Objects/codeobject.c:117: size=1760 KiB (+871 KiB), count=11267 (+5578),
average=160 B
#8: Objects/dictobject.c:399: size=784 KiB (+627 KiB), count=10040 (+8028),
average=80 B
#9: Objects/listobject.c:159: size=420 KiB (+382 KiB), count=5386 (+4891),
average=80 B
#10: Objects/frameobject.c:649: size=1705 KiB (+257 KiB), count=3374 (+505),
average=517 B
#11: ???:?: size=388 KiB (+161 KiB), count=588 (+240), average=676 B
#12: Objects/weakrefobject.c:36: size=241 KiB (+138 KiB), count=2579 (+1482),
average=96 B
#13: Objects/dictobject.c:420: size=135 KiB (+112 KiB), count=2031 (+1736),
average=68 B
#14: Objects/classobject.c:59: size=109 KiB (+105 KiB), count=1400 (+1345),
average=80 B
#15: Objects/unicodeobject.c:727: size=188 KiB (+86 KiB), count=1237 (+687),
average=156 B
37 more: size=828 KiB (+315 KiB), count=8421 (+5281), average=100 B
Total Python memory: size=29 MiB (+16 MiB), count=212766 (+117312), average=145
B
Total process memory: size=68 MiB (+22 MiB) (ignore tracemalloc: 0 B)
---
I also had to modify the following GC functions to get more accurate
information:
- _PyObject_GC_Malloc(size)
- _PyObject_GC_New(tp)
- _PyObject_GC_NewVar(tp, nitems)
- PyObject_GC_Del(op)
For example, PyTuple_New() calls PyObject_GC_NewVar() to allocate its memory.
With my patch, you get "Objects/tupleobject.c:104" instead of a generic
"Modules/gcmodule.c:1717".
----------
Added file: http://bugs.python.org/file30537/py_setallocators-filename.patch
_______________________________________
Python tracker <rep...@bugs.python.org>
<http://bugs.python.org/issue3329>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
