python: setup.py: how NOT to install C extensions used only by tests
I have a module that has a tests/ directory which contains a C extension that's only used by test cases. I don't want to install it. I'm building it as a setuptools.Extension() added to setup() using the ext_modules argument. The test directory is not added to setup's packages argument. Even though none of the python sources from the tests/ directory gets installed when running setup.py install, the extension binary (and nothing else) is installed into site-packages/tests/. How can I prohibit setuptools from doing it? Best Regards, Bartosz Golaszewski -- https://mail.python.org/mailman/listinfo/python-list
Tracking a memory leak in C extension - interpreting the output of PYTHONMALLOCSTATS
ocation of 432 bytes. I launched gdb and set the following breakpoint: 'b PyMem_Malloc if size == 432' but the breakpoint was never triggered. I tried the same for PyObject_Malloc and still nothing. How do I use the info produced by PYTHONMALLOCSTATS do get to the culprit of the leak? Is there anything wrong in my reasoning here? Best regards, Bartosz Golaszewski [1] https://git.kernel.org/pub/scm/libs/libgpiod/libgpiod.git/ -- https://mail.python.org/mailman/listinfo/python-list
Re: Tracking a memory leak in C extension - interpreting the output of PYTHONMALLOCSTATS
2018-07-24 12:09 GMT+02:00 Bartosz Golaszewski : > 2018-07-23 21:51 GMT+02:00 Thomas Jollans : >> On 23/07/18 20:02, Bartosz Golaszewski wrote: >>> Hi! >> >> Hey! >> >>> A user recently reported a memory leak in python bindings (C extension >>> module) to a C library[1] I wrote. I've been trying to fix it since >>> but so far without success. Since I'm probably dealing with a space >>> leak rather than actual memory leak, valgrind didn't help much even >>> when using malloc as allocator. I'm now trying to use >>> PYTHONMALLOCSTATS but need some help on how to interpret the output >>> emitted it's enabled. >> >> Oh dear. >> >>> >>> [snip] >>> >>> The number of pools in arena 53 continuously grows. Its size column >>> says: 432. I couldn't find any documentation on what it means but I >>> assume it's an allocation of 432 bytes. [...] >> >> I had a quick look at the code (because what else does one do for fun); >> I don't understand much, but what I can tell you is that >> (a) yes, that is an allocation size in bytes, and >> (b) as you can see, it uses intervals of 8. This means that pool 53 >> is used for allocations of 424 < nbytes <= 432 bytes. Maybe your >> breakpoint needs tweaking. >> (c) Try breaking on _PyObject_Malloc or pymalloc_alloc. I think they're >> called by both PyMem_Malloc and PyObject_Malloc. >> >> int _PyObject_DebugMallocStats(FILE *out) >> >> https://github.com/python/cpython/blob/b18f8bc1a77193c372d79afa79b284028a2842d7/Objects/obmalloc.c#L2435 >> >> static int pymalloc_alloc(void *ctx, void **ptr_p, size_t nbytes) >> >> https://github.com/python/cpython/blob/b18f8bc1a77193c372d79afa79b284028a2842d7/Objects/obmalloc.c#L1327 >> >> >> Have fun debugging! >> >> -- Thomas >> >> [snip!] > > I don't see any other allocation of this size. Can this be some bug in > the interpreter? > > Bart Ok so this is strange: I can fix the leak if I explicitly call PyObject_Free() on the leaking object which is created by "calling" its type. Is this normal? Shouldn't Py_DECREF() be enough? The relevant dealloc callback is called from Py_DECREF() but the object's memory is not freed. Bart -- https://mail.python.org/mailman/listinfo/python-list
Re: Tracking a memory leak in C extension - interpreting the output of PYTHONMALLOCSTATS
2018-07-23 21:51 GMT+02:00 Thomas Jollans : > On 23/07/18 20:02, Bartosz Golaszewski wrote: >> Hi! > > Hey! > >> A user recently reported a memory leak in python bindings (C extension >> module) to a C library[1] I wrote. I've been trying to fix it since >> but so far without success. Since I'm probably dealing with a space >> leak rather than actual memory leak, valgrind didn't help much even >> when using malloc as allocator. I'm now trying to use >> PYTHONMALLOCSTATS but need some help on how to interpret the output >> emitted it's enabled. > > Oh dear. > >> >> [snip] >> >> The number of pools in arena 53 continuously grows. Its size column >> says: 432. I couldn't find any documentation on what it means but I >> assume it's an allocation of 432 bytes. [...] > > I had a quick look at the code (because what else does one do for fun); > I don't understand much, but what I can tell you is that > (a) yes, that is an allocation size in bytes, and > (b) as you can see, it uses intervals of 8. This means that pool 53 > is used for allocations of 424 < nbytes <= 432 bytes. Maybe your > breakpoint needs tweaking. > (c) Try breaking on _PyObject_Malloc or pymalloc_alloc. I think they're > called by both PyMem_Malloc and PyObject_Malloc. > > int _PyObject_DebugMallocStats(FILE *out) > > https://github.com/python/cpython/blob/b18f8bc1a77193c372d79afa79b284028a2842d7/Objects/obmalloc.c#L2435 > > static int pymalloc_alloc(void *ctx, void **ptr_p, size_t nbytes) > > https://github.com/python/cpython/blob/b18f8bc1a77193c372d79afa79b284028a2842d7/Objects/obmalloc.c#L1327 > > > Have fun debugging! > > -- Thomas > > >> >> How do I use the info produced by PYTHONMALLOCSTATS do get to the >> culprit of the leak? Is there anything wrong in my reasoning here? >> >> Best regards, >> Bartosz Golaszewski >> >> [1] https://git.kernel.org/pub/scm/libs/libgpiod/libgpiod.git/ >> > > -- > https://mail.python.org/mailman/listinfo/python-list Thanks for the hints! I've been able to pinpoint the allocation in question to this line: https://git.kernel.org/pub/scm/libs/libgpiod/libgpiod.git/tree/bindings/python/gpiodmodule.c?h=next#n1238 with the following stack trace: #0 _PyObject_Malloc (ctx=0x0, nbytes=432) at Objects/obmalloc.c:1523 #1 0x55614c38 in _PyMem_DebugRawAlloc (ctx=0x55a3c340 <_PyMem_Debug+96>, nbytes=400, use_calloc=0) at Objects/obmalloc.c:1998 #2 0x556238c5 in PyType_GenericAlloc (type=0x76e06820 , nitems=0) at Objects/typeobject.c:972 #3 0x55627ba5 in type_call (type=0x76e06820 , args=0x76e21910, kwds=0x0) at Objects/typeobject.c:929 #4 0x555cc666 in PyObject_Call (kwargs=0x0, args=, callable=0x76e06820 ) at Objects/call.c:245 #5 PyEval_CallObjectWithKeywords (kwargs=0x0, args=, callable=0x76e06820 ) at Objects/call.c:826 #6 PyObject_CallObject (callable=0x76e06820 , args=) at Objects/call.c:834 #7 0x76c008dd in gpiod_LineToLineBulk (line=line@entry=0x75bbd240) at gpiodmodule.c:1238 #8 0x76c009af in gpiod_Line_set_value (self=0x75bbd240, args=) at gpiodmodule.c:442 #9 0x555c9ef8 in _PyMethodDef_RawFastCallKeywords (method=0x76e06280 , self=self@entry=0x75bbd240, args=args@entry=0x55b15e18, nargs=nargs@entry=1, kwnames=kwnames@entry=0x0) at Objects/call.c:694 #10 0x55754db9 in _PyMethodDescr_FastCallKeywords (descrobj=0x76e344d0, args=args@entry=0x55b15e10, nargs=nargs@entry=2, kwnames=kwnames@entry=0x0) at Objects/descrobject.c:288 #11 0x555b7fcd in call_function (kwnames=0x0, oparg=2, pp_stack=) at Python/ceval.c:4581 #12 _PyEval_EvalFrameDefault (f=, throwflag=) at Python/ceval.c:3176 #13 0x55683b7c in PyEval_EvalFrameEx (throwflag=0, f=0x55b15ca0) at Python/ceval.c:536 #14 _PyEval_EvalCodeWithName (_co=_co@entry=0x77e50460, globals=globals@entry=0x77f550e8, locals=locals@entry=0x77e50460, args=args@entry=0x0, argcount=argcount@entry=0, kwnames=kwnames@entry=0x0, kwargs=0x0, kwcount=0, kwstep=2, defs=0x0, defcount=0, kwdefs=0x0, closure=0x0, name=0x0, qualname=0x0) at Python/ceval.c:3941 #15 0x55683ca3 in PyEval_EvalCodeEx (closure=0x0, kwdefs=0x0, defcount=0, defs=0x0, kwcount=0, kws=0x0, argcount=0, args=0x0, locals=locals@entry=0x77e50460, globals=globals@entry=0x77f550e8, _co=_co@entry=0x77e50460) at Python/ceval.c:3970 #16 PyEval_EvalCode (co=co@entry=0x77e50460, globals=globals@entry=0x77efcc50, locals=locals@entry=0x77efcc50) at Python/ceval.c:513 #17 0x556bb099 in run_mod (arena=0x77f550e8, flags=0x7fffe1a0, locals=0x77efcc50, globals=0x77efcc50
Re: Tracking a memory leak in C extension - interpreting the output of PYTHONMALLOCSTATS
2018-07-24 13:30 GMT+02:00 Bartosz Golaszewski : > 2018-07-24 12:09 GMT+02:00 Bartosz Golaszewski : >> 2018-07-23 21:51 GMT+02:00 Thomas Jollans : >>> On 23/07/18 20:02, Bartosz Golaszewski wrote: >>>> Hi! >>> >>> Hey! >>> >>>> A user recently reported a memory leak in python bindings (C extension >>>> module) to a C library[1] I wrote. I've been trying to fix it since >>>> but so far without success. Since I'm probably dealing with a space >>>> leak rather than actual memory leak, valgrind didn't help much even >>>> when using malloc as allocator. I'm now trying to use >>>> PYTHONMALLOCSTATS but need some help on how to interpret the output >>>> emitted it's enabled. >>> >>> Oh dear. >>> >>>> >>>> [snip] >>>> >>>> The number of pools in arena 53 continuously grows. Its size column >>>> says: 432. I couldn't find any documentation on what it means but I >>>> assume it's an allocation of 432 bytes. [...] >>> >>> I had a quick look at the code (because what else does one do for fun); >>> I don't understand much, but what I can tell you is that >>> (a) yes, that is an allocation size in bytes, and >>> (b) as you can see, it uses intervals of 8. This means that pool 53 >>> is used for allocations of 424 < nbytes <= 432 bytes. Maybe your >>> breakpoint needs tweaking. >>> (c) Try breaking on _PyObject_Malloc or pymalloc_alloc. I think they're >>> called by both PyMem_Malloc and PyObject_Malloc. >>> >>> int _PyObject_DebugMallocStats(FILE *out) >>> >>> https://github.com/python/cpython/blob/b18f8bc1a77193c372d79afa79b284028a2842d7/Objects/obmalloc.c#L2435 >>> >>> static int pymalloc_alloc(void *ctx, void **ptr_p, size_t nbytes) >>> >>> https://github.com/python/cpython/blob/b18f8bc1a77193c372d79afa79b284028a2842d7/Objects/obmalloc.c#L1327 >>> >>> >>> Have fun debugging! >>> >>> -- Thomas >>> >>> > > [snip!] > >> >> I don't see any other allocation of this size. Can this be some bug in >> the interpreter? >> >> Bart > > Ok so this is strange: I can fix the leak if I explicitly call > PyObject_Free() on the leaking object which is created by "calling" > its type. Is this normal? Shouldn't Py_DECREF() be enough? The > relevant dealloc callback is called from Py_DECREF() but the object's > memory is not freed. > > Bart Ok I've found the problem and it's my fault. From tp_dealloc's documentation: --- The destructor function should free all references which the instance owns, free all memory buffers owned by the instance (using the freeing function corresponding to the allocation function used to allocate the buffer), and finally (as its last action) call the type’s tp_free function. --- I'm not calling the tp_free function... Best regards, Bartosz Golaszewski -- https://mail.python.org/mailman/listinfo/python-list
Defining a Python enum in a C extension - am I doing this right?
", foobar_descr);
Py_DECREF(modname);
if (!sub_enum_type) {
Py_DECREF(module);
return NULL;
}
ret = PyModule_AddObject(module, "FooBar", sub_enum_type);
if (ret < 0) {
Py_DECREF(sub_enum_type);
Py_DECREF(module);
return NULL;
}
return module;
}
Basically I'm calling the EnumMeta's __prepare__ method directly to
create a correct classdict and then I call the PyType_Type object too
to create the sub-type.
This works and AFAICT results in a class that behaves exactly as
expected, but... am I doing this right? Any feedback is appreciated.
I want to use this in a real project, namely the v2 of libgpiod python
bindings[1] so it's important to me to get this right.
Best regards,
Bartosz Golaszewski
[1] https://git.kernel.org/pub/scm/libs/libgpiod/libgpiod.git/
--
https://mail.python.org/mailman/listinfo/python-list
Re: Defining a Python enum in a C extension - am I doing this right?
On Fri, Jul 23, 2021 at 5:08 PM MRAB wrote:
>
> On 2021-07-23 09:20, Bartosz Golaszewski wrote:
> > Hi!
> >
> > I'm working on a Python C extension and I would like to expose a
> > custom enum (as in: a class inheriting from enum.Enum) that would be
> > entirely defined in C.
> >
> > It turned out to not be a trivial task and the regular mechanism for
> > inheritance using .tp_base doesn't work - most likely due to the
> > Enum's meta class not being pulled in.
> >
> > Basically I'm trying to do this:
> >
> [snip]
> >
> > static PyObject *make_bases(PyObject *enum_mod)
> > {
> > PyObject *enum_type, *bases;
> >
> > enum_type = PyObject_GetAttrString(enum_mod, "Enum");
> > if (!enum_type)
> > return NULL;
> >
> > bases = PyTuple_Pack(1, enum_type); /* Steals reference. */
>
> PyTuple_Pack doesn't steal references, as far as I can tell.
>
Right, the doc says it's equivalent to Py_BuildValue("(OO...)", ...)
and it does increase the reference count on stored objects. It doesn't
answer the main question though. :)
Bartosz
> > if (!bases)
> > Py_DECREF(enum_type);
> >
> > return bases;
> > }
> >
> [snip]
> --
> https://mail.python.org/mailman/listinfo/python-list
--
https://mail.python.org/mailman/listinfo/python-list
Re: Defining a Python enum in a C extension - am I doing this right?
On Sat, Jul 24, 2021 at 6:55 AM Dan Stromberg wrote: > > > On Fri, Jul 23, 2021 at 1:20 AM Bartosz Golaszewski wrote: >> >> Hi! >> >> I'm working on a Python C extension and I would like to expose a >> custom enum (as in: a class inheriting from enum.Enum) that would be >> entirely defined in C. > > > I'm probably missing something obvious, but why would you write new code in C > when you can just use Cython? Cython is a lot easier, and quite fast, and > should (eventually?) allow compiling to HPY instead of just "the" C extension > module interface. > > https://news.ycombinator.com/item?id=26627683 > I'm the author and maintainer of libgpiod - the user-space library and tools for using the linux GPIO character device. The core library is written in C but we're also exposing C++ and Python bindings (with more language bindings planned). The python bindings are written as a C extension module and look like this: https://git.kernel.org/pub/scm/libs/libgpiod/libgpiod.git/tree/bindings/python/gpiodmodule.c. We're in the process of writing the (backward incompatible) version 2 of the library in order to support the new kernel features and the C API has changed a lot so we're also rewriting the bindings. Among others: all bitwise flags have now been converted to enums, hence my question. In C++ we'll use scoped enum classes and I'd like to do a similar thing in python. What I'm doing is not aimed at using C for speed but for calling the C APIs. I know I could use SWIG but in order to make the interface elegant, it would have to be packaged in proper Python classes anyway, creating another layer of code so I prefer to just use C. Bart -- https://mail.python.org/mailman/listinfo/python-list
Re: Defining a Python enum in a C extension - am I doing this right?
On Fri, Jul 30, 2021 at 2:41 PM Serhiy Storchaka wrote: > > 23.07.21 11:20, Bartosz Golaszewski пише: > > I'm working on a Python C extension and I would like to expose a > > custom enum (as in: a class inheriting from enum.Enum) that would be > > entirely defined in C. > > I think that it would be much easier to define it in Python, and then > either import a Python module in your C code, or exec a Python code as a > string. > You mean: evaluate a string like this: ''' import enum class FooBar(enum.Enum): FOO = 1 BAR = 2 BAZ = 3 ''' And then pull in the FooBar type from the resulting dictionary into my C code? Sounds good actually. I think I'll be able to add the FooBar type to another type's tp_dict too - because some enums I want to create will be nested in other classes. Bart -- https://mail.python.org/mailman/listinfo/python-list
Re: Defining a Python enum in a C extension - am I doing this right?
On Sat, Jul 31, 2021 at 3:01 PM Bartosz Golaszewski wrote:
>
> On Fri, Jul 30, 2021 at 2:41 PM Serhiy Storchaka wrote:
> >
> > 23.07.21 11:20, Bartosz Golaszewski пише:
> > > I'm working on a Python C extension and I would like to expose a
> > > custom enum (as in: a class inheriting from enum.Enum) that would be
> > > entirely defined in C.
> >
> > I think that it would be much easier to define it in Python, and then
> > either import a Python module in your C code, or exec a Python code as a
> > string.
> >
>
> You mean: evaluate a string like this:
>
> '''
> import enum
>
> class FooBar(enum.Enum):
> FOO = 1
> BAR = 2
> BAZ = 3
> '''
>
> And then pull in the FooBar type from the resulting dictionary into my
> C code? Sounds good actually. I think I'll be able to add the FooBar
> type to another type's tp_dict too - because some enums I want to
> create will be nested in other classes.
>
> Bart
Just a follow-up: this is how I did it eventually:
```
#include
typedef struct {
PyObject_HEAD;
} dummy_object;
PyDoc_STRVAR(dummy_type_doc, "Dummy type in which the enum will be nested.");
static PyTypeObject dummy_type = {
PyVarObject_HEAD_INIT(NULL, 0)
.tp_name = "pycenum.DummyType",
.tp_basicsize = sizeof(dummy_object),
.tp_flags = Py_TPFLAGS_DEFAULT,
.tp_doc = dummy_type_doc,
.tp_new = PyType_GenericNew,
.tp_dealloc = (destructor)PyObject_Del,
};
PyDoc_STRVAR(module_doc,
"C extension module defining a class inheriting from enum.Enum.");
static PyModuleDef module_def = {
PyModuleDef_HEAD_INIT,
.m_name = "pycenum",
.m_doc = module_doc,
.m_size = -1,
};
static int add_foobar_enum(PyObject *module)
{
static const char *foobar_src =
"class FooBar(enum.Enum):\n"
"FOO = 1\n"
"BAR = 2\n"
"BAZ = 3\n";
PyObject *main_mod, *main_dict, *enum_mod, *result, *foobar_type;
int ret;
main_mod = PyImport_AddModule("__main__");
if (!main_mod)
return -1;
main_dict = PyModule_GetDict(main_mod);
if (!main_dict) {
Py_DECREF(main_mod);
return -1;
}
enum_mod = PyImport_ImportModule("enum");
if (!enum_mod) {
Py_DECREF(main_mod);
return -1;
}
ret = PyDict_SetItemString(main_dict, "enum", enum_mod);
Py_DECREF(enum_mod);
if (ret) {
Py_DECREF(main_mod);
return -1;
}
result = PyRun_String(foobar_src, Py_single_input,
main_dict, main_dict);
if (!result) {
Py_DECREF(main_mod);
return -1;
}
foobar_type = PyDict_GetItemString(main_dict, "FooBar");
if (!foobar_type) {
Py_DECREF(main_mod);
return -1;
}
ret = PyDict_SetItemString(dummy_type.tp_dict, "FooBar", foobar_type);
Py_DECREF(foobar_type);
Py_DECREF(main_mod);
if (ret)
return -1;
PyType_Modified(&dummy_type);
return ret;
}
PyMODINIT_FUNC PyInit_pycenum(void)
{
PyObject *module;
int ret;
module = PyModule_Create(&module_def);
if (!module)
return NULL;
ret = PyModule_AddStringConstant(module, "__version__", "0.0.1");
if (ret) {
Py_DECREF(module);
return NULL;
}
ret = PyType_Ready(&dummy_type);
if (ret) {
Py_DECREF(module);
return NULL;
}
ret = add_foobar_enum(module);
if (ret) {
Py_DECREF(module);
return NULL;
}
Py_INCREF(&dummy_type);
ret = PyModule_AddObject(module, "DummyType", (PyObject *)&dummy_type);
if (ret) {
Py_DECREF(&dummy_type);
Py_DECREF(module);
return NULL;
}
return module;
}
```
Bart
--
https://mail.python.org/mailman/listinfo/python-list
Re: Defining a Python enum in a C extension - am I doing this right?
On Fri, Aug 6, 2021 at 9:02 PM Serhiy Storchaka wrote: > > 03.08.21 13:03, Bartosz Golaszewski пише: > > Just a follow-up: this is how I did it eventually: > > I think it can be simpler. > > 1. No need to create the __main__ module. You can just create a dict. If > some attributes are required (e.g. __name__) it is easy to set them in > the Python code (__name__ = 'pycenum'). > > 2. No need to import the enum module in the C code. It is easier to do > it in the Python code. > Don't you need the '__import__' attribute for that? It's linked to point #1. > 3. Can you define your DummyType in the Python code instead of the C > code? Even if you need to create in the C code, it may be better to > define your enum class inside a dummy DummyType. It will set correct > __qualname__ of the enum class: > This code is a PoC for the real code I want to use it in where the type embedding the enum is defined in C. > __name__ = 'pycenum' > import enum > class DummyType: > class FooBar(enum.Enum): > ... > > 4. Did you consider idea of making DummyType a heap-allocated type? It > may be easy to create new type with PyType_FromModuleAndSpec(), and it > is more stable API, and the type is not immutable, so it is easy to add > new attributes with PyObject_SetAttrString(). > Makes sense, thanks, I'll try it. Bart -- https://mail.python.org/mailman/listinfo/python-list
