Re: PyGILState_Release called twice in embedded application
On 23-03-2023 13:33, Barry Scott wrote:
On 23 Mar 2023, at 08:46, Arnaud Loonstra wrote:
Hi all,
I'm running in a crash due to a ResourceWarning (some socket is not closed in a
used module) after calling PyGILState_Release.
I'm running Python in a native thread (so a thread created by C not Python).
I'm acquiring the GIL through PyGILState_Ensure before doing any CPYthon calls
and releasing when finished using PyGILState_Release.
This has worked fine. But now I'm using a python module (openai) which doesn't
close its socket correctly which results in a ResourceWarning which triggers an
assert.
In the backtrace (below) I can see PyGILState_Release is called again. (7)
while I've already called it (126).
I can make the crash go away by adding
import warnings
warnings.simplefilter("ignore", ResourceWarning)
to my python code. But I'd rather prevent this from happening in the first
place.
Any suggestion very welcomed cause I'm puzzled.
What 3rd party C extension are you running with?
I'd be surprised if the cpython code was the issue.
Barry
I'm not using any 3rd party extension myself. But the issue is caused by
the openai module using the requests module which is not closing sockets:
https://github.com/openai/openai-python/issues/140
I'm not aware what C extensions they might use.
Btw, I've tested this with python 3.8 and 3.11.2.
Rg,
Arnaud
--
https://mail.python.org/mailman/listinfo/python-list
PyGILState_Release called twice in embedded application
Hi all,
I'm running in a crash due to a ResourceWarning (some socket is not
closed in a used module) after calling PyGILState_Release.
I'm running Python in a native thread (so a thread created by C not
Python). I'm acquiring the GIL through PyGILState_Ensure before doing
any CPYthon calls and releasing when finished using PyGILState_Release.
This has worked fine. But now I'm using a python module (openai) which
doesn't close its socket correctly which results in a ResourceWarning
which triggers an assert.
In the backtrace (below) I can see PyGILState_Release is called again.
(7) while I've already called it (126).
I can make the crash go away by adding
import warnings
warnings.simplefilter("ignore", ResourceWarning)
to my python code. But I'd rather prevent this from happening in the
first place.
Any suggestion very welcomed cause I'm puzzled.
Rg,
Arnaud
1 __pthread_kill_implementation pthread_kill.c 44
0x76fcaccc
2 __pthread_kill_internalpthread_kill.c 78
0x76fcad2f
3 __GI_raise raise.c 26
0x76f7bef2
4 __GI_abort abort.c 79
0x76f66472
5 __assert_fail_base assert.c 92
0x76f66395
6 __GI___assert_fail assert.c 101
0x76f74df2
7 PyGILState_Release pystate.c1742
0x77abcf9f
8 tracemalloc_raw_alloc _tracemalloc.c 779
0x77afa803
9 tracemalloc_raw_malloc _tracemalloc.c 789
0x77afa844
10 PyMem_RawMallocobmalloc.c 586
0x779dab8d
11 decode_current_locale fileutils.c 472
0x77ada4ab
12 _Py_DecodeLocaleEx fileutils.c 598
0x77adafce
13 unicode_decode_locale unicodeobject.c 3970
0x77a13b65
14 PyUnicode_DecodeLocale unicodeobject.c 4012
0x77a18c79
15 PyErr_SetFromErrnoWithFilenameObjects errors.c 772
0x77a94178
16 PyErr_SetFromErrnoWithFilenameObject errors.c 750
0x77a942ce
17 posix_path_object_errorposixmodule.c1709
0x77ae4201
18 path_object_error posixmodule.c1719
0x77ae420f
19 path_error posixmodule.c1737
0x77ae4221
20 posix_do_stat posixmodule.c2565
0x77af42f7
21 os_stat_impl posixmodule.c2897
0x77af4491
22 os_statposixmodule.c.h 71
0x77af45e5
23 cfunction_vectorcall_FASTCALL_KEYWORDS methodobject.c 443
0x779d2ece
24 _PyObject_VectorcallTstate pycore_call.h92
0x7798b579
25 PyObject_Vectorcallcall.c 299
0x7798b65e
26 _PyEval_EvalFrameDefault ceval.c 4772
0x77a75453
27 _PyEval_EvalFrame pycore_ceval.h 73
0x77a79887
28 _PyEval_Vector ceval.c 6435
0x77a799b1
29 _PyFunction_Vectorcall call.c 393
0x7798b215
30 _PyObject_VectorcallTstate pycore_call.h92
0x7798b579
31 PyObject_CallOneArgcall.c 376
0x7798b6cd
32 call_show_warning _warnings.c 609
0x77a31d24
33 warn_explicit _warnings.c 746
0x77a3223b
34 do_warn_warnings.c 947
0x77a3236f
35 warn_unicode _warnings.c 1106
0x77a32414
36 _PyErr_WarnFormatV _warnings.c 1126
0x77a32481
37 PyErr_ResourceWarning _warnings.c 1177
0x77a32d38
38 sock_finalize socketmodule.c 5073
0x76840cf3
39 PyObject_CallFinalizer object.c 208
0x779d6449
40 PyObject_CallFinalizerFromDealloc object.c 226
0x779d75ac
41 subtype_dealloctypeobject.c 1382
0x779ecf81
42 _Py_Deallocobject.c 2390
0x779d6c38
43 Py_DECREF object.h 527
0x779c116e
44 Py_XDECREF object.h 602
0x779c118d
45 free_keys_object dictobject.c 664
0x779c120a
46 dictkeys_decrefdictobject.c 324
0x779c130f
47 dict_dealloc dictobject.c 2378
0x779c2eeb
48 _Py_Deallocobject.c 2390
0x779d6c38
49 Py_DECREF object.h 5
Re: Embedding Python crash on PyTuple_New
On 24-11-2021 01:46, MRAB wrote:
On 2021-11-23 20:25, Arnaud Loonstra wrote:
On 23-11-2021 18:31, MRAB wrote:
On 2021-11-23 16:04, Arnaud Loonstra wrote:
On 23-11-2021 16:37, MRAB wrote:
On 2021-11-23 15:17, MRAB wrote:
On 2021-11-23 14:44, Arnaud Loonstra wrote:
On 23-11-2021 15:34, MRAB wrote:
On 2021-11-23 12:07, Arnaud Loonstra wrote:
Hi,
I've got Python embedded successfully in a program up until now
as I'm
now running into weird GC related segfaults. I'm currently
trying to
debug this but my understanding of CPython limits me here.
I'm creating a Tuple in C but it crashes on creating it after a
while.
It doesn't make sense which makes me wonder something else must be
happening? Could be it just crashes here because the GC is
cleaning up
stuff completely unrelated to the allocation of the new tuple?
How can I
troubleshoot this?
I've got CPython compiled with --with-valgrind --without-pymalloc
--with-pydebug
In C I'm creating a tuple with the following method:
static PyObject *
s_py_zosc_tuple(pythonactor_t *self, zosc_t *oscmsg)
{
assert(self);
assert(oscmsg);
char *format = zosc_format(oscmsg);
PyObject *rettuple = PyTuple_New((Py_ssize_t)
strlen(format) );
It segfaults here (frame 16) after 320 times (consistently)
1 __GI_raise raise.c 49 0x772c4e71
2 __GI_abort abort.c 79 0x772ae536
3 fatal_error pylifecycle.c 2183 0x77d84b4f
4 Py_FatalError pylifecycle.c 2193 0x77d878b2
5 _PyObject_AssertFailed object.c 2200 0x77c93cf2
6 visit_decref gcmodule.c 378 0x77dadfd5
7 tupletraverse tupleobject.c 623 0x77ca3e81
8 subtract_refs gcmodule.c 406 0x77dad340
9 collect gcmodule.c 1054 0x77dae838
10 collect_with_callback gcmodule.c 1240 0x77daf17b
11 collect_generations gcmodule.c 1262 0x77daf3f6
12 _PyObject_GC_Alloc gcmodule.c 1977 0x77daf4f2
13 _PyObject_GC_Malloc gcmodule.c 1987 0x77dafebc
14 _PyObject_GC_NewVar gcmodule.c 2016 0x77daffa5
15 PyTuple_New tupleobject.c 118 0x77ca4da7
16 s_py_zosc_tuple pythonactor.c 366 0x5568cc82
17 pythonactor_socket pythonactor.c 664 0x5568dac7
18 pythonactor_handle_msg pythonactor.c 862 0x5568e472
19 pythonactor_handler pythonactor.c 828 0x5568e2e2
20 sphactor_actor_run sphactor_actor.c 855 0x558cb268
...
Any pointer really appreciated.
[snip]
Basically, yes, but I won't be surprised if it was due to too few
INCREFs or too many DECREFs somewhere.
https://github.com/hku-ect/gazebosc/blob/505b30c46bf3f78d188c3f575c80e294d3db7e5d/Actors/pythonactor.c#L286
Incidentally, in s_py_zosc_tuple, you're not doing "assert(rc == 0);"
after "after zosc_pop_float" or "zosc_pop_double".
Thanks for those pointers! I think your intuition is right. I might have
found the bugger. In s_py_zosc I call Py_DECREF on pAddress and pData.
However they are acquired by PyTuple_GetItem which returns a borrowed
reference. I think pAddress and pData are then also 'decrefed' when the
pReturn tuple which contains pAddress and pData is 'decrefed'?
Yes, members of a container are DECREFed when the container is destroyed.
It's bad practice for a function to DECREF its arguments unless the
function's sole purpose is cleanup because the function won't know where
the arguments came from.
I'm finding it out now. What strikes me was how hard it was to debug
this. I think it was caused because I INCREFed the return object. I
guess I did that to workaround the wrong DECREF data in the return
object. However that caused a hell to debug. I'm really curious what the
best practices are for debugging embedded CPython.
Thanks big time for your feedback!
--
https://mail.python.org/mailman/listinfo/python-list
Re: Embedding Python crash on PyTuple_New
On 23-11-2021 18:31, MRAB wrote:
On 2021-11-23 16:04, Arnaud Loonstra wrote:
On 23-11-2021 16:37, MRAB wrote:
On 2021-11-23 15:17, MRAB wrote:
On 2021-11-23 14:44, Arnaud Loonstra wrote:
On 23-11-2021 15:34, MRAB wrote:
On 2021-11-23 12:07, Arnaud Loonstra wrote:
Hi,
I've got Python embedded successfully in a program up until now
as I'm
now running into weird GC related segfaults. I'm currently trying to
debug this but my understanding of CPython limits me here.
I'm creating a Tuple in C but it crashes on creating it after a
while.
It doesn't make sense which makes me wonder something else must be
happening? Could be it just crashes here because the GC is
cleaning up
stuff completely unrelated to the allocation of the new tuple?
How can I
troubleshoot this?
I've got CPython compiled with --with-valgrind --without-pymalloc
--with-pydebug
In C I'm creating a tuple with the following method:
static PyObject *
s_py_zosc_tuple(pythonactor_t *self, zosc_t *oscmsg)
{
assert(self);
assert(oscmsg);
char *format = zosc_format(oscmsg);
PyObject *rettuple = PyTuple_New((Py_ssize_t)
strlen(format) );
It segfaults here (frame 16) after 320 times (consistently)
1 __GI_raise raise.c 49 0x772c4e71
2 __GI_abort abort.c 79 0x772ae536
3 fatal_error pylifecycle.c 2183 0x77d84b4f
4 Py_FatalError pylifecycle.c 2193 0x77d878b2
5 _PyObject_AssertFailed object.c 2200 0x77c93cf2
6 visit_decref gcmodule.c 378 0x77dadfd5
7 tupletraverse tupleobject.c 623 0x77ca3e81
8 subtract_refs gcmodule.c 406 0x77dad340
9 collect gcmodule.c 1054 0x77dae838
10 collect_with_callback gcmodule.c 1240 0x77daf17b
11 collect_generations gcmodule.c 1262 0x77daf3f6
12 _PyObject_GC_Alloc gcmodule.c 1977 0x77daf4f2
13 _PyObject_GC_Malloc gcmodule.c 1987 0x77dafebc
14 _PyObject_GC_NewVar gcmodule.c 2016 0x77daffa5
15 PyTuple_New tupleobject.c 118 0x77ca4da7
16 s_py_zosc_tuple pythonactor.c 366 0x5568cc82
17 pythonactor_socket pythonactor.c 664 0x5568dac7
18 pythonactor_handle_msg pythonactor.c 862 0x5568e472
19 pythonactor_handler pythonactor.c 828 0x5568e2e2
20 sphactor_actor_run sphactor_actor.c 855 0x558cb268
...
Any pointer really appreciated.
You're creating a tuple that'll have the same number of members as
the length of a string? That looks strange to me.
How are you setting the tuple's members?
It's from a serialisation format called OSC. The string describes the
type of bytes, every character is a type.
I'm creating the tuple as follows:
PyObject *rettuple = PyTuple_New((Py_ssize_t) strlen(format) );
Then I iterate the OSC message using the format string, (just showing
handling an int (i))
char type = '0';
Py_ssize_t pos = 0;
const void *data = zosc_first(oscmsg, &type);
while(data)
{
switch (type)
{
case('i'):
{
int32_t val = 9;
int rc = zosc_pop_int32(oscmsg, &val);
assert(rc == 0);
PyObject *o = PyLong_FromLong((long)val);
assert( o );
rc = PyTuple_SetItem(rettuple, pos, o);
assert(rc == 0);
break;
}
Full code is here:
https://github.com/hku-ect/gazebosc/blob/822452dfa27201db274d37ce09e835d98fe500b2/Actors/pythonactor.c#L360
Looking at that code, you have:
PyObject *o = Py_BuildValue("s#", str, 1);
what I'd check is the type of the 1 that you're passing. Wouldn't the
compiler assume that it's an int?
The format string tells the function to expect a Py_ssize_t, but how
would the compiler know that?
Looking at https://www.mankier.com/3/zosc, it says for 'T' and 'F'
"(no value required)", but you're doing:
int rc = zosc_pop_bool(oscmsg, &bl);
If no value is required, is there a bool there to be popped?
The value is only required to set a user provided boolean to the value
in the message. There's no boolean value encoded in the message, only
the T and F in the format string.
With regards to the Py_BuildValue("s#", str, 1);, that's a valid point.
I'll fix that. However in the segfaults I'm testing that code is not
touched.
You can use "C" as a format string for Py_BuildValue to convert a C int
representing a character to a Python string.
I'm now testing different parts of the code to see if it runs stable.
I've found it runs stable if I do not process the returned tuple.
PyObject *pReturn = PyObject_CallMethod(self-&
Re: Embedding Python crash on PyTuple_New
On 23-11-2021 16:37, MRAB wrote:
On 2021-11-23 15:17, MRAB wrote:
On 2021-11-23 14:44, Arnaud Loonstra wrote:
On 23-11-2021 15:34, MRAB wrote:
On 2021-11-23 12:07, Arnaud Loonstra wrote:
Hi,
I've got Python embedded successfully in a program up until now as I'm
now running into weird GC related segfaults. I'm currently trying to
debug this but my understanding of CPython limits me here.
I'm creating a Tuple in C but it crashes on creating it after a while.
It doesn't make sense which makes me wonder something else must be
happening? Could be it just crashes here because the GC is cleaning up
stuff completely unrelated to the allocation of the new tuple? How
can I
troubleshoot this?
I've got CPython compiled with --with-valgrind --without-pymalloc
--with-pydebug
In C I'm creating a tuple with the following method:
static PyObject *
s_py_zosc_tuple(pythonactor_t *self, zosc_t *oscmsg)
{
assert(self);
assert(oscmsg);
char *format = zosc_format(oscmsg);
PyObject *rettuple = PyTuple_New((Py_ssize_t) strlen(format) );
It segfaults here (frame 16) after 320 times (consistently)
1 __GI_raise raise.c 49 0x772c4e71
2 __GI_abort abort.c 79 0x772ae536
3 fatal_error pylifecycle.c 2183 0x77d84b4f
4 Py_FatalError pylifecycle.c 2193 0x77d878b2
5 _PyObject_AssertFailed object.c 2200 0x77c93cf2
6 visit_decref gcmodule.c 378 0x77dadfd5
7 tupletraverse tupleobject.c 623 0x77ca3e81
8 subtract_refs gcmodule.c 406 0x77dad340
9 collect gcmodule.c 1054 0x77dae838
10 collect_with_callback gcmodule.c 1240 0x77daf17b
11 collect_generations gcmodule.c 1262 0x77daf3f6
12 _PyObject_GC_Alloc gcmodule.c 1977 0x77daf4f2
13 _PyObject_GC_Malloc gcmodule.c 1987 0x77dafebc
14 _PyObject_GC_NewVar gcmodule.c 2016 0x77daffa5
15 PyTuple_New tupleobject.c 118 0x77ca4da7
16 s_py_zosc_tuple pythonactor.c 366 0x5568cc82
17 pythonactor_socket pythonactor.c 664 0x5568dac7
18 pythonactor_handle_msg pythonactor.c 862 0x5568e472
19 pythonactor_handler pythonactor.c 828 0x5568e2e2
20 sphactor_actor_run sphactor_actor.c 855 0x558cb268
...
Any pointer really appreciated.
You're creating a tuple that'll have the same number of members as
the length of a string? That looks strange to me.
How are you setting the tuple's members?
It's from a serialisation format called OSC. The string describes the
type of bytes, every character is a type.
I'm creating the tuple as follows:
PyObject *rettuple = PyTuple_New((Py_ssize_t) strlen(format) );
Then I iterate the OSC message using the format string, (just showing
handling an int (i))
char type = '0';
Py_ssize_t pos = 0;
const void *data = zosc_first(oscmsg, &type);
while(data)
{
switch (type)
{
case('i'):
{
int32_t val = 9;
int rc = zosc_pop_int32(oscmsg, &val);
assert(rc == 0);
PyObject *o = PyLong_FromLong((long)val);
assert( o );
rc = PyTuple_SetItem(rettuple, pos, o);
assert(rc == 0);
break;
}
Full code is here:
https://github.com/hku-ect/gazebosc/blob/822452dfa27201db274d37ce09e835d98fe500b2/Actors/pythonactor.c#L360
Looking at that code, you have:
PyObject *o = Py_BuildValue("s#", str, 1);
what I'd check is the type of the 1 that you're passing. Wouldn't the
compiler assume that it's an int?
The format string tells the function to expect a Py_ssize_t, but how
would the compiler know that?
Looking at https://www.mankier.com/3/zosc, it says for 'T' and 'F' "(no
value required)", but you're doing:
int rc = zosc_pop_bool(oscmsg, &bl);
If no value is required, is there a bool there to be popped?
The value is only required to set a user provided boolean to the value
in the message. There's no boolean value encoded in the message, only
the T and F in the format string.
With regards to the Py_BuildValue("s#", str, 1);, that's a valid point.
I'll fix that. However in the segfaults I'm testing that code is not
touched.
I'm now testing different parts of the code to see if it runs stable.
I've found it runs stable if I do not process the returned tuple.
PyObject *pReturn = PyObject_CallMethod(self->pyinstance,
"handleSocket", "sOsss",
oscaddress,
py_osctuple,
ev->type, ev->name, strdup(ev->u
Re: Embedding Python crash on PyTuple_New
On 23-11-2021 15:34, MRAB wrote:
On 2021-11-23 12:07, Arnaud Loonstra wrote:
Hi,
I've got Python embedded successfully in a program up until now as I'm
now running into weird GC related segfaults. I'm currently trying to
debug this but my understanding of CPython limits me here.
I'm creating a Tuple in C but it crashes on creating it after a while.
It doesn't make sense which makes me wonder something else must be
happening? Could be it just crashes here because the GC is cleaning up
stuff completely unrelated to the allocation of the new tuple? How can I
troubleshoot this?
I've got CPython compiled with --with-valgrind --without-pymalloc
--with-pydebug
In C I'm creating a tuple with the following method:
static PyObject *
s_py_zosc_tuple(pythonactor_t *self, zosc_t *oscmsg)
{
assert(self);
assert(oscmsg);
char *format = zosc_format(oscmsg);
PyObject *rettuple = PyTuple_New((Py_ssize_t) strlen(format) );
It segfaults here (frame 16) after 320 times (consistently)
1 __GI_raise raise.c 49 0x772c4e71
2 __GI_abort abort.c 79 0x772ae536
3 fatal_error pylifecycle.c 2183 0x77d84b4f
4 Py_FatalError pylifecycle.c 2193 0x77d878b2
5 _PyObject_AssertFailed object.c 2200 0x77c93cf2
6 visit_decref gcmodule.c 378 0x77dadfd5
7 tupletraverse tupleobject.c 623 0x77ca3e81
8 subtract_refs gcmodule.c 406 0x77dad340
9 collect gcmodule.c 1054 0x77dae838
10 collect_with_callback gcmodule.c 1240 0x77daf17b
11 collect_generations gcmodule.c 1262 0x77daf3f6
12 _PyObject_GC_Alloc gcmodule.c 1977 0x77daf4f2
13 _PyObject_GC_Malloc gcmodule.c 1987 0x77dafebc
14 _PyObject_GC_NewVar gcmodule.c 2016 0x77daffa5
15 PyTuple_New tupleobject.c 118 0x77ca4da7
16 s_py_zosc_tuple pythonactor.c 366 0x5568cc82
17 pythonactor_socket pythonactor.c 664 0x5568dac7
18 pythonactor_handle_msg pythonactor.c 862 0x5568e472
19 pythonactor_handler pythonactor.c 828 0x5568e2e2
20 sphactor_actor_run sphactor_actor.c 855 0x558cb268
...
Any pointer really appreciated.
You're creating a tuple that'll have the same number of members as the
length of a string? That looks strange to me.
How are you setting the tuple's members?
It's from a serialisation format called OSC. The string describes the
type of bytes, every character is a type.
I'm creating the tuple as follows:
PyObject *rettuple = PyTuple_New((Py_ssize_t) strlen(format) );
Then I iterate the OSC message using the format string, (just showing
handling an int (i))
char type = '0';
Py_ssize_t pos = 0;
const void *data = zosc_first(oscmsg, &type);
while(data)
{
switch (type)
{
case('i'):
{
int32_t val = 9;
int rc = zosc_pop_int32(oscmsg, &val);
assert(rc == 0);
PyObject *o = PyLong_FromLong((long)val);
assert( o );
rc = PyTuple_SetItem(rettuple, pos, o);
assert(rc == 0);
break;
}
Full code is here:
https://github.com/hku-ect/gazebosc/blob/822452dfa27201db274d37ce09e835d98fe500b2/Actors/pythonactor.c#L360
--
https://mail.python.org/mailman/listinfo/python-list
Re: Embedding Python crash on PyTuple_New
On 23-11-2021 13:07, Arnaud Loonstra wrote:
Hi,
I've got Python embedded successfully in a program up until now as I'm
now running into weird GC related segfaults. I'm currently trying to
debug this but my understanding of CPython limits me here.
I'm creating a Tuple in C but it crashes on creating it after a while.
It doesn't make sense which makes me wonder something else must be
happening? Could be it just crashes here because the GC is cleaning up
stuff completely unrelated to the allocation of the new tuple? How can I
troubleshoot this?
I've got CPython compiled with --with-valgrind --without-pymalloc
--with-pydebug
In C I'm creating a tuple with the following method:
static PyObject *
s_py_zosc_tuple(pythonactor_t *self, zosc_t *oscmsg)
{
assert(self);
assert(oscmsg);
char *format = zosc_format(oscmsg);
PyObject *rettuple = PyTuple_New((Py_ssize_t) strlen(format) );
It segfaults here (frame 16) after 320 times (consistently)
1 __GI_raise raise.c 49 0x772c4e71
2 __GI_abort abort.c 79 0x772ae536
3 fatal_error pylifecycle.c 2183 0x77d84b4f
4 Py_FatalError pylifecycle.c 2193 0x77d878b2
5 _PyObject_AssertFailed object.c 2200 0x77c93cf2
6 visit_decref gcmodule.c 378 0x77dadfd5
7 tupletraverse tupleobject.c 623 0x77ca3e81
8 subtract_refs gcmodule.c 406 0x77dad340
9 collect gcmodule.c 1054 0x77dae838
10 collect_with_callback gcmodule.c 1240 0x77daf17b
11 collect_generations gcmodule.c 1262 0x77daf3f6
12 _PyObject_GC_Alloc gcmodule.c 1977 0x77daf4f2
13 _PyObject_GC_Malloc gcmodule.c 1987 0x77dafebc
14 _PyObject_GC_NewVar gcmodule.c 2016 0x77daffa5
15 PyTuple_New tupleobject.c 118 0x77ca4da7
16 s_py_zosc_tuple pythonactor.c 366 0x5568cc82
17 pythonactor_socket pythonactor.c 664 0x5568dac7
18 pythonactor_handle_msg pythonactor.c 862 0x5568e472
19 pythonactor_handler pythonactor.c 828 0x5568e2e2
20 sphactor_actor_run sphactor_actor.c 855 0x558cb268
...
Any pointer really appreciated.
I've found enabling PYTHONTRACEMALLOC=1 in the environment gives me a
pointer to where to offending block was allocated.
I'm calling this method from C:
18def handleSocket(self, addr, data, type, name, uuid):
19if addr == "/pulse":
20self.lampval += 1
21return (addr, [0,0])
Modules/gcmodule.c:108: gc_decref: Assertion "gc_get_refs(g) > 0"
failed: refcount is too small
Memory block allocated at (most recent call first):
File "/home/arnaud/src/build-gazebosc-Desktop-Debug/bin/lampen.py",
line 21
object address : 0x7fffd81154c0
object refcount : 1
object type : 0x77f3df20
object type name: list
object repr : [117, 0]
Fatal Python error: _PyObject_AssertFailed
Python runtime state: initialized
Current thread 0x72481640 (most recent call first):
File "/home/arnaud/src/build-gazebosc-Desktop-Debug/bin/lampen.py",
line 21 in handleSocket
Thread 0x75d288c0 (most recent call first):
Now it clearly says the refcount is 1 so I'm puzzling what it means?
Rg,
Arnaud
--
https://mail.python.org/mailman/listinfo/python-list
Embedding Python crash on PyTuple_New
Hi,
I've got Python embedded successfully in a program up until now as I'm
now running into weird GC related segfaults. I'm currently trying to
debug this but my understanding of CPython limits me here.
I'm creating a Tuple in C but it crashes on creating it after a while.
It doesn't make sense which makes me wonder something else must be
happening? Could be it just crashes here because the GC is cleaning up
stuff completely unrelated to the allocation of the new tuple? How can I
troubleshoot this?
I've got CPython compiled with --with-valgrind --without-pymalloc
--with-pydebug
In C I'm creating a tuple with the following method:
static PyObject *
s_py_zosc_tuple(pythonactor_t *self, zosc_t *oscmsg)
{
assert(self);
assert(oscmsg);
char *format = zosc_format(oscmsg);
PyObject *rettuple = PyTuple_New((Py_ssize_t) strlen(format) );
It segfaults here (frame 16) after 320 times (consistently)
1 __GI_raise raise.c 49 0x772c4e71
2 __GI_abort abort.c 79 0x772ae536
3 fatal_errorpylifecycle.c2183 0x77d84b4f
4 Py_FatalError pylifecycle.c2193 0x77d878b2
5 _PyObject_AssertFailed object.c 2200 0x77c93cf2
6 visit_decref gcmodule.c 378 0x77dadfd5
7 tupletraverse tupleobject.c623 0x77ca3e81
8 subtract_refs gcmodule.c 406 0x77dad340
9 collectgcmodule.c 1054 0x77dae838
10 collect_with_callback gcmodule.c 1240 0x77daf17b
11 collect_generationsgcmodule.c 1262 0x77daf3f6
12 _PyObject_GC_Alloc gcmodule.c 1977 0x77daf4f2
13 _PyObject_GC_Mallocgcmodule.c 1987 0x77dafebc
14 _PyObject_GC_NewVargcmodule.c 2016 0x77daffa5
15 PyTuple_Newtupleobject.c118 0x77ca4da7
16 s_py_zosc_tuplepythonactor.c366 0x5568cc82
17 pythonactor_socket pythonactor.c664 0x5568dac7
18 pythonactor_handle_msg pythonactor.c862 0x5568e472
19 pythonactor_handlerpythonactor.c828 0x5568e2e2
20 sphactor_actor_run sphactor_actor.c 855 0x558cb268
...
Any pointer really appreciated.
Rg,
Arnaud
--
https://mail.python.org/mailman/listinfo/python-list
Re: return a ctypes object to C
On 31-10-2019 15:39, Arnaud Loonstra wrote:
On 31-10-2019 14:44, Thomas Jollans wrote:
On 31/10/2019 14.13, Arnaud Loonstra wrote:
On 30-10-2019 09:32, Arnaud Loonstra wrote:
Hi all,
I'm trying to wrap my head around the ctypes API. I have a C
structure I wish to create in Python and then return from python to C.
So a python method is called from C and needs to return an object
which we then process in C again.
I have a binding to access and create the C methods and structures so
in Python I can call the Zmsg() constructor. I now need to return this.
My python test method is simply:
def actor_test( *args, **kwargs):
print("test")
msg = Zmsg()
frame = Zframe(b"Hello", 5)
msg.prepend(frame)
return msg
the method is called from C as follows:
PyObject *pReturn = PyObject_CallObject(pFunc, NULL);
This correctly calls the method. However the returned object is of
course a PyObject*. The debugger says it's
"" PyObject
[class] ""
[super class] ""
[meta type] ""
ob_refcnt 1 Py_ssize_t
However how I can I get it back to the original C type (zmsg_t *)
Any help really appreciated.
What I've found so far is that I can return the address of the ctypes
object.
msg = Zmsg()
frame = Zframe(b"Hello", 5)
msg.prepend(frame)
return addressof(msg._as_parameter_.contents)
In C I can then cast it back to the original type.
PyObject *pReturn = PyObject_CallObject(pFunc, NULL);
assert(pReturn);
long bla = PyLong_AsLong(pReturn);
zmsg_t* test = (zmsg_t *)bla;
assert(test);
char *hello = zmsg_popstr(test);
assert(hello);
assert(streq(hello, "Hello"));
This works, I'm not sure if this is the right way. It also creates a
complicated setup with the garbage collector.
Anybody better ideas?
You've already got a complicated setup with your ctypes objects...
If you're using the Python C API anyway, why would you use ctypes at
all? You can create custom Python types in C to wrap your C pointers.
Alternatively: if you're using ctypes anyway, why use the Python C API
at all? You can create C function pointers from Python functions with
ctypes.
If you're mixing two different ways of interfacing Python and C, the
result will ALWAYS be messy. Better to stick to one. Personally, I
prefer cffi or cython depending on the situation, as I find them clearer
and easier to use than ctypes. Using the Python C API directly is
usually the hardest to understand and the easiest to get wrong.
-- Thomas
Hi Thomas,
I have an engine running which can call handlers. These handlers return
a zmsg_t (a message) which the engine then can process.
In this engine we have Python embedded and I want to use a Python method
as a handler. To embed Python we need to use the Python C API. To
construct a zmsg_t type in Python we need to call the corresponding C
method and we use ctypes to do that.
I'm using a ctypes binding because it already exists, it's here:
https://github.com/zeromq/czmq/blob/d6283985ba52fd8c3f8fbdc7cd5c08372ff69ca1/bindings/python/czmq/_czmq_ctypes.py#L4392
I know I can use cffi for example but that's IMHO just a ctypes
alternative.
Are you saying it's better to create some approach to create a zmsg_t
using the Python C API?
Please enlighten me if this should be done differently.
Rg,
Arnaud
Ok, you're reply triggered me to do some more reading into the C-API.
I've found I can do it all using the C-API. Especially this guide was
very helpful for my situation:
https://docs.python.org/3/extending/newtypes_tutorial.html
Thnx,
Arnaud
--
https://mail.python.org/mailman/listinfo/python-list
Re: return a ctypes object to C
On 31-10-2019 14:44, Thomas Jollans wrote:
On 31/10/2019 14.13, Arnaud Loonstra wrote:
On 30-10-2019 09:32, Arnaud Loonstra wrote:
Hi all,
I'm trying to wrap my head around the ctypes API. I have a C
structure I wish to create in Python and then return from python to C.
So a python method is called from C and needs to return an object
which we then process in C again.
I have a binding to access and create the C methods and structures so
in Python I can call the Zmsg() constructor. I now need to return this.
My python test method is simply:
def actor_test( *args, **kwargs):
print("test")
msg = Zmsg()
frame = Zframe(b"Hello", 5)
msg.prepend(frame)
return msg
the method is called from C as follows:
PyObject *pReturn = PyObject_CallObject(pFunc, NULL);
This correctly calls the method. However the returned object is of
course a PyObject*. The debugger says it's
"" PyObject
[class] ""
[super class] ""
[meta type] ""
ob_refcnt 1 Py_ssize_t
However how I can I get it back to the original C type (zmsg_t *)
Any help really appreciated.
What I've found so far is that I can return the address of the ctypes
object.
msg = Zmsg()
frame = Zframe(b"Hello", 5)
msg.prepend(frame)
return addressof(msg._as_parameter_.contents)
In C I can then cast it back to the original type.
PyObject *pReturn = PyObject_CallObject(pFunc, NULL);
assert(pReturn);
long bla = PyLong_AsLong(pReturn);
zmsg_t* test = (zmsg_t *)bla;
assert(test);
char *hello = zmsg_popstr(test);
assert(hello);
assert(streq(hello, "Hello"));
This works, I'm not sure if this is the right way. It also creates a
complicated setup with the garbage collector.
Anybody better ideas?
You've already got a complicated setup with your ctypes objects...
If you're using the Python C API anyway, why would you use ctypes at
all? You can create custom Python types in C to wrap your C pointers.
Alternatively: if you're using ctypes anyway, why use the Python C API
at all? You can create C function pointers from Python functions with
ctypes.
If you're mixing two different ways of interfacing Python and C, the
result will ALWAYS be messy. Better to stick to one. Personally, I
prefer cffi or cython depending on the situation, as I find them clearer
and easier to use than ctypes. Using the Python C API directly is
usually the hardest to understand and the easiest to get wrong.
-- Thomas
Hi Thomas,
I have an engine running which can call handlers. These handlers return
a zmsg_t (a message) which the engine then can process.
In this engine we have Python embedded and I want to use a Python method
as a handler. To embed Python we need to use the Python C API. To
construct a zmsg_t type in Python we need to call the corresponding C
method and we use ctypes to do that.
I'm using a ctypes binding because it already exists, it's here:
https://github.com/zeromq/czmq/blob/d6283985ba52fd8c3f8fbdc7cd5c08372ff69ca1/bindings/python/czmq/_czmq_ctypes.py#L4392
I know I can use cffi for example but that's IMHO just a ctypes
alternative.
Are you saying it's better to create some approach to create a zmsg_t
using the Python C API?
Please enlighten me if this should be done differently.
Rg,
Arnaud
--
https://mail.python.org/mailman/listinfo/python-list
Re: return a ctypes object to C
On 30-10-2019 09:32, Arnaud Loonstra wrote:
Hi all,
I'm trying to wrap my head around the ctypes API. I have a C structure I
wish to create in Python and then return from python to C.
So a python method is called from C and needs to return an object which
we then process in C again.
I have a binding to access and create the C methods and structures so in
Python I can call the Zmsg() constructor. I now need to return this.
My python test method is simply:
def actor_test( *args, **kwargs):
print("test")
msg = Zmsg()
frame = Zframe(b"Hello", 5)
msg.prepend(frame)
return msg
the method is called from C as follows:
PyObject *pReturn = PyObject_CallObject(pFunc, NULL);
This correctly calls the method. However the returned object is of
course a PyObject*. The debugger says it's
"" PyObject
[class] ""
[super class] ""
[meta type] ""
ob_refcnt 1 Py_ssize_t
However how I can I get it back to the original C type (zmsg_t *)
Any help really appreciated.
What I've found so far is that I can return the address of the ctypes
object.
msg = Zmsg()
frame = Zframe(b"Hello", 5)
msg.prepend(frame)
return addressof(msg._as_parameter_.contents)
In C I can then cast it back to the original type.
PyObject *pReturn = PyObject_CallObject(pFunc, NULL);
assert(pReturn);
long bla = PyLong_AsLong(pReturn);
zmsg_t* test = (zmsg_t *)bla;
assert(test);
char *hello = zmsg_popstr(test);
assert(hello);
assert(streq(hello, "Hello"));
This works, I'm not sure if this is the right way. It also creates a
complicated setup with the garbage collector.
Anybody better ideas?
Rg,
Arnaud
--
https://mail.python.org/mailman/listinfo/python-list
return a ctypes object to C
Hi all,
I'm trying to wrap my head around the ctypes API. I have a C structure I
wish to create in Python and then return from python to C.
So a python method is called from C and needs to return an object which
we then process in C again.
I have a binding to access and create the C methods and structures so in
Python I can call the Zmsg() constructor. I now need to return this.
My python test method is simply:
def actor_test( *args, **kwargs):
print("test")
msg = Zmsg()
frame = Zframe(b"Hello", 5)
msg.prepend(frame)
return msg
the method is called from C as follows:
PyObject *pReturn = PyObject_CallObject(pFunc, NULL);
This correctly calls the method. However the returned object is of
course a PyObject*. The debugger says it's
"" PyObject
[class] ""
[super class] ""
[meta type] ""
ob_refcnt 1 Py_ssize_t
However how I can I get it back to the original C type (zmsg_t *)
Any help really appreciated.
Rg,
Arnaud
--
https://mail.python.org/mailman/listinfo/python-list
Embedding Python (3.7) on OSX crash on Py_Initialize when run as bundle
Hi, I'm trying to wrap my head around an OSX app which has python embedded. This works fine if I run the build on the same machine as it is build on. However when using Travis to build the app I can only get it to run from the terminal. A double click on the appdir (or using open) results in a crash. trace: 0 libsystem_kernel.dylib 0x7fffb9d6cd42 __pthread_kill + 10 1 libsystem_pthread.dylib 0x7fffb9e5a457 pthread_kill + 90 2 libsystem_c.dylib 0x7fffb9cd24bb __abort + 140 3 libsystem_c.dylib 0x7fffb9cd242f abort + 144 4 com.your-company-name.www 0x000108c6674e fatal_error + 606 5 com.your-company-name.www 0x000108c65b3a _Py_FatalInitError + 26 6 com.your-company-name.www 0x000108c65bbe Py_Initialize + 126 I can't figure out what is going wrong. I can only guess that's there's something preventing it to access the python files? The build log on travis is here: https://travis-ci.org/hku-ect/NatNet2OSCbridge/jobs/499348594 It's artefact is here: https://pong.hku.nl/~buildbot/nightly/NatNet2OSCbridge_osx_7ceb.zip Python is build with the following flags: ./configure --prefix $HOME/openFrameworks/apps/devApps/$APPNAME/bin/python --disable-shared --with-openssl=$(brew --prefix openssl); Anybody any pointers or advice? Rg, Arnaud -- https://mail.python.org/mailman/listinfo/python-list
