Re: PyGILState_Release called twice in embedded application
> On 23 Mar 2023, at 14:34, Arnaud Loonstra wrote:
>
> On 23-03-2023 13:33, Barry Scott wrote:
On 23 Mar 2023, at 08:46, Arnaud Loonstra wrote:
>>>
>>> Hi all,
>>>
>>> I'm running in a crash due to a ResourceWarning (some socket is not closed
>>> in a used module) after calling PyGILState_Release.
>>>
>>> I'm running Python in a native thread (so a thread created by C not
>>> Python). I'm acquiring the GIL through PyGILState_Ensure before doing any
>>> CPYthon calls and releasing when finished using PyGILState_Release.
>>>
>>> This has worked fine. But now I'm using a python module (openai) which
>>> doesn't close its socket correctly which results in a ResourceWarning which
>>> triggers an assert.
>>>
>>> In the backtrace (below) I can see PyGILState_Release is called again. (7)
>>> while I've already called it (126).
>>>
>>> I can make the crash go away by adding
>>>
>>> import warnings
>>> warnings.simplefilter("ignore", ResourceWarning)
>>>
>>> to my python code. But I'd rather prevent this from happening in the first
>>> place.
>>>
>>> Any suggestion very welcomed cause I'm puzzled.
>> What 3rd party C extension are you running with?
>> I'd be surprised if the cpython code was the issue.
>> Barry
>
> I'm not using any 3rd party extension myself. But the issue is caused by the
> openai module using the requests module which is not closing sockets:
>
> https://github.com/openai/openai-python/issues/140
>
> I'm not aware what C extensions they might use.
In gdb use info shared to see what is loaded.
Leaving a socket open will not cause a logic error as you are encountering.
I expect a bug in C/C++ cod3 that is not part of python itself.
You are using 3rd party code, requests and openai that you name.
Barry
>
> Btw, I've tested this with python 3.8 and 3.11.2.
>
> Rg,
>
> Arnaud
>
--
https://mail.python.org/mailman/listinfo/python-list
Re: PyGILState_Release called twice in embedded application
On 23-03-2023 13:33, Barry Scott wrote:
On 23 Mar 2023, at 08:46, Arnaud Loonstra wrote:
Hi all,
I'm running in a crash due to a ResourceWarning (some socket is not closed in a
used module) after calling PyGILState_Release.
I'm running Python in a native thread (so a thread created by C not Python).
I'm acquiring the GIL through PyGILState_Ensure before doing any CPYthon calls
and releasing when finished using PyGILState_Release.
This has worked fine. But now I'm using a python module (openai) which doesn't
close its socket correctly which results in a ResourceWarning which triggers an
assert.
In the backtrace (below) I can see PyGILState_Release is called again. (7)
while I've already called it (126).
I can make the crash go away by adding
import warnings
warnings.simplefilter("ignore", ResourceWarning)
to my python code. But I'd rather prevent this from happening in the first
place.
Any suggestion very welcomed cause I'm puzzled.
What 3rd party C extension are you running with?
I'd be surprised if the cpython code was the issue.
Barry
I'm not using any 3rd party extension myself. But the issue is caused by
the openai module using the requests module which is not closing sockets:
https://github.com/openai/openai-python/issues/140
I'm not aware what C extensions they might use.
Btw, I've tested this with python 3.8 and 3.11.2.
Rg,
Arnaud
--
https://mail.python.org/mailman/listinfo/python-list
Re: PyGILState_Release called twice in embedded application
> On 23 Mar 2023, at 08:46, Arnaud Loonstra wrote:
>
> Hi all,
>
> I'm running in a crash due to a ResourceWarning (some socket is not closed in
> a used module) after calling PyGILState_Release.
>
> I'm running Python in a native thread (so a thread created by C not Python).
> I'm acquiring the GIL through PyGILState_Ensure before doing any CPYthon
> calls and releasing when finished using PyGILState_Release.
>
> This has worked fine. But now I'm using a python module (openai) which
> doesn't close its socket correctly which results in a ResourceWarning which
> triggers an assert.
>
> In the backtrace (below) I can see PyGILState_Release is called again. (7)
> while I've already called it (126).
>
> I can make the crash go away by adding
>
> import warnings
> warnings.simplefilter("ignore", ResourceWarning)
>
> to my python code. But I'd rather prevent this from happening in the first
> place.
>
> Any suggestion very welcomed cause I'm puzzled.
What 3rd party C extension are you running with?
I'd be surprised if the cpython code was the issue.
Barry
>
> Rg,
>
> Arnaud
>
> 1 __pthread_kill_implementation pthread_kill.c 44
> 0x76fcaccc
> 2 __pthread_kill_internalpthread_kill.c 78
> 0x76fcad2f
> 3 __GI_raise raise.c 26
> 0x76f7bef2
> 4 __GI_abort abort.c 79
> 0x76f66472
> 5 __assert_fail_base assert.c 92
> 0x76f66395
> 6 __GI___assert_fail assert.c 101
> 0x76f74df2
> 7 PyGILState_Release pystate.c1742
> 0x77abcf9f
> 8 tracemalloc_raw_alloc _tracemalloc.c 779
> 0x77afa803
> 9 tracemalloc_raw_malloc _tracemalloc.c 789
> 0x77afa844
> 10 PyMem_RawMallocobmalloc.c 586
> 0x779dab8d
> 11 decode_current_locale fileutils.c 472
> 0x77ada4ab
> 12 _Py_DecodeLocaleEx fileutils.c 598
> 0x77adafce
> 13 unicode_decode_locale unicodeobject.c 3970
> 0x77a13b65
> 14 PyUnicode_DecodeLocale unicodeobject.c 4012
> 0x77a18c79
> 15 PyErr_SetFromErrnoWithFilenameObjects errors.c 772
> 0x77a94178
> 16 PyErr_SetFromErrnoWithFilenameObject errors.c 750
> 0x77a942ce
> 17 posix_path_object_errorposixmodule.c1709
> 0x77ae4201
> 18 path_object_error posixmodule.c1719
> 0x77ae420f
> 19 path_error posixmodule.c1737
> 0x77ae4221
> 20 posix_do_stat posixmodule.c2565
> 0x77af42f7
> 21 os_stat_impl posixmodule.c2897
> 0x77af4491
> 22 os_statposixmodule.c.h 71
> 0x77af45e5
> 23 cfunction_vectorcall_FASTCALL_KEYWORDS methodobject.c 443
> 0x779d2ece
> 24 _PyObject_VectorcallTstate pycore_call.h92
> 0x7798b579
> 25 PyObject_Vectorcallcall.c 299
> 0x7798b65e
> 26 _PyEval_EvalFrameDefault ceval.c 4772
> 0x77a75453
> 27 _PyEval_EvalFrame pycore_ceval.h 73
> 0x77a79887
> 28 _PyEval_Vector ceval.c 6435
> 0x77a799b1
> 29 _PyFunction_Vectorcall call.c 393
> 0x7798b215
> 30 _PyObject_VectorcallTstate pycore_call.h92
> 0x7798b579
> 31 PyObject_CallOneArgcall.c 376
> 0x7798b6cd
> 32 call_show_warning _warnings.c 609
> 0x77a31d24
> 33 warn_explicit _warnings.c 746
> 0x77a3223b
> 34 do_warn_warnings.c 947
> 0x77a3236f
> 35 warn_unicode _warnings.c 1106
> 0x77a32414
> 36 _PyErr_WarnFormatV _warnings.c 1126
> 0x77a32481
> 37 PyErr_ResourceWarning _warnings.c 1177
> 0x77a32d38
> 38 sock_finalize socketmodule.c 5073
> 0x76840cf3
> 39 PyObject_CallFinalizer object.c 208
> 0x779d6449
> 40 PyObject_CallFinalizerFromDealloc object.c 226
> 0x779d75ac
> 41 subtype_dealloctypeobject.c 1382
> 0x779ecf81
> 42 _Py_Deallocobject.c 2390
> 0x779d6c38
> 43 Py_DECREF object.h 527
> 0x779c116e
> 44 Py_XDECREF object.h 602
> 0x779c118d
>
PyGILState_Release called twice in embedded application
Hi all,
I'm running in a crash due to a ResourceWarning (some socket is not
closed in a used module) after calling PyGILState_Release.
I'm running Python in a native thread (so a thread created by C not
Python). I'm acquiring the GIL through PyGILState_Ensure before doing
any CPYthon calls and releasing when finished using PyGILState_Release.
This has worked fine. But now I'm using a python module (openai) which
doesn't close its socket correctly which results in a ResourceWarning
which triggers an assert.
In the backtrace (below) I can see PyGILState_Release is called again.
(7) while I've already called it (126).
I can make the crash go away by adding
import warnings
warnings.simplefilter("ignore", ResourceWarning)
to my python code. But I'd rather prevent this from happening in the
first place.
Any suggestion very welcomed cause I'm puzzled.
Rg,
Arnaud
1 __pthread_kill_implementation pthread_kill.c 44
0x76fcaccc
2 __pthread_kill_internalpthread_kill.c 78
0x76fcad2f
3 __GI_raise raise.c 26
0x76f7bef2
4 __GI_abort abort.c 79
0x76f66472
5 __assert_fail_base assert.c 92
0x76f66395
6 __GI___assert_fail assert.c 101
0x76f74df2
7 PyGILState_Release pystate.c1742
0x77abcf9f
8 tracemalloc_raw_alloc _tracemalloc.c 779
0x77afa803
9 tracemalloc_raw_malloc _tracemalloc.c 789
0x77afa844
10 PyMem_RawMallocobmalloc.c 586
0x779dab8d
11 decode_current_locale fileutils.c 472
0x77ada4ab
12 _Py_DecodeLocaleEx fileutils.c 598
0x77adafce
13 unicode_decode_locale unicodeobject.c 3970
0x77a13b65
14 PyUnicode_DecodeLocale unicodeobject.c 4012
0x77a18c79
15 PyErr_SetFromErrnoWithFilenameObjects errors.c 772
0x77a94178
16 PyErr_SetFromErrnoWithFilenameObject errors.c 750
0x77a942ce
17 posix_path_object_errorposixmodule.c1709
0x77ae4201
18 path_object_error posixmodule.c1719
0x77ae420f
19 path_error posixmodule.c1737
0x77ae4221
20 posix_do_stat posixmodule.c2565
0x77af42f7
21 os_stat_impl posixmodule.c2897
0x77af4491
22 os_statposixmodule.c.h 71
0x77af45e5
23 cfunction_vectorcall_FASTCALL_KEYWORDS methodobject.c 443
0x779d2ece
24 _PyObject_VectorcallTstate pycore_call.h92
0x7798b579
25 PyObject_Vectorcallcall.c 299
0x7798b65e
26 _PyEval_EvalFrameDefault ceval.c 4772
0x77a75453
27 _PyEval_EvalFrame pycore_ceval.h 73
0x77a79887
28 _PyEval_Vector ceval.c 6435
0x77a799b1
29 _PyFunction_Vectorcall call.c 393
0x7798b215
30 _PyObject_VectorcallTstate pycore_call.h92
0x7798b579
31 PyObject_CallOneArgcall.c 376
0x7798b6cd
32 call_show_warning _warnings.c 609
0x77a31d24
33 warn_explicit _warnings.c 746
0x77a3223b
34 do_warn_warnings.c 947
0x77a3236f
35 warn_unicode _warnings.c 1106
0x77a32414
36 _PyErr_WarnFormatV _warnings.c 1126
0x77a32481
37 PyErr_ResourceWarning _warnings.c 1177
0x77a32d38
38 sock_finalize socketmodule.c 5073
0x76840cf3
39 PyObject_CallFinalizer object.c 208
0x779d6449
40 PyObject_CallFinalizerFromDealloc object.c 226
0x779d75ac
41 subtype_dealloctypeobject.c 1382
0x779ecf81
42 _Py_Deallocobject.c 2390
0x779d6c38
43 Py_DECREF object.h 527
0x779c116e
44 Py_XDECREF object.h 602
0x779c118d
45 free_keys_object dictobject.c 664
0x779c120a
46 dictkeys_decrefdictobject.c 324
0x779c130f
47 dict_dealloc dictobject.c 2378
0x779c2eeb
48 _Py_Deallocobject.c 2390
0x779d6c38
49 Py_DECREF object.h
