jvstein commented on issue #122:
URL:
https://github.com/apache/pulsar-client-python/issues/122#issuecomment-1561828733
Attached. For what it's worth, this is the stack from my real code that's
deadlocking not the exact example above. The fact that I'm using an avro schema
might be relevant as there would likely be some additional communication with
the schema registry. The schema itself isn't that complicated and has been used
before on the cluster.
```python
schema_file = "/path/to/my.avsc"
schema_definition = load_schema(schema_file)
schema = AvroSchema(None, schema_definition=schema_definition)
client = pulsar.Client('pulsar://my-pulsar-host:6650')
producer = client.create_producer(
topic="persistent://tenant/namespace/topic",
schema=schema,
block_if_queue_full=True,
batching_enabled=True,
batching_max_publish_delay_ms=10
)
```
<details>
<summary>thread apply all bt output</summary>
```
Thread 4 (Thread 0x7ffff48ad640 (LWP 1067857) "python"):
#0 0x00007ffff7dadfde in epoll_wait (epfd=8, events=0x7ffff48ac4f0,
maxevents=128, timeout=-1) at ../sysdeps/unix/sysv/linux/epoll_wait.c:30
#1 0x00007ffff682dad5 in boost::asio::detail::epoll_reactor::run(long,
boost::asio::detail::op_queue<boost::asio::detail::scheduler_operation>&) ()
from
/tmp/.venv/lib/python3.9/site-packages/pulsar_client.libs/libpulsar-df34d1a6.so
#2 0x00007ffff6836443 in
boost::asio::detail::scheduler::run(boost::system::error_code&) () from
/tmp/.venv/lib/python3.9/site-packages/pulsar_client.libs/libpulsar-df34d1a6.so
#3 0x00007ffff68c325b in
pulsar::ExecutorService::start()::{lambda()#1}::operator()() const () from
/tmp/.venv/lib/python3.9/site-packages/pulsar_client.libs/libpulsar-df34d1a6.so
#4 0x00007ffff6e4b650 in execute_native_thread_routine () from
/tmp/.venv/lib/python3.9/site-packages/pulsar_client.libs/libpulsar-df34d1a6.so
#5 0x00007ffff7d1cb43 in start_thread (arg=<optimized out>) at
./nptl/pthread_create.c:442
#6 0x00007ffff7daea00 in clone3 () at
../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
Thread 3 (Thread 0x7ffff50ae640 (LWP 1067856) "python"):
#0 __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0,
op=393, expected=0, futex_word=0x555555bd163c) at ./nptl/futex-internal.c:57
#1 __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0,
clockid=0, expected=0, futex_word=0x555555bd163c) at ./nptl/futex-internal.c:87
#2 __GI___futex_abstimed_wait_cancelable64
(futex_word=futex_word@entry=0x555555bd163c, expected=expected@entry=0,
clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at
./nptl/futex-internal.c:139
#3 0x00007ffff7d1bac1 in __pthread_cond_wait_common (abstime=0x0,
clockid=0, mutex=0x555555bd15d8, cond=0x555555bd1610) at
./nptl/pthread_cond_wait.c:503
#4 ___pthread_cond_wait (cond=0x555555bd1610, mutex=0x555555bd15d8) at
./nptl/pthread_cond_wait.c:627
#5 0x00007ffff68365bc in
boost::asio::detail::scheduler::run(boost::system::error_code&) () from
/tmp/.venv/lib/python3.9/site-packages/pulsar_client.libs/libpulsar-df34d1a6.so
#6 0x00007ffff6836b3a in
boost::asio::detail::posix_thread::func<boost::asio::detail::resolver_service_base::work_scheduler_runner>::run()
() from
/tmp/.venv/lib/python3.9/site-packages/pulsar_client.libs/libpulsar-df34d1a6.so
#7 0x00007ffff682cdf0 in boost_asio_detail_posix_thread_function () from
/tmp/.venv/lib/python3.9/site-packages/pulsar_client.libs/libpulsar-df34d1a6.so
#8 0x00007ffff7d1cb43 in start_thread (arg=<optimized out>) at
./nptl/pthread_create.c:442
#9 0x00007ffff7daea00 in clone3 () at
../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
Thread 2 (Thread 0x7ffff58af640 (LWP 1067855) "python"):
#0 __futex_abstimed_wait_common64 (private=1, cancel=true,
abstime=0x7ffff58ae120, op=137, expected=0, futex_word=0x555555919788
<_PyRuntime+424>) at ./nptl/futex-internal.c:57
#1 __futex_abstimed_wait_common (cancel=true, private=1,
abstime=0x7ffff58ae120, clockid=1435604832, expected=0,
futex_word=0x555555919788 <_PyRuntime+424>) at ./nptl/futex-internal.c:87
#2 __GI___futex_abstimed_wait_cancelable64
(futex_word=futex_word@entry=0x555555919788 <_PyRuntime+424>,
expected=expected@entry=0, clockid=clockid@entry=1,
abstime=abstime@entry=0x7ffff58ae120, private=private@entry=0) at
./nptl/futex-internal.c:139
#3 0x00007ffff7d1bf1b in __pthread_cond_wait_common
(abstime=0x7ffff58ae120, clockid=1, mutex=0x555555919790 <_PyRuntime+432>,
cond=0x555555919760 <_PyRuntime+384>) at ./nptl/pthread_cond_wait.c:503
#4 ___pthread_cond_timedwait64 (cond=cond@entry=0x555555919760
<_PyRuntime+384>, mutex=mutex@entry=0x555555919790 <_PyRuntime+432>,
abstime=abstime@entry=0x7ffff58ae120) at ./nptl/pthread_cond_wait.c:652
#5 0x000055555567704f in PyCOND_TIMEDWAIT (us=<optimized out>,
mut=0x555555919790 <_PyRuntime+432>, cond=0x555555919760 <_PyRuntime+384>) at
Python/condvar.h:73
#6 take_gil (tstate=tstate@entry=0x7ffff002b290) at Python/ceval_gil.h:247
#7 0x0000555555677a12 in PyEval_AcquireThread (tstate=0x7ffff002b290) at
Python/ceval.c:385
#8 0x00007ffff713f5d5 in pybind11::gil_scoped_acquire::gil_scoped_acquire()
() from
/tmp/.venv/lib/python3.9/site-packages/_pulsar.cpython-39-x86_64-linux-gnu.so
#9 0x00007ffff7199cbc in std::_Function_handler<void (pulsar::Result,
pulsar::MessageId const&), pybind11::detail::type_caster<std::function<void
(pulsar::Result, pulsar::MessageId const&)>, void>::load(pybind11::handle,
bool)::func_wrapper>::_M_manager(std::_Any_data&, std::_Any_data const&,
std::_Manager_operation) () from
/tmp/.venv/lib/python3.9/site-packages/_pulsar.cpython-39-x86_64-linux-gnu.so
#10 0x00007ffff6934cb2 in std::function<void (pulsar::Result,
pulsar::MessageId const&)>::function(std::function<void (pulsar::Result,
pulsar::MessageId const&)> const&) () from
/tmp/.venv/lib/python3.9/site-packages/pulsar_client.libs/libpulsar-df34d1a6.so
#11 0x00007ffff692a5cc in std::_Function_handler<void (pulsar::Result,
pulsar::MessageId const&), pulsar::ProducerImpl::sendAsync(pulsar::Message
const&, std::function<void (pulsar::Result, pulsar::MessageId
const&)>)::{lambda(pulsar::Result, pulsar::MessageId
const&)#1}>::_M_manager(std::_Any_data&, std::_Any_data const&,
std::_Manager_operation) () from
/tmp/.venv/lib/python3.9/site-packages/pulsar_client.libs/libpulsar-df34d1a6.so
#12 0x00007ffff68de83f in
pulsar::MessageAndCallbackBatch::createSendCallback() const () from
/tmp/.venv/lib/python3.9/site-packages/pulsar_client.libs/libpulsar-df34d1a6.so
#13 0x00007ffff67ff43d in
pulsar::BatchMessageContainerBase::createOpSendMsgHelper(pulsar::OpSendMsg&,
std::function<void (pulsar::Result)> const&, pulsar::MessageAndCallbackBatch
const&) const () from
/tmp/.venv/lib/python3.9/site-packages/pulsar_client.libs/libpulsar-df34d1a6.so
#14 0x00007ffff680075c in
pulsar::BatchMessageContainerBase::processAndClear(std::function<void
(pulsar::Result, pulsar::OpSendMsg const&)>, std::function<void
(pulsar::Result)>) () from
/tmp/.venv/lib/python3.9/site-packages/pulsar_client.libs/libpulsar-df34d1a6.so
#15 0x00007ffff692ab88 in
pulsar::ProducerImpl::batchMessageAndSend(std::function<void (pulsar::Result)>
const&) () from
/tmp/.venv/lib/python3.9/site-packages/pulsar_client.libs/libpulsar-df34d1a6.so
#16 0x00007ffff692b215 in
pulsar::ProducerImpl::sendAsyncWithStatsUpdate(pulsar::Message const&,
std::function<void (pulsar::Result, pulsar::MessageId const&)>
const&)::{lambda(boost::system::error_code
const&)#2}::operator()(boost::system::error_code const&) const () from
/tmp/.venv/lib/python3.9/site-packages/pulsar_client.libs/libpulsar-df34d1a6.so
#17 0x00007ffff692bb85 in
boost::asio::detail::wait_handler<pulsar::ProducerImpl::sendAsyncWithStatsUpdate(pulsar::Message
const&, std::function<void (pulsar::Result, pulsar::MessageId const&)>
const&)::{lambda(boost::system::error_code const&)#2},
boost::asio::any_io_executor>::do_complete(void*,
boost::asio::detail::scheduler_operation*, boost::system::error_code const&,
unsigned long) () from
/tmp/.venv/lib/python3.9/site-packages/pulsar_client.libs/libpulsar-df34d1a6.so
#18 0x00007ffff68367a2 in
boost::asio::detail::scheduler::run(boost::system::error_code&) () from
/tmp/.venv/lib/python3.9/site-packages/pulsar_client.libs/libpulsar-df34d1a6.so
#19 0x00007ffff68c325b in
pulsar::ExecutorService::start()::{lambda()#1}::operator()() const () from
/tmp/.venv/lib/python3.9/site-packages/pulsar_client.libs/libpulsar-df34d1a6.so
#20 0x00007ffff6e4b650 in execute_native_thread_routine () from
/tmp/.venv/lib/python3.9/site-packages/pulsar_client.libs/libpulsar-df34d1a6.so
#21 0x00007ffff7d1cb43 in start_thread (arg=<optimized out>) at
./nptl/pthread_create.c:442
#22 0x00007ffff7daea00 in clone3 () at
../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
Thread 1 (Thread 0x7ffff7c85740 (LWP 1067852) "python"):
#0 futex_wait (private=0, expected=2, futex_word=0x7ffff0002eb0) at
../sysdeps/nptl/futex-internal.h:146
#1 __GI___lll_lock_wait (futex=futex@entry=0x7ffff0002eb0, private=0) at
./nptl/lowlevellock.c:49
#2 0x00007ffff7d20082 in lll_mutex_lock_optimized (mutex=0x7ffff0002eb0) at
./nptl/pthread_mutex_lock.c:48
#3 ___pthread_mutex_lock (mutex=0x7ffff0002eb0) at
./nptl/pthread_mutex_lock.c:93
#4 0x00007ffff692be05 in
pulsar::ProducerImpl::flushAsync(std::function<void (pulsar::Result)>) () from
/tmp/.venv/lib/python3.9/site-packages/pulsar_client.libs/libpulsar-df34d1a6.so
#5 0x00007ffff6915e03 in
pulsar::PartitionedProducerImpl::flushAsync(std::function<void
(pulsar::Result)>) () from
/tmp/.venv/lib/python3.9/site-packages/pulsar_client.libs/libpulsar-df34d1a6.so
#6 0x00007ffff6924aaf in pulsar::Producer::flushAsync(std::function<void
(pulsar::Result)>) () from
/tmp/.venv/lib/python3.9/site-packages/pulsar_client.libs/libpulsar-df34d1a6.so
#7 0x00007ffff71970b8 in std::_Function_handler<void (std::function<void
(pulsar::Result)>),
Producer_flush(pulsar::Producer&)::{lambda(std::function<void
(pulsar::Result)>)#1}>::_M_invoke(std::_Any_data const&, std::function<void
(pulsar::Result)>&&) () from
/tmp/.venv/lib/python3.9/site-packages/_pulsar.cpython-39-x86_64-linux-gnu.so
#8 0x00007ffff71a3a19 in waitForAsyncResult(std::function<void
(std::function<void (pulsar::Result)>)>) () from
/tmp/.venv/lib/python3.9/site-packages/_pulsar.cpython-39-x86_64-linux-gnu.so
#9 0x00007ffff71968f7 in Producer_flush(pulsar::Producer&) () from
/tmp/.venv/lib/python3.9/site-packages/_pulsar.cpython-39-x86_64-linux-gnu.so
#10 0x00007ffff7199d83 in pybind11::cpp_function::initialize<void
(*&)(pulsar::Producer&), void, pulsar::Producer&, pybind11::name,
pybind11::is_method, pybind11::sibling, char [108]>(void
(*&)(pulsar::Producer&), void (*)(pulsar::Producer&), pybind11::name const&,
pybind11::is_method const&, pybind11::sibling const&, char const (&)
[108])::{lambda(pybind11::detail::function_call&)#3}::_FUN(pybind11::detail::function_call&)
() from
/tmp/.venv/lib/python3.9/site-packages/_pulsar.cpython-39-x86_64-linux-gnu.so
#11 0x00007ffff71459e7 in pybind11::cpp_function::dispatcher(_object*,
_object*, _object*) () from
/tmp/.venv/lib/python3.9/site-packages/_pulsar.cpython-39-x86_64-linux-gnu.so
#12 0x0000555555776563 in cfunction_call (func=0x7ffff64190e0,
args=<optimized out>, kwargs=<optimized out>) at Objects/methodobject.c:543
#13 0x00005555555c7bdc in _PyObject_MakeTpCall (tstate=0x5555559247f0,
callable=0x7ffff64190e0, args=<optimized out>, nargs=<optimized out>,
keywords=0x0) at Objects/call.c:191
#14 0x00005555557617c2 in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=1,
args=0x7ffff58b39d8, callable=0x7ffff64190e0, tstate=0x5555559247f0) at
./Include/cpython/abstract.h:116
#15 _PyObject_VectorcallTstate (kwnames=0x0, nargsf=1, args=0x7ffff58b39d8,
callable=0x7ffff64190e0, tstate=0x5555559247f0) at
./Include/cpython/abstract.h:103
#16 method_vectorcall (method=<optimized out>, args=0x7ffff58b39e0,
nargsf=<optimized out>, kwnames=0x0) at Objects/classobject.c:53
#17 0x00005555555b7af8 in _PyObject_VectorcallTstate (kwnames=0x0,
nargsf=<optimized out>, args=0x7ffff58b39e0, callable=0x7ffff5914a00,
tstate=0x5555559247f0) at ./Include/cpython/abstract.h:118
#18 PyObject_Vectorcall (kwnames=0x0, nargsf=<optimized out>,
args=0x7ffff58b39e0, callable=0x7ffff5914a00) at
./Include/cpython/abstract.h:127
#19 call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic
pointer>, tstate=<optimized out>) at Python/ceval.c:5077
#20 _PyEval_EvalFrameDefault (tstate=<optimized out>, f=<optimized out>,
throwflag=<optimized out>) at Python/ceval.c:3489
#21 0x00005555555aff1b in _PyEval_EvalFrame (throwflag=0, f=0x7ffff58b3860,
tstate=0x5555559247f0) at ./Include/internal/pycore_ceval.h:40
#22 function_code_fastcall (tstate=0x5555559247f0, co=<optimized out>,
args=<optimized out>, nargs=1, globals=<optimized out>) at Objects/call.c:330
#23 0x00005555555b84a2 in _PyObject_VectorcallTstate (kwnames=0x0,
nargsf=<optimized out>, args=0x7ffff5915bf8, callable=0x7ffff6227b80,
tstate=0x5555559247f0) at ./Include/cpython/abstract.h:118
#24 PyObject_Vectorcall (kwnames=0x0, nargsf=<optimized out>,
args=0x7ffff5915bf8, callable=0x7ffff6227b80) at
./Include/cpython/abstract.h:127
#25 call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic
pointer>, tstate=<optimized out>) at Python/ceval.c:5077
#26 _PyEval_EvalFrameDefault (tstate=<optimized out>, f=<optimized out>,
throwflag=<optimized out>) at Python/ceval.c:3506
#27 0x0000555555678ad4 in _PyEval_EvalFrame (throwflag=0, f=0x7ffff5915a40,
tstate=0x5555559247f0) at ./Include/internal/pycore_ceval.h:40
#28 _PyEval_EvalCode (tstate=tstate@entry=0x5555559247f0,
_co=_co@entry=0x7ffff75d93a0, globals=<optimized out>, locals=locals@entry=0x0,
args=<optimized out>, argcount=0, kwnames=0x0, kwargs=0x555555977920,
kwcount=0, kwstep=1, defs=0x0, defcount=0, kwdefs=0x0, closure=0x0,
name=0x7ffff756fdb0, qualname=0x7ffff756fdb0) at Python/ceval.c:4329
#29 0x00005555555c8dce in _PyFunction_Vectorcall (func=<optimized out>,
stack=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>) at
Objects/call.c:396
#30 0x00005555555b6955 in _PyObject_VectorcallTstate (kwnames=0x0,
nargsf=<optimized out>, args=0x555555977920, callable=0x7ffff7687160,
tstate=0x5555559247f0) at ./Include/cpython/abstract.h:118
#31 PyObject_Vectorcall (kwnames=0x0, nargsf=<optimized out>,
args=0x555555977920, callable=0x7ffff7687160) at
./Include/cpython/abstract.h:127
#32 call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=<synthetic
pointer>, tstate=<optimized out>) at Python/ceval.c:5077
#33 _PyEval_EvalFrameDefault (tstate=<optimized out>, f=<optimized out>,
throwflag=<optimized out>) at Python/ceval.c:3520
#34 0x0000555555678ad4 in _PyEval_EvalFrame (throwflag=0, f=0x5555559777b0,
tstate=0x5555559247f0) at ./Include/internal/pycore_ceval.h:40
#35 _PyEval_EvalCode (tstate=0x5555559247f0, _co=_co@entry=0x7ffff75e10e0,
globals=globals@entry=0x7ffff7644a80, locals=locals@entry=0x7ffff7644a80,
args=args@entry=0x0, argcount=argcount@entry=0, kwnames=0x0, kwargs=0x0,
kwcount=0, kwstep=2, defs=0x0, defcount=0, kwdefs=0x0, closure=0x0, name=0x0,
qualname=0x0) at Python/ceval.c:4329
#36 0x0000555555678dfa in _PyEval_EvalCodeWithName (qualname=0x0, name=0x0,
closure=0x0, kwdefs=0x0, defcount=0, defs=0x0, kwstep=2, kwcount=0, kwargs=0x0,
kwnames=0x0, argcount=0, args=0x0, locals=0x7ffff7644a80,
globals=0x7ffff7644a80, _co=0x7ffff75e10e0) at Python/ceval.c:4361
#37 PyEval_EvalCodeEx (closure=0x0, kwdefs=0x0, defcount=0, defs=0x0,
kwcount=0, kws=0x0, argcount=0, args=0x0, locals=0x7ffff7644a80,
globals=0x7ffff7644a80, _co=0x7ffff75e10e0) at Python/ceval.c:4377
#38 PyEval_EvalCode (co=co@entry=0x7ffff75e10e0,
globals=globals@entry=0x7ffff7644a80, locals=locals@entry=0x7ffff7644a80) at
Python/ceval.c:828
#39 0x00005555556b908c in run_eval_code_obj (locals=0x7ffff7644a80,
globals=0x7ffff7644a80, co=0x7ffff75e10e0, tstate=0x5555559247f0) at
Python/pythonrun.c:1221
#40 run_mod (mod=<optimized out>, filename=filename@entry=0x7ffff76b9570,
globals=globals@entry=0x7ffff7644a80, locals=locals@entry=0x7ffff7644a80,
flags=flags@entry=0x7fffffffc7d8, arena=arena@entry=0x7ffff7693910) at
Python/pythonrun.c:1242
#41 0x00005555556bb0d0 in pyrun_file (flags=0x7fffffffc7d8,
closeit=<optimized out>, locals=0x7ffff7644a80, globals=0x7ffff7644a80,
start=257, filename=0x7ffff76b9570, fp=0x555555925310) at
Python/pythonrun.c:1140
#42 pyrun_simple_file (flags=0x7fffffffc7d8, closeit=<optimized out>,
filename=0x7ffff76b9570, fp=0x555555925310) at Python/pythonrun.c:450
#43 PyRun_SimpleFileExFlags (fp=fp@entry=0x555555925310, filename=<optimized
out>, closeit=closeit@entry=1, flags=flags@entry=0x7fffffffc7d8) at
Python/pythonrun.c:483
#44 0x00005555556bb6cc in PyRun_AnyFileExFlags (fp=fp@entry=0x555555925310,
filename=<optimized out>, closeit=closeit@entry=1,
flags=flags@entry=0x7fffffffc7d8) at Python/pythonrun.c:92
#45 0x00005555555babef in pymain_run_file (cf=0x7fffffffc7d8,
config=0x555555926020) at Modules/main.c:373
#46 pymain_run_python (exitcode=exitcode@entry=0x7fffffffc900) at
Modules/main.c:598
#47 0x00005555555bb290 in Py_RunMain () at Modules/main.c:677
#48 pymain_main (args=0x7fffffffc8c0) at Modules/main.c:707
#49 Py_BytesMain (argc=<optimized out>, argv=<optimized out>) at
Modules/main.c:731
#50 0x00007ffff7cb1d90 in __libc_start_call_main
(main=main@entry=0x5555555afeb0 <main>, argc=argc@entry=2,
argv=argv@entry=0x7fffffffca48) at ../sysdeps/nptl/libc_start_call_main.h:58
#51 0x00007ffff7cb1e40 in __libc_start_main_impl (main=0x5555555afeb0
<main>, argc=2, argv=0x7fffffffca48, init=<optimized out>, fini=<optimized
out>, rtld_fini=<optimized out>, stack_end=0x7fffffffca38) at
../csu/libc-start.c:392
#52 0x00005555555b9e75 in _start ()
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
