acelyc111 opened a new issue #5728: URL: https://github.com/apache/incubator-doris/issues/5728
**Describe the bug** When prometheus get metrics from BE, it timeouted. I checked the http API, it timeouted indeed. **Trouble shooting** The dead lock is cause by multi-locks, there are some related thread stack: ``` Thread 430 (Thread 0x7f79569fc700 (LWP 78827)): #0 0x00007f7959b6e1bd in __lll_lock_wait () from /lib64/libpthread.so.0 #1 0x00007f7959b69d02 in _L_lock_791 () from /lib64/libpthread.so.0 #2 0x00007f7959b69c08 in pthread_mutex_lock () from /lib64/libpthread.so.0 #3 0x00000000014ddaa1 in doris::Mutex::lock() () at ../src/util/mutex.cpp:107 #4 0x00000000017fe293 in lock_guard (__m=..., this=<synthetic pointer>) at /opt/rh/devtoolset-10/root/usr/include/c++/10/bits/std_mutex.h:158 #5 operator() (__closure=<optimized out>) at ../src/agent/task_worker_pool.cpp:103 ////////// acquier REPORT_DISK_STATE-task_worker_pool lock #6 operator() (__closure=<optimized out>, __closure=<optimized out>) at ../src/agent/task_worker_pool.cpp:103 #7 __invoke_impl<void, doris::TaskWorkerPool::TaskWorkerPool(doris::TaskWorkerPool::TaskWorkerType, doris::ExecEnv*, const doris::TMasterInfo&)::<lambda()>&> (__f=...) at /opt/rh/devtoolset-10/root/usr/include/c++/10/bits/invoke.h:60 #8 __invoke_r<void, doris::TaskWorkerPool::TaskWorkerPool(doris::TaskWorkerPool::TaskWorkerType, doris::ExecEnv*, const doris::TMasterInfo&)::<lambda()>&> (__fn=...) at /opt/rh/devtoolset-10/root/usr/include/c++/10/bits/invoke.h:110 #9 std::_Function_handler<void(), doris::TaskWorkerPool::TaskWorkerPool(doris::TaskWorkerPool::TaskWorkerType, doris::ExecEnv*, const doris::TMasterInfo&)::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...) at /opt/rh/devtoolset-10/root/usr/include/c++/10/bits/std_function.h:291 #10 0x000000000147ccd2 in operator() (this=0x9ea3e28) at /opt/rh/devtoolset-10/root/usr/include/c++/10/bits/std_function.h:617 #11 trigger_hook_unlocked (force=true, this=0x2b8e38900) at ../src/util/metrics.cpp:255 #12 doris::MetricRegistry::trigger_all_hooks(bool) const () at ../src/util/metrics.cpp:304 //////// got metrics lock #13 0x00000000010b3edf in doris::Daemon::calculate_metrics_thread() () at ../src/util/doris_metrics.h:206 #14 0x00000000014e2b93 in operator() (this=0x7099698) at /opt/rh/devtoolset-10/root/usr/include/c++/10/bits/std_function.h:617 #15 doris::Thread::supervise_thread(void*) () at ../src/util/thread.cpp:385 #16 0x00007f7959b67dc5 in start_thread () from /lib64/libpthread.so.0 #17 0x00007f795959473d in clone () from /lib64/libc.so.6 ``` ``` Thread 198 (Thread 0x7f78df10d700 (LWP 80151)): #0 0x00007f7959b6e1bd in __lll_lock_wait () from /lib64/libpthread.so.0 #1 0x00007f7959b69d02 in _L_lock_791 () from /lib64/libpthread.so.0 #2 0x00007f7959b69c08 in pthread_mutex_lock () from /lib64/libpthread.so.0 #3 0x000000000119bf50 in __gthread_mutex_lock (__mutex=0x75e6428) at /opt/rh/devtoolset-10/root/usr/include/c++/10/x86_64-redhat-linux/bits/gthr-default.h:749 #4 lock (this=0x75e6428) at /opt/rh/devtoolset-10/root/usr/include/c++/10/bits/std_mutex.h:100 #5 lock_guard (__m=..., this=<synthetic pointer>) at /opt/rh/devtoolset-10/root/usr/include/c++/10/bits/std_mutex.h:159 #6 doris::StorageEngine::get_all_data_dir_info(std::vector<doris::DataDirInfo, std::allocator<doris::DataDirInfo> >*, bool) () at ../src/olap/storage_engine.cpp:360 #7 0x000000000180eeb0 in doris::TaskWorkerPool::_report_disk_state_worker_thread_callback() () at ../src/runtime/exec_env.h:134 //////// got REPORT_DISK_STATE-task_worker_pool lock(after _worker_thread_condition_variable.wait_for returned), and acquier StorageEngine::_store_lock #8 0x00000000014e86e7 in operator() (this=0x7784458) at /opt/rh/devtoolset-10/root/usr/include/c++/10/bits/std_function.h:617 #9 run (this=0x7784450) at ../src/util/threadpool.cpp:42 #10 doris::ThreadPool::dispatch_thread() () at ../src/util/threadpool.cpp:548 #11 0x00000000014e2b93 in operator() (this=0x8a71d58) at /opt/rh/devtoolset-10/root/usr/include/c++/10/bits/std_function.h:617 #12 doris::Thread::supervise_thread(void*) () at ../src/util/thread.cpp:385 #13 0x00007f7959b67dc5 in start_thread () from /lib64/libpthread.so.0 #14 0x00007f795959473d in clone () from /lib64/libc.so.6 ``` ``` Thread 259 (Thread 0x7f7901151700 (LWP 80082)): #0 0x00007f7959b6b03e in pthread_rwlock_wrlock () from /lib64/libpthread.so.0 #1 0x00000000014ddfb1 in doris::RWMutex::wrlock() () at ../src/util/mutex.cpp:179 #2 0x00000000011d8751 in WriteLock (try_lock=false, mutex=0x7579ac0, this=<synthetic pointer>) at ../src/util/mutex.h:160 #3 doris::TabletManager::drop_tablets_on_error_root_path(std::vector<doris::TabletInfo, std::allocator<doris::TabletInfo> > const&) () at ../src/olap/tablet_manager.cpp:566 ///// acquier TabletManager::_tablets_shards lock #4 0x0000000001199560 in doris::StorageEngine::_delete_tablets_on_unused_root_path() () at /opt/rh/devtoolset-10/root/usr/include/c++/10/bits/unique_ptr.h:421 ////// got StorageEngine _store_lock #5 0x00000000011995ea in doris::StorageEngine::_start_disk_stat_monitor() () at ../src/olap/storage_engine.cpp:384 #6 0x000000000118dc48 in doris::StorageEngine::_disk_stat_monitor_thread_callback() () at ../src/olap/olap_server.cpp:196 #7 0x00000000014e2b93 in operator() (this=0x7915a58) at /opt/rh/devtoolset-10/root/usr/include/c++/10/bits/std_function.h:617 #8 doris::Thread::supervise_thread(void*) () at ../src/util/thread.cpp:385 #9 0x00007f7959b67dc5 in start_thread () from /lib64/libpthread.so.0 #10 0x00007f795959473d in clone () from /lib64/libc.so.6 ``` ``` Thread 204 (Thread 0x7f78e2113700 (LWP 80145)): #0 0x00007f79595792e7 in sched_yield () from /lib64/libc.so.6 #1 0x00000000014b7305 in doris::SpinLock::slow_acquire (this=<optimized out>) at ../src/util/spinlock.cc:33 #2 0x00000000014811c6 in lock (this=0x3acbf38 <doris::DorisMetrics::instance()::instance+888>) at ../src/util/spinlock.h:36 #3 lock_guard (__m=..., this=<synthetic pointer>) at /opt/rh/devtoolset-10/root/usr/include/c++/10/bits/std_mutex.h:159 #4 doris::MetricRegistry::register_entity(std::string const&, std::unordered_map<std::string, std::string, std::hash<std::string>, std::equal_to<std::string>, std::allocator<std::pair<std::string const, std::string> > > const&, doris::MetricEntityType) () at ../src/util/metrics.cpp:265 /////// acquier metrics lock #5 0x00000000012245a8 in doris::BaseTablet::BaseTablet(std::shared_ptr<doris::TabletMeta>, doris::DataDir*) () at ../src/gutil/strings/substitute.h:178 #6 0x00000000011bffb7 in doris::Tablet::Tablet(std::shared_ptr<doris::TabletMeta>, doris::DataDir*, std::string const&) () at /opt/rh/devtoolset-10/root/usr/include/c++/10/ext/atomicity.h:100 #7 0x00000000011c040f in construct<doris::Tablet, std::shared_ptr<doris::TabletMeta>&, doris::DataDir*&> (this=<optimized out>, __p=0xe7205c10) at /opt/rh/devtoolset-10/root/usr/include/c++/10/new:175 #8 construct<doris::Tablet, std::shared_ptr<doris::TabletMeta>&, doris::DataDir*&> (__p=0xe7205c10, __a=...) at /opt/rh/devtoolset-10/root/usr/include/c++/10/bits/alloc_traits.h:512 #9 _Sp_counted_ptr_inplace<std::shared_ptr<doris::TabletMeta>&, doris::DataDir*&> (__a=..., this=0xe7205c00) at /opt/rh/devtoolset-10/root/usr/include/c++/10/bits/shared_ptr_base.h:551 #10 __shared_count<doris::Tablet, std::allocator<doris::Tablet>, std::shared_ptr<doris::TabletMeta>&, doris::DataDir*&> (__a=..., __p=@0x7f78e2109490: 0x0, this=0x7f78e2109498) at /opt/rh/devtoolset-10/root/usr/include/c++/10/bits/shared_ptr_base.h:682 #11 __shared_ptr<std::allocator<doris::Tablet>, std::shared_ptr<doris::TabletMeta>&, doris::DataDir*&> (__tag=..., this=0x7f78e2109490) at /opt/rh/devtoolset-10/root/usr/include/c++/10/bits/shared_ptr_base.h:1371 #12 shared_ptr<std::allocator<doris::Tablet>, std::shared_ptr<doris::TabletMeta>&, doris::DataDir*&> (__tag=..., this=0x7f78e2109490) at /opt/rh/devtoolset-10/root/usr/include/c++/10/bits/shared_ptr.h:408 #13 allocate_shared<doris::Tablet, std::allocator<doris::Tablet>, std::shared_ptr<doris::TabletMeta>&, doris::DataDir*&> (__a=...) at /opt/rh/devtoolset-10/root/usr/include/c++/10/bits/shared_ptr.h:860 #14 make_shared<doris::Tablet, std::shared_ptr<doris::TabletMeta>&, doris::DataDir*&> () at /opt/rh/devtoolset-10/root/usr/include/c++/10/bits/shared_ptr.h:876 #15 doris::Tablet::create_tablet_from_meta(std::shared_ptr<doris::TabletMeta>, doris::DataDir*) () at ../src/olap/tablet.cpp:61 #16 0x00000000011d8e22 in doris::TabletManager::load_tablet_from_meta(doris::DataDir*, long, int, std::string const&, bool, bool, bool, bool) () at /opt/rh/devtoolset-10/root/usr/include/c++/10/ext/atomicity.h:100 ///// got TabletManager::_tablets_shards lock #17 0x00000000011dbee3 in doris::TabletManager::load_tablet_from_dir(doris::DataDir*, long, int, std::string const&, bool, bool) () at ../src/olap/tablet_manager.cpp:917 #18 0x00000000019ec9ab in doris::EngineCloneTask::_do_clone() () at /opt/rh/devtoolset-10/root/usr/include/c++/10/sstream:831 #19 0x00000000019ed22d in doris::EngineCloneTask::execute (this=0x7f78e2109b90) at ../src/olap/task/engine_clone_task.cpp:64 #20 0x000000000119d9a1 in doris::StorageEngine::execute_task(doris::EngineTask*) () at /opt/rh/devtoolset-10/root/usr/include/c++/10/ext/atomicity.h:97 #21 0x000000000180a390 in doris::TaskWorkerPool::_clone_worker_thread_callback() () at ../src/runtime/exec_env.h:134 #22 0x00000000014e86e7 in operator() (this=0x7785158) at /opt/rh/devtoolset-10/root/usr/include/c++/10/bits/std_function.h:617 #23 run (this=0x7785150) at ../src/util/threadpool.cpp:42 #24 doris::ThreadPool::dispatch_thread() () at ../src/util/threadpool.cpp:548 #25 0x00000000014e2b93 in operator() (this=0x8a718d8) at /opt/rh/devtoolset-10/root/usr/include/c++/10/bits/std_function.h:617 #26 doris::Thread::supervise_thread(void*) () at ../src/util/thread.cpp:385 #27 0x00007f7959b67dc5 in start_thread () from /lib64/libpthread.so.0 #28 0x00007f795959473d in clone () from /lib64/libc.so.6 ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
