Hi Doug, Thanks for a quick patch. I'm running the tests for nearly 4 hours now and the problem does not seem to be occurring. I will have a double confirmed that after I finish my testing. Thanks again for the patch!
Mateusz On Sat, Mar 21, 2009 at 5:26 PM, Doug Judd <[email protected]> wrote: > Hi Mateusz, > > This is a 5-way deadlock between threads 15, 11, 6, 5, and 3. I've > attached a patch, please give it a try and let me know if it fixes the > problem. The fix is also in my 'pu' branch. Thanks! > > - Doug > > On Sat, Mar 21, 2009 at 4:30 AM, Mateusz Berezecki <[email protected]> > wrote: >> >> Hi Doug, >> >> Took a while but finally got it (Thread #3 ) >> >> (gdb) thread apply all where >> >> Thread 28 (Thread 0x421db950 (LWP 2944)): >> #0 0x00007f90c9d80288 in epoll_wait () from /lib/libc.so.6 >> #1 0x000000000077fe32 in Hypertable::ReactorRunner::operator() >> (this=0x19ca2d8) at >> /home/mateusz/hypertable/src/cc/AsyncComm/ReactorRunner.cc:70 >> #2 0x000000000077f18a in >> boost::detail::thread_data<Hypertable::ReactorRunner>::run >> (this=0x19ca1d0) at >> /usr/local/include/boost/thread/detail/thread.hpp:56 >> #3 0x00007f90cb6b992f in thread_proxy () from >> /usr/local/lib/libboost_thread-mt.so >> #4 0x00007f90ca7bc3ea in start_thread () from /lib/libpthread.so.0 >> #5 0x00007f90c9d7fcbd in clone () from /lib/libc.so.6 >> #6 0x0000000000000000 in ?? () >> >> Thread 27 (Thread 0x429dc950 (LWP 2945)): >> #0 0x00007f90c9d80288 in epoll_wait () from /lib/libc.so.6 >> #1 0x000000000077fe32 in Hypertable::ReactorRunner::operator() >> (this=0x19cf4d8) at >> /home/mateusz/hypertable/src/cc/AsyncComm/ReactorRunner.cc:70 >> #2 0x000000000077f18a in >> boost::detail::thread_data<Hypertable::ReactorRunner>::run >> (this=0x19cf3d0) at >> /usr/local/include/boost/thread/detail/thread.hpp:56 >> #3 0x00007f90cb6b992f in thread_proxy () from >> /usr/local/lib/libboost_thread-mt.so >> #4 0x00007f90ca7bc3ea in start_thread () from /lib/libpthread.so.0 >> #5 0x00007f90c9d7fcbd in clone () from /lib/libc.so.6 >> #6 0x0000000000000000 in ?? () >> >> Thread 26 (Thread 0x431dd950 (LWP 2946)): >> #0 0x00007f90c9d80288 in epoll_wait () from /lib/libc.so.6 >> #1 0x000000000077fe32 in Hypertable::ReactorRunner::operator() >> (this=0x19cf808) at >> /home/mateusz/hypertable/src/cc/AsyncComm/ReactorRunner.cc:70 >> #2 0x000000000077f18a in >> boost::detail::thread_data<Hypertable::ReactorRunner>::run >> (this=0x19cf700) at >> /usr/local/include/boost/thread/detail/thread.hpp:56 >> #3 0x00007f90cb6b992f in thread_proxy () from >> /usr/local/lib/libboost_thread-mt.so >> #4 0x00007f90ca7bc3ea in start_thread () from /lib/libpthread.so.0 >> #5 0x00007f90c9d7fcbd in clone () from /lib/libc.so.6 >> #6 0x0000000000000000 in ?? () >> >> Thread 25 (Thread 0x439de950 (LWP 2947)): >> #0 0x00007f90c9d80288 in epoll_wait () from /lib/libc.so.6 >> #1 0x000000000077fe32 in Hypertable::ReactorRunner::operator() >> (this=0x19ce868) at >> /home/mateusz/hypertable/src/cc/AsyncComm/ReactorRunner.cc:70 >> #2 0x000000000077f18a in >> boost::detail::thread_data<Hypertable::ReactorRunner>::run >> (this=0x19ce760) at >> /usr/local/include/boost/thread/detail/thread.hpp:56 >> #3 0x00007f90cb6b992f in thread_proxy () from >> /usr/local/lib/libboost_thread-mt.so >> #4 0x00007f90ca7bc3ea in start_thread () from /lib/libpthread.so.0 >> #5 0x00007f90c9d7fcbd in clone () from /lib/libc.so.6 >> #6 0x0000000000000000 in ?? () >> >> Thread 24 (Thread 0x441df950 (LWP 2948)): >> #0 0x00007f90ca7c02d9 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib/libpthread.so.0 >> #1 0x00000000005e35d8 in >> boost::condition_variable_any::wait<boost::unique_lock<boost::mutex> > >> (this=0x19ceb20, m...@0x441df090) >> at /usr/local/include/boost/thread/pthread/condition_variable.hpp:84 >> #2 0x000000000076ff63 in Hypertable::ConnectionManager::operator() >> (this=0x19cece8) at >> /home/mateusz/hypertable/src/cc/AsyncComm/ConnectionManager.cc:308 >> #3 0x00000000005e33d0 in >> boost::detail::thread_data<Hypertable::ConnectionManager>::run >> (this=0x19cebe0) at >> /usr/local/include/boost/thread/detail/thread.hpp:56 >> #4 0x00007f90cb6b992f in thread_proxy () from >> /usr/local/lib/libboost_thread-mt.so >> #5 0x00007f90ca7bc3ea in start_thread () from /lib/libpthread.so.0 >> #6 0x00007f90c9d7fcbd in clone () from /lib/libc.so.6 >> #7 0x0000000000000000 in ?? () >> >> Thread 23 (Thread 0x449e0950 (LWP 2949)): >> ---Type <return> to continue, or q <return> to quit--- >> #0 0x00007f90ca7c02d9 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib/libpthread.so.0 >> #1 0x00000000005e35d8 in >> boost::condition_variable_any::wait<boost::unique_lock<boost::mutex> > >> (this=0x19cef90, m...@0x449e0060) >> at /usr/local/include/boost/thread/pthread/condition_variable.hpp:84 >> #2 0x00000000005e36f4 in >> Hypertable::ApplicationQueue::Worker::operator() (this=0x19cf148) at >> /home/mateusz/hypertable/src/cc/AsyncComm/ApplicationQueue.h:98 >> #3 0x00000000005e3ab0 in >> boost::detail::thread_data<Hypertable::ApplicationQueue::Worker>::run >> (this=0x19cf040) at >> /usr/local/include/boost/thread/detail/thread.hpp:56 >> #4 0x00007f90cb6b992f in thread_proxy () from >> /usr/local/lib/libboost_thread-mt.so >> #5 0x00007f90ca7bc3ea in start_thread () from /lib/libpthread.so.0 >> #6 0x00007f90c9d7fcbd in clone () from /lib/libc.so.6 >> #7 0x0000000000000000 in ?? () >> >> Thread 22 (Thread 0x451e1950 (LWP 2951)): >> #0 0x00007f90ca7c02d9 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib/libpthread.so.0 >> #1 0x00000000005e35d8 in >> boost::condition_variable_any::wait<boost::unique_lock<boost::mutex> > >> (this=0x19cef90, m...@0x451e1060) >> at /usr/local/include/boost/thread/pthread/condition_variable.hpp:84 >> #2 0x00000000005e36f4 in >> Hypertable::ApplicationQueue::Worker::operator() (this=0x19d0978) at >> /home/mateusz/hypertable/src/cc/AsyncComm/ApplicationQueue.h:98 >> #3 0x00000000005e3ab0 in >> boost::detail::thread_data<Hypertable::ApplicationQueue::Worker>::run >> (this=0x19d0870) at >> /usr/local/include/boost/thread/detail/thread.hpp:56 >> #4 0x00007f90cb6b992f in thread_proxy () from >> /usr/local/lib/libboost_thread-mt.so >> #5 0x00007f90ca7bc3ea in start_thread () from /lib/libpthread.so.0 >> #6 0x00007f90c9d7fcbd in clone () from /lib/libc.so.6 >> #7 0x0000000000000000 in ?? () >> >> Thread 21 (Thread 0x459e2950 (LWP 2952)): >> #0 0x00007f90ca7c02d9 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib/libpthread.so.0 >> #1 0x00000000005e35d8 in >> boost::condition_variable_any::wait<boost::unique_lock<boost::mutex> > >> (this=0x19cef90, m...@0x459e2060) >> at /usr/local/include/boost/thread/pthread/condition_variable.hpp:84 >> #2 0x00000000005e36f4 in >> Hypertable::ApplicationQueue::Worker::operator() (this=0x19d0c68) at >> /home/mateusz/hypertable/src/cc/AsyncComm/ApplicationQueue.h:98 >> #3 0x00000000005e3ab0 in >> boost::detail::thread_data<Hypertable::ApplicationQueue::Worker>::run >> (this=0x19d0b60) at >> /usr/local/include/boost/thread/detail/thread.hpp:56 >> #4 0x00007f90cb6b992f in thread_proxy () from >> /usr/local/lib/libboost_thread-mt.so >> #5 0x00007f90ca7bc3ea in start_thread () from /lib/libpthread.so.0 >> #6 0x00007f90c9d7fcbd in clone () from /lib/libc.so.6 >> #7 0x0000000000000000 in ?? () >> >> Thread 20 (Thread 0x461e3950 (LWP 2954)): >> #0 0x00007f90ca7c02d9 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib/libpthread.so.0 >> #1 0x00000000005e35d8 in >> boost::condition_variable_any::wait<boost::unique_lock<boost::mutex> > >> (this=0x19cef90, m...@0x461e3060) >> at /usr/local/include/boost/thread/pthread/condition_variable.hpp:84 >> #2 0x00000000005e36f4 in >> Hypertable::ApplicationQueue::Worker::operator() (this=0x19d0f58) at >> /home/mateusz/hypertable/src/cc/AsyncComm/ApplicationQueue.h:98 >> #3 0x00000000005e3ab0 in >> boost::detail::thread_data<Hypertable::ApplicationQueue::Worker>::run >> (this=0x19d0e50) at >> /usr/local/include/boost/thread/detail/thread.hpp:56 >> #4 0x00007f90cb6b992f in thread_proxy () from >> /usr/local/lib/libboost_thread-mt.so >> #5 0x00007f90ca7bc3ea in start_thread () from /lib/libpthread.so.0 >> #6 0x00007f90c9d7fcbd in clone () from /lib/libc.so.6 >> #7 0x0000000000000000 in ?? () >> >> Thread 19 (Thread 0x469e4950 (LWP 2955)): >> #0 0x00007f90ca7c02d9 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib/libpthread.so.0 >> #1 0x00000000005e35d8 in >> boost::condition_variable_any::wait<boost::unique_lock<boost::mutex> > >> (this=0x19cef90, m...@0x469e4060) >> at /usr/local/include/boost/thread/pthread/condition_variable.hpp:84 >> #2 0x00000000005e36f4 in >> Hypertable::ApplicationQueue::Worker::operator() (this=0x19d1248) at >> /home/mateusz/hypertable/src/cc/AsyncComm/ApplicationQueue.h:98 >> #3 0x00000000005e3ab0 in >> boost::detail::thread_data<Hypertable::ApplicationQueue::Worker>::run >> (this=0x19d1140) at >> /usr/local/include/boost/thread/detail/thread.hpp:56 >> ---Type <return> to continue, or q <return> to quit--- >> #4 0x00007f90cb6b992f in thread_proxy () from >> /usr/local/lib/libboost_thread-mt.so >> #5 0x00007f90ca7bc3ea in start_thread () from /lib/libpthread.so.0 >> #6 0x00007f90c9d7fcbd in clone () from /lib/libc.so.6 >> #7 0x0000000000000000 in ?? () >> >> Thread 18 (Thread 0x471e5950 (LWP 2957)): >> #0 0x00007f90ca7c02d9 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib/libpthread.so.0 >> #1 0x00000000005e35d8 in >> boost::condition_variable_any::wait<boost::unique_lock<boost::mutex> > >> (this=0x19cef90, m...@0x471e5060) >> at /usr/local/include/boost/thread/pthread/condition_variable.hpp:84 >> #2 0x00000000005e36f4 in >> Hypertable::ApplicationQueue::Worker::operator() (this=0x19d1538) at >> /home/mateusz/hypertable/src/cc/AsyncComm/ApplicationQueue.h:98 >> #3 0x00000000005e3ab0 in >> boost::detail::thread_data<Hypertable::ApplicationQueue::Worker>::run >> (this=0x19d1430) at >> /usr/local/include/boost/thread/detail/thread.hpp:56 >> #4 0x00007f90cb6b992f in thread_proxy () from >> /usr/local/lib/libboost_thread-mt.so >> #5 0x00007f90ca7bc3ea in start_thread () from /lib/libpthread.so.0 >> #6 0x00007f90c9d7fcbd in clone () from /lib/libc.so.6 >> #7 0x0000000000000000 in ?? () >> >> Thread 17 (Thread 0x479e6950 (LWP 2958)): >> #0 0x00007f90ca7c02d9 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib/libpthread.so.0 >> #1 0x00000000005e35d8 in >> boost::condition_variable_any::wait<boost::unique_lock<boost::mutex> > >> (this=0x19cef90, m...@0x479e6060) >> at /usr/local/include/boost/thread/pthread/condition_variable.hpp:84 >> #2 0x00000000005e36f4 in >> Hypertable::ApplicationQueue::Worker::operator() (this=0x7f90c4000b58) >> at /home/mateusz/hypertable/src/cc/AsyncComm/ApplicationQueue.h:98 >> #3 0x00000000005e3ab0 in >> boost::detail::thread_data<Hypertable::ApplicationQueue::Worker>::run >> (this=0x7f90c4000a50) at >> /usr/local/include/boost/thread/detail/thread.hpp:56 >> #4 0x00007f90cb6b992f in thread_proxy () from >> /usr/local/lib/libboost_thread-mt.so >> #5 0x00007f90ca7bc3ea in start_thread () from /lib/libpthread.so.0 >> #6 0x00007f90c9d7fcbd in clone () from /lib/libc.so.6 >> #7 0x0000000000000000 in ?? () >> >> Thread 16 (Thread 0x481e7950 (LWP 2960)): >> #0 0x00007f90ca7c02d9 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib/libpthread.so.0 >> #1 0x00000000005e35d8 in >> boost::condition_variable_any::wait<boost::unique_lock<boost::mutex> > >> (this=0x19cef90, m...@0x481e7060) >> at /usr/local/include/boost/thread/pthread/condition_variable.hpp:84 >> #2 0x00000000005e36f4 in >> Hypertable::ApplicationQueue::Worker::operator() (this=0x7f90c4000e28) >> at /home/mateusz/hypertable/src/cc/AsyncComm/ApplicationQueue.h:98 >> #3 0x00000000005e3ab0 in >> boost::detail::thread_data<Hypertable::ApplicationQueue::Worker>::run >> (this=0x7f90c4000d20) at >> /usr/local/include/boost/thread/detail/thread.hpp:56 >> #4 0x00007f90cb6b992f in thread_proxy () from >> /usr/local/lib/libboost_thread-mt.so >> #5 0x00007f90ca7bc3ea in start_thread () from /lib/libpthread.so.0 >> #6 0x00007f90c9d7fcbd in clone () from /lib/libc.so.6 >> #7 0x0000000000000000 in ?? () >> >> Thread 15 (Thread 0x489e8950 (LWP 2961)): >> #0 0x00007f90ca7c2b04 in __lll_lock_wait () from /lib/libpthread.so.0 >> #1 0x00007f90ca7be1a0 in _L_lock_102 () from /lib/libpthread.so.0 >> #2 0x00007f90ca7bdaae in pthread_mutex_lock () from /lib/libpthread.so.0 >> #3 0x00000000005dd21f in boost::mutex::lock (this=0x7f90bd02b018) at >> /usr/local/include/boost/thread/pthread/mutex.hpp:50 >> #4 0x000000000066ba18 in >> Hypertable::LiveFileTracker::update_files_column (this=0x7f90bd02aff0) >> at >> /home/mateusz/hypertable/src/cc/Hypertable/RangeServer/LiveFileTracker.cc:99 >> #5 0x000000000064bb26 in Hypertable::AccessGroup::release_files >> (this=0x7f90bd02aec0, fil...@0x7f907d525a98) >> at >> /home/mateusz/hypertable/src/cc/Hypertable/RangeServer/AccessGroup.cc:622 >> #6 0x000000000065b2c8 in >> Hypertable::CellStoreReleaseCallback::operator() (this=0x7f907d525a90) >> at >> /home/mateusz/hypertable/src/cc/Hypertable/RangeServer/CellStoreReleaseCallback.cc:38 >> #7 0x00000000006703b2 in ~MergeScanner (this=0x7f907d525940) at >> /home/mateusz/hypertable/src/cc/Hypertable/RangeServer/MergeScanner.cc:60 >> ---Type <return> to continue, or q <return> to quit--- >> #8 0x000000000067035e in ~MergeScanner (this=0x62184c0) at >> /home/mateusz/hypertable/src/cc/Hypertable/RangeServer/MergeScanner.cc:58 >> #9 0x00000000005d844b in Hypertable::intrusive_ptr_release >> (rc=0x62184c0) at >> /home/mateusz/hypertable/src/cc/Common/ReferenceCount.h:73 >> #10 0x00000000005e8bfd in ~intrusive_ptr (this=0x489e5cf0) at >> /usr/local/include/boost/intrusive_ptr.hpp:101 >> #11 0x00000000005fec2b in Hypertable::RangeServer::create_scanner >> (this=0x7f90c40147a0, cb=0x489e6f70, table=0x489e6fa0, >> range_spec=0x489e6f90, scan_spec=0x489e6f00) >> at >> /home/mateusz/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:689 >> #12 0x0000000000643833 in Hypertable::RequestHandlerCreateScanner::run >> (this=0x7f90868d9030) >> at >> /home/mateusz/hypertable/src/cc/Hypertable/RangeServer/RequestHandlerCreateScanner.cc:52 >> #13 0x00000000005e39d4 in >> Hypertable::ApplicationQueue::Worker::operator() (this=0x7f90c40010f8) >> at /home/mateusz/hypertable/src/cc/AsyncComm/ApplicationQueue.h:142 >> #14 0x00000000005e3ab0 in >> boost::detail::thread_data<Hypertable::ApplicationQueue::Worker>::run >> (this=0x7f90c4000ff0) at >> /usr/local/include/boost/thread/detail/thread.hpp:56 >> #15 0x00007f90cb6b992f in thread_proxy () from >> /usr/local/lib/libboost_thread-mt.so >> #16 0x00007f90ca7bc3ea in start_thread () from /lib/libpthread.so.0 >> #17 0x00007f90c9d7fcbd in clone () from /lib/libc.so.6 >> #18 0x0000000000000000 in ?? () >> >> Thread 14 (Thread 0x491e9950 (LWP 2962)): >> #0 0x00007f90ca7c02d9 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib/libpthread.so.0 >> #1 0x00000000005e35d8 in >> boost::condition_variable_any::wait<boost::unique_lock<boost::mutex> > >> (this=0x19cef90, m...@0x491e9060) >> at /usr/local/include/boost/thread/pthread/condition_variable.hpp:84 >> #2 0x00000000005e36f4 in >> Hypertable::ApplicationQueue::Worker::operator() (this=0x7f90c40013c8) >> at /home/mateusz/hypertable/src/cc/AsyncComm/ApplicationQueue.h:98 >> #3 0x00000000005e3ab0 in >> boost::detail::thread_data<Hypertable::ApplicationQueue::Worker>::run >> (this=0x7f90c40012c0) at >> /usr/local/include/boost/thread/detail/thread.hpp:56 >> #4 0x00007f90cb6b992f in thread_proxy () from >> /usr/local/lib/libboost_thread-mt.so >> #5 0x00007f90ca7bc3ea in start_thread () from /lib/libpthread.so.0 >> #6 0x00007f90c9d7fcbd in clone () from /lib/libc.so.6 >> #7 0x0000000000000000 in ?? () >> >> Thread 13 (Thread 0x499ea950 (LWP 2963)): >> #0 0x00007f90ca7c02d9 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib/libpthread.so.0 >> #1 0x00000000005e35d8 in >> boost::condition_variable_any::wait<boost::unique_lock<boost::mutex> > >> (this=0x19cef90, m...@0x499ea060) >> at /usr/local/include/boost/thread/pthread/condition_variable.hpp:84 >> #2 0x00000000005e36f4 in >> Hypertable::ApplicationQueue::Worker::operator() (this=0x7f90c4001698) >> at /home/mateusz/hypertable/src/cc/AsyncComm/ApplicationQueue.h:98 >> #3 0x00000000005e3ab0 in >> boost::detail::thread_data<Hypertable::ApplicationQueue::Worker>::run >> (this=0x7f90c4001590) at >> /usr/local/include/boost/thread/detail/thread.hpp:56 >> #4 0x00007f90cb6b992f in thread_proxy () from >> /usr/local/lib/libboost_thread-mt.so >> #5 0x00007f90ca7bc3ea in start_thread () from /lib/libpthread.so.0 >> #6 0x00007f90c9d7fcbd in clone () from /lib/libc.so.6 >> #7 0x0000000000000000 in ?? () >> >> Thread 12 (Thread 0x4a1eb950 (LWP 2964)): >> #0 0x00007f90ca7c02d9 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib/libpthread.so.0 >> #1 0x00000000005e35d8 in >> boost::condition_variable_any::wait<boost::unique_lock<boost::mutex> > >> (this=0x19cef90, m...@0x4a1eb060) >> at /usr/local/include/boost/thread/pthread/condition_variable.hpp:84 >> #2 0x00000000005e36f4 in >> Hypertable::ApplicationQueue::Worker::operator() (this=0x7f90c4001968) >> at /home/mateusz/hypertable/src/cc/AsyncComm/ApplicationQueue.h:98 >> #3 0x00000000005e3ab0 in >> boost::detail::thread_data<Hypertable::ApplicationQueue::Worker>::run >> (this=0x7f90c4001860) at >> /usr/local/include/boost/thread/detail/thread.hpp:56 >> #4 0x00007f90cb6b992f in thread_proxy () from >> /usr/local/lib/libboost_thread-mt.so >> #5 0x00007f90ca7bc3ea in start_thread () from /lib/libpthread.so.0 >> #6 0x00007f90c9d7fcbd in clone () from /lib/libc.so.6 >> #7 0x0000000000000000 in ?? () >> >> Thread 11 (Thread 0x4a9ec950 (LWP 2965)): >> #0 0x00007f90ca7c2b04 in __lll_lock_wait () from /lib/libpthread.so.0 >> ---Type <return> to continue, or q <return> to quit--- >> #1 0x00007f90ca7be1a0 in _L_lock_102 () from /lib/libpthread.so.0 >> #2 0x00007f90ca7bdaae in pthread_mutex_lock () from /lib/libpthread.so.0 >> #3 0x00000000005dd21f in boost::mutex::lock (this=0x7f90bd02aff0) at >> /usr/local/include/boost/thread/pthread/mutex.hpp:50 >> #4 0x00000000005dd2b4 in boost::unique_lock<boost::mutex>::lock >> (this=0x4a9e95e0) at /usr/local/include/boost/thread/locks.hpp:338 >> #5 0x00000000005dd2f2 in unique_lock (this=0x4a9e95e0, >> m...@0x7f90bd02aff0) at /usr/local/include/boost/thread/locks.hpp:224 >> #6 0x000000000066afc2 in Hypertable::LiveFileTracker::add_references >> (this=0x7f90bd02aff0, fil...@0x4a9e9658) >> at >> /home/mateusz/hypertable/src/cc/Hypertable/RangeServer/LiveFileTracker.cc:45 >> #7 0x000000000064ff42 in Hypertable::AccessGroup::create_scanner >> (this=0x7f90bd02aec0, scan_conte...@0x4a9e9cd0) >> at >> /home/mateusz/hypertable/src/cc/Hypertable/RangeServer/AccessGroup.cc:181 >> #8 0x00000000006364a2 in Hypertable::Range::create_scanner >> (this=0x7f90bcf7ee60, scan_c...@0x4a9e9cd0) at >> /home/mateusz/hypertable/src/cc/Hypertable/RangeServer/Range.cc:309 >> #9 0x00000000005fe23b in Hypertable::RangeServer::create_scanner >> (this=0x7f90c40147a0, cb=0x4a9eaf70, table=0x4a9eafa0, >> range_spec=0x4a9eaf90, scan_spec=0x4a9eaf00) >> at >> /home/mateusz/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:654 >> #10 0x0000000000643833 in Hypertable::RequestHandlerCreateScanner::run >> (this=0x7f90a51293e0) >> at >> /home/mateusz/hypertable/src/cc/Hypertable/RangeServer/RequestHandlerCreateScanner.cc:52 >> #11 0x00000000005e39d4 in >> Hypertable::ApplicationQueue::Worker::operator() (this=0x7f90c4001c38) >> at /home/mateusz/hypertable/src/cc/AsyncComm/ApplicationQueue.h:142 >> #12 0x00000000005e3ab0 in >> boost::detail::thread_data<Hypertable::ApplicationQueue::Worker>::run >> (this=0x7f90c4001b30) at >> /usr/local/include/boost/thread/detail/thread.hpp:56 >> #13 0x00007f90cb6b992f in thread_proxy () from >> /usr/local/lib/libboost_thread-mt.so >> #14 0x00007f90ca7bc3ea in start_thread () from /lib/libpthread.so.0 >> #15 0x00007f90c9d7fcbd in clone () from /lib/libc.so.6 >> #16 0x0000000000000000 in ?? () >> >> Thread 10 (Thread 0x4b1ed950 (LWP 2967)): >> #0 0x00007f90ca7c02d9 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib/libpthread.so.0 >> #1 0x00000000005e35d8 in >> boost::condition_variable_any::wait<boost::unique_lock<boost::mutex> > >> (this=0x19cef90, m...@0x4b1ed060) >> at /usr/local/include/boost/thread/pthread/condition_variable.hpp:84 >> #2 0x00000000005e36f4 in >> Hypertable::ApplicationQueue::Worker::operator() (this=0x7f90c4001f08) >> at /home/mateusz/hypertable/src/cc/AsyncComm/ApplicationQueue.h:98 >> #3 0x00000000005e3ab0 in >> boost::detail::thread_data<Hypertable::ApplicationQueue::Worker>::run >> (this=0x7f90c4001e00) at >> /usr/local/include/boost/thread/detail/thread.hpp:56 >> #4 0x00007f90cb6b992f in thread_proxy () from >> /usr/local/lib/libboost_thread-mt.so >> #5 0x00007f90ca7bc3ea in start_thread () from /lib/libpthread.so.0 >> #6 0x00007f90c9d7fcbd in clone () from /lib/libc.so.6 >> #7 0x0000000000000000 in ?? () >> >> Thread 9 (Thread 0x4b9ee950 (LWP 2968)): >> #0 0x00007f90ca7c02d9 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib/libpthread.so.0 >> #1 0x00000000005e35d8 in >> boost::condition_variable_any::wait<boost::unique_lock<boost::mutex> > >> (this=0x19cef90, m...@0x4b9ee060) >> at /usr/local/include/boost/thread/pthread/condition_variable.hpp:84 >> #2 0x00000000005e36f4 in >> Hypertable::ApplicationQueue::Worker::operator() (this=0x7f90c40021d8) >> at /home/mateusz/hypertable/src/cc/AsyncComm/ApplicationQueue.h:98 >> #3 0x00000000005e3ab0 in >> boost::detail::thread_data<Hypertable::ApplicationQueue::Worker>::run >> (this=0x7f90c40020d0) at >> /usr/local/include/boost/thread/detail/thread.hpp:56 >> #4 0x00007f90cb6b992f in thread_proxy () from >> /usr/local/lib/libboost_thread-mt.so >> #5 0x00007f90ca7bc3ea in start_thread () from /lib/libpthread.so.0 >> #6 0x00007f90c9d7fcbd in clone () from /lib/libc.so.6 >> #7 0x0000000000000000 in ?? () >> >> Thread 8 (Thread 0x4c1ef950 (LWP 2969)): >> #0 0x00007f90ca7c02d9 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib/libpthread.so.0 >> #1 0x00000000005e35d8 in >> boost::condition_variable_any::wait<boost::unique_lock<boost::mutex> > >> (this=0x19cef90, m...@0x4c1ef060) >> at /usr/local/include/boost/thread/pthread/condition_variable.hpp:84 >> #2 0x00000000005e36f4 in >> Hypertable::ApplicationQueue::Worker::operator() (this=0x7f90c40024a8) >> at /home/mateusz/hypertable/src/cc/AsyncComm/ApplicationQueue.h:98 >> #3 0x00000000005e3ab0 in >> boost::detail::thread_data<Hypertable::ApplicationQueue::Worker>::run >> (this=0x7f90c40023a0) at >> /usr/local/include/boost/thread/detail/thread.hpp:56 >> ---Type <return> to continue, or q <return> to quit--- >> #4 0x00007f90cb6b992f in thread_proxy () from >> /usr/local/lib/libboost_thread-mt.so >> #5 0x00007f90ca7bc3ea in start_thread () from /lib/libpthread.so.0 >> #6 0x00007f90c9d7fcbd in clone () from /lib/libc.so.6 >> #7 0x0000000000000000 in ?? () >> >> Thread 7 (Thread 0x4c9f0950 (LWP 2970)): >> #0 0x00007f90ca7c02d9 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib/libpthread.so.0 >> #1 0x00000000005e35d8 in >> boost::condition_variable_any::wait<boost::unique_lock<boost::mutex> > >> (this=0x19cef90, m...@0x4c9f0060) >> at /usr/local/include/boost/thread/pthread/condition_variable.hpp:84 >> #2 0x00000000005e36f4 in >> Hypertable::ApplicationQueue::Worker::operator() (this=0x7f90c4002778) >> at /home/mateusz/hypertable/src/cc/AsyncComm/ApplicationQueue.h:98 >> #3 0x00000000005e3ab0 in >> boost::detail::thread_data<Hypertable::ApplicationQueue::Worker>::run >> (this=0x7f90c4002670) at >> /usr/local/include/boost/thread/detail/thread.hpp:56 >> #4 0x00007f90cb6b992f in thread_proxy () from >> /usr/local/lib/libboost_thread-mt.so >> #5 0x00007f90ca7bc3ea in start_thread () from /lib/libpthread.so.0 >> #6 0x00007f90c9d7fcbd in clone () from /lib/libc.so.6 >> #7 0x0000000000000000 in ?? () >> >> Thread 6 (Thread 0x4d1f1950 (LWP 2971)): >> #0 0x00007f90ca7c2b04 in __lll_lock_wait () from /lib/libpthread.so.0 >> #1 0x00007f90ca7be1a0 in _L_lock_102 () from /lib/libpthread.so.0 >> #2 0x00007f90ca7bdaae in pthread_mutex_lock () from /lib/libpthread.so.0 >> #3 0x00000000005dd21f in boost::mutex::lock (this=0x7f90bd02aee0) at >> /usr/local/include/boost/thread/pthread/mutex.hpp:50 >> #4 0x000000000063d8d9 in Hypertable::AccessGroup::lock >> (this=0x7f90bd02aec0) at >> /home/mateusz/hypertable/src/cc/Hypertable/RangeServer/AccessGroup.h:84 >> #5 0x000000000062ee6f in Hypertable::Range::lock >> (this=0x7f90bcf7ee60) at >> /home/mateusz/hypertable/src/cc/Hypertable/RangeServer/Range.cc:912 >> #6 0x0000000000617c93 in Hypertable::Locker<Hypertable::Range>::lock >> (this=0x4d1ee910) at /home/mateusz/hypertable/src/cc/Common/Mutex.h:52 >> #7 0x0000000000617ce5 in Locker (this=0x4d1ee910, >> mut...@0x7f90bcf7ee60, init_lock=true) at >> /home/mateusz/hypertable/src/cc/Common/Mutex.h:40 >> #8 0x00000000005f681d in Hypertable::RangeServer::update >> (this=0x7f90c40147a0, cb=0x4d1eff70, table=0x4d1effa0, count=232, >> buff...@0x4d1eff90) >> at >> /home/mateusz/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:1405 >> #9 0x0000000000649909 in Hypertable::RequestHandlerUpdate::run >> (this=0x7f902e1f8900) at >> >> /home/mateusz/hypertable/src/cc/Hypertable/RangeServer/RequestHandlerUpdate.cc:54 >> #10 0x00000000005e39d4 in >> Hypertable::ApplicationQueue::Worker::operator() (this=0x7f90c4002a48) >> at /home/mateusz/hypertable/src/cc/AsyncComm/ApplicationQueue.h:142 >> #11 0x00000000005e3ab0 in >> boost::detail::thread_data<Hypertable::ApplicationQueue::Worker>::run >> (this=0x7f90c4002940) at >> /usr/local/include/boost/thread/detail/thread.hpp:56 >> #12 0x00007f90cb6b992f in thread_proxy () from >> /usr/local/lib/libboost_thread-mt.so >> #13 0x00007f90ca7bc3ea in start_thread () from /lib/libpthread.so.0 >> #14 0x00007f90c9d7fcbd in clone () from /lib/libc.so.6 >> #15 0x0000000000000000 in ?? () >> >> Thread 5 (Thread 0x4d9f2950 (LWP 2972)): >> #0 0x00007f90ca7c2b04 in __lll_lock_wait () from /lib/libpthread.so.0 >> #1 0x00007f90ca7be1a0 in _L_lock_102 () from /lib/libpthread.so.0 >> #2 0x00007f90ca7bdaae in pthread_mutex_lock () from /lib/libpthread.so.0 >> #3 0x00000000005dd21f in boost::mutex::lock (this=0x7f90c4014938) at >> /usr/local/include/boost/thread/pthread/mutex.hpp:50 >> #4 0x00000000005f6429 in Hypertable::RangeServer::update >> (this=0x7f90c40147a0, cb=0x4d9f0f70, table=0x4d9f0fa0, count=1, >> buff...@0x4d9f0f90) >> at >> /home/mateusz/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:1372 >> #5 0x0000000000649909 in Hypertable::RequestHandlerUpdate::run >> (this=0x7f904ad6ea00) at >> >> /home/mateusz/hypertable/src/cc/Hypertable/RangeServer/RequestHandlerUpdate.cc:54 >> #6 0x00000000005e39d4 in >> Hypertable::ApplicationQueue::Worker::operator() (this=0x7f90c4002d18) >> at /home/mateusz/hypertable/src/cc/AsyncComm/ApplicationQueue.h:142 >> #7 0x00000000005e3ab0 in >> boost::detail::thread_data<Hypertable::ApplicationQueue::Worker>::run >> (this=0x7f90c4002c10) at >> /usr/local/include/boost/thread/detail/thread.hpp:56 >> #8 0x00007f90cb6b992f in thread_proxy () from >> /usr/local/lib/libboost_thread-mt.so >> #9 0x00007f90ca7bc3ea in start_thread () from /lib/libpthread.so.0 >> #10 0x00007f90c9d7fcbd in clone () from /lib/libc.so.6 >> #11 0x0000000000000000 in ?? () >> ---Type <return> to continue, or q <return> to quit--- >> >> Thread 4 (Thread 0x4e1f3950 (LWP 2973)): >> #0 0x00007f90ca7c02d9 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib/libpthread.so.0 >> #1 0x00000000005e35d8 in >> boost::condition_variable_any::wait<boost::unique_lock<boost::mutex> > >> (this=0x19cef90, m...@0x4e1f3060) >> at /usr/local/include/boost/thread/pthread/condition_variable.hpp:84 >> #2 0x00000000005e36f4 in >> Hypertable::ApplicationQueue::Worker::operator() (this=0x7f90c4002fe8) >> at /home/mateusz/hypertable/src/cc/AsyncComm/ApplicationQueue.h:98 >> #3 0x00000000005e3ab0 in >> boost::detail::thread_data<Hypertable::ApplicationQueue::Worker>::run >> (this=0x7f90c4002ee0) at >> /usr/local/include/boost/thread/detail/thread.hpp:56 >> #4 0x00007f90cb6b992f in thread_proxy () from >> /usr/local/lib/libboost_thread-mt.so >> #5 0x00007f90ca7bc3ea in start_thread () from /lib/libpthread.so.0 >> #6 0x00007f90c9d7fcbd in clone () from /lib/libc.so.6 >> #7 0x0000000000000000 in ?? () >> >> Thread 3 (Thread 0x4e9f4950 (LWP 2974)): >> #0 0x00007f90ca7c055d in pthread_cond_timedwait@@GLIBC_2.3.2 () from >> /lib/libpthread.so.0 >> #1 0x000000000061b819 in >> boost::condition_variable_any::timed_wait<boost::unique_lock<boost::mutex> >> > (this=0x7f90c4036dc8, m...@0x4e9efa80, wait_unt...@0x4e9efa00) >> at /usr/local/include/boost/thread/pthread/condition_variable.hpp:110 >> #2 0x000000000061b922 in >> boost::condition_variable_any::timed_wait<boost::unique_lock<boost::mutex> >> > (this=0x7f90c4036dc8, m...@0x4e9efa80, wait_unt...@0x4e9efa70) >> at /usr/local/include/boost/thread/pthread/condition_variable.hpp:127 >> #3 0x00000000006e6153 in >> Hypertable::TableMutatorCompletionCounter::wait_for_completion >> (this=0x7f90c4036da0, tim...@0x4e9efea0) >> at >> /home/mateusz/hypertable/src/cc/Hypertable/Lib/TableMutatorCompletionCounter.h:70 >> #4 0x00000000006e0076 in >> Hypertable::TableMutatorScatterBuffer::wait_for_completion >> (this=0x7f90c4036d10, tim...@0x4e9efea0) >> at >> /home/mateusz/hypertable/src/cc/Hypertable/Lib/TableMutatorScatterBuffer.cc:287 >> #5 0x00000000006d910a in >> Hypertable::TableMutator::wait_for_previous_buffer >> (this=0x7f90c4036c40, tim...@0x4e9efea0) >> at /home/mateusz/hypertable/src/cc/Hypertable/Lib/TableMutator.cc:280 >> #6 0x00000000006db79d in Hypertable::TableMutator::flush >> (this=0x7f90c4036c40) at >> /home/mateusz/hypertable/src/cc/Hypertable/Lib/TableMutator.cc:230 >> #7 0x00000000006749d6 in Hypertable::MetadataNormal::write_files >> (this=0x4e9f0000, ag_na...@0x7f90bd02b070, fil...@0x4e9f0220) >> at >> /home/mateusz/hypertable/src/cc/Hypertable/RangeServer/MetadataNormal.cc:103 >> #8 0x000000000066bb00 in >> Hypertable::LiveFileTracker::update_files_column (this=0x7f90bd02aff0) >> at >> /home/mateusz/hypertable/src/cc/Hypertable/RangeServer/LiveFileTracker.cc:109 >> #9 0x000000000064f3dd in Hypertable::AccessGroup::run_compaction >> (this=0x7f90bd02aec0, major=true) at >> /home/mateusz/hypertable/src/cc/Hypertable/RangeServer/AccessGroup.cc:510 >> #10 0x0000000000631c43 in Hypertable::Range::split_compact_and_shrink >> (this=0x7f90bcf7ee60) at >> /home/mateusz/hypertable/src/cc/Hypertable/RangeServer/Range.cc:550 >> #11 0x0000000000635e55 in Hypertable::Range::split >> (this=0x7f90bcf7ee60) at >> /home/mateusz/hypertable/src/cc/Hypertable/RangeServer/Range.cc:363 >> #12 0x000000000062ec63 in Hypertable::MaintenanceTaskSplit::execute >> (this=0x7f8fc2d957e0) at >> >> /home/mateusz/hypertable/src/cc/Hypertable/RangeServer/MaintenanceTaskSplit.cc:41 >> #13 0x000000000061bb56 in >> Hypertable::MaintenanceQueue::Worker::operator() (this=0x7f90c4015d98) >> at >> /home/mateusz/hypertable/src/cc/Hypertable/RangeServer/MaintenanceQueue.h:114 >> #14 0x000000000061c024 in >> boost::detail::thread_data<Hypertable::MaintenanceQueue::Worker>::run >> (this=0x7f90c4015c90) at >> /usr/local/include/boost/thread/detail/thread.hpp:56 >> #15 0x00007f90cb6b992f in thread_proxy () from >> /usr/local/lib/libboost_thread-mt.so >> #16 0x00007f90ca7bc3ea in start_thread () from /lib/libpthread.so.0 >> #17 0x00007f90c9d7fcbd in clone () from /lib/libc.so.6 >> #18 0x0000000000000000 in ?? () >> >> Thread 2 (Thread 0x4f1f5950 (LWP 2975)): >> #0 0x00007f90ca7c02d9 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib/libpthread.so.0 >> #1 0x00000000005e35d8 in >> boost::condition_variable_any::wait<boost::unique_lock<boost::mutex> > >> (this=0x7f90c40158d8, m...@0x4f1f5030) >> at /usr/local/include/boost/thread/pthread/condition_variable.hpp:84 >> #2 0x000000000061b9c2 in >> Hypertable::MaintenanceQueue::Worker::operator() (this=0x7f90c4016028) >> at >> /home/mateusz/hypertable/src/cc/Hypertable/RangeServer/MaintenanceQueue.h:93 >> #3 0x000000000061c024 in >> boost::detail::thread_data<Hypertable::MaintenanceQueue::Worker>::run >> (this=0x7f90c4015f20) at >> /usr/local/include/boost/thread/detail/thread.hpp:56 >> #4 0x00007f90cb6b992f in thread_proxy () from >> /usr/local/lib/libboost_thread-mt.so >> #5 0x00007f90ca7bc3ea in start_thread () from /lib/libpthread.so.0 >> #6 0x00007f90c9d7fcbd in clone () from /lib/libc.so.6 >> #7 0x0000000000000000 in ?? () >> ---Type <return> to continue, or q <return> to quit--- >> >> Thread 1 (Thread 0x7f90cc59f6f0 (LWP 2943)): >> #0 0x00007f90ca7c02d9 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib/libpthread.so.0 >> #1 0x00007f90cb6b8340 in boost::thread::join () from >> /usr/local/lib/libboost_thread-mt.so >> #2 0x00000000005e1feb in boost::thread_group::join_all >> (this=0x19ceff0) at >> /usr/local/include/boost/thread/detail/thread.hpp:541 >> #3 0x00000000005e207e in Hypertable::ApplicationQueue::join >> (this=0x19ceec0) at >> /home/mateusz/hypertable/src/cc/AsyncComm/ApplicationQueue.h:209 >> #4 0x00000000005d7c1d in main (argc=3, argv=0x7fffd46f9d58) at >> /home/mateusz/hypertable/src/cc/Hypertable/RangeServer/main.cc:92 >> #0 0x00007f90ca7c02d9 in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib/libpthread.so.0 >> (gdb) >> >> >> Mateusz >> >> > > > > > --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Hypertable Development" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/hypertable-dev?hl=en -~----------~----~----~----~------~----~------~--~---
