Hi Doug, It seems that the problem is now gone. I tested things thoroughly and it looks like we got one bug less.
Mateusz On Sat, Mar 21, 2009 at 9:35 PM, Mateusz Berezecki <[email protected]> wrote: > Hi Doug, > > Thanks for a quick patch. I'm running the tests for nearly 4 hours now > and the problem does not seem to be occurring. I will have a double > confirmed that after I finish my testing. Thanks again for the patch! > > > Mateusz > > On Sat, Mar 21, 2009 at 5:26 PM, Doug Judd <[email protected]> wrote: >> Hi Mateusz, >> >> This is a 5-way deadlock between threads 15, 11, 6, 5, and 3. I've >> attached a patch, please give it a try and let me know if it fixes the >> problem. The fix is also in my 'pu' branch. Thanks! >> >> - Doug >> >> On Sat, Mar 21, 2009 at 4:30 AM, Mateusz Berezecki <[email protected]> >> wrote: >>> >>> Hi Doug, >>> >>> Took a while but finally got it (Thread #3 ) >>> >>> (gdb) thread apply all where >>> >>> Thread 28 (Thread 0x421db950 (LWP 2944)): >>> #0 0x00007f90c9d80288 in epoll_wait () from /lib/libc.so.6 >>> #1 0x000000000077fe32 in Hypertable::ReactorRunner::operator() >>> (this=0x19ca2d8) at >>> /home/mateusz/hypertable/src/cc/AsyncComm/ReactorRunner.cc:70 >>> #2 0x000000000077f18a in >>> boost::detail::thread_data<Hypertable::ReactorRunner>::run >>> (this=0x19ca1d0) at >>> /usr/local/include/boost/thread/detail/thread.hpp:56 >>> #3 0x00007f90cb6b992f in thread_proxy () from >>> /usr/local/lib/libboost_thread-mt.so >>> #4 0x00007f90ca7bc3ea in start_thread () from /lib/libpthread.so.0 >>> #5 0x00007f90c9d7fcbd in clone () from /lib/libc.so.6 >>> #6 0x0000000000000000 in ?? () >>> >>> Thread 27 (Thread 0x429dc950 (LWP 2945)): >>> #0 0x00007f90c9d80288 in epoll_wait () from /lib/libc.so.6 >>> #1 0x000000000077fe32 in Hypertable::ReactorRunner::operator() >>> (this=0x19cf4d8) at >>> /home/mateusz/hypertable/src/cc/AsyncComm/ReactorRunner.cc:70 >>> #2 0x000000000077f18a in >>> boost::detail::thread_data<Hypertable::ReactorRunner>::run >>> (this=0x19cf3d0) at >>> /usr/local/include/boost/thread/detail/thread.hpp:56 >>> #3 0x00007f90cb6b992f in thread_proxy () from >>> /usr/local/lib/libboost_thread-mt.so >>> #4 0x00007f90ca7bc3ea in start_thread () from /lib/libpthread.so.0 >>> #5 0x00007f90c9d7fcbd in clone () from /lib/libc.so.6 >>> #6 0x0000000000000000 in ?? () >>> >>> Thread 26 (Thread 0x431dd950 (LWP 2946)): >>> #0 0x00007f90c9d80288 in epoll_wait () from /lib/libc.so.6 >>> #1 0x000000000077fe32 in Hypertable::ReactorRunner::operator() >>> (this=0x19cf808) at >>> /home/mateusz/hypertable/src/cc/AsyncComm/ReactorRunner.cc:70 >>> #2 0x000000000077f18a in >>> boost::detail::thread_data<Hypertable::ReactorRunner>::run >>> (this=0x19cf700) at >>> /usr/local/include/boost/thread/detail/thread.hpp:56 >>> #3 0x00007f90cb6b992f in thread_proxy () from >>> /usr/local/lib/libboost_thread-mt.so >>> #4 0x00007f90ca7bc3ea in start_thread () from /lib/libpthread.so.0 >>> #5 0x00007f90c9d7fcbd in clone () from /lib/libc.so.6 >>> #6 0x0000000000000000 in ?? () >>> >>> Thread 25 (Thread 0x439de950 (LWP 2947)): >>> #0 0x00007f90c9d80288 in epoll_wait () from /lib/libc.so.6 >>> #1 0x000000000077fe32 in Hypertable::ReactorRunner::operator() >>> (this=0x19ce868) at >>> /home/mateusz/hypertable/src/cc/AsyncComm/ReactorRunner.cc:70 >>> #2 0x000000000077f18a in >>> boost::detail::thread_data<Hypertable::ReactorRunner>::run >>> (this=0x19ce760) at >>> /usr/local/include/boost/thread/detail/thread.hpp:56 >>> #3 0x00007f90cb6b992f in thread_proxy () from >>> /usr/local/lib/libboost_thread-mt.so >>> #4 0x00007f90ca7bc3ea in start_thread () from /lib/libpthread.so.0 >>> #5 0x00007f90c9d7fcbd in clone () from /lib/libc.so.6 >>> #6 0x0000000000000000 in ?? () >>> >>> Thread 24 (Thread 0x441df950 (LWP 2948)): >>> #0 0x00007f90ca7c02d9 in pthread_cond_wait@@GLIBC_2.3.2 () from >>> /lib/libpthread.so.0 >>> #1 0x00000000005e35d8 in >>> boost::condition_variable_any::wait<boost::unique_lock<boost::mutex> > >>> (this=0x19ceb20, m...@0x441df090) >>> at /usr/local/include/boost/thread/pthread/condition_variable.hpp:84 >>> #2 0x000000000076ff63 in Hypertable::ConnectionManager::operator() >>> (this=0x19cece8) at >>> /home/mateusz/hypertable/src/cc/AsyncComm/ConnectionManager.cc:308 >>> #3 0x00000000005e33d0 in >>> boost::detail::thread_data<Hypertable::ConnectionManager>::run >>> (this=0x19cebe0) at >>> /usr/local/include/boost/thread/detail/thread.hpp:56 >>> #4 0x00007f90cb6b992f in thread_proxy () from >>> /usr/local/lib/libboost_thread-mt.so >>> #5 0x00007f90ca7bc3ea in start_thread () from /lib/libpthread.so.0 >>> #6 0x00007f90c9d7fcbd in clone () from /lib/libc.so.6 >>> #7 0x0000000000000000 in ?? () >>> >>> Thread 23 (Thread 0x449e0950 (LWP 2949)): >>> ---Type <return> to continue, or q <return> to quit--- >>> #0 0x00007f90ca7c02d9 in pthread_cond_wait@@GLIBC_2.3.2 () from >>> /lib/libpthread.so.0 >>> #1 0x00000000005e35d8 in >>> boost::condition_variable_any::wait<boost::unique_lock<boost::mutex> > >>> (this=0x19cef90, m...@0x449e0060) >>> at /usr/local/include/boost/thread/pthread/condition_variable.hpp:84 >>> #2 0x00000000005e36f4 in >>> Hypertable::ApplicationQueue::Worker::operator() (this=0x19cf148) at >>> /home/mateusz/hypertable/src/cc/AsyncComm/ApplicationQueue.h:98 >>> #3 0x00000000005e3ab0 in >>> boost::detail::thread_data<Hypertable::ApplicationQueue::Worker>::run >>> (this=0x19cf040) at >>> /usr/local/include/boost/thread/detail/thread.hpp:56 >>> #4 0x00007f90cb6b992f in thread_proxy () from >>> /usr/local/lib/libboost_thread-mt.so >>> #5 0x00007f90ca7bc3ea in start_thread () from /lib/libpthread.so.0 >>> #6 0x00007f90c9d7fcbd in clone () from /lib/libc.so.6 >>> #7 0x0000000000000000 in ?? () >>> >>> Thread 22 (Thread 0x451e1950 (LWP 2951)): >>> #0 0x00007f90ca7c02d9 in pthread_cond_wait@@GLIBC_2.3.2 () from >>> /lib/libpthread.so.0 >>> #1 0x00000000005e35d8 in >>> boost::condition_variable_any::wait<boost::unique_lock<boost::mutex> > >>> (this=0x19cef90, m...@0x451e1060) >>> at /usr/local/include/boost/thread/pthread/condition_variable.hpp:84 >>> #2 0x00000000005e36f4 in >>> Hypertable::ApplicationQueue::Worker::operator() (this=0x19d0978) at >>> /home/mateusz/hypertable/src/cc/AsyncComm/ApplicationQueue.h:98 >>> #3 0x00000000005e3ab0 in >>> boost::detail::thread_data<Hypertable::ApplicationQueue::Worker>::run >>> (this=0x19d0870) at >>> /usr/local/include/boost/thread/detail/thread.hpp:56 >>> #4 0x00007f90cb6b992f in thread_proxy () from >>> /usr/local/lib/libboost_thread-mt.so >>> #5 0x00007f90ca7bc3ea in start_thread () from /lib/libpthread.so.0 >>> #6 0x00007f90c9d7fcbd in clone () from /lib/libc.so.6 >>> #7 0x0000000000000000 in ?? () >>> >>> Thread 21 (Thread 0x459e2950 (LWP 2952)): >>> #0 0x00007f90ca7c02d9 in pthread_cond_wait@@GLIBC_2.3.2 () from >>> /lib/libpthread.so.0 >>> #1 0x00000000005e35d8 in >>> boost::condition_variable_any::wait<boost::unique_lock<boost::mutex> > >>> (this=0x19cef90, m...@0x459e2060) >>> at /usr/local/include/boost/thread/pthread/condition_variable.hpp:84 >>> #2 0x00000000005e36f4 in >>> Hypertable::ApplicationQueue::Worker::operator() (this=0x19d0c68) at >>> /home/mateusz/hypertable/src/cc/AsyncComm/ApplicationQueue.h:98 >>> #3 0x00000000005e3ab0 in >>> boost::detail::thread_data<Hypertable::ApplicationQueue::Worker>::run >>> (this=0x19d0b60) at >>> /usr/local/include/boost/thread/detail/thread.hpp:56 >>> #4 0x00007f90cb6b992f in thread_proxy () from >>> /usr/local/lib/libboost_thread-mt.so >>> #5 0x00007f90ca7bc3ea in start_thread () from /lib/libpthread.so.0 >>> #6 0x00007f90c9d7fcbd in clone () from /lib/libc.so.6 >>> #7 0x0000000000000000 in ?? () >>> >>> Thread 20 (Thread 0x461e3950 (LWP 2954)): >>> #0 0x00007f90ca7c02d9 in pthread_cond_wait@@GLIBC_2.3.2 () from >>> /lib/libpthread.so.0 >>> #1 0x00000000005e35d8 in >>> boost::condition_variable_any::wait<boost::unique_lock<boost::mutex> > >>> (this=0x19cef90, m...@0x461e3060) >>> at /usr/local/include/boost/thread/pthread/condition_variable.hpp:84 >>> #2 0x00000000005e36f4 in >>> Hypertable::ApplicationQueue::Worker::operator() (this=0x19d0f58) at >>> /home/mateusz/hypertable/src/cc/AsyncComm/ApplicationQueue.h:98 >>> #3 0x00000000005e3ab0 in >>> boost::detail::thread_data<Hypertable::ApplicationQueue::Worker>::run >>> (this=0x19d0e50) at >>> /usr/local/include/boost/thread/detail/thread.hpp:56 >>> #4 0x00007f90cb6b992f in thread_proxy () from >>> /usr/local/lib/libboost_thread-mt.so >>> #5 0x00007f90ca7bc3ea in start_thread () from /lib/libpthread.so.0 >>> #6 0x00007f90c9d7fcbd in clone () from /lib/libc.so.6 >>> #7 0x0000000000000000 in ?? () >>> >>> Thread 19 (Thread 0x469e4950 (LWP 2955)): >>> #0 0x00007f90ca7c02d9 in pthread_cond_wait@@GLIBC_2.3.2 () from >>> /lib/libpthread.so.0 >>> #1 0x00000000005e35d8 in >>> boost::condition_variable_any::wait<boost::unique_lock<boost::mutex> > >>> (this=0x19cef90, m...@0x469e4060) >>> at /usr/local/include/boost/thread/pthread/condition_variable.hpp:84 >>> #2 0x00000000005e36f4 in >>> Hypertable::ApplicationQueue::Worker::operator() (this=0x19d1248) at >>> /home/mateusz/hypertable/src/cc/AsyncComm/ApplicationQueue.h:98 >>> #3 0x00000000005e3ab0 in >>> boost::detail::thread_data<Hypertable::ApplicationQueue::Worker>::run >>> (this=0x19d1140) at >>> /usr/local/include/boost/thread/detail/thread.hpp:56 >>> ---Type <return> to continue, or q <return> to quit--- >>> #4 0x00007f90cb6b992f in thread_proxy () from >>> /usr/local/lib/libboost_thread-mt.so >>> #5 0x00007f90ca7bc3ea in start_thread () from /lib/libpthread.so.0 >>> #6 0x00007f90c9d7fcbd in clone () from /lib/libc.so.6 >>> #7 0x0000000000000000 in ?? () >>> >>> Thread 18 (Thread 0x471e5950 (LWP 2957)): >>> #0 0x00007f90ca7c02d9 in pthread_cond_wait@@GLIBC_2.3.2 () from >>> /lib/libpthread.so.0 >>> #1 0x00000000005e35d8 in >>> boost::condition_variable_any::wait<boost::unique_lock<boost::mutex> > >>> (this=0x19cef90, m...@0x471e5060) >>> at /usr/local/include/boost/thread/pthread/condition_variable.hpp:84 >>> #2 0x00000000005e36f4 in >>> Hypertable::ApplicationQueue::Worker::operator() (this=0x19d1538) at >>> /home/mateusz/hypertable/src/cc/AsyncComm/ApplicationQueue.h:98 >>> #3 0x00000000005e3ab0 in >>> boost::detail::thread_data<Hypertable::ApplicationQueue::Worker>::run >>> (this=0x19d1430) at >>> /usr/local/include/boost/thread/detail/thread.hpp:56 >>> #4 0x00007f90cb6b992f in thread_proxy () from >>> /usr/local/lib/libboost_thread-mt.so >>> #5 0x00007f90ca7bc3ea in start_thread () from /lib/libpthread.so.0 >>> #6 0x00007f90c9d7fcbd in clone () from /lib/libc.so.6 >>> #7 0x0000000000000000 in ?? () >>> >>> Thread 17 (Thread 0x479e6950 (LWP 2958)): >>> #0 0x00007f90ca7c02d9 in pthread_cond_wait@@GLIBC_2.3.2 () from >>> /lib/libpthread.so.0 >>> #1 0x00000000005e35d8 in >>> boost::condition_variable_any::wait<boost::unique_lock<boost::mutex> > >>> (this=0x19cef90, m...@0x479e6060) >>> at /usr/local/include/boost/thread/pthread/condition_variable.hpp:84 >>> #2 0x00000000005e36f4 in >>> Hypertable::ApplicationQueue::Worker::operator() (this=0x7f90c4000b58) >>> at /home/mateusz/hypertable/src/cc/AsyncComm/ApplicationQueue.h:98 >>> #3 0x00000000005e3ab0 in >>> boost::detail::thread_data<Hypertable::ApplicationQueue::Worker>::run >>> (this=0x7f90c4000a50) at >>> /usr/local/include/boost/thread/detail/thread.hpp:56 >>> #4 0x00007f90cb6b992f in thread_proxy () from >>> /usr/local/lib/libboost_thread-mt.so >>> #5 0x00007f90ca7bc3ea in start_thread () from /lib/libpthread.so.0 >>> #6 0x00007f90c9d7fcbd in clone () from /lib/libc.so.6 >>> #7 0x0000000000000000 in ?? () >>> >>> Thread 16 (Thread 0x481e7950 (LWP 2960)): >>> #0 0x00007f90ca7c02d9 in pthread_cond_wait@@GLIBC_2.3.2 () from >>> /lib/libpthread.so.0 >>> #1 0x00000000005e35d8 in >>> boost::condition_variable_any::wait<boost::unique_lock<boost::mutex> > >>> (this=0x19cef90, m...@0x481e7060) >>> at /usr/local/include/boost/thread/pthread/condition_variable.hpp:84 >>> #2 0x00000000005e36f4 in >>> Hypertable::ApplicationQueue::Worker::operator() (this=0x7f90c4000e28) >>> at /home/mateusz/hypertable/src/cc/AsyncComm/ApplicationQueue.h:98 >>> #3 0x00000000005e3ab0 in >>> boost::detail::thread_data<Hypertable::ApplicationQueue::Worker>::run >>> (this=0x7f90c4000d20) at >>> /usr/local/include/boost/thread/detail/thread.hpp:56 >>> #4 0x00007f90cb6b992f in thread_proxy () from >>> /usr/local/lib/libboost_thread-mt.so >>> #5 0x00007f90ca7bc3ea in start_thread () from /lib/libpthread.so.0 >>> #6 0x00007f90c9d7fcbd in clone () from /lib/libc.so.6 >>> #7 0x0000000000000000 in ?? () >>> >>> Thread 15 (Thread 0x489e8950 (LWP 2961)): >>> #0 0x00007f90ca7c2b04 in __lll_lock_wait () from /lib/libpthread.so.0 >>> #1 0x00007f90ca7be1a0 in _L_lock_102 () from /lib/libpthread.so.0 >>> #2 0x00007f90ca7bdaae in pthread_mutex_lock () from /lib/libpthread.so.0 >>> #3 0x00000000005dd21f in boost::mutex::lock (this=0x7f90bd02b018) at >>> /usr/local/include/boost/thread/pthread/mutex.hpp:50 >>> #4 0x000000000066ba18 in >>> Hypertable::LiveFileTracker::update_files_column (this=0x7f90bd02aff0) >>> at >>> /home/mateusz/hypertable/src/cc/Hypertable/RangeServer/LiveFileTracker.cc:99 >>> #5 0x000000000064bb26 in Hypertable::AccessGroup::release_files >>> (this=0x7f90bd02aec0, fil...@0x7f907d525a98) >>> at >>> /home/mateusz/hypertable/src/cc/Hypertable/RangeServer/AccessGroup.cc:622 >>> #6 0x000000000065b2c8 in >>> Hypertable::CellStoreReleaseCallback::operator() (this=0x7f907d525a90) >>> at >>> /home/mateusz/hypertable/src/cc/Hypertable/RangeServer/CellStoreReleaseCallback.cc:38 >>> #7 0x00000000006703b2 in ~MergeScanner (this=0x7f907d525940) at >>> /home/mateusz/hypertable/src/cc/Hypertable/RangeServer/MergeScanner.cc:60 >>> ---Type <return> to continue, or q <return> to quit--- >>> #8 0x000000000067035e in ~MergeScanner (this=0x62184c0) at >>> /home/mateusz/hypertable/src/cc/Hypertable/RangeServer/MergeScanner.cc:58 >>> #9 0x00000000005d844b in Hypertable::intrusive_ptr_release >>> (rc=0x62184c0) at >>> /home/mateusz/hypertable/src/cc/Common/ReferenceCount.h:73 >>> #10 0x00000000005e8bfd in ~intrusive_ptr (this=0x489e5cf0) at >>> /usr/local/include/boost/intrusive_ptr.hpp:101 >>> #11 0x00000000005fec2b in Hypertable::RangeServer::create_scanner >>> (this=0x7f90c40147a0, cb=0x489e6f70, table=0x489e6fa0, >>> range_spec=0x489e6f90, scan_spec=0x489e6f00) >>> at >>> /home/mateusz/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:689 >>> #12 0x0000000000643833 in Hypertable::RequestHandlerCreateScanner::run >>> (this=0x7f90868d9030) >>> at >>> /home/mateusz/hypertable/src/cc/Hypertable/RangeServer/RequestHandlerCreateScanner.cc:52 >>> #13 0x00000000005e39d4 in >>> Hypertable::ApplicationQueue::Worker::operator() (this=0x7f90c40010f8) >>> at /home/mateusz/hypertable/src/cc/AsyncComm/ApplicationQueue.h:142 >>> #14 0x00000000005e3ab0 in >>> boost::detail::thread_data<Hypertable::ApplicationQueue::Worker>::run >>> (this=0x7f90c4000ff0) at >>> /usr/local/include/boost/thread/detail/thread.hpp:56 >>> #15 0x00007f90cb6b992f in thread_proxy () from >>> /usr/local/lib/libboost_thread-mt.so >>> #16 0x00007f90ca7bc3ea in start_thread () from /lib/libpthread.so.0 >>> #17 0x00007f90c9d7fcbd in clone () from /lib/libc.so.6 >>> #18 0x0000000000000000 in ?? () >>> >>> Thread 14 (Thread 0x491e9950 (LWP 2962)): >>> #0 0x00007f90ca7c02d9 in pthread_cond_wait@@GLIBC_2.3.2 () from >>> /lib/libpthread.so.0 >>> #1 0x00000000005e35d8 in >>> boost::condition_variable_any::wait<boost::unique_lock<boost::mutex> > >>> (this=0x19cef90, m...@0x491e9060) >>> at /usr/local/include/boost/thread/pthread/condition_variable.hpp:84 >>> #2 0x00000000005e36f4 in >>> Hypertable::ApplicationQueue::Worker::operator() (this=0x7f90c40013c8) >>> at /home/mateusz/hypertable/src/cc/AsyncComm/ApplicationQueue.h:98 >>> #3 0x00000000005e3ab0 in >>> boost::detail::thread_data<Hypertable::ApplicationQueue::Worker>::run >>> (this=0x7f90c40012c0) at >>> /usr/local/include/boost/thread/detail/thread.hpp:56 >>> #4 0x00007f90cb6b992f in thread_proxy () from >>> /usr/local/lib/libboost_thread-mt.so >>> #5 0x00007f90ca7bc3ea in start_thread () from /lib/libpthread.so.0 >>> #6 0x00007f90c9d7fcbd in clone () from /lib/libc.so.6 >>> #7 0x0000000000000000 in ?? () >>> >>> Thread 13 (Thread 0x499ea950 (LWP 2963)): >>> #0 0x00007f90ca7c02d9 in pthread_cond_wait@@GLIBC_2.3.2 () from >>> /lib/libpthread.so.0 >>> #1 0x00000000005e35d8 in >>> boost::condition_variable_any::wait<boost::unique_lock<boost::mutex> > >>> (this=0x19cef90, m...@0x499ea060) >>> at /usr/local/include/boost/thread/pthread/condition_variable.hpp:84 >>> #2 0x00000000005e36f4 in >>> Hypertable::ApplicationQueue::Worker::operator() (this=0x7f90c4001698) >>> at /home/mateusz/hypertable/src/cc/AsyncComm/ApplicationQueue.h:98 >>> #3 0x00000000005e3ab0 in >>> boost::detail::thread_data<Hypertable::ApplicationQueue::Worker>::run >>> (this=0x7f90c4001590) at >>> /usr/local/include/boost/thread/detail/thread.hpp:56 >>> #4 0x00007f90cb6b992f in thread_proxy () from >>> /usr/local/lib/libboost_thread-mt.so >>> #5 0x00007f90ca7bc3ea in start_thread () from /lib/libpthread.so.0 >>> #6 0x00007f90c9d7fcbd in clone () from /lib/libc.so.6 >>> #7 0x0000000000000000 in ?? () >>> >>> Thread 12 (Thread 0x4a1eb950 (LWP 2964)): >>> #0 0x00007f90ca7c02d9 in pthread_cond_wait@@GLIBC_2.3.2 () from >>> /lib/libpthread.so.0 >>> #1 0x00000000005e35d8 in >>> boost::condition_variable_any::wait<boost::unique_lock<boost::mutex> > >>> (this=0x19cef90, m...@0x4a1eb060) >>> at /usr/local/include/boost/thread/pthread/condition_variable.hpp:84 >>> #2 0x00000000005e36f4 in >>> Hypertable::ApplicationQueue::Worker::operator() (this=0x7f90c4001968) >>> at /home/mateusz/hypertable/src/cc/AsyncComm/ApplicationQueue.h:98 >>> #3 0x00000000005e3ab0 in >>> boost::detail::thread_data<Hypertable::ApplicationQueue::Worker>::run >>> (this=0x7f90c4001860) at >>> /usr/local/include/boost/thread/detail/thread.hpp:56 >>> #4 0x00007f90cb6b992f in thread_proxy () from >>> /usr/local/lib/libboost_thread-mt.so >>> #5 0x00007f90ca7bc3ea in start_thread () from /lib/libpthread.so.0 >>> #6 0x00007f90c9d7fcbd in clone () from /lib/libc.so.6 >>> #7 0x0000000000000000 in ?? () >>> >>> Thread 11 (Thread 0x4a9ec950 (LWP 2965)): >>> #0 0x00007f90ca7c2b04 in __lll_lock_wait () from /lib/libpthread.so.0 >>> ---Type <return> to continue, or q <return> to quit--- >>> #1 0x00007f90ca7be1a0 in _L_lock_102 () from /lib/libpthread.so.0 >>> #2 0x00007f90ca7bdaae in pthread_mutex_lock () from /lib/libpthread.so.0 >>> #3 0x00000000005dd21f in boost::mutex::lock (this=0x7f90bd02aff0) at >>> /usr/local/include/boost/thread/pthread/mutex.hpp:50 >>> #4 0x00000000005dd2b4 in boost::unique_lock<boost::mutex>::lock >>> (this=0x4a9e95e0) at /usr/local/include/boost/thread/locks.hpp:338 >>> #5 0x00000000005dd2f2 in unique_lock (this=0x4a9e95e0, >>> m...@0x7f90bd02aff0) at /usr/local/include/boost/thread/locks.hpp:224 >>> #6 0x000000000066afc2 in Hypertable::LiveFileTracker::add_references >>> (this=0x7f90bd02aff0, fil...@0x4a9e9658) >>> at >>> /home/mateusz/hypertable/src/cc/Hypertable/RangeServer/LiveFileTracker.cc:45 >>> #7 0x000000000064ff42 in Hypertable::AccessGroup::create_scanner >>> (this=0x7f90bd02aec0, scan_conte...@0x4a9e9cd0) >>> at >>> /home/mateusz/hypertable/src/cc/Hypertable/RangeServer/AccessGroup.cc:181 >>> #8 0x00000000006364a2 in Hypertable::Range::create_scanner >>> (this=0x7f90bcf7ee60, scan_c...@0x4a9e9cd0) at >>> /home/mateusz/hypertable/src/cc/Hypertable/RangeServer/Range.cc:309 >>> #9 0x00000000005fe23b in Hypertable::RangeServer::create_scanner >>> (this=0x7f90c40147a0, cb=0x4a9eaf70, table=0x4a9eafa0, >>> range_spec=0x4a9eaf90, scan_spec=0x4a9eaf00) >>> at >>> /home/mateusz/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:654 >>> #10 0x0000000000643833 in Hypertable::RequestHandlerCreateScanner::run >>> (this=0x7f90a51293e0) >>> at >>> /home/mateusz/hypertable/src/cc/Hypertable/RangeServer/RequestHandlerCreateScanner.cc:52 >>> #11 0x00000000005e39d4 in >>> Hypertable::ApplicationQueue::Worker::operator() (this=0x7f90c4001c38) >>> at /home/mateusz/hypertable/src/cc/AsyncComm/ApplicationQueue.h:142 >>> #12 0x00000000005e3ab0 in >>> boost::detail::thread_data<Hypertable::ApplicationQueue::Worker>::run >>> (this=0x7f90c4001b30) at >>> /usr/local/include/boost/thread/detail/thread.hpp:56 >>> #13 0x00007f90cb6b992f in thread_proxy () from >>> /usr/local/lib/libboost_thread-mt.so >>> #14 0x00007f90ca7bc3ea in start_thread () from /lib/libpthread.so.0 >>> #15 0x00007f90c9d7fcbd in clone () from /lib/libc.so.6 >>> #16 0x0000000000000000 in ?? () >>> >>> Thread 10 (Thread 0x4b1ed950 (LWP 2967)): >>> #0 0x00007f90ca7c02d9 in pthread_cond_wait@@GLIBC_2.3.2 () from >>> /lib/libpthread.so.0 >>> #1 0x00000000005e35d8 in >>> boost::condition_variable_any::wait<boost::unique_lock<boost::mutex> > >>> (this=0x19cef90, m...@0x4b1ed060) >>> at /usr/local/include/boost/thread/pthread/condition_variable.hpp:84 >>> #2 0x00000000005e36f4 in >>> Hypertable::ApplicationQueue::Worker::operator() (this=0x7f90c4001f08) >>> at /home/mateusz/hypertable/src/cc/AsyncComm/ApplicationQueue.h:98 >>> #3 0x00000000005e3ab0 in >>> boost::detail::thread_data<Hypertable::ApplicationQueue::Worker>::run >>> (this=0x7f90c4001e00) at >>> /usr/local/include/boost/thread/detail/thread.hpp:56 >>> #4 0x00007f90cb6b992f in thread_proxy () from >>> /usr/local/lib/libboost_thread-mt.so >>> #5 0x00007f90ca7bc3ea in start_thread () from /lib/libpthread.so.0 >>> #6 0x00007f90c9d7fcbd in clone () from /lib/libc.so.6 >>> #7 0x0000000000000000 in ?? () >>> >>> Thread 9 (Thread 0x4b9ee950 (LWP 2968)): >>> #0 0x00007f90ca7c02d9 in pthread_cond_wait@@GLIBC_2.3.2 () from >>> /lib/libpthread.so.0 >>> #1 0x00000000005e35d8 in >>> boost::condition_variable_any::wait<boost::unique_lock<boost::mutex> > >>> (this=0x19cef90, m...@0x4b9ee060) >>> at /usr/local/include/boost/thread/pthread/condition_variable.hpp:84 >>> #2 0x00000000005e36f4 in >>> Hypertable::ApplicationQueue::Worker::operator() (this=0x7f90c40021d8) >>> at /home/mateusz/hypertable/src/cc/AsyncComm/ApplicationQueue.h:98 >>> #3 0x00000000005e3ab0 in >>> boost::detail::thread_data<Hypertable::ApplicationQueue::Worker>::run >>> (this=0x7f90c40020d0) at >>> /usr/local/include/boost/thread/detail/thread.hpp:56 >>> #4 0x00007f90cb6b992f in thread_proxy () from >>> /usr/local/lib/libboost_thread-mt.so >>> #5 0x00007f90ca7bc3ea in start_thread () from /lib/libpthread.so.0 >>> #6 0x00007f90c9d7fcbd in clone () from /lib/libc.so.6 >>> #7 0x0000000000000000 in ?? () >>> >>> Thread 8 (Thread 0x4c1ef950 (LWP 2969)): >>> #0 0x00007f90ca7c02d9 in pthread_cond_wait@@GLIBC_2.3.2 () from >>> /lib/libpthread.so.0 >>> #1 0x00000000005e35d8 in >>> boost::condition_variable_any::wait<boost::unique_lock<boost::mutex> > >>> (this=0x19cef90, m...@0x4c1ef060) >>> at /usr/local/include/boost/thread/pthread/condition_variable.hpp:84 >>> #2 0x00000000005e36f4 in >>> Hypertable::ApplicationQueue::Worker::operator() (this=0x7f90c40024a8) >>> at /home/mateusz/hypertable/src/cc/AsyncComm/ApplicationQueue.h:98 >>> #3 0x00000000005e3ab0 in >>> boost::detail::thread_data<Hypertable::ApplicationQueue::Worker>::run >>> (this=0x7f90c40023a0) at >>> /usr/local/include/boost/thread/detail/thread.hpp:56 >>> ---Type <return> to continue, or q <return> to quit--- >>> #4 0x00007f90cb6b992f in thread_proxy () from >>> /usr/local/lib/libboost_thread-mt.so >>> #5 0x00007f90ca7bc3ea in start_thread () from /lib/libpthread.so.0 >>> #6 0x00007f90c9d7fcbd in clone () from /lib/libc.so.6 >>> #7 0x0000000000000000 in ?? () >>> >>> Thread 7 (Thread 0x4c9f0950 (LWP 2970)): >>> #0 0x00007f90ca7c02d9 in pthread_cond_wait@@GLIBC_2.3.2 () from >>> /lib/libpthread.so.0 >>> #1 0x00000000005e35d8 in >>> boost::condition_variable_any::wait<boost::unique_lock<boost::mutex> > >>> (this=0x19cef90, m...@0x4c9f0060) >>> at /usr/local/include/boost/thread/pthread/condition_variable.hpp:84 >>> #2 0x00000000005e36f4 in >>> Hypertable::ApplicationQueue::Worker::operator() (this=0x7f90c4002778) >>> at /home/mateusz/hypertable/src/cc/AsyncComm/ApplicationQueue.h:98 >>> #3 0x00000000005e3ab0 in >>> boost::detail::thread_data<Hypertable::ApplicationQueue::Worker>::run >>> (this=0x7f90c4002670) at >>> /usr/local/include/boost/thread/detail/thread.hpp:56 >>> #4 0x00007f90cb6b992f in thread_proxy () from >>> /usr/local/lib/libboost_thread-mt.so >>> #5 0x00007f90ca7bc3ea in start_thread () from /lib/libpthread.so.0 >>> #6 0x00007f90c9d7fcbd in clone () from /lib/libc.so.6 >>> #7 0x0000000000000000 in ?? () >>> >>> Thread 6 (Thread 0x4d1f1950 (LWP 2971)): >>> #0 0x00007f90ca7c2b04 in __lll_lock_wait () from /lib/libpthread.so.0 >>> #1 0x00007f90ca7be1a0 in _L_lock_102 () from /lib/libpthread.so.0 >>> #2 0x00007f90ca7bdaae in pthread_mutex_lock () from /lib/libpthread.so.0 >>> #3 0x00000000005dd21f in boost::mutex::lock (this=0x7f90bd02aee0) at >>> /usr/local/include/boost/thread/pthread/mutex.hpp:50 >>> #4 0x000000000063d8d9 in Hypertable::AccessGroup::lock >>> (this=0x7f90bd02aec0) at >>> /home/mateusz/hypertable/src/cc/Hypertable/RangeServer/AccessGroup.h:84 >>> #5 0x000000000062ee6f in Hypertable::Range::lock >>> (this=0x7f90bcf7ee60) at >>> /home/mateusz/hypertable/src/cc/Hypertable/RangeServer/Range.cc:912 >>> #6 0x0000000000617c93 in Hypertable::Locker<Hypertable::Range>::lock >>> (this=0x4d1ee910) at /home/mateusz/hypertable/src/cc/Common/Mutex.h:52 >>> #7 0x0000000000617ce5 in Locker (this=0x4d1ee910, >>> mut...@0x7f90bcf7ee60, init_lock=true) at >>> /home/mateusz/hypertable/src/cc/Common/Mutex.h:40 >>> #8 0x00000000005f681d in Hypertable::RangeServer::update >>> (this=0x7f90c40147a0, cb=0x4d1eff70, table=0x4d1effa0, count=232, >>> buff...@0x4d1eff90) >>> at >>> /home/mateusz/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:1405 >>> #9 0x0000000000649909 in Hypertable::RequestHandlerUpdate::run >>> (this=0x7f902e1f8900) at >>> >>> /home/mateusz/hypertable/src/cc/Hypertable/RangeServer/RequestHandlerUpdate.cc:54 >>> #10 0x00000000005e39d4 in >>> Hypertable::ApplicationQueue::Worker::operator() (this=0x7f90c4002a48) >>> at /home/mateusz/hypertable/src/cc/AsyncComm/ApplicationQueue.h:142 >>> #11 0x00000000005e3ab0 in >>> boost::detail::thread_data<Hypertable::ApplicationQueue::Worker>::run >>> (this=0x7f90c4002940) at >>> /usr/local/include/boost/thread/detail/thread.hpp:56 >>> #12 0x00007f90cb6b992f in thread_proxy () from >>> /usr/local/lib/libboost_thread-mt.so >>> #13 0x00007f90ca7bc3ea in start_thread () from /lib/libpthread.so.0 >>> #14 0x00007f90c9d7fcbd in clone () from /lib/libc.so.6 >>> #15 0x0000000000000000 in ?? () >>> >>> Thread 5 (Thread 0x4d9f2950 (LWP 2972)): >>> #0 0x00007f90ca7c2b04 in __lll_lock_wait () from /lib/libpthread.so.0 >>> #1 0x00007f90ca7be1a0 in _L_lock_102 () from /lib/libpthread.so.0 >>> #2 0x00007f90ca7bdaae in pthread_mutex_lock () from /lib/libpthread.so.0 >>> #3 0x00000000005dd21f in boost::mutex::lock (this=0x7f90c4014938) at >>> /usr/local/include/boost/thread/pthread/mutex.hpp:50 >>> #4 0x00000000005f6429 in Hypertable::RangeServer::update >>> (this=0x7f90c40147a0, cb=0x4d9f0f70, table=0x4d9f0fa0, count=1, >>> buff...@0x4d9f0f90) >>> at >>> /home/mateusz/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:1372 >>> #5 0x0000000000649909 in Hypertable::RequestHandlerUpdate::run >>> (this=0x7f904ad6ea00) at >>> >>> /home/mateusz/hypertable/src/cc/Hypertable/RangeServer/RequestHandlerUpdate.cc:54 >>> #6 0x00000000005e39d4 in >>> Hypertable::ApplicationQueue::Worker::operator() (this=0x7f90c4002d18) >>> at /home/mateusz/hypertable/src/cc/AsyncComm/ApplicationQueue.h:142 >>> #7 0x00000000005e3ab0 in >>> boost::detail::thread_data<Hypertable::ApplicationQueue::Worker>::run >>> (this=0x7f90c4002c10) at >>> /usr/local/include/boost/thread/detail/thread.hpp:56 >>> #8 0x00007f90cb6b992f in thread_proxy () from >>> /usr/local/lib/libboost_thread-mt.so >>> #9 0x00007f90ca7bc3ea in start_thread () from /lib/libpthread.so.0 >>> #10 0x00007f90c9d7fcbd in clone () from /lib/libc.so.6 >>> #11 0x0000000000000000 in ?? () >>> ---Type <return> to continue, or q <return> to quit--- >>> >>> Thread 4 (Thread 0x4e1f3950 (LWP 2973)): >>> #0 0x00007f90ca7c02d9 in pthread_cond_wait@@GLIBC_2.3.2 () from >>> /lib/libpthread.so.0 >>> #1 0x00000000005e35d8 in >>> boost::condition_variable_any::wait<boost::unique_lock<boost::mutex> > >>> (this=0x19cef90, m...@0x4e1f3060) >>> at /usr/local/include/boost/thread/pthread/condition_variable.hpp:84 >>> #2 0x00000000005e36f4 in >>> Hypertable::ApplicationQueue::Worker::operator() (this=0x7f90c4002fe8) >>> at /home/mateusz/hypertable/src/cc/AsyncComm/ApplicationQueue.h:98 >>> #3 0x00000000005e3ab0 in >>> boost::detail::thread_data<Hypertable::ApplicationQueue::Worker>::run >>> (this=0x7f90c4002ee0) at >>> /usr/local/include/boost/thread/detail/thread.hpp:56 >>> #4 0x00007f90cb6b992f in thread_proxy () from >>> /usr/local/lib/libboost_thread-mt.so >>> #5 0x00007f90ca7bc3ea in start_thread () from /lib/libpthread.so.0 >>> #6 0x00007f90c9d7fcbd in clone () from /lib/libc.so.6 >>> #7 0x0000000000000000 in ?? () >>> >>> Thread 3 (Thread 0x4e9f4950 (LWP 2974)): >>> #0 0x00007f90ca7c055d in pthread_cond_timedwait@@GLIBC_2.3.2 () from >>> /lib/libpthread.so.0 >>> #1 0x000000000061b819 in >>> boost::condition_variable_any::timed_wait<boost::unique_lock<boost::mutex> >>> > (this=0x7f90c4036dc8, m...@0x4e9efa80, wait_unt...@0x4e9efa00) >>> at /usr/local/include/boost/thread/pthread/condition_variable.hpp:110 >>> #2 0x000000000061b922 in >>> boost::condition_variable_any::timed_wait<boost::unique_lock<boost::mutex> >>> > (this=0x7f90c4036dc8, m...@0x4e9efa80, wait_unt...@0x4e9efa70) >>> at /usr/local/include/boost/thread/pthread/condition_variable.hpp:127 >>> #3 0x00000000006e6153 in >>> Hypertable::TableMutatorCompletionCounter::wait_for_completion >>> (this=0x7f90c4036da0, tim...@0x4e9efea0) >>> at >>> /home/mateusz/hypertable/src/cc/Hypertable/Lib/TableMutatorCompletionCounter.h:70 >>> #4 0x00000000006e0076 in >>> Hypertable::TableMutatorScatterBuffer::wait_for_completion >>> (this=0x7f90c4036d10, tim...@0x4e9efea0) >>> at >>> /home/mateusz/hypertable/src/cc/Hypertable/Lib/TableMutatorScatterBuffer.cc:287 >>> #5 0x00000000006d910a in >>> Hypertable::TableMutator::wait_for_previous_buffer >>> (this=0x7f90c4036c40, tim...@0x4e9efea0) >>> at /home/mateusz/hypertable/src/cc/Hypertable/Lib/TableMutator.cc:280 >>> #6 0x00000000006db79d in Hypertable::TableMutator::flush >>> (this=0x7f90c4036c40) at >>> /home/mateusz/hypertable/src/cc/Hypertable/Lib/TableMutator.cc:230 >>> #7 0x00000000006749d6 in Hypertable::MetadataNormal::write_files >>> (this=0x4e9f0000, ag_na...@0x7f90bd02b070, fil...@0x4e9f0220) >>> at >>> /home/mateusz/hypertable/src/cc/Hypertable/RangeServer/MetadataNormal.cc:103 >>> #8 0x000000000066bb00 in >>> Hypertable::LiveFileTracker::update_files_column (this=0x7f90bd02aff0) >>> at >>> /home/mateusz/hypertable/src/cc/Hypertable/RangeServer/LiveFileTracker.cc:109 >>> #9 0x000000000064f3dd in Hypertable::AccessGroup::run_compaction >>> (this=0x7f90bd02aec0, major=true) at >>> /home/mateusz/hypertable/src/cc/Hypertable/RangeServer/AccessGroup.cc:510 >>> #10 0x0000000000631c43 in Hypertable::Range::split_compact_and_shrink >>> (this=0x7f90bcf7ee60) at >>> /home/mateusz/hypertable/src/cc/Hypertable/RangeServer/Range.cc:550 >>> #11 0x0000000000635e55 in Hypertable::Range::split >>> (this=0x7f90bcf7ee60) at >>> /home/mateusz/hypertable/src/cc/Hypertable/RangeServer/Range.cc:363 >>> #12 0x000000000062ec63 in Hypertable::MaintenanceTaskSplit::execute >>> (this=0x7f8fc2d957e0) at >>> >>> /home/mateusz/hypertable/src/cc/Hypertable/RangeServer/MaintenanceTaskSplit.cc:41 >>> #13 0x000000000061bb56 in >>> Hypertable::MaintenanceQueue::Worker::operator() (this=0x7f90c4015d98) >>> at >>> /home/mateusz/hypertable/src/cc/Hypertable/RangeServer/MaintenanceQueue.h:114 >>> #14 0x000000000061c024 in >>> boost::detail::thread_data<Hypertable::MaintenanceQueue::Worker>::run >>> (this=0x7f90c4015c90) at >>> /usr/local/include/boost/thread/detail/thread.hpp:56 >>> #15 0x00007f90cb6b992f in thread_proxy () from >>> /usr/local/lib/libboost_thread-mt.so >>> #16 0x00007f90ca7bc3ea in start_thread () from /lib/libpthread.so.0 >>> #17 0x00007f90c9d7fcbd in clone () from /lib/libc.so.6 >>> #18 0x0000000000000000 in ?? () >>> >>> Thread 2 (Thread 0x4f1f5950 (LWP 2975)): >>> #0 0x00007f90ca7c02d9 in pthread_cond_wait@@GLIBC_2.3.2 () from >>> /lib/libpthread.so.0 >>> #1 0x00000000005e35d8 in >>> boost::condition_variable_any::wait<boost::unique_lock<boost::mutex> > >>> (this=0x7f90c40158d8, m...@0x4f1f5030) >>> at /usr/local/include/boost/thread/pthread/condition_variable.hpp:84 >>> #2 0x000000000061b9c2 in >>> Hypertable::MaintenanceQueue::Worker::operator() (this=0x7f90c4016028) >>> at >>> /home/mateusz/hypertable/src/cc/Hypertable/RangeServer/MaintenanceQueue.h:93 >>> #3 0x000000000061c024 in >>> boost::detail::thread_data<Hypertable::MaintenanceQueue::Worker>::run >>> (this=0x7f90c4015f20) at >>> /usr/local/include/boost/thread/detail/thread.hpp:56 >>> #4 0x00007f90cb6b992f in thread_proxy () from >>> /usr/local/lib/libboost_thread-mt.so >>> #5 0x00007f90ca7bc3ea in start_thread () from /lib/libpthread.so.0 >>> #6 0x00007f90c9d7fcbd in clone () from /lib/libc.so.6 >>> #7 0x0000000000000000 in ?? () >>> ---Type <return> to continue, or q <return> to quit--- >>> >>> Thread 1 (Thread 0x7f90cc59f6f0 (LWP 2943)): >>> #0 0x00007f90ca7c02d9 in pthread_cond_wait@@GLIBC_2.3.2 () from >>> /lib/libpthread.so.0 >>> #1 0x00007f90cb6b8340 in boost::thread::join () from >>> /usr/local/lib/libboost_thread-mt.so >>> #2 0x00000000005e1feb in boost::thread_group::join_all >>> (this=0x19ceff0) at >>> /usr/local/include/boost/thread/detail/thread.hpp:541 >>> #3 0x00000000005e207e in Hypertable::ApplicationQueue::join >>> (this=0x19ceec0) at >>> /home/mateusz/hypertable/src/cc/AsyncComm/ApplicationQueue.h:209 >>> #4 0x00000000005d7c1d in main (argc=3, argv=0x7fffd46f9d58) at >>> /home/mateusz/hypertable/src/cc/Hypertable/RangeServer/main.cc:92 >>> #0 0x00007f90ca7c02d9 in pthread_cond_wait@@GLIBC_2.3.2 () from >>> /lib/libpthread.so.0 >>> (gdb) >>> >>> >>> Mateusz >>> >>> >> >> >> >> >> > --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Hypertable Development" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/hypertable-dev?hl=en -~----------~----~----~----~------~----~------~--~---
