I've also ran a RangeServer inside a gdb and here's the stack trace of
the offending thread. The other threads' stack traces follow below
1237587949 INFO Hypertable.RangeServer :
(/home/mateusz/hypertable/src/cc/Hypertable/RangeServer/AccessGroup.cc:350)
Starting Major Compaction of wikipedia_index[..community](default)
1237587965 INFO Hypertable.RangeServer :
(/home/mateusz/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:2019)
Memory Usage: 457293331 bytes
1237587965 INFO Hypertable.RangeServer :
(/home/mateusz/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:2076)
Cleaning log (threshold=2482879892)
1237587985 INFO Hypertable.RangeServer :
(/home/mateusz/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:2019)
Memory Usage: 457293331 bytes
1237588005 INFO Hypertable.RangeServer :
(/home/mateusz/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:2019)
Memory Usage: 457293331 bytes
1237588025 INFO Hypertable.RangeServer :
(/home/mateusz/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:2019)
Memory Usage: 457293331 bytes
1237588045 INFO Hypertable.RangeServer :
(/home/mateusz/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:2019)
Memory Usage: 457293331 bytes
1237588065 INFO Hypertable.RangeServer :
(/home/mateusz/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:2019)
Memory Usage: 457293331 bytes
1237588085 WARN Hypertable.RangeServer :
(/home/mateusz/hypertable/src/cc/Hypertable/RangeServer/ScannerMap.cc:82)
Destroying scanner 1 because it has not been used in 120000
milliseconds
1237588085 INFO Hypertable.RangeServer :
(/home/mateusz/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:2019)
Memory Usage: 457293331 bytes
1237588105 INFO Hypertable.RangeServer :
(/home/mateusz/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:2019)
Memory Usage: 457293331 bytes
1237588125 INFO Hypertable.RangeServer :
(/home/mateusz/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:2019)
Memory Usage: 457293331 bytes
1237588132 ERROR Hypertable.RangeServer : handle_exceptions
(/home/mateusz/hypertable/src/cc/Hypertable/Lib/TableMutator.cc:47):
Hypertable::Exception: waiting for previous buffer - HYPERTABLE
request timeout
at void
Hypertable::TableMutator::wait_for_previous_buffer(Hypertable::Timer&)
(/home/mateusz/hypertable/src/cc/Hypertable/Lib/TableMutator.cc:300)
at bool
Hypertable::TableMutatorCompletionCounter::wait_for_completion(Hypertable::Timer&)
(/home/mateusz/hypertable/src/cc/Hypertable/Lib/TableMutatorCompletionCounter.h:71):
1237588132 ERROR Hypertable.RangeServer : update_files_column
(/home/mateusz/hypertable/src/cc/Hypertable/RangeServer/LiveFileTracker.cc:114):
Problem updating 'Files' column of METADATA: Hypertable::Exception:
waiting for previous buffer - HYPERTABLE request timeout
at void
Hypertable::TableMutator::wait_for_previous_buffer(Hypertable::Timer&)
(/home/mateusz/hypertable/src/cc/Hypertable/Lib/TableMutator.cc:300)
at bool
Hypertable::TableMutatorCompletionCounter::wait_for_completion(Hypertable::Timer&)
(/home/mateusz/hypertable/src/cc/Hypertable/Lib/TableMutatorCompletionCounter.h:71):
Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x436d9950 (LWP 22296)]
0x00007f910d3f02e0 in ?? ()
(gdb)
(gdb) bt
#0 0x00007f910d3f02e0 in ?? ()
#1 0x0000000000777027 in Hypertable::IOHandler::deliver_event
(this=0x7f911407a200, event=0x10773630, dh=0x7f910cbe4e80) at
/home/mateusz/hypertable/src/cc/AsyncComm/IOHandler.h:109
#2 0x0000000000778dbc in Hypertable::Reactor::handle_timeouts
(this=0x197ed80, next_timeo...@0x436d8ff0) at
/home/mateusz/hypertable/src/cc/AsyncComm/Reactor.cc:150
#3 0x000000000077fde8 in Hypertable::ReactorRunner::operator()
(this=0x197ef68) at
/home/mateusz/hypertable/src/cc/AsyncComm/ReactorRunner.cc:103
#4 0x000000000077f18a in
boost::detail::thread_data<Hypertable::ReactorRunner>::run
(this=0x197ee60) at
/usr/local/include/boost/thread/detail/thread.hpp:56
#5 0x00007f911b5ae92f in thread_proxy () from
/usr/local/lib/libboost_thread-mt.so
#6 0x00007f911a6b13ea in start_thread () from /lib/libpthread.so.0
#7 0x00007f9119c74cbd in clone () from /lib/libc.so.6
#8 0x0000000000000000 in ?? ()
All threads' stack traces
(gdb) thread apply all where
Thread 28 (Thread 0x4e6ef950 (LWP 22319)):
#0 0x00007f911a6b7b04 in __lll_lock_wait () from /lib/libpthread.so.0
#1 0x00007f911a6b31a0 in _L_lock_102 () from /lib/libpthread.so.0
#2 0x00007f911a6b2aae in pthread_mutex_lock () from /lib/libpthread.so.0
#3 0x00000000005dd21f in boost::mutex::lock (this=0x7f9110cef470) at
/usr/local/include/boost/thread/pthread/mutex.hpp:50
#4 0x00000000005dd2b4 in boost::unique_lock<boost::mutex>::lock
(this=0x4e6ec660) at /usr/local/include/boost/thread/locks.hpp:338
#5 0x00000000005dd2f2 in unique_lock (this=0x4e6ec660,
m...@0x7f9110cef470) at /usr/local/include/boost/thread/locks.hpp:224
#6 0x000000000064f62e in Hypertable::AccessGroup::run_compaction
(this=0x7f9110cef450, major=true) at
/home/mateusz/hypertable/src/cc/Hypertable/RangeServer/AccessGroup.cc:519
#7 0x0000000000631c43 in Hypertable::Range::split_compact_and_shrink
(this=0x7f9110ceef80) at
/home/mateusz/hypertable/src/cc/Hypertable/RangeServer/Range.cc:550
#8 0x0000000000635e55 in Hypertable::Range::split
(this=0x7f9110ceef80) at
/home/mateusz/hypertable/src/cc/Hypertable/RangeServer/Range.cc:363
#9 0x000000000062ec63 in Hypertable::MaintenanceTaskSplit::execute
(this=0x7f9114018e40) at
/home/mateusz/hypertable/src/cc/Hypertable/RangeServer/MaintenanceTaskSplit.cc:41
#10 0x000000000061bb56 in
Hypertable::MaintenanceQueue::Worker::operator() (this=0x7f9114016e38)
at /home/mateusz/hypertable/src/cc/Hypertable/RangeServer/MaintenanceQueue.h:114
#11 0x000000000061c024 in
boost::detail::thread_data<Hypertable::MaintenanceQueue::Worker>::run
(this=0x7f9114016d30) at
/usr/local/include/boost/thread/detail/thread.hpp:56
#12 0x00007f911b5ae92f in thread_proxy () from
/usr/local/lib/libboost_thread-mt.so
#13 0x00007f911a6b13ea in start_thread () from /lib/libpthread.so.0
#14 0x00007f9119c74cbd in clone () from /lib/libc.so.6
#15 0x0000000000000000 in ?? ()
Thread 27 (Thread 0x4deee950 (LWP 22318)):
#0 0x00007f911a6b7b04 in __lll_lock_wait () from /lib/libpthread.so.0
#1 0x00007f911a6b31a0 in _L_lock_102 () from /lib/libpthread.so.0
#2 0x00007f911a6b2aae in pthread_mutex_lock () from /lib/libpthread.so.0
#3 0x00000000005dd21f in boost::mutex::lock (this=0x7f9110ceefc8) at
/usr/local/include/boost/thread/pthread/mutex.hpp:50
#4 0x00000000005dd2b4 in boost::unique_lock<boost::mutex>::lock
(this=0x4deedd10) at /usr/local/include/boost/thread/locks.hpp:338
#5 0x00000000005dd2f2 in unique_lock (this=0x4deedd10,
m...@0x7f9110ceefc8) at /usr/local/include/boost/thread/locks.hpp:224
#6 0x00000000006360d9 in
Hypertable::Range::get_compaction_priority_data (this=0x7f9110ceef80,
priority_data_vect...@0x4deeddc0) at
/home/mateusz/hypertable/src/cc/Hypertable/RangeServer/Range.cc:325
#7 0x00000000005ebf49 in
Hypertable::RangeServer::schedule_log_cleanup_compactions
(this=0x7f91140155b0, range_v...@0x4deedef0, log=0x7f910c6ec800,
prune_threshold=2482879892)
at
/home/mateusz/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:2094
#8 0x00000000005ec7bc in Hypertable::RangeServer::log_cleanup
(this=0x7f91140155b0) at
/home/mateusz/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:2079
#9 0x000000000062ea53 in
Hypertable::MaintenanceTaskLogCleanup::execute (this=0x7f910d3f06e0)
at
/home/mateusz/hypertable/src/cc/Hypertable/RangeServer/MaintenanceTaskLogCleanup.cc:42
#10 0x000000000061bb56 in
Hypertable::MaintenanceQueue::Worker::operator() (this=0x7f9114016ba8)
at /home/mateusz/hypertable/src/cc/Hypertable/RangeServer/MaintenanceQueue.h:114
#11 0x000000000061c024 in
boost::detail::thread_data<Hypertable::MaintenanceQueue::Worker>::run
(this=0x7f9114016aa0) at
/usr/local/include/boost/thread/detail/thread.hpp:56
#12 0x00007f911b5ae92f in thread_proxy () from
/usr/local/lib/libboost_thread-mt.so
#13 0x00007f911a6b13ea in start_thread () from /lib/libpthread.so.0
#14 0x00007f9119c74cbd in clone () from /lib/libc.so.6
#15 0x0000000000000000 in ?? ()
Thread 26 (Thread 0x4d6ed950 (LWP 22317)):
#0 0x00007f911a6b7b04 in __lll_lock_wait () from /lib/libpthread.so.0
#1 0x00007f911a6b31a0 in _L_lock_102 () from /lib/libpthread.so.0
#2 0x00007f911a6b2aae in pthread_mutex_lock () from /lib/libpthread.so.0
#3 0x00000000005dd21f in boost::mutex::lock (this=0x7f9114015748) at
/usr/local/include/boost/thread/pthread/mutex.hpp:50
#4 0x00000000005f6429 in Hypertable::RangeServer::update
(this=0x7f91140155b0, cb=0x4d6ebf70, table=0x4d6ebfa0, count=1,
buff...@0x4d6ebf90)
at
/home/mateusz/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:1372
#5 0x0000000000649909 in Hypertable::RequestHandlerUpdate::run
(this=0x7f9114021990) at
/home/mateusz/hypertable/src/cc/Hypertable/RangeServer/RequestHandlerUpdate.cc:54
#6 0x00000000005e39d4 in
Hypertable::ApplicationQueue::Worker::operator() (this=0x7f9114003df8)
at /home/mateusz/hypertable/src/cc/AsyncComm/ApplicationQueue.h:142
#7 0x00000000005e3ab0 in
boost::detail::thread_data<Hypertable::ApplicationQueue::Worker>::run
(this=0x7f9114003cf0) at
/usr/local/include/boost/thread/detail/thread.hpp:56
#8 0x00007f911b5ae92f in thread_proxy () from
/usr/local/lib/libboost_thread-mt.so
#9 0x00007f911a6b13ea in start_thread () from /lib/libpthread.so.0
#10 0x00007f9119c74cbd in clone () from /lib/libc.so.6
#11 0x0000000000000000 in ?? ()
---Type <return> to continue, or q <return> to quit---
Thread 25 (Thread 0x4ceec950 (LWP 22316)):
#0 0x00007f911a6b52d9 in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib/libpthread.so.0
#1 0x00000000005e35d8 in
boost::condition_variable_any::wait<boost::unique_lock<boost::mutex> >
(this=0x197e6a0, m...@0x4ceec060) at
/usr/local/include/boost/thread/pthread/condition_variable.hpp:84
#2 0x00000000005e36f4 in
Hypertable::ApplicationQueue::Worker::operator() (this=0x7f9114003b28)
at /home/mateusz/hypertable/src/cc/AsyncComm/ApplicationQueue.h:98
#3 0x00000000005e3ab0 in
boost::detail::thread_data<Hypertable::ApplicationQueue::Worker>::run
(this=0x7f9114003a20) at
/usr/local/include/boost/thread/detail/thread.hpp:56
#4 0x00007f911b5ae92f in thread_proxy () from
/usr/local/lib/libboost_thread-mt.so
#5 0x00007f911a6b13ea in start_thread () from /lib/libpthread.so.0
#6 0x00007f9119c74cbd in clone () from /lib/libc.so.6
#7 0x0000000000000000 in ?? ()
Thread 24 (Thread 0x4c6eb950 (LWP 22315)):
#0 0x00007f911a6b52d9 in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib/libpthread.so.0
#1 0x00000000005e35d8 in
boost::condition_variable_any::wait<boost::unique_lock<boost::mutex> >
(this=0x197e6a0, m...@0x4c6eb060) at
/usr/local/include/boost/thread/pthread/condition_variable.hpp:84
#2 0x00000000005e36f4 in
Hypertable::ApplicationQueue::Worker::operator() (this=0x7f9114003858)
at /home/mateusz/hypertable/src/cc/AsyncComm/ApplicationQueue.h:98
#3 0x00000000005e3ab0 in
boost::detail::thread_data<Hypertable::ApplicationQueue::Worker>::run
(this=0x7f9114003750) at
/usr/local/include/boost/thread/detail/thread.hpp:56
#4 0x00007f911b5ae92f in thread_proxy () from
/usr/local/lib/libboost_thread-mt.so
#5 0x00007f911a6b13ea in start_thread () from /lib/libpthread.so.0
#6 0x00007f9119c74cbd in clone () from /lib/libc.so.6
#7 0x0000000000000000 in ?? ()
Thread 23 (Thread 0x4beea950 (LWP 22314)):
#0 0x00007f911a6b7b04 in __lll_lock_wait () from /lib/libpthread.so.0
#1 0x00007f911a6b31a0 in _L_lock_102 () from /lib/libpthread.so.0
#2 0x00007f911a6b2aae in pthread_mutex_lock () from /lib/libpthread.so.0
#3 0x00000000005dd21f in boost::mutex::lock (this=0x197ed90) at
/usr/local/include/boost/thread/pthread/mutex.hpp:50
#4 0x00000000005dd2b4 in boost::unique_lock<boost::mutex>::lock
(this=0x4bee3ab0) at /usr/local/include/boost/thread/locks.hpp:338
#5 0x00000000005dd2f2 in unique_lock (this=0x4bee3ab0, m...@0x197ed90)
at /usr/local/include/boost/thread/locks.hpp:224
#6 0x0000000000776ce3 in Hypertable::Reactor::add_request
(this=0x197ed80, id=10688, handler=0x7f911407a200, dh=0x7f9114019ed0,
expi...@0x4bee3b20) at
/home/mateusz/hypertable/src/cc/AsyncComm/Reactor.h:57
#7 0x0000000000775156 in Hypertable::IOHandlerData::send_message
(this=0x7f911407a200, c...@0x4bee3d80, timeout_ms=180000,
disp_handler=0x7f9114019ed0)
at /home/mateusz/hypertable/src/cc/AsyncComm/IOHandlerData.cc:431
#8 0x0000000000765231 in Hypertable::Comm::send_request
(this=0x1984020, ad...@0x7f911401e620, timeout_ms=180000,
cbuf_p...@0x4bee3d80, resp_handler=0x7f9114019ed0)
at /home/mateusz/hypertable/src/cc/AsyncComm/Comm.cc:230
#9 0x0000000000708f5e in Hypertable::RangeServerClient::send_message
(this=0x7f911401e460, ad...@0x7f911401e620, c...@0x4bee3d80,
handler=0x7f9114019ed0)
at /home/mateusz/hypertable/src/cc/Hypertable/Lib/RangeServerClient.cc:309
#10 0x000000000070c404 in Hypertable::RangeServerClient::update
(this=0x7f911401e460, ad...@0x7f911401e620, tab...@0x7f911401e480,
count=1, buff...@0x7f911401e610, handler=0x7f9114019ed0)
at /home/mateusz/hypertable/src/cc/Hypertable/Lib/RangeServerClient.cc:77
#11 0x00000000006e0fbe in Hypertable::TableMutatorScatterBuffer::send
(this=0x7f911401e430) at
/home/mateusz/hypertable/src/cc/Hypertable/Lib/TableMutatorScatterBuffer.cc:259
#12 0x00000000006db77b in Hypertable::TableMutator::flush
(this=0x7f9114018f60) at
/home/mateusz/hypertable/src/cc/Hypertable/Lib/TableMutator.cc:228
#13 0x00000000006749d6 in Hypertable::MetadataNormal::write_files
(this=0x4bee40a0, ag_na...@0x7f9110cef600, fil...@0x4bee42c0)
at
/home/mateusz/hypertable/src/cc/Hypertable/RangeServer/MetadataNormal.cc:103
#14 0x000000000066bb00 in
Hypertable::LiveFileTracker::update_files_column (this=0x7f9110cef580)
at /home/mateusz/hypertable/src/cc/Hypertable/RangeServer/LiveFileTracker.cc:109
#15 0x000000000064bb26 in Hypertable::AccessGroup::release_files
(this=0x7f9110cef450, fil...@0x7f90f9b6f908) at
/home/mateusz/hypertable/src/cc/Hypertable/RangeServer/AccessGroup.cc:622
#16 0x000000000065b2c8 in
Hypertable::CellStoreReleaseCallback::operator() (this=0x7f90f9b6f900)
at
/home/mateusz/hypertable/src/cc/Hypertable/RangeServer/CellStoreReleaseCallback.cc:38
#17 0x00000000006703b2 in ~MergeScanner (this=0x7f90f9b6f7b0) at
/home/mateusz/hypertable/src/cc/Hypertable/RangeServer/MergeScanner.cc:60
#18 0x000000000067035e in ~MergeScanner (this=0x7f911403ce00) at
/home/mateusz/hypertable/src/cc/Hypertable/RangeServer/MergeScanner.cc:58
#19 0x00000000005d844b in Hypertable::intrusive_ptr_release
(rc=0x7f911403ce00) at
/home/mateusz/hypertable/src/cc/Common/ReferenceCount.h:73
#20 0x00000000005e8bfd in ~intrusive_ptr (this=0x4bee7cf0) at
/usr/local/include/boost/intrusive_ptr.hpp:101
#21 0x00000000005fec2b in Hypertable::RangeServer::create_scanner
(this=0x7f91140155b0, cb=0x4bee8f70, table=0x4bee8fa0,
range_spec=0x4bee8f90, scan_spec=0x4bee8f00)
at /home/mateusz/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:689
#22 0x0000000000643833 in Hypertable::RequestHandlerCreateScanner::run
(this=0x7f910f394de0) at
/home/mateusz/hypertable/src/cc/Hypertable/RangeServer/RequestHandlerCreateScanner.cc:52
#23 0x00000000005e39d4 in
Hypertable::ApplicationQueue::Worker::operator() (this=0x7f9114003588)
at /home/mateusz/hypertable/src/cc/AsyncComm/ApplicationQueue.h:142
---Type <return> to continue, or q <return> to quit---
#24 0x00000000005e3ab0 in
boost::detail::thread_data<Hypertable::ApplicationQueue::Worker>::run
(this=0x7f9114003480) at
/usr/local/include/boost/thread/detail/thread.hpp:56
#25 0x00007f911b5ae92f in thread_proxy () from
/usr/local/lib/libboost_thread-mt.so
#26 0x00007f911a6b13ea in start_thread () from /lib/libpthread.so.0
#27 0x00007f9119c74cbd in clone () from /lib/libc.so.6
#28 0x0000000000000000 in ?? ()
Thread 22 (Thread 0x4b6e9950 (LWP 22313)):
#0 0x00007f911a6b7b04 in __lll_lock_wait () from /lib/libpthread.so.0
#1 0x00007f911a6b31a0 in _L_lock_102 () from /lib/libpthread.so.0
#2 0x00007f911a6b2aae in pthread_mutex_lock () from /lib/libpthread.so.0
#3 0x00000000005dd21f in boost::mutex::lock (this=0x7f9114015720) at
/usr/local/include/boost/thread/pthread/mutex.hpp:50
#4 0x00000000005f4db1 in Hypertable::RangeServer::update
(this=0x7f91140155b0, cb=0x4b6e7f70, table=0x4b6e7fa0, count=109,
buff...@0x4b6e7f90)
at
/home/mateusz/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:1157
#5 0x0000000000649909 in Hypertable::RequestHandlerUpdate::run
(this=0x7f911411eee0) at
/home/mateusz/hypertable/src/cc/Hypertable/RangeServer/RequestHandlerUpdate.cc:54
#6 0x00000000005e39d4 in
Hypertable::ApplicationQueue::Worker::operator() (this=0x7f91140032b8)
at /home/mateusz/hypertable/src/cc/AsyncComm/ApplicationQueue.h:142
#7 0x00000000005e3ab0 in
boost::detail::thread_data<Hypertable::ApplicationQueue::Worker>::run
(this=0x7f91140031b0) at
/usr/local/include/boost/thread/detail/thread.hpp:56
#8 0x00007f911b5ae92f in thread_proxy () from
/usr/local/lib/libboost_thread-mt.so
#9 0x00007f911a6b13ea in start_thread () from /lib/libpthread.so.0
#10 0x00007f9119c74cbd in clone () from /lib/libc.so.6
#11 0x0000000000000000 in ?? ()
Thread 21 (Thread 0x4aee8950 (LWP 22312)):
#0 0x00007f911a6b52d9 in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib/libpthread.so.0
#1 0x00000000005e35d8 in
boost::condition_variable_any::wait<boost::unique_lock<boost::mutex> >
(this=0x197e6a0, m...@0x4aee8060) at
/usr/local/include/boost/thread/pthread/condition_variable.hpp:84
#2 0x00000000005e36f4 in
Hypertable::ApplicationQueue::Worker::operator() (this=0x7f9114002fe8)
at /home/mateusz/hypertable/src/cc/AsyncComm/ApplicationQueue.h:98
#3 0x00000000005e3ab0 in
boost::detail::thread_data<Hypertable::ApplicationQueue::Worker>::run
(this=0x7f9114002ee0) at
/usr/local/include/boost/thread/detail/thread.hpp:56
#4 0x00007f911b5ae92f in thread_proxy () from
/usr/local/lib/libboost_thread-mt.so
#5 0x00007f911a6b13ea in start_thread () from /lib/libpthread.so.0
#6 0x00007f9119c74cbd in clone () from /lib/libc.so.6
#7 0x0000000000000000 in ?? ()
Thread 20 (Thread 0x4a6e7950 (LWP 22311)):
#0 0x0000000000659fbe in std::_Rb_tree<Hypertable::SerializedKey
const, std::pair<Hypertable::SerializedKey const, unsigned int>,
std::_Select1st<std::pair<Hypertable::SerializedKey const, unsigned
int> >, std::less<Hypertable::SerializedKey const>,
Hypertable::CellCachePoolAllocator<std::pair<Hypertable::SerializedKey
const, unsigned int> > >::_M_begin (this=0xb96f610)
at /usr/include/c++/4.3/bits/stl_tree.h:453
#1 0x000000000065aafc in std::_Rb_tree<Hypertable::SerializedKey
const, std::pair<Hypertable::SerializedKey const, unsigned int>,
std::_Select1st<std::pair<Hypertable::SerializedKey const, unsigned
int> >, std::less<Hypertable::SerializedKey const>,
Hypertable::CellCachePoolAllocator<std::pair<Hypertable::SerializedKey
const, unsigned int> > >::_M_insert_unique (this=0xb96f610,
_...@0x4a6e3150)
at /usr/include/c++/4.3/bits/stl_tree.h:1135
#2 0x000000000065ad01 in std::map<Hypertable::SerializedKey const,
unsigned int, std::less<Hypertable::SerializedKey const>,
Hypertable::CellCachePoolAllocator<std::pair<Hypertable::SerializedKey
const, unsigned int> > >::insert (this=0xb96f610, _...@0x4a6e3150) at
/usr/include/c++/4.3/bits/stl_map.h:469
#3 0x000000000065998c in Hypertable::CellCache::add (this=0xb96f570,
k...@0x4a6e4710, value={ptr = 0x11efbc07 "\020�C\233"}) at
/home/mateusz/hypertable/src/cc/Hypertable/RangeServer/CellCache.cc:63
#4 0x000000000064a03e in Hypertable::AccessGroup::add
(this=0x7f9110cef450, k...@0x4a6e4710, value={ptr = 0x11efbc07
"\020�C\233"})
at /home/mateusz/hypertable/src/cc/Hypertable/RangeServer/AccessGroup.cc:157
#5 0x0000000000636850 in Hypertable::Range::add (this=0x7f9110ceef80,
k...@0x4a6e4710, value={ptr = 0x11efbc07 "\020�C\233"}) at
/home/mateusz/hypertable/src/cc/Hypertable/RangeServer/Range.cc:285
#6 0x00000000005f6919 in Hypertable::RangeServer::update
(this=0x7f91140155b0, cb=0x4a6e5f70, table=0x4a6e5fa0, count=2385,
buff...@0x4a6e5f90)
at
/home/mateusz/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:1416
#7 0x0000000000649909 in Hypertable::RequestHandlerUpdate::run
(this=0x7f9114021d90) at
/home/mateusz/hypertable/src/cc/Hypertable/RangeServer/RequestHandlerUpdate.cc:54
#8 0x00000000005e39d4 in
Hypertable::ApplicationQueue::Worker::operator() (this=0x7f9114002d18)
at /home/mateusz/hypertable/src/cc/AsyncComm/ApplicationQueue.h:142
#9 0x00000000005e3ab0 in
boost::detail::thread_data<Hypertable::ApplicationQueue::Worker>::run
(this=0x7f9114002c10) at
/usr/local/include/boost/thread/detail/thread.hpp:56
#10 0x00007f911b5ae92f in thread_proxy () from
/usr/local/lib/libboost_thread-mt.so
#11 0x00007f911a6b13ea in start_thread () from /lib/libpthread.so.0
#12 0x00007f9119c74cbd in clone () from /lib/libc.so.6
---Type <return> to continue, or q <return> to quit---
#13 0x0000000000000000 in ?? ()
Thread 19 (Thread 0x49ee6950 (LWP 22310)):
#0 0x00007f911a6b52d9 in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib/libpthread.so.0
#1 0x00000000005e35d8 in
boost::condition_variable_any::wait<boost::unique_lock<boost::mutex> >
(this=0x197e6a0, m...@0x49ee6060) at
/usr/local/include/boost/thread/pthread/condition_variable.hpp:84
#2 0x00000000005e36f4 in
Hypertable::ApplicationQueue::Worker::operator() (this=0x7f9114002a48)
at /home/mateusz/hypertable/src/cc/AsyncComm/ApplicationQueue.h:98
#3 0x00000000005e3ab0 in
boost::detail::thread_data<Hypertable::ApplicationQueue::Worker>::run
(this=0x7f9114002940) at
/usr/local/include/boost/thread/detail/thread.hpp:56
#4 0x00007f911b5ae92f in thread_proxy () from
/usr/local/lib/libboost_thread-mt.so
#5 0x00007f911a6b13ea in start_thread () from /lib/libpthread.so.0
#6 0x00007f9119c74cbd in clone () from /lib/libc.so.6
#7 0x0000000000000000 in ?? ()
Thread 18 (Thread 0x496e5950 (LWP 22309)):
#0 0x00007f911a6b52d9 in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib/libpthread.so.0
#1 0x00000000005e35d8 in
boost::condition_variable_any::wait<boost::unique_lock<boost::mutex> >
(this=0x197e6a0, m...@0x496e5060) at
/usr/local/include/boost/thread/pthread/condition_variable.hpp:84
#2 0x00000000005e36f4 in
Hypertable::ApplicationQueue::Worker::operator() (this=0x7f9114002778)
at /home/mateusz/hypertable/src/cc/AsyncComm/ApplicationQueue.h:98
#3 0x00000000005e3ab0 in
boost::detail::thread_data<Hypertable::ApplicationQueue::Worker>::run
(this=0x7f9114002670) at
/usr/local/include/boost/thread/detail/thread.hpp:56
#4 0x00007f911b5ae92f in thread_proxy () from
/usr/local/lib/libboost_thread-mt.so
#5 0x00007f911a6b13ea in start_thread () from /lib/libpthread.so.0
#6 0x00007f9119c74cbd in clone () from /lib/libc.so.6
#7 0x0000000000000000 in ?? ()
Thread 17 (Thread 0x48ee4950 (LWP 22308)):
#0 0x00007f9119c0a890 in ?? () from /lib/libc.so.6
#1 0x00007f9119c0c658 in malloc () from /lib/libc.so.6
#2 0x00007f911a46489d in operator new () from /usr/lib/libstdc++.so.6
#3 0x00007f911a442011 in std::string::_Rep::_S_create () from
/usr/lib/libstdc++.so.6
#4 0x00007f911a4428bb in std::string::_Rep::_M_clone () from
/usr/lib/libstdc++.so.6
#5 0x00007f911a443541 in std::string::reserve () from /usr/lib/libstdc++.so.6
#6 0x00007f911a4437a3 in std::string::append () from /usr/lib/libstdc++.so.6
#7 0x00000000005e6327 in std::operator+<char, std::char_traits<char>,
std::allocator<char> > (__l...@0x7f9109df60e0, __rhs=0x7b6625 ";\n")
at /usr/include/c++/4.3/bits/basic_string.h:2124
#8 0x000000000066b63a in
Hypertable::LiveFileTracker::update_files_column (this=0x7f9110cef580)
at /home/mateusz/hypertable/src/cc/Hypertable/RangeServer/LiveFileTracker.cc:87
#9 0x000000000064bb26 in Hypertable::AccessGroup::release_files
(this=0x7f9110cef450, fil...@0xbf1d0a8) at
/home/mateusz/hypertable/src/cc/Hypertable/RangeServer/AccessGroup.cc:622
#10 0x000000000065b2c8 in
Hypertable::CellStoreReleaseCallback::operator() (this=0xbf1d0a0) at
/home/mateusz/hypertable/src/cc/Hypertable/RangeServer/CellStoreReleaseCallback.cc:38
#11 0x00000000006703b2 in ~MergeScanner (this=0xbf1cf50) at
/home/mateusz/hypertable/src/cc/Hypertable/RangeServer/MergeScanner.cc:60
#12 0x000000000067035e in ~MergeScanner (this=0xbf1cdd0) at
/home/mateusz/hypertable/src/cc/Hypertable/RangeServer/MergeScanner.cc:58
#13 0x00000000005d844b in Hypertable::intrusive_ptr_release
(rc=0xbf1cdd0) at
/home/mateusz/hypertable/src/cc/Common/ReferenceCount.h:73
#14 0x00000000005e8bfd in ~intrusive_ptr (this=0x48ee1cf0) at
/usr/local/include/boost/intrusive_ptr.hpp:101
#15 0x00000000005fec2b in Hypertable::RangeServer::create_scanner
(this=0x7f91140155b0, cb=0x48ee2f70, table=0x48ee2fa0,
range_spec=0x48ee2f90, scan_spec=0x48ee2f00)
at /home/mateusz/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:689
#16 0x0000000000643833 in Hypertable::RequestHandlerCreateScanner::run
(this=0x7f910d9b5240) at
/home/mateusz/hypertable/src/cc/Hypertable/RangeServer/RequestHandlerCreateScanner.cc:52
#17 0x00000000005e39d4 in
Hypertable::ApplicationQueue::Worker::operator() (this=0x7f91140024a8)
at /home/mateusz/hypertable/src/cc/AsyncComm/ApplicationQueue.h:142
#18 0x00000000005e3ab0 in
boost::detail::thread_data<Hypertable::ApplicationQueue::Worker>::run
(this=0x7f91140023a0) at
/usr/local/include/boost/thread/detail/thread.hpp:56
#19 0x00007f911b5ae92f in thread_proxy () from
/usr/local/lib/libboost_thread-mt.so
#20 0x00007f911a6b13ea in start_thread () from /lib/libpthread.so.0
#21 0x00007f9119c74cbd in clone () from /lib/libc.so.6
#22 0x0000000000000000 in ?? ()
Thread 16 (Thread 0x486e3950 (LWP 22307)):
#0 0x00007f911a6b52d9 in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib/libpthread.so.0
#1 0x00000000005e35d8 in
boost::condition_variable_any::wait<boost::unique_lock<boost::mutex> >
(this=0x197e6a0, m...@0x486e3060) at
/usr/local/include/boost/thread/pthread/condition_variable.hpp:84
#2 0x00000000005e36f4 in
Hypertable::ApplicationQueue::Worker::operator() (this=0x7f91140021d8)
at /home/mateusz/hypertable/src/cc/AsyncComm/ApplicationQueue.h:98
---Type <return> to continue, or q <return> to quit---
#3 0x00000000005e3ab0 in
boost::detail::thread_data<Hypertable::ApplicationQueue::Worker>::run
(this=0x7f91140020d0) at
/usr/local/include/boost/thread/detail/thread.hpp:56
#4 0x00007f911b5ae92f in thread_proxy () from
/usr/local/lib/libboost_thread-mt.so
#5 0x00007f911a6b13ea in start_thread () from /lib/libpthread.so.0
#6 0x00007f9119c74cbd in clone () from /lib/libc.so.6
#7 0x0000000000000000 in ?? ()
Thread 15 (Thread 0x47ee2950 (LWP 22306)):
#0 0x00007f911a6b52d9 in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib/libpthread.so.0
#1 0x00000000005e35d8 in
boost::condition_variable_any::wait<boost::unique_lock<boost::mutex> >
(this=0x197e6a0, m...@0x47ee2060) at
/usr/local/include/boost/thread/pthread/condition_variable.hpp:84
#2 0x00000000005e36f4 in
Hypertable::ApplicationQueue::Worker::operator() (this=0x7f9114001f08)
at /home/mateusz/hypertable/src/cc/AsyncComm/ApplicationQueue.h:98
#3 0x00000000005e3ab0 in
boost::detail::thread_data<Hypertable::ApplicationQueue::Worker>::run
(this=0x7f9114001e00) at
/usr/local/include/boost/thread/detail/thread.hpp:56
#4 0x00007f911b5ae92f in thread_proxy () from
/usr/local/lib/libboost_thread-mt.so
#5 0x00007f911a6b13ea in start_thread () from /lib/libpthread.so.0
#6 0x00007f9119c74cbd in clone () from /lib/libc.so.6
#7 0x0000000000000000 in ?? ()
Thread 14 (Thread 0x476e1950 (LWP 22305)):
#0 0x00007f911a6b52d9 in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib/libpthread.so.0
#1 0x00000000005e35d8 in
boost::condition_variable_any::wait<boost::unique_lock<boost::mutex> >
(this=0x197e6a0, m...@0x476e1060) at
/usr/local/include/boost/thread/pthread/condition_variable.hpp:84
#2 0x00000000005e36f4 in
Hypertable::ApplicationQueue::Worker::operator() (this=0x7f9114001c38)
at /home/mateusz/hypertable/src/cc/AsyncComm/ApplicationQueue.h:98
#3 0x00000000005e3ab0 in
boost::detail::thread_data<Hypertable::ApplicationQueue::Worker>::run
(this=0x7f9114001b30) at
/usr/local/include/boost/thread/detail/thread.hpp:56
#4 0x00007f911b5ae92f in thread_proxy () from
/usr/local/lib/libboost_thread-mt.so
#5 0x00007f911a6b13ea in start_thread () from /lib/libpthread.so.0
#6 0x00007f9119c74cbd in clone () from /lib/libc.so.6
#7 0x0000000000000000 in ?? ()
Thread 13 (Thread 0x46ee0950 (LWP 22304)):
#0 0x00007f911a6b52d9 in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib/libpthread.so.0
#1 0x00000000005e35d8 in
boost::condition_variable_any::wait<boost::unique_lock<boost::mutex> >
(this=0x197e6a0, m...@0x46ee0060) at
/usr/local/include/boost/thread/pthread/condition_variable.hpp:84
#2 0x00000000005e36f4 in
Hypertable::ApplicationQueue::Worker::operator() (this=0x7f9114001968)
at /home/mateusz/hypertable/src/cc/AsyncComm/ApplicationQueue.h:98
#3 0x00000000005e3ab0 in
boost::detail::thread_data<Hypertable::ApplicationQueue::Worker>::run
(this=0x7f9114001860) at
/usr/local/include/boost/thread/detail/thread.hpp:56
#4 0x00007f911b5ae92f in thread_proxy () from
/usr/local/lib/libboost_thread-mt.so
#5 0x00007f911a6b13ea in start_thread () from /lib/libpthread.so.0
#6 0x00007f9119c74cbd in clone () from /lib/libc.so.6
#7 0x0000000000000000 in ?? ()
Thread 12 (Thread 0x466df950 (LWP 22303)):
#0 0x00007f911a6b52d9 in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib/libpthread.so.0
#1 0x00000000005e35d8 in
boost::condition_variable_any::wait<boost::unique_lock<boost::mutex> >
(this=0x197e6a0, m...@0x466df060) at
/usr/local/include/boost/thread/pthread/condition_variable.hpp:84
#2 0x00000000005e36f4 in
Hypertable::ApplicationQueue::Worker::operator() (this=0x7f9114001698)
at /home/mateusz/hypertable/src/cc/AsyncComm/ApplicationQueue.h:98
#3 0x00000000005e3ab0 in
boost::detail::thread_data<Hypertable::ApplicationQueue::Worker>::run
(this=0x7f9114001590) at
/usr/local/include/boost/thread/detail/thread.hpp:56
#4 0x00007f911b5ae92f in thread_proxy () from
/usr/local/lib/libboost_thread-mt.so
#5 0x00007f911a6b13ea in start_thread () from /lib/libpthread.so.0
#6 0x00007f9119c74cbd in clone () from /lib/libc.so.6
#7 0x0000000000000000 in ?? ()
Thread 11 (Thread 0x410e7950 (LWP 22302)):
#0 0x00007f911a6b52d9 in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib/libpthread.so.0
#1 0x00000000005e35d8 in
boost::condition_variable_any::wait<boost::unique_lock<boost::mutex> >
(this=0x197e6a0, m...@0x410e7060) at
/usr/local/include/boost/thread/pthread/condition_variable.hpp:84
#2 0x00000000005e36f4 in
Hypertable::ApplicationQueue::Worker::operator() (this=0x7f91140013c8)
at /home/mateusz/hypertable/src/cc/AsyncComm/ApplicationQueue.h:98
#3 0x00000000005e3ab0 in
boost::detail::thread_data<Hypertable::ApplicationQueue::Worker>::run
(this=0x7f91140012c0) at
/usr/local/include/boost/thread/detail/thread.hpp:56
#4 0x00007f911b5ae92f in thread_proxy () from
/usr/local/lib/libboost_thread-mt.so
---Type <return> to continue, or q <return> to quit---
#5 0x00007f911a6b13ea in start_thread () from /lib/libpthread.so.0
#6 0x00007f9119c74cbd in clone () from /lib/libc.so.6
#7 0x0000000000000000 in ?? ()
Thread 10 (Thread 0x45ede950 (LWP 22301)):
#0 0x00007f911a6b52d9 in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib/libpthread.so.0
#1 0x00000000005e35d8 in
boost::condition_variable_any::wait<boost::unique_lock<boost::mutex> >
(this=0x197e6a0, m...@0x45ede060) at
/usr/local/include/boost/thread/pthread/condition_variable.hpp:84
#2 0x00000000005e36f4 in
Hypertable::ApplicationQueue::Worker::operator() (this=0x7f91140010f8)
at /home/mateusz/hypertable/src/cc/AsyncComm/ApplicationQueue.h:98
#3 0x00000000005e3ab0 in
boost::detail::thread_data<Hypertable::ApplicationQueue::Worker>::run
(this=0x7f9114000ff0) at
/usr/local/include/boost/thread/detail/thread.hpp:56
#4 0x00007f911b5ae92f in thread_proxy () from
/usr/local/lib/libboost_thread-mt.so
#5 0x00007f911a6b13ea in start_thread () from /lib/libpthread.so.0
#6 0x00007f9119c74cbd in clone () from /lib/libc.so.6
#7 0x0000000000000000 in ?? ()
Thread 9 (Thread 0x456dd950 (LWP 22300)):
#0 0x00007f911a6b52d9 in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib/libpthread.so.0
#1 0x00000000005e35d8 in
boost::condition_variable_any::wait<boost::unique_lock<boost::mutex> >
(this=0x197e6a0, m...@0x456dd060) at
/usr/local/include/boost/thread/pthread/condition_variable.hpp:84
#2 0x00000000005e36f4 in
Hypertable::ApplicationQueue::Worker::operator() (this=0x7f9114000e28)
at /home/mateusz/hypertable/src/cc/AsyncComm/ApplicationQueue.h:98
#3 0x00000000005e3ab0 in
boost::detail::thread_data<Hypertable::ApplicationQueue::Worker>::run
(this=0x7f9114000d20) at
/usr/local/include/boost/thread/detail/thread.hpp:56
#4 0x00007f911b5ae92f in thread_proxy () from
/usr/local/lib/libboost_thread-mt.so
#5 0x00007f911a6b13ea in start_thread () from /lib/libpthread.so.0
#6 0x00007f9119c74cbd in clone () from /lib/libc.so.6
#7 0x0000000000000000 in ?? ()
Thread 8 (Thread 0x44edc950 (LWP 22299)):
#0 0x00007f911a6b52d9 in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib/libpthread.so.0
#1 0x00000000005e35d8 in
boost::condition_variable_any::wait<boost::unique_lock<boost::mutex> >
(this=0x197e6a0, m...@0x44edc060) at
/usr/local/include/boost/thread/pthread/condition_variable.hpp:84
#2 0x00000000005e36f4 in
Hypertable::ApplicationQueue::Worker::operator() (this=0x7f9114000b58)
at /home/mateusz/hypertable/src/cc/AsyncComm/ApplicationQueue.h:98
#3 0x00000000005e3ab0 in
boost::detail::thread_data<Hypertable::ApplicationQueue::Worker>::run
(this=0x7f9114000a50) at
/usr/local/include/boost/thread/detail/thread.hpp:56
#4 0x00007f911b5ae92f in thread_proxy () from
/usr/local/lib/libboost_thread-mt.so
#5 0x00007f911a6b13ea in start_thread () from /lib/libpthread.so.0
#6 0x00007f9119c74cbd in clone () from /lib/libc.so.6
#7 0x0000000000000000 in ?? ()
Thread 7 (Thread 0x446db950 (LWP 22298)):
#0 0x00007f911a6b52d9 in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib/libpthread.so.0
#1 0x00000000005e35d8 in
boost::condition_variable_any::wait<boost::unique_lock<boost::mutex> >
(this=0x197e6a0, m...@0x446db060) at
/usr/local/include/boost/thread/pthread/condition_variable.hpp:84
#2 0x00000000005e36f4 in
Hypertable::ApplicationQueue::Worker::operator() (this=0x1980768) at
/home/mateusz/hypertable/src/cc/AsyncComm/ApplicationQueue.h:98
#3 0x00000000005e3ab0 in
boost::detail::thread_data<Hypertable::ApplicationQueue::Worker>::run
(this=0x1980660) at
/usr/local/include/boost/thread/detail/thread.hpp:56
#4 0x00007f911b5ae92f in thread_proxy () from
/usr/local/lib/libboost_thread-mt.so
#5 0x00007f911a6b13ea in start_thread () from /lib/libpthread.so.0
#6 0x00007f9119c74cbd in clone () from /lib/libc.so.6
#7 0x0000000000000000 in ?? ()
Thread 6 (Thread 0x43eda950 (LWP 22297)):
#0 0x00007f911a6b52d9 in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib/libpthread.so.0
#1 0x00000000005e35d8 in
boost::condition_variable_any::wait<boost::unique_lock<boost::mutex> >
(this=0x197e230, m...@0x43eda090) at
/usr/local/include/boost/thread/pthread/condition_variable.hpp:84
#2 0x000000000076ff63 in Hypertable::ConnectionManager::operator()
(this=0x197e3f8) at
/home/mateusz/hypertable/src/cc/AsyncComm/ConnectionManager.cc:308
#3 0x00000000005e33d0 in
boost::detail::thread_data<Hypertable::ConnectionManager>::run
(this=0x197e2f0) at
/usr/local/include/boost/thread/detail/thread.hpp:56
#4 0x00007f911b5ae92f in thread_proxy () from
/usr/local/lib/libboost_thread-mt.so
#5 0x00007f911a6b13ea in start_thread () from /lib/libpthread.so.0
#6 0x00007f9119c74cbd in clone () from /lib/libc.so.6
---Type <return> to continue, or q <return> to quit---
#7 0x0000000000000000 in ?? ()
Thread 5 (Thread 0x436d9950 (LWP 22296)):
#0 0x00007f910d3f02e0 in ?? ()
#1 0x0000000000777027 in Hypertable::IOHandler::deliver_event
(this=0x7f911407a200, event=0x10773630, dh=0x7f910cbe4e80) at
/home/mateusz/hypertable/src/cc/AsyncComm/IOHandler.h:109
#2 0x0000000000778dbc in Hypertable::Reactor::handle_timeouts
(this=0x197ed80, next_timeo...@0x436d8ff0) at
/home/mateusz/hypertable/src/cc/AsyncComm/Reactor.cc:150
#3 0x000000000077fde8 in Hypertable::ReactorRunner::operator()
(this=0x197ef68) at
/home/mateusz/hypertable/src/cc/AsyncComm/ReactorRunner.cc:103
#4 0x000000000077f18a in
boost::detail::thread_data<Hypertable::ReactorRunner>::run
(this=0x197ee60) at
/usr/local/include/boost/thread/detail/thread.hpp:56
#5 0x00007f911b5ae92f in thread_proxy () from
/usr/local/lib/libboost_thread-mt.so
#6 0x00007f911a6b13ea in start_thread () from /lib/libpthread.so.0
#7 0x00007f9119c74cbd in clone () from /lib/libc.so.6
#8 0x0000000000000000 in ?? ()
Thread 4 (Thread 0x42ed8950 (LWP 22295)):
#0 0x00007f9119c75288 in epoll_wait () from /lib/libc.so.6
#1 0x000000000077fe32 in Hypertable::ReactorRunner::operator()
(this=0x197ebf8) at
/home/mateusz/hypertable/src/cc/AsyncComm/ReactorRunner.cc:70
#2 0x000000000077f18a in
boost::detail::thread_data<Hypertable::ReactorRunner>::run
(this=0x197eaf0) at
/usr/local/include/boost/thread/detail/thread.hpp:56
#3 0x00007f911b5ae92f in thread_proxy () from
/usr/local/lib/libboost_thread-mt.so
#4 0x00007f911a6b13ea in start_thread () from /lib/libpthread.so.0
#5 0x00007f9119c74cbd in clone () from /lib/libc.so.6
#6 0x0000000000000000 in ?? ()
Thread 3 (Thread 0x426d7950 (LWP 22294)):
#0 0x00007f9119c75288 in epoll_wait () from /lib/libc.so.6
#1 0x000000000077fe32 in Hypertable::ReactorRunner::operator()
(this=0x1985518) at
/home/mateusz/hypertable/src/cc/AsyncComm/ReactorRunner.cc:70
#2 0x000000000077f18a in
boost::detail::thread_data<Hypertable::ReactorRunner>::run
(this=0x1985410) at
/usr/local/include/boost/thread/detail/thread.hpp:56
#3 0x00007f911b5ae92f in thread_proxy () from
/usr/local/lib/libboost_thread-mt.so
#4 0x00007f911a6b13ea in start_thread () from /lib/libpthread.so.0
#5 0x00007f9119c74cbd in clone () from /lib/libc.so.6
#6 0x0000000000000000 in ?? ()
Thread 2 (Thread 0x41ed6950 (LWP 22293)):
#0 0x00007f9119c75288 in epoll_wait () from /lib/libc.so.6
#1 0x000000000077fe32 in Hypertable::ReactorRunner::operator()
(this=0x197f7b8) at
/home/mateusz/hypertable/src/cc/AsyncComm/ReactorRunner.cc:70
#2 0x000000000077f18a in
boost::detail::thread_data<Hypertable::ReactorRunner>::run
(this=0x197f6b0) at
/usr/local/include/boost/thread/detail/thread.hpp:56
#3 0x00007f911b5ae92f in thread_proxy () from
/usr/local/lib/libboost_thread-mt.so
#4 0x00007f911a6b13ea in start_thread () from /lib/libpthread.so.0
#5 0x00007f9119c74cbd in clone () from /lib/libc.so.6
#6 0x0000000000000000 in ?? ()
Thread 1 (Thread 0x7f911c4946f0 (LWP 22290)):
#0 0x00007f911a6b52d9 in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib/libpthread.so.0
#1 0x00007f911b5ad340 in boost::thread::join () from
/usr/local/lib/libboost_thread-mt.so
#2 0x00000000005e1feb in boost::thread_group::join_all
(this=0x197e700) at
/usr/local/include/boost/thread/detail/thread.hpp:541
#3 0x00000000005e207e in Hypertable::ApplicationQueue::join
(this=0x197e5d0) at
/home/mateusz/hypertable/src/cc/AsyncComm/ApplicationQueue.h:209
#4 0x00000000005d7c1d in main (argc=3, argv=0x7fff245ee988) at
/home/mateusz/hypertable/src/cc/Hypertable/RangeServer/main.cc:92
#0 0x00007f910d3f02e0 in ?? ()
(gdb) where
#0 0x00007f910d3f02e0 in ?? ()
#1 0x0000000000777027 in Hypertable::IOHandler::deliver_event
(this=0x7f911407a200, event=0x10773630, dh=0x7f910cbe4e80) at
/home/mateusz/hypertable/src/cc/AsyncComm/IOHandler.h:109
#2 0x0000000000778dbc in Hypertable::Reactor::handle_timeouts
(this=0x197ed80, next_timeo...@0x436d8ff0) at
/home/mateusz/hypertable/src/cc/AsyncComm/Reactor.cc:150
#3 0x000000000077fde8 in Hypertable::ReactorRunner::operator()
(this=0x197ef68) at
/home/mateusz/hypertable/src/cc/AsyncComm/ReactorRunner.cc:103
#4 0x000000000077f18a in
boost::detail::thread_data<Hypertable::ReactorRunner>::run
(this=0x197ee60) at
/usr/local/include/boost/thread/detail/thread.hpp:56
#5 0x00007f911b5ae92f in thread_proxy () from
/usr/local/lib/libboost_thread-mt.so
#6 0x00007f911a6b13ea in start_thread () from /lib/libpthread.so.0
#7 0x00007f9119c74cbd in clone () from /lib/libc.so.6
#8 0x0000000000000000 in ?? ()
(gdb) up
#1 0x0000000000777027 in Hypertable::IOHandler::deliver_event
(this=0x7f911407a200, event=0x10773630, dh=0x7f910cbe4e80) at
/home/mateusz/hypertable/src/cc/AsyncComm/IOHandler.h:109
109 dh->handle(event_ptr);
(gdb) p dh
$1 = (class Hypertable::DispatchHandler *) 0x7f910cbe4e80
(gdb)
I'll try recompiling with Debug mode enabled to see if I can get more
details out of it
Mateusz
2009/3/20 Mateusz Berezecki <[email protected]>:
> Hi list readers,
>
> I've checked out the latest source code and re-ran my indexing
> application for the wikipedia corpus. It seems the code is much more
> stable now and the database can withstand insertions for much longer
> periods of time. But the usual bug is still there:
>
> 1237586373 INFO Hypertable.RangeServer :
> (/home/mateusz/hypertable/src/cc/Hypertable/RangeServer/AccessGroup.cc:350)
> Starting Major Compaction of
> wikipedia_index[..http://en.wikipedia.org/w/index.php?title=culture_of_taiwan&diff=114390379&oldid=114389700].](default)
> 1237586377 INFO Hypertable.RangeServer :
> (/home/mateusz/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:2019)
> Memory Usage: 298054887 bytes
> 1237586377 INFO Hypertable.RangeServer :
> (/home/mateusz/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:2076)
> Cleaning log (threshold=3114270720)
> 1237586397 INFO Hypertable.RangeServer :
> (/home/mateusz/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:2019)
> Memory Usage: 298089509 bytes
> 1237586397 INFO Hypertable.RangeServer :
> (/home/mateusz/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:2076)
> Cleaning log (threshold=200000000)
> 1237586417 INFO Hypertable.RangeServer :
> (/home/mateusz/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:2019)
> Memory Usage: 298089509 bytes
> 1237586437 INFO Hypertable.RangeServer :
> (/home/mateusz/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:2019)
> Memory Usage: 298089509 bytes
> 1237586457 INFO Hypertable.RangeServer :
> (/home/mateusz/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:2019)
> Memory Usage: 298089509 bytes
> 1237586477 WARN Hypertable.RangeServer :
> (/home/mateusz/hypertable/src/cc/Hypertable/RangeServer/ScannerMap.cc:82)
> Destroying scanner 3 because it has not been used in 120000
> milliseconds
> 1237586477 INFO Hypertable.RangeServer :
> (/home/mateusz/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:2019)
> Memory Usage: 298089509 bytes
> 1237586497 INFO Hypertable.RangeServer :
> (/home/mateusz/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:2019)
> Memory Usage: 298089509 bytes
> 1237586517 INFO Hypertable.RangeServer :
> (/home/mateusz/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:2019)
> Memory Usage: 298089509 bytes
> 1237586537 INFO Hypertable.RangeServer :
> (/home/mateusz/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:2019)
> Memory Usage: 298089509 bytes
> 1237586557 INFO Hypertable.RangeServer :
> (/home/mateusz/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:2019)
> Memory Usage: 298089509 bytes
> 1237586558 ERROR Hypertable.RangeServer : handle_exceptions
> (/home/mateusz/hypertable/src/cc/Hypertable/Lib/TableMutator.cc:47):
> Hypertable::Exception: waiting for previous buffer - HYPERTABLE
> request timeout
> at void
> Hypertable::TableMutator::wait_for_previous_buffer(Hypertable::Timer&)
> (/home/mateusz/hypertable/src/cc/Hypertable/Lib/TableMutator.cc:300)
> at bool
> Hypertable::TableMutatorCompletionCounter::wait_for_completion(Hypertable::Timer&)
> (/home/mateusz/hypertable/src/cc/Hypertable/Lib/TableMutatorCompletionCounter.h:71):
> 1237586558 ERROR Hypertable.RangeServer : update_files_column
> (/home/mateusz/hypertable/src/cc/Hypertable/RangeServer/LiveFileTracker.cc:114):
> Problem updating 'Files' column of METADATA: Hypertable::Exception:
> waiting for previous buffer - HYPERTABLE request timeout
> at void
> Hypertable::TableMutator::wait_for_previous_buffer(Hypertable::Timer&)
> (/home/mateusz/hypertable/src/cc/Hypertable/Lib/TableMutator.cc:300)
> at bool
> Hypertable::TableMutatorCompletionCounter::wait_for_completion(Hypertable::Timer&)
> (/home/mateusz/hypertable/src/cc/Hypertable/Lib/TableMutatorCompletionCounter.h:71):
>
>
> When restarting RangeServer after this crash the following messages
> appear in the log
>
> 1237587141 INFO Hypertable.RangeServer :
> (/home/mateusz/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:1722)
> Successfully replay loaded range
> wikipedia_index[..http://en.wikipedia.org/w/index.php?title=culture_of_taiwan&diff=114390379&oldid=114389700].]
> 1237587141 INFO Hypertable.RangeServer :
> (/home/mateusz/hypertable/src/cc/Hypertable/RangeServer/Range.cc:230)
> Loading CellStore
> /hypertable/tables/wikipedia_index/default/AB2A0D28DE6B77FFDD6C72AF/cs1
> 1237587141 INFO Hypertable.RangeServer :
> (/home/mateusz/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:1722)
> Successfully replay loaded range
> wikipedia_index[http://en.wikipedia.org/w/index.php?title=culture_of_taiwan&diff=114390379&oldid=114389700]...mythology]
> 1237587141 INFO Hypertable.RangeServer :
> (/home/mateusz/hypertable/src/cc/Hypertable/RangeServer/Range.cc:230)
> Loading CellStore
> /hypertable/tables/wikipedia_index/default/AB2A0D28DE6B77FFDD6C72AF/cs1
> 1237587141 INFO Hypertable.RangeServer :
> (/home/mateusz/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:1722)
> Successfully replay loaded range wikipedia_index[mythology..��]
> 1237587142 INFO Hypertable.RangeServer :
> (/home/mateusz/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:520)
> Replayed 40 blocks of updates from
> '/hypertable/servers/172.16.0.19_38060/log/user'
> 1237587142 INFO Hypertable.RangeServer :
> (/home/mateusz/hypertable/src/cc/Hypertable/RangeServer/Range.cc:978)
> Replayed 12403 updates (12 blocks) from split log
> '/hypertable/servers/172.16.0.19_38060/log/82DF41632DA476065CE29979'
> into
> wikipedia_index[..http://en.wikipedia.org/w/index.php?title=culture_of_taiwan&diff=114390379&oldid=114389700].]
> 1237587142 INFO Hypertable.RangeServer :
> (/home/mateusz/hypertable/src/cc/Hypertable/RangeServer/Range.cc:834)
> Restored range state to SPLIT_LOG_INSTALLED (split point='community'
> split
> log='/hypertable/servers/172.16.0.19_38060/log/82DF41632DA476065CE29979')
> 1237587145 INFO Hypertable.RangeServer :
> (/home/mateusz/hypertable/src/cc/Hypertable/RangeServer/AccessGroup.cc:350)
> Starting Major Compaction of
> wikipedia_index[..http://en.wikipedia.org/w/index.php?title=culture_of_taiwan&diff=114390379&oldid=114389700].](default)
> 1237587149 INFO Hypertable.RangeServer :
> (/home/mateusz/hypertable/src/cc/Hypertable/RangeServer/AccessGroup.cc:515)
> Finished Compaction of
> wikipedia_index[..http://en.wikipedia.org/w/index.php?title=culture_of_taiwan&diff=114390379&oldid=114389700].](default)
> 1237587149 INFO Hypertable.RangeServer :
> (/home/mateusz/hypertable/src/cc/Hypertable/RangeServer/Range.cc:726)
> Reporting newly split off range
> wikipedia_index[community..http://en.wikipedia.org/w/index.php?title=culture_of_taiwan&diff=114390379&oldid=114389700].]
> to Master
> 1237587149 INFO Hypertable.RangeServer :
> (/home/mateusz/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:798)
> Loading range: {TableIdentifier: name='wikipedia_index' id=4
> generation=1} {RangeSpec: start='community'
> end='http://en.wikipedia.org/w/index.php?title=culture_of_taiwan&diff=114390379&oldid=114389700].'}
> 1237587149 INFO Hypertable.RangeServer :
> (/home/mateusz/hypertable/src/cc/Hypertable/RangeServer/Range.cc:230)
> Loading CellStore
> /hypertable/tables/wikipedia_index/default/A4CDB32FC10A364A233092C2/cs1
> 1237587149 INFO Hypertable.RangeServer :
> (/home/mateusz/hypertable/src/cc/Hypertable/RangeServer/Range.cc:978)
> Replayed 12403 updates (12 blocks) from split log
> '/hypertable/servers/172.16.0.19_38060/log/82DF41632DA476065CE29979'
> into
> wikipedia_index[community..http://en.wikipedia.org/w/index.php?title=culture_of_taiwan&diff=114390379&oldid=114389700].]
> 1237587149 INFO Hypertable.RangeServer :
> (/home/mateusz/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:958)
> Successfully loaded range
> wikipedia_index[community..http://en.wikipedia.org/w/index.php?title=culture_of_taiwan&diff=114390379&oldid=114389700].]
> 1237587149 INFO Hypertable.RangeServer :
> (/home/mateusz/hypertable/src/cc/Hypertable/RangeServer/Range.cc:381)
> Split Complete. New Range end_row=
> 1237587165 INFO Hypertable.RangeServer :
> (/home/mateusz/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:2019)
> Memory Usage: 15055844 bytes
> 1237587165 INFO Hypertable.RangeServer :
> (/home/mateusz/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:2076)
> Cleaning log (threshold=200000000)
>
>
> It seems to me that the major compaction of the range started but
> never finished in the first run.
> I'm attaching logs to this message as they may be helpful. The
> symptoms are as before, CPU usage drops to zero, memory usage is at
> 25% and everything timeouts after a while.
>
>
> Anything I can do to help debugging this error, please bug me ?
>
> Mateusz
>
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Hypertable Development" group.
To post to this group, send email to [email protected]
To unsubscribe from this group, send email to
[email protected]
For more options, visit this group at
http://groups.google.com/group/hypertable-dev?hl=en
-~----------~----~----~----~------~----~------~--~---
