Hi All,
I meet this strange problem while loading 500GB log data into 7
Hypertable Range Servers, one range server dies with a core dump.
According to gdb output, the core is generated due to segmentation
fault in memcpy(). However, I can access both the source and
destination address in gdb:
$ gdb -c core.333 Hypertable.RangeServer
[...]
(gdb) where
#0 0x000000302af71900 in memcpy () from /lib64/tls/libc.so.6
#1 0x00000000006354d8 in Hypertable::InetAddr::initialize
(addr=0x284d0f0, host=0x2ab184da68 "10.65.25.163", port=38060)
at /home/ht4/src/hypertable-0.9.0.10-0903/src/cc/Common/
InetAddr.cc:71
#2 0x00000000005e855a in Hypertable::LocationCache::location_to_addr
(location=0x1311fd8 "10.65.25.163_38060", [EMAIL PROTECTED])
at /home/ht4/src/hypertable-0.9.0.10-0903/src/cc/Hypertable/Lib/
LocationCache.cc:256
#3 0x00000000005ce19c in Hypertable::TableMutatorScatterBuffer::set
(this=0xff4c900, [EMAIL PROTECTED], value=0x165e5c18,
value_len=639, [EMAIL PROTECTED])
at /home/ht4/src/hypertable-0.9.0.10-0903/src/cc/Hypertable/Lib/
TableMutatorScatterBuffer.cc:72
#4 0x00000000005cc05b in Hypertable::TableMutator::set
(this=0x2ab21ec240, timestamp=0, [EMAIL PROTECTED], value=0x165e5c18,
value_len=639) at /home/ht4/src/hypertable-0.9.0.10-0903/src/cc/
Hypertable/Lib/TableMutator.cc:93
#5 0x00000000005856f7 in Hypertable::MetadataNormal::write_files
(this=0x5221bc50, [EMAIL PROTECTED], [EMAIL PROTECTED])
at /home/ht4/src/hypertable-0.9.0.10-0903/src/cc/Hypertable/
RangeServer/MetadataNormal.cc:100
#6 0x000000000056f0e5 in Hypertable::AccessGroup::update_files_column
(this=0x1bc2a00)
at /home/ht4/src/hypertable-0.9.0.10-0903/src/cc/Hypertable/
RangeServer/AccessGroup.cc:571
#7 0x000000000056de6f in Hypertable::AccessGroup::run_compaction
(this=0x1bc2a00, timestamp=
{logical = 1222151526557033001, real = 1222151526557728000},
major=false)
at /home/ht4/src/hypertable-0.9.0.10-0903/src/cc/Hypertable/
RangeServer/AccessGroup.cc:422
#8 0x000000000055d5dc in Hypertable::Range::run_compaction
(this=0x19a1000, major=false)
at /home/ht4/src/hypertable-0.9.0.10-0903/src/cc/Hypertable/
RangeServer/Range.cc:681
#9 0x000000000055d4e9 in Hypertable::Range::compact (this=0x19a1000,
major=false)
at /home/ht4/src/hypertable-0.9.0.10-0903/src/cc/Hypertable/
RangeServer/Range.cc:666
#10 0x0000000000556ea9 in
Hypertable::MaintenanceTaskCompaction::execute (this=0x1036150)
at /home/ht4/src/hypertable-0.9.0.10-0903/src/cc/Hypertable/
RangeServer/MaintenanceTaskCompaction.cc:38
#11 0x0000000000541a4f in
Hypertable::MaintenanceQueue::Worker::operator() (this=0x5221c1a8)
at /home/ht4/src/hypertable-0.9.0.10-0903/src/cc/Hypertable/
RangeServer/MaintenanceQueue.h:108
#12 0x0000000000541885 in
boost::detail::function::void_function_obj_invoker0<Hypertable::MaintenanceQueue::Worker,
void>::invoke (
[EMAIL PROTECTED])
at /home/ht4/src/hypertable-0.9.0.10-0903/src/cc/boost-1_34-fix/
boost/function/function_template.hpp:158
#13 0x0000002a95b8adc7 in boost::function0<void,
std::allocator<boost::function_base> >::operator() ()
from /home/ht4/hypertable/lib/libboost_thread-gcc34-mt-1_34_1.so.
1.34.1
#14 0x0000002a95b8a407 in boost::thread_group::join_all () from /home/
ht4/hypertable/lib/libboost_thread-gcc34-mt-1_34_1.so.1.34.1
#15 0x000000302b80610a in start_thread () from /lib64/tls/
libpthread.so.0
#16 0x000000302afc6003 in clone () from /lib64/tls/libc.so.6
#17 0x0000000000000000 in ?? ()
(gdb) frame 1
#1 0x00000000006354d8 in Hypertable::InetAddr::initialize
(addr=0x284d0f0, host=0x2ab184da68 "10.65.25.163", port=38060)
at /home/ht4/src/hypertable-0.9.0.10-0903/src/cc/Common/
InetAddr.cc:71
71 memcpy(&addr->sin_addr.s_addr, he->h_addr_list[0],
sizeof(uint32_t));
(gdb) x/x &addr->sin_addr.s_addr
0x284d0f4: 0x00000000
(gdb) x/x he->h_addr_list[0]
0xa6dc00: 0xa319410a
The same problem happened twice today, in the same line of code, with
the same stack trace, but on a different server.
Does anybody have any idea how this happens?
Donald
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Hypertable Development" group.
To post to this group, send email to [email protected]
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at
http://groups.google.com/group/hypertable-dev?hl=en
-~----------~----~----~----~------~----~------~--~---
