Hi,
is this 0.9.6.5 and always reproducible after you created the empty hdfs
directory as you described in one of your previous mails?
can you run the RangeServer in valgrind?
ht start-rangeserver.sh --valgrind
this creates a log file vg.rs.{pid} which may contain more information
about the invalid memory access.
bye
Christoph
2013/1/7 richardman <[email protected]>
> Hi,all
> we can got murderer , the member this->m_send_queue of
> Hypertable::IOHandlerData, rangeServer crash when m_send_queue
> be Destructor, thanks, Expecting a reply
>
> (gdb) p &this->m_send_queue
> $4 = (
> std::list<boost::intrusive_ptr<Hypertable::CommBuf>,
> std::allocator<boost::intrusive_ptr<Hypertable::CommBuf> > > *) 0x516fd98
> (gdb)
>
>
> 2013/1/7 richardman <[email protected]>
>
>> Hi,all
>>
>> the gdb debug log as follow:
>> ////////////////////////////////////begin////////////////////////
>> 1357564314 INFO Hypertable.RangeServer :
>> (/home/hadoop/temp/hypertable-0.9.6.5/src/cc/Hypertable/RangeServer/MaintenanceScheduler.cc:276)
>> Memory Allocation: BlockCache=5.42% BlockIndex=0.53% BloomFilter=6.75%
>> CellCache=64.58% ShadowCache=0.00% QueryCache=22.72%
>> 1357564314 INFO Hypertable.RangeServer :
>> (/home/hadoop/temp/hypertable-0.9.6.5/src/cc/Hypertable/RangeServer/RangeServer.cc:3777)
>> Memory Usage: 172245191 bytes
>> 1357564329 INFO Hypertable.RangeServer :
>> (/home/hadoop/temp/hypertable-0.9.6.5/src/cc/AsyncComm/IOHandler.h:95)
>> Event: type=CONNECTION_ESTABLISHED from=1.164.167.224:64707
>> 1357564330 ERROR Hypertable.RangeServer :
>> (/home/hadoop/temp/hypertable-0.9.6.5/src/cc/AsyncComm/IOHandlerData.cc:237)
>> socket read(37, len=1953718478) failure : Bad address
>> 1357564330 INFO Hypertable.RangeServer :
>> (/home/hadoop/temp/hypertable-0.9.6.5/src/cc/Hypertable/RangeServer/ConnectionHandler.cc:215)
>> Event: type=DISCONNECT from=1.164.167.224:64707
>>
>> Program received signal SIGSEGV, Segmentation fault.
>> [Switching to Thread 0x7ffff3a65700 (LWP 21510)]
>> 0x0000000000e18324 in
>> std::_List_base<boost::intrusive_ptr<Hypertable::CommBuf>,
>> std::allocator<boost::intrusive_ptr<Hypertable::CommBuf> > >::_M_clear
>> (this=0x516fd98) at /usr/include/c++/4.4/bits/list.tcc:72
>> 72 __cur = static_cast<_Node*>(__cur->_M_next);
>> (gdb) where
>> #0 0x0000000000e18324 in
>> std::_List_base<boost::intrusive_ptr<Hypertable::CommBuf>,
>> std::allocator<boost::intrusive_ptr<Hypertable::CommBuf> > >::_M_clear
>> (this=0x516fd98) at /usr/include/c++/4.4/bits/list.tcc:72
>> #1 0x0000000000e1674d in ~_List_base (this=0x516fd98, __in_chrg=<value
>> optimized out>)
>> at /usr/include/c++/4.4/bits/stl_list.h:360
>> #2 0x0000000000e13200 in ~list (this=0x516fd98, __in_chrg=<value
>> optimized out>)
>> at /usr/include/c++/4.4/bits/stl_list.h:418
>> #3 0x0000000000e29d4d in ~IOHandlerData (this=0x516fc60,
>> __in_chrg=<value optimized out>)
>> at
>> /home/hadoop/temp/hypertable-0.9.6.5/src/cc/AsyncComm/IOHandlerData.h:55
>> #4 0x00000000009f05b0 in Hypertable::intrusive_ptr_release (rc=0x516fc60)
>> at
>> /home/hadoop/temp/hypertable-0.9.6.5/src/cc/Common/ReferenceCount.h:73
>> #5 0x0000000000e16f49 in ~intrusive_ptr (this=0x7ffff3a63f10,
>> __in_chrg=<value optimized out>)
>> at /usr/include/boost/smart_ptr/intrusive_ptr.hpp:101
>> #6 0x0000000000e3eab7 in Hypertable::HandlerMap::purge_handler
>> (this=0x134e000, handler=0x516fc60)
>> at
>> /home/hadoop/temp/hypertable-0.9.6.5/src/cc/AsyncComm/HandlerMap.h:275
>> #7 0x0000000000e3e760 in
>> Hypertable::ReactorRunner::cleanup_and_remove_handlers
>> (this=0x7fffec001060,
>> handlers=...) at
>> /home/hadoop/temp/hypertable-0.9.6.5/src/cc/AsyncComm/ReactorRunner.cc:326
>> #8 0x0000000000e3deff in Hypertable::ReactorRunner::operator()
>> (this=0x7fffec001060)
>> at
>> /home/hadoop/temp/hypertable-0.9.6.5/src/cc/AsyncComm/ReactorRunner.cc:171
>> #9 0x0000000000e3cd5a in
>> boost::detail::thread_data<Hypertable::ReactorRunner>::run
>> (this=0x7fffec000f30)
>> at /usr/include/boost/thread/detail/thread.hpp:56
>> #10 0x00007ffff7324200 in thread_proxy () from
>> /usr/lib/libboost_thread.so.1.42.0
>> #11 0x00007ffff64778ca in start_thread (arg=<value optimized out>) at
>> pthread_create.c:300
>> #12 0x00007ffff5371b6d in clone () at
>> ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
>> #13 0x0000000000000000 in ?? ()
>> (gdb)
>>
>> ////////////////////////////////////end//////////////////////////
>>
>>
>>
>> 2013/1/7 richardman <[email protected]>
>>
>>> thanks, I should debug the RangeServer later and feedback to you ;
>>>
>>>
>>> 2013/1/6 Christoph Rupp <[email protected]>
>>>
>>>> Does it really crash? this is just a normal error message. You have to
>>>> figure out which client has this IP (could be another RangeServer, a
>>>> Hyperspace machine, the Master, a ThriftBroker or an application using the
>>>> native C++ API). If it's an unknown IP then maybe setup a firewall to
>>>> protect your cluster from external requests.
>>>>
>>>> bye
>>>> Christoph
>>>>
>>>> 2013/1/6 richardman <[email protected]>
>>>>
>>>>> Hi,all
>>>>> Hypertable.RangeServer will crash, running for some time, the
>>>>> log is as follows:
>>>>>
>>>>> CPU cores count=4
>>>>> CephBroker.MonAddr=10.0.1.245:6789
>>>>> DfsBroker.Local.Root=fs/local
>>>>> DfsBroker.Port=38030
>>>>> HdfsBroker.fs.default.name=hdfs://192.168.0.245:9000
>>>>> Hyperspace.GracePeriod=200000
>>>>> Hyperspace.KeepAlive.Interval=30000
>>>>> Hyperspace.Lease.Interval=1000000
>>>>> Hyperspace.Replica.Dir=hyperspace
>>>>> Hyperspace.Replica.Host=[192.168.0.245]
>>>>> Hyperspace.Replica.Port=38040
>>>>> Hypertable.Master.Port=38050
>>>>> Hypertable.RangeServer.CellStore.SkipNotFound=true
>>>>> Hypertable.RangeServer.LowMemoryLimit.Percentage=10
>>>>> Hypertable.RangeServer.MaintenanceThreads=10
>>>>> Hypertable.RangeServer.MemoryLimit=4096000000
>>>>> Hypertable.RangeServer.MemoryLimit.Percentage=25
>>>>> Hypertable.RangeServer.Port=38060
>>>>> Hypertable.RangeServer.Range.SplitSize=512000000
>>>>> Hypertable.RangeServer.Reactors=4
>>>>> Hypertable.RangeServer.Workers=50
>>>>> Hypertable.Verbose=true
>>>>> ThriftBroker.Port=38080
>>>>> config=/opt/hypertable/0.9.6.5/conf/hypertable.cfg
>>>>> dfs-port=38030
>>>>> grace-period=200000
>>>>> hs-host=[192.168.0.245]
>>>>> hs-port=38040
>>>>> keepalive=30000
>>>>> lease-interval=1000000
>>>>> master-port=38050
>>>>> pidfile=/opt/hypertable/current/run/Hypertable.RangeServer.pid
>>>>> port=38060
>>>>> reactors=4
>>>>> verbose=true
>>>>> workers=50
>>>>> 1357412412 INFO Hypertable.RangeServer :
>>>>> (/home/hadoop/temp/hypertable-0.9.6.5/src/cc/Hypertable/RangeServer/HyperspaceSessionHandler.cc:31)
>>>>> Hyperspace session state change: SAFE
>>>>> drive count = 1
>>>>> maintenance threads = 10
>>>>> 1357412413 INFO Hypertable.RangeServer :
>>>>> (/home/hadoop/temp/hypertable-0.9.6.5/src/cc/Hypertable/RangeServer/ConnectionHandler.cc:215)
>>>>> Event: type=CONNECTION_ESTABLISHED from=192.168.0.245:38050
>>>>> 1357412413 INFO Hypertable.RangeServer :
>>>>> (/home/hadoop/temp/hypertable-0.9.6.5/src/cc/Hypertable/RangeServer/RangeServer.cc:502)
>>>>> log_dir=/hypertable/servers/rs1/log
>>>>> 1357412413 INFO Hypertable.RangeServer :
>>>>> (/home/hadoop/temp/hypertable-0.9.6.5/src/cc/AsyncComm/IOHandler.h:95)
>>>>> Event: type=CONNECTION_ESTABLISHED from=192.168.0.245:33108
>>>>> 1357412413 INFO Hypertable.RangeServer :
>>>>> (/home/hadoop/temp/hypertable-0.9.6.5/src/cc/AsyncComm/IOHandler.h:95)
>>>>> Event: type=CONNECTION_ESTABLISHED from=192.168.0.245:48072
>>>>> 1357412413 INFO Hypertable.RangeServer :
>>>>> (/home/hadoop/temp/hypertable-0.9.6.5/src/cc/AsyncComm/IOHandler.h:95)
>>>>> Event: type=CONNECTION_ESTABLISHED from=192.168.0.245:51749
>>>>> 1357412414 INFO Hypertable.RangeServer :
>>>>> (/home/hadoop/temp/hypertable-0.9.6.5/src/cc/AsyncComm/IOHandler.h:95)
>>>>> Event: type=CONNECTION_ESTABLISHED from=192.168.0.245:47547
>>>>> 1357412414 INFO Hypertable.RangeServer :
>>>>> (/home/hadoop/temp/hypertable-0.9.6.5/src/cc/Hypertable/RangeServer/ConnectionHandler.cc:215)
>>>>> Event: type=DISCONNECT from=192.168.0.245:47547
>>>>> 1357412414 INFO Hypertable.RangeServer :
>>>>> (/home/hadoop/temp/hypertable-0.9.6.5/src/cc/Hypertable/RangeServer/Range.cc:269)
>>>>> Loading CellStore 0/0/default/4MmZEeUI1pJuk6SA/cs12
>>>>> 1357412414 INFO Hypertable.RangeServer :
>>>>> (/home/hadoop/temp/hypertable-0.9.6.5/src/cc/Hypertable/RangeServer/Range.cc:269)
>>>>> Loading CellStore 0/0/location/4MmZEeUI1pJuk6SA/cs13
>>>>> 1357412414 INFO Hypertable.RangeServer :
>>>>> (/home/hadoop/temp/hypertable-0.9.6.5/src/cc/Hypertable/RangeServer/TableInfo.cc:252)
>>>>> Adding range 0/0[..0/0:ÿÿ] to TableInfo
>>>>>
>>>>> .....................skip......follow is the last log
>>>>>
>>>>> 1357420036 INFO Hypertable.RangeServer :
>>>>> (/home/hadoop/temp/hypertable-0.9.6.5/src/cc/Hypertable/RangeServer/MaintenanceScheduler.cc:271)
>>>>> Memory Statistics (MB): VM=1099.50, RSS=167.14, tracked=142.00,
>>>>> computed=188.75 limit=3906.25
>>>>> 1357420036 INFO Hypertable.RangeServer :
>>>>> (/home/hadoop/temp/hypertable-0.9.6.5/src/cc/Hypertable/RangeServer/MaintenanceScheduler.cc:276)
>>>>> Memory Allocation: BlockCache=0.74% BlockIndex=0.35% BloomFilter=5.80%
>>>>> CellCache=67.84% ShadowCache=0.00% QueryCache=25.26%
>>>>> 1357420036 INFO Hypertable.RangeServer :
>>>>> (/home/hadoop/temp/hypertable-0.9.6.5/src/cc/Hypertable/RangeServer/RangeServer.cc:3777)
>>>>> Memory Usage: 148895531 bytes
>>>>> 1357420045 INFO Hypertable.RangeServer :
>>>>> (/home/hadoop/temp/hypertable-0.9.6.5/src/cc/AsyncComm/IOHandler.h:95)
>>>>> Event: type=CONNECTION_ESTABLISHED from=80.179.7.132:2564
>>>>> 1357420056 INFO Hypertable.RangeServer :
>>>>> (/home/hadoop/temp/hypertable-0.9.6.5/src/cc/Hypertable/RangeServer/RSStats.h:83)
>>>>> Maintenance stats scans=(0 0 0 0.000000) updates=(0 0 0 0.000000 0)
>>>>> 1357420056 INFO Hypertable.RangeServer :
>>>>> (/home/hadoop/temp/hypertable-0.9.6.5/src/cc/Hypertable/RangeServer/MaintenanceScheduler.cc:271)
>>>>> Memory Statistics (MB): VM=1099.50, RSS=167.14, tracked=142.00,
>>>>> computed=188.75 limit=3906.25
>>>>> 1357420056 INFO Hypertable.RangeServer :
>>>>> (/home/hadoop/temp/hypertable-0.9.6.5/src/cc/Hypertable/RangeServer/MaintenanceScheduler.cc:276)
>>>>> Memory Allocation: BlockCache=0.74% BlockIndex=0.35% BloomFilter=5.80%
>>>>> CellCache=67.84% ShadowCache=0.00% QueryCache=25.26%
>>>>> 1357420056 INFO Hypertable.RangeServer :
>>>>> (/home/hadoop/temp/hypertable-0.9.6.5/src/cc/Hypertable/RangeServer/RangeServer.cc:3777)
>>>>> Memory Usage: 148895531 bytes
>>>>> 1357420057 INFO Hypertable.RangeServer :
>>>>> (/home/hadoop/temp/hypertable-0.9.6.5/src/cc/Hypertable/RangeServer/RangeServer.cc:3038)
>>>>> Entering get_statistics()
>>>>> 1357420057 INFO Hypertable.RangeServer :
>>>>> (/home/hadoop/temp/hypertable-0.9.6.5/src/cc/Hypertable/RangeServer/RangeServer.cc:3283)
>>>>> Exiting get_statistics()
>>>>> 1357420067 ERROR Hypertable.RangeServer :
>>>>> (/home/hadoop/temp/hypertable-0.9.6.5/src/cc/AsyncComm/IOHandlerData.cc:237)
>>>>> socket read(33, len=50331562) failure : Bad address
>>>>> 1357420067 INFO Hypertable.RangeServer :
>>>>> (/home/hadoop/temp/hypertable-0.9.6.5/src/cc/Hypertable/RangeServer/ConnectionHandler.cc:215)
>>>>> Event: type=DISCONNECT from=80.179.7.132:2564
>>>>>
>>>>> ///////////////////////////End////////////////////////////////////
>>>>> the rangeserver want to connect to a IP 80.179.7.132:2564
>>>>> but my master server in 192.168.0.245,what happen?
>>>>>
>>>>> --
>>>>> You received this message because you are subscribed to the Google
>>>>> Groups "Hypertable Development" group.
>>>>> To post to this group, send email to [email protected].
>>>>> To unsubscribe from this group, send email to
>>>>> [email protected].
>>>>> For more options, visit this group at
>>>>> http://groups.google.com/group/hypertable-dev?hl=en.
>>>>>
>>>>
>>>> --
>>>> You received this message because you are subscribed to the Google
>>>> Groups "Hypertable Development" group.
>>>> To post to this group, send email to [email protected].
>>>> To unsubscribe from this group, send email to
>>>> [email protected].
>>>> For more options, visit this group at
>>>> http://groups.google.com/group/hypertable-dev?hl=en.
>>>>
>>>
>>>
>>
> --
> You received this message because you are subscribed to the Google Groups
> "Hypertable Development" group.
> To post to this group, send email to [email protected].
> To unsubscribe from this group, send email to
> [email protected].
> For more options, visit this group at
> http://groups.google.com/group/hypertable-dev?hl=en.
>
--
You received this message because you are subscribed to the Google Groups
"Hypertable Development" group.
To post to this group, send email to [email protected].
To unsubscribe from this group, send email to
[email protected].
For more options, visit this group at
http://groups.google.com/group/hypertable-dev?hl=en.
