yes, I did gdb on dfs1 and dfs3. But I do not see anything related to my
problem. even I can not see anything.(:
log files is the following:
dfs3-> DfsBroker.hadoop.log
Oct 30, 2012 11:57:55 AM org.hypertable.DfsBroker.hadoop.HdfsBroker Open
INFO: Opening file '/hypertable/tables/2/4/default/qyoNKN5rd__dbHKv/cs0'
flags=5 bs=0 handle = 179
Oct 30, 2012 11:57:55 AM org.hypertable.DfsBroker.hadoop.HdfsBroker Open
INFO: Opening file '/hypertable/tables/2/4/default/qyoNKN5rd__dbHKv/cs1'
flags=5 bs=0 handle = 180
Oct 30, 2012 12:48:10 PM org.hypertable.DfsBroker.hadoop.HdfsBroker Close
INFO: Closing input stream for file
/hypertable/tables/2/4/default/qyoNKN5rd__dbHKv/cs0 handle 179
Oct 30, 2012 12:48:10 PM org.hypertable.DfsBroker.hadoop.HdfsBroker Close
INFO: Closing input stream for file
/hypertable/tables/2/4/default/qyoNKN5rd__dbHKv/cs1 handle 180
Oct 30, 2012 3:45:08 PM org.hypertable.DfsBroker.hadoop.HdfsBroker Close
INFO: Closing output file /hypertable/servers/rs1/log/user/60 handle 172
Oct 30, 2012 3:45:08 PM org.hypertable.DfsBroker.hadoop.HdfsBroker Create
INFO: Creating file '/hypertable/servers/rs1/log/user/61' handle = 181
Oct 30, 2012 3:45:08 PM org.hypertable.DfsBroker.hadoop.HdfsBroker Create
INFO: Created file '/hypertable/servers/rs1/log/user/61' handle = 181
Oct 30, 2012 3:45:13 PM org.hypertable.DfsBroker.hadoop.HdfsBroker Close
INFO: Closing output file /hypertable/servers/rs1/log/user/61 handle 181
Oct 30, 2012 3:45:13 PM org.hypertable.DfsBroker.hadoop.HdfsBroker Create
INFO: Creating file '/hypertable/servers/rs1/log/user/62' handle = 182
Oct 30, 2012 3:45:13 PM org.hypertable.DfsBroker.hadoop.HdfsBroker Create
INFO: Created file '/hypertable/servers/rs1/log/user/62' handle = 182
Oct 30, 2012 3:45:17 PM org.hypertable.DfsBroker.hadoop.HdfsBroker Close
INFO: Closing output file /hypertable/servers/rs1/log/user/62 handle 182
....
....
Exception in thread "ApplicationQueueThread 11"
java.lang.IllegalAccessError:
org/apache/hadoop/hdfs/DFSClient$DFSDataInputStream
at
org.hypertable.DfsBroker.hadoop.HdfsBroker.Length(HdfsBroker.java:399)
at
org.hypertable.DfsBroker.hadoop.RequestHandlerLength.run(RequestHandlerLength.java:53)
at
org.hypertable.AsyncComm.ApplicationQueue$Worker.run(ApplicationQueue.java:98)
at java.lang.Thread.run(Thread.java:679)
Oct 30, 2012 3:56:16 PM org.hypertable.DfsBroker.hadoop.HdfsBroker Remove
INFO: Removing file '/hypertable/servers/rs1/log/user/66
Oct 30, 2012 3:56:16 PM org.hypertable.DfsBroker.hadoop.HdfsBroker Close
INFO: Closing noverify input stream for file
/hypertable/tables/2/10/default/qyoNKN5rd__dbHKv/cs0 handle 205
Oct 30, 2012 3:56:16 PM org.hypertable.DfsBroker.hadoop.HdfsBroker Close
INFO: Closing noverify input stream for file
/hypertable/tables/2/10/meta/qyoNKN5rd__dbHKv/cs1 handle 204
Oct 30, 2012 3:56:16 PM org.hypertable.DfsBroker.hadoop.HdfsBroker Mkdirs
INFO: Making directory '/hypertable/tables/2/10/default/qyoNKN5rd__dbHKv'
Oct 30, 2012 3:56:16 PM org.hypertable.DfsBroker.hadoop.HdfsBroker Mkdirs
INFO: Making directory '/hypertable/tables/2/10/meta/qyoNKN5rd__dbHKv'
Oct 30, 2012 3:56:16 PM org.hypertable.DfsBroker.hadoop.HdfsBroker Remove
INFO: Removing file '/hypertable/servers/rs1/log/user/67
Oct 30, 2012 3:56:16 PM org.hypertable.DfsBroker.hadoop.HdfsBroker Exists
INFO: Testing for existence of file
'/hypertable/tables/2/10/default/qyoNKN5rd__dbHKv
Oct 30, 2012 3:56:16 PM org.hypertable.DfsBroker.hadoop.HdfsBroker Remove
INFO: Removing file '/hypertable/servers/rs1/log/user/68
Oct 30, 2012 3:56:16 PM org.hypertable.DfsBroker.hadoop.HdfsBroker Readdir
INFO: Readdir('/hypertable/tables/2/10/default/qyoNKN5rd__dbHKv')
Oct 30, 2012 3:56:16 PM org.hypertable.DfsBroker.hadoop.HdfsBroker Exists
INFO: Testing for existence of file
'/hypertable/tables/2/10/meta/qyoNKN5rd__dbHKv
Oct 30, 2012 3:56:16 PM org.hypertable.DfsBroker.hadoop.HdfsBroker Readdir
INFO: Readdir('/hypertable/tables/2/10/meta/qyoNKN5rd__dbHKv')
Oct 30, 2012 3:56:16 PM org.hypertable.DfsBroker.hadoop.HdfsBroker Length
INFO: Getting length of file
'/hypertable/tables/2/10/meta/qyoNKN5rd__dbHKv/cs1' (accurate: false)
Oct 30, 2012 3:56:16 PM org.hypertable.DfsBroker.hadoop.HdfsBroker Open
INFO: Opening file '/hypertable/tables/2/10/meta/qyoNKN5rd__dbHKv/cs1'
flags=1 bs=0 handle = 206
Oct 30, 2012 3:56:17 PM org.hypertable.DfsBroker.hadoop.HdfsBroker Length
INFO: Getting length of file
'/hypertable/tables/2/10/default/qyoNKN5rd__dbHKv/cs0' (accurate: false)
Oct 30, 2012 3:56:17 PM org.hypertable.DfsBroker.hadoop.HdfsBroker Open
INFO: Opening file '/hypertable/tables/2/10/default/qyoNKN5rd__dbHKv/cs0'
flags=1 bs=0 handle = 207
Oct 30, 2012 3:56:17 PM org.hypertable.DfsBroker.hadoop.HdfsBroker Readdir
INFO:
Readdir('/hypertable/servers/rs1/log/2/10/Eei6XLwtMtWXrj85-1351604749')
Oct 30, 2012 3:56:17 PM org.hypertable.DfsBroker.hadoop.HdfsBroker Length
INFO: Getting length of file
'/hypertable/servers/rs1/log/2/10/Eei6XLwtMtWXrj85-1351604749/0' (accurate:
true)
.....
...
...
Hypertable.Master.log
1351605377 INFO Hypertable.Master :
(/root/src/hypertable/src/cc/Hypertable/Master/OperationRelinquishAcknowledge.cc:51)
Entering RelinquishAcknowledge-67895
2/10[71b73697-4c90-4fba-a699-cc1f446b1d41..��] source=rs1 state=INITIAL
1351605377 INFO Hypertable.Master :
(/root/src/hypertable/src/cc/Hypertable/Master/OperationRelinquishAcknowledge.cc:78)
Leaving RelinquishAcknowledge-67895
2/10[71b73697-4c90-4fba-a699-cc1f446b1d41..��] from rs1
1351605377 INFO Hypertable.Master :
(/root/src/hypertable/src/cc/Hypertable/Master/OperationGatherStatistics.cc:100)
Leaving GatherStatistics-67873
1351605387 INFO Hypertable.Master :
(/root/src/hypertable/src/cc/Hypertable/Master/OperationGatherStatistics.cc:57)
Entering GatherStatistics-67896 state=INITIAL
1351605387 INFO Hypertable.Master :
(/root/src/hypertable/src/cc/Hypertable/Master/OperationCollectGarbage.cc:38)
Entering CollectGarbage-67897
1351605387 INFO Hypertable.Master :
(/root/src/hypertable/src/cc/Hypertable/Master/OperationLoadBalancer.cc:53)
Entering LoadBalancer-67898
1351605387 INFO Hypertable.Master :
(/root/src/hypertable/src/cc/Hypertable/Master/OperationLoadBalancer.cc:72)
Leaving LoadBalancer-67898
1351605417 WARN Hypertable.Master :
(/root/src/hypertable/src/cc/Hypertable/Master/ConnectionHandler.cc:220)
*Dropping
OperationGatherStatistics because another one is outstanding*
1351605447 WARN Hypertable.Master :
(/root/src/hypertable/src/cc/Hypertable/Master/ConnectionHandler.cc:220)
Dropping OperationGatherStatistics because another one is outstanding
1351605477 WARN Hypertable.Master :
(/root/src/hypertable/src/cc/Hypertable/Master/ConnectionHandler.cc:220)
Dropping OperationGatherStatistics because another one is outstanding
...
...
rangeserver.log
1351605376 INFO Hypertable.RangeServer :
(/root/src/hypertable/src/cc/Hypertable/Lib/CommitLog.cc:350)
purge('/hypertable/servers/rs1/log/user') breaking because
1351604755205931230 >= 1351604749409371001
1351605376 INFO Hypertable.RangeServer :
(/root/src/hypertable/src/cc/Hypertable/RangeServer/MaintenanceScheduler.cc:271)
Memory Statistics (MB): VM=3907.13, RSS=3070.39, tracked=440.93,
computed=1399.96 limit=4795.20
1351605376 INFO Hypertable.RangeServer :
(/root/src/hypertable/src/cc/Hypertable/RangeServer/MaintenanceScheduler.cc:276)
Memory Allocation: BlockCache=0.00% BlockIndex=0.07% BloomFilter=1.21%
CellCache=30.21% ShadowCache=0.00% QueryCache=68.50%
1351605376 INFO Hypertable.RangeServer :
(/root/src/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:3777)
Memory Usage: 462345025 bytes
1351605376 INFO Hypertable.RangeServer :
(/root/src/hypertable/src/cc/Hypertable/RangeServer/MetaLogEntityTaskAcknowledgeRelinquish.cc:40)
relinquish_acknowledge(rs1 2/10[71b73697-4c90-4fba-a699-cc1f446b1d41..��])
1351605376 INFO Hypertable.RangeServer :
(/root/src/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:1400)
Successfully created scanner (id=0) on table '0/0', returning 4 k/v pairs,
more=0
1351605376 INFO Hypertable.RangeServer :
(/root/src/hypertable/src/cc/Hypertable/RangeServer/Range.cc:269) Loading
CellStore 2/10/meta/qyoNKN5rd__dbHKv/cs1
1351605377 INFO Hypertable.RangeServer :
(/root/src/hypertable/src/cc/Hypertable/RangeServer/Range.cc:269) Loading
CellStore 2/10/default/qyoNKN5rd__dbHKv/cs0
1351605387 INFO Hypertable.RangeServer :
(/root/src/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:3038)
Entering get_statistics()
1351605394 INFO Hypertable.RangeServer :
(/root/src/hypertable/src/cc/Hypertable/RangeServer/RSStats.h:83)
Maintenance stats scans=(7 31 1667 0.000002) updates=(276 2170351
2710090640 3.387613 0)
bin/ht shell
Failed:
(f93e6298-0906-4e29-81ea-942667efc22d,anahtarkelime,1349738298383020001) -
HYPERTABLE request timeout
Failed: (f93e6298-0906-4e29-81ea-942667efc22d,headers,1349738298458034001)
- HYPERTABLE request timeout
Failed: (f93e6298-0906-4e29-81ea-942667efc22d,icerik,1349738298318529001) -
HYPERTABLE request timeout
Failed: (f93e70cd-a513-406b-86dd-53d993cd3616,url,1349333861927562001) -
HYPERTABLE request timeout
Failed:
(f93e70cd-a513-406b-86dd-53d993cd3616,sayfabasligi,1349333861960247001) -
HYPERTABLE request timeout
*rs1(dfs3) last error "Hypertable no response" on the monitoring. * i
restart rs1 on dfs3. so RangeServer pid change. So all tables disappeared
on monitoring and i can not scan. when i scan : select * from <table> :
1351759073 ERROR hypertable :
(/root/src/hypertable/src/cc/AsyncComm/IOHandlerData.cc:632) connect()
completion error - Connection refused
1351759073 INFO hypertable :
(/root/src/hypertable/src/cc/AsyncComm/ConnectionManager.cc:359) Event:
type=DISCONNECT from=dfs3:38060; Problem connecting to Root RangeServer,
will retry in 3000 milliseconds...
1351759076 ERROR hypertable :
(/root/src/hypertable/src/cc/AsyncComm/IOHandlerData.cc:632) connect()
completion error - Connection refused
1351759076 INFO hypertable :
(/root/src/hypertable/src/cc/AsyncComm/ConnectionManager.cc:359) Event:
type=DISCONNECT from=dfs3:38060; Problem connecting to Root RangeServer,
will retry in 3000 milliseconds...
if i do the following:
cap cleandb
cap start,
i lost all of the data...
2012/10/30 Doug Judd <[email protected]>
> First check to see if there are any errors in the DfsBroker.hadoop.log
> file on dfs3. If so, that might indicate that HDFS is having problems with
> the load being placed on it. If there are no errors in the
> DfsBroker.hadoop.log file on rs3, then try your test again and when the
> RangeServer deadlocks, do the following:
>
> 1. On dfs3, figure out what the process ID of the RangeServer is (e.g.
> with the 'ps' command)
>
> 2. Attach to the RangeServer process in gdb with the following command:
>
> /opt/hypertable/current/bin/ht gdb
> /opt/hypertable/current/bin/Hypertable.RangeServer <pid-of-range-server>
>
> 3. Issue the following gdb command and post the output:
>
> (gdb) thread apply all where
>
>
> That should give us more information about what's going on.
>
> - Doug
>
>
> On Tue, Oct 30, 2012 at 6:34 AM, Hatice Çobanoğlu <
> [email protected]> wrote:
>
>> hi,
>> my hypertable.cfg file is:
>>
>> #
>> # hypertable.cfg
>> #
>>
>> # HDFS Broker
>> HdfsBroker.Hadoop.ConfDir=/hadoop/conf
>>
>> # Ceph Broker
>> CephBroker.MonAddr=10.0.1.245:6789
>>
>> # Local Broker
>> DfsBroker.Local.Root=fs/local
>>
>> # DFS Broker - for clients
>> DfsBroker.Port=38030
>>
>> # Hyperspace
>> Hyperspace.Replica.Host=dfs1
>> Hyperspace.Replica.Port=38040
>> Hyperspace.Replica.Dir=hyperspace
>> Hyperspace.Replica.Workers=20
>>
>> # Hypertable.Master
>> Hypertable.Master.Port=38050
>>
>> # Hypertable.RangeServer
>> Hypertable.RangeServer.Port=38060
>>
>> Hyperspace.KeepAlive.Interval=30000
>> Hyperspace.Lease.Interval=1000000
>> Hyperspace.GracePeriod=200000
>>
>> Hypertable.RangeServer.Range.SplitSize=2GB
>> Hypertable.RangeServer.QueryCache.MaxMemory=4GB
>> Hypertable.RangeServer.CellStore.SkipNotFound=true
>> Hypertable.RangeServer.Maintenance.Interval=300000
>> Hypertable.RangeServer.IgnoreClockSkewErrors=true
>> Hypertable.CommitLog.SkipErrors=true
>> Hypertable.RangeServer.Scanner.Ttl=7200000
>>
>> # ThriftBroker
>> ThriftBroker.Port=38080
>>
>> config file is:
>>
>> set :source_machine, "dfs1"
>> set :install_dir, "/opt/hypertable"
>> set :hypertable_version, "0.9.6.4"
>> set :default_pkg, "/tmp/hypertable-0.9.6.4-linux-x86_64.deb"
>> set :default_dfs, "hadoop"
>> set :default_config, "/opt/hypertable/current/conf/hypertable.cfg"
>>
>> role :source, "dfs1"
>> role :master, "dfs1"
>> role :hyperspace, "dfs1"
>> role :slave, "dfs3"
>> role :localhost, "dfs1"
>> role :spare
>>
>>
>>
>> 2012/10/30 Hatice Çobanoğlu <[email protected]>
>>
>>> hi,
>>> i m running on
>>>
>>> Ram: 32gb
>>>
>>> CPU Core: 8
>>>
>>> İşletim Sistemi: Ubuntu 12.04 LTS
>>>
>>> Disk: 105+673gb
>>>
>>>
>>> and RangeServer running on
>>>
>>>
>>> Ram: 32gb
>>>
>>> CPU Core: 8
>>>
>>> İşletim Sistemi: Ubuntu 12.04 LTS
>>>
>>> Disk: 105+673gb
>>>
>>>
>>>
>>> when RangeServer is deadlocked, ht system hangs. i restart RangeServer
>>> but not used.
>>>
>>>
>>> 2012/10/19 Doug Judd <[email protected]>
>>>
>>>> What kind of system are you running on (OS, architecture, amount of
>>>> RAM, # of disks)? If it looks like the RangeServer is deadlocked, it would
>>>> be good to get a stack trace of it while it is deadlocked. If you're
>>>> running on a 64-bit CentOS or Redhat system you can use the tool 'pstack'.
>>>> Otherwise, you should be able to get a stack trace with the gdb command
>>>> 'thread apply all where'.
>>>>
>>>> - Doug
>>>>
>>>> On Fri, Oct 19, 2012 at 5:41 AM, Hatice Çobanoğlu <
>>>> [email protected]> wrote:
>>>>
>>>>> Hi,
>>>>> when running Hypertable on HDFS, load data infile hangs. To get around
>>>>> this problem, added a --no-log-sync option but this not solve my
>>>>> problem.
>>>>> it is deadlock in the same place.
>>>>>
>>>>> (i using hypertable 0.9.6.4 version)
>>>>>
>>>>> Loading 2,097,140,225 bytes of input data...
>>>>>
>>>>> 0% 10 20 30 40 50 60 70 80 90 100%
>>>>>
>>>>> |----|----|----|----|----|----|----|----|----|----|----|----|----|----|----|
>>>>> ******************************************
>>>>>
>>>>> range server log:
>>>>>
>>>>> 1350649914 INFO Hypertable.RangeServer :
>>>>> (/root/src/hypertable/src/cc/Hypertable/RangeServer/MaintenanceScheduler.cc:276)
>>>>> Memory Allocation: BlockCache=0.00% BlockIndex=0.06% BloomFilter=1.03%
>>>>> CellCache=40.78% ShadowCache=0.00% QueryCache=58.13%
>>>>> 1350649914 INFO Hypertable.RangeServer :
>>>>> (/root/src/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:3777)
>>>>> Memory Usage: 724344597 bytes
>>>>> 1350649923 INFO Hypertable.RangeServer :
>>>>> (/root/src/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:1400)
>>>>> Successfully created scanner (id=0) on table '0/0', returning 2 k/v pairs,
>>>>> more=0
>>>>> 1350649934 INFO Hypertable.RangeServer :
>>>>> (/root/src/hypertable/src/cc/Hypertable/RangeServer/RSStats.h:83)
>>>>> Maintenance stats scans=(18 44 2118 0.000007) updates=(16 0 0 0.000000 0)
>>>>> 1350649934 INFO Hypertable.RangeServer :
>>>>> (/root/src/hypertable/src/cc/Hypertable/RangeServer/MaintenanceScheduler.cc:271)
>>>>> Memory Statistics (MB): VM=3907.13, RSS=3069.55, tracked=690.79,
>>>>> computed=1649.82 limit=4795.20
>>>>> 1350649934 INFO Hypertable.RangeServer :
>>>>> (/root/src/hypertable/src/cc/Hypertable/RangeServer/MaintenanceScheduler.cc:276)
>>>>> Memory Allocation: BlockCache=0.00% BlockIndex=0.06% BloomFilter=1.03%
>>>>> CellCache=40.78% ShadowCache=0.00% QueryCache=58.13%
>>>>> 1350649934 INFO Hypertable.RangeServer :
>>>>> (/root/src/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:3777)
>>>>> Memory Usage: 724344597 bytes
>>>>> 1350649934 INFO Hypertable.RangeServer :
>>>>> (/root/src/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:3038)
>>>>> Entering get_statistics()
>>>>> 1350649934 INFO Hypertable.RangeServer :
>>>>> (/root/src/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:3283)
>>>>> Exiting get_statistics()
>>>>> ......
>>>>> ....
>>>>> ......
>>>>>
>>>>> after...
>>>>>
>>>>> Failed:
>>>>> (ef6d5480-9216-424c-a9c9-4dc25ff9e2c1,headers,1349278165372653001) -
>>>>> HYPERTABLE request timeout
>>>>> Failed:
>>>>> (ef6d5480-9216-424c-a9c9-4dc25ff9e2c1,icerik,1349278165358924001) -
>>>>> HYPERTABLE request timeout
>>>>> Failed: (ef6d6758-8291-421c-974d-8d9d01b81e76,url,1349742562322557001)
>>>>> - HYPERTABLE request timeout
>>>>>
>>>>>
>>>>> --
>>>>> You received this message because you are subscribed to the Google
>>>>> Groups "Hypertable Development" group.
>>>>> To post to this group, send email to [email protected].
>>>>> To unsubscribe from this group, send email to
>>>>> [email protected].
>>>>> For more options, visit this group at
>>>>> http://groups.google.com/group/hypertable-dev?hl=en.
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Doug Judd
>>>> CEO, Hypertable Inc.
>>>>
>>>> --
>>>> You received this message because you are subscribed to the Google
>>>> Groups "Hypertable Development" group.
>>>> To post to this group, send email to [email protected].
>>>> To unsubscribe from this group, send email to
>>>> [email protected].
>>>> For more options, visit this group at
>>>> http://groups.google.com/group/hypertable-dev?hl=en.
>>>>
>>>
>>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Hypertable Development" group.
>> To post to this group, send email to [email protected].
>> To unsubscribe from this group, send email to
>> [email protected].
>> For more options, visit this group at
>> http://groups.google.com/group/hypertable-dev?hl=en.
>>
>
>
>
> --
> Doug Judd
> CEO, Hypertable Inc.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Hypertable Development" group.
> To post to this group, send email to [email protected].
> To unsubscribe from this group, send email to
> [email protected].
> For more options, visit this group at
> http://groups.google.com/group/hypertable-dev?hl=en.
>
--
You received this message because you are subscribed to the Google Groups
"Hypertable Development" group.
To post to this group, send email to [email protected].
To unsubscribe from this group, send email to
[email protected].
For more options, visit this group at
http://groups.google.com/group/hypertable-dev?hl=en.
