That exception on the DfsBroker log looks suspicious. What were the log lines immediately before it? There should be one that includes "Getting length of file ...".
The 'cap cleandb' command will wipe your database clean. If you just want to stop the services, use 'cap stop'. - Doug On Thu, Nov 1, 2012 at 2:02 AM, Hatice Çobanoğlu < [email protected]> wrote: > yes, I did gdb on dfs1 and dfs3. But I do not see anything related to my > problem. even I can not see anything.(: > > log files is the following: > > dfs3-> DfsBroker.hadoop.log > > Oct 30, 2012 11:57:55 AM org.hypertable.DfsBroker.hadoop.HdfsBroker Open > INFO: Opening file '/hypertable/tables/2/4/default/qyoNKN5rd__dbHKv/cs0' > flags=5 bs=0 handle = 179 > Oct 30, 2012 11:57:55 AM org.hypertable.DfsBroker.hadoop.HdfsBroker Open > INFO: Opening file '/hypertable/tables/2/4/default/qyoNKN5rd__dbHKv/cs1' > flags=5 bs=0 handle = 180 > Oct 30, 2012 12:48:10 PM org.hypertable.DfsBroker.hadoop.HdfsBroker Close > INFO: Closing input stream for file > /hypertable/tables/2/4/default/qyoNKN5rd__dbHKv/cs0 handle 179 > Oct 30, 2012 12:48:10 PM org.hypertable.DfsBroker.hadoop.HdfsBroker Close > INFO: Closing input stream for file > /hypertable/tables/2/4/default/qyoNKN5rd__dbHKv/cs1 handle 180 > Oct 30, 2012 3:45:08 PM org.hypertable.DfsBroker.hadoop.HdfsBroker Close > INFO: Closing output file /hypertable/servers/rs1/log/user/60 handle 172 > Oct 30, 2012 3:45:08 PM org.hypertable.DfsBroker.hadoop.HdfsBroker Create > INFO: Creating file '/hypertable/servers/rs1/log/user/61' handle = 181 > Oct 30, 2012 3:45:08 PM org.hypertable.DfsBroker.hadoop.HdfsBroker Create > INFO: Created file '/hypertable/servers/rs1/log/user/61' handle = 181 > Oct 30, 2012 3:45:13 PM org.hypertable.DfsBroker.hadoop.HdfsBroker Close > INFO: Closing output file /hypertable/servers/rs1/log/user/61 handle 181 > Oct 30, 2012 3:45:13 PM org.hypertable.DfsBroker.hadoop.HdfsBroker Create > INFO: Creating file '/hypertable/servers/rs1/log/user/62' handle = 182 > Oct 30, 2012 3:45:13 PM org.hypertable.DfsBroker.hadoop.HdfsBroker Create > INFO: Created file '/hypertable/servers/rs1/log/user/62' handle = 182 > Oct 30, 2012 3:45:17 PM org.hypertable.DfsBroker.hadoop.HdfsBroker Close > INFO: Closing output file /hypertable/servers/rs1/log/user/62 handle 182 > .... > .... > Exception in thread "ApplicationQueueThread 11" > java.lang.IllegalAccessError: > org/apache/hadoop/hdfs/DFSClient$DFSDataInputStream > at > org.hypertable.DfsBroker.hadoop.HdfsBroker.Length(HdfsBroker.java:399) > at > org.hypertable.DfsBroker.hadoop.RequestHandlerLength.run(RequestHandlerLength.java:53) > at > org.hypertable.AsyncComm.ApplicationQueue$Worker.run(ApplicationQueue.java:98) > at java.lang.Thread.run(Thread.java:679) > > Oct 30, 2012 3:56:16 PM org.hypertable.DfsBroker.hadoop.HdfsBroker Remove > INFO: Removing file '/hypertable/servers/rs1/log/user/66 > Oct 30, 2012 3:56:16 PM org.hypertable.DfsBroker.hadoop.HdfsBroker Close > INFO: Closing noverify input stream for file > /hypertable/tables/2/10/default/qyoNKN5rd__dbHKv/cs0 handle 205 > Oct 30, 2012 3:56:16 PM org.hypertable.DfsBroker.hadoop.HdfsBroker Close > INFO: Closing noverify input stream for file > /hypertable/tables/2/10/meta/qyoNKN5rd__dbHKv/cs1 handle 204 > Oct 30, 2012 3:56:16 PM org.hypertable.DfsBroker.hadoop.HdfsBroker Mkdirs > INFO: Making directory '/hypertable/tables/2/10/default/qyoNKN5rd__dbHKv' > Oct 30, 2012 3:56:16 PM org.hypertable.DfsBroker.hadoop.HdfsBroker Mkdirs > INFO: Making directory '/hypertable/tables/2/10/meta/qyoNKN5rd__dbHKv' > Oct 30, 2012 3:56:16 PM org.hypertable.DfsBroker.hadoop.HdfsBroker Remove > INFO: Removing file '/hypertable/servers/rs1/log/user/67 > Oct 30, 2012 3:56:16 PM org.hypertable.DfsBroker.hadoop.HdfsBroker Exists > INFO: Testing for existence of file > '/hypertable/tables/2/10/default/qyoNKN5rd__dbHKv > Oct 30, 2012 3:56:16 PM org.hypertable.DfsBroker.hadoop.HdfsBroker Remove > INFO: Removing file '/hypertable/servers/rs1/log/user/68 > Oct 30, 2012 3:56:16 PM org.hypertable.DfsBroker.hadoop.HdfsBroker Readdir > INFO: Readdir('/hypertable/tables/2/10/default/qyoNKN5rd__dbHKv') > Oct 30, 2012 3:56:16 PM org.hypertable.DfsBroker.hadoop.HdfsBroker Exists > INFO: Testing for existence of file > '/hypertable/tables/2/10/meta/qyoNKN5rd__dbHKv > Oct 30, 2012 3:56:16 PM org.hypertable.DfsBroker.hadoop.HdfsBroker Readdir > INFO: Readdir('/hypertable/tables/2/10/meta/qyoNKN5rd__dbHKv') > Oct 30, 2012 3:56:16 PM org.hypertable.DfsBroker.hadoop.HdfsBroker Length > INFO: Getting length of file > '/hypertable/tables/2/10/meta/qyoNKN5rd__dbHKv/cs1' (accurate: false) > Oct 30, 2012 3:56:16 PM org.hypertable.DfsBroker.hadoop.HdfsBroker Open > INFO: Opening file '/hypertable/tables/2/10/meta/qyoNKN5rd__dbHKv/cs1' > flags=1 bs=0 handle = 206 > Oct 30, 2012 3:56:17 PM org.hypertable.DfsBroker.hadoop.HdfsBroker Length > INFO: Getting length of file > '/hypertable/tables/2/10/default/qyoNKN5rd__dbHKv/cs0' (accurate: false) > Oct 30, 2012 3:56:17 PM org.hypertable.DfsBroker.hadoop.HdfsBroker Open > INFO: Opening file '/hypertable/tables/2/10/default/qyoNKN5rd__dbHKv/cs0' > flags=1 bs=0 handle = 207 > Oct 30, 2012 3:56:17 PM org.hypertable.DfsBroker.hadoop.HdfsBroker Readdir > INFO: > Readdir('/hypertable/servers/rs1/log/2/10/Eei6XLwtMtWXrj85-1351604749') > Oct 30, 2012 3:56:17 PM org.hypertable.DfsBroker.hadoop.HdfsBroker Length > INFO: Getting length of file > '/hypertable/servers/rs1/log/2/10/Eei6XLwtMtWXrj85-1351604749/0' (accurate: > true) > ..... > ... > ... > > Hypertable.Master.log > > 1351605377 INFO Hypertable.Master : > (/root/src/hypertable/src/cc/Hypertable/Master/OperationRelinquishAcknowledge.cc:51) > Entering RelinquishAcknowledge-67895 > 2/10[71b73697-4c90-4fba-a699-cc1f446b1d41..��] source=rs1 state=INITIAL > 1351605377 INFO Hypertable.Master : > (/root/src/hypertable/src/cc/Hypertable/Master/OperationRelinquishAcknowledge.cc:78) > Leaving RelinquishAcknowledge-67895 > 2/10[71b73697-4c90-4fba-a699-cc1f446b1d41..��] from rs1 > 1351605377 INFO Hypertable.Master : > (/root/src/hypertable/src/cc/Hypertable/Master/OperationGatherStatistics.cc:100) > Leaving GatherStatistics-67873 > 1351605387 INFO Hypertable.Master : > (/root/src/hypertable/src/cc/Hypertable/Master/OperationGatherStatistics.cc:57) > Entering GatherStatistics-67896 state=INITIAL > 1351605387 INFO Hypertable.Master : > (/root/src/hypertable/src/cc/Hypertable/Master/OperationCollectGarbage.cc:38) > Entering CollectGarbage-67897 > 1351605387 INFO Hypertable.Master : > (/root/src/hypertable/src/cc/Hypertable/Master/OperationLoadBalancer.cc:53) > Entering LoadBalancer-67898 > 1351605387 INFO Hypertable.Master : > (/root/src/hypertable/src/cc/Hypertable/Master/OperationLoadBalancer.cc:72) > Leaving LoadBalancer-67898 > 1351605417 WARN Hypertable.Master : > (/root/src/hypertable/src/cc/Hypertable/Master/ConnectionHandler.cc:220) > *Dropping > OperationGatherStatistics because another one is outstanding* > 1351605447 WARN Hypertable.Master : > (/root/src/hypertable/src/cc/Hypertable/Master/ConnectionHandler.cc:220) > Dropping OperationGatherStatistics because another one is outstanding > 1351605477 WARN Hypertable.Master : > (/root/src/hypertable/src/cc/Hypertable/Master/ConnectionHandler.cc:220) > Dropping OperationGatherStatistics because another one is outstanding > ... > ... > > rangeserver.log > > 1351605376 INFO Hypertable.RangeServer : > (/root/src/hypertable/src/cc/Hypertable/Lib/CommitLog.cc:350) > purge('/hypertable/servers/rs1/log/user') breaking because > 1351604755205931230 >= 1351604749409371001 > 1351605376 INFO Hypertable.RangeServer : > (/root/src/hypertable/src/cc/Hypertable/RangeServer/MaintenanceScheduler.cc:271) > Memory Statistics (MB): VM=3907.13, RSS=3070.39, tracked=440.93, > computed=1399.96 limit=4795.20 > 1351605376 INFO Hypertable.RangeServer : > (/root/src/hypertable/src/cc/Hypertable/RangeServer/MaintenanceScheduler.cc:276) > Memory Allocation: BlockCache=0.00% BlockIndex=0.07% BloomFilter=1.21% > CellCache=30.21% ShadowCache=0.00% QueryCache=68.50% > 1351605376 INFO Hypertable.RangeServer : > (/root/src/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:3777) > Memory Usage: 462345025 bytes > 1351605376 INFO Hypertable.RangeServer : > (/root/src/hypertable/src/cc/Hypertable/RangeServer/MetaLogEntityTaskAcknowledgeRelinquish.cc:40) > relinquish_acknowledge(rs1 2/10[71b73697-4c90-4fba-a699-cc1f446b1d41..��]) > 1351605376 INFO Hypertable.RangeServer : > (/root/src/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:1400) > Successfully created scanner (id=0) on table '0/0', returning 4 k/v pairs, > more=0 > 1351605376 INFO Hypertable.RangeServer : > (/root/src/hypertable/src/cc/Hypertable/RangeServer/Range.cc:269) Loading > CellStore 2/10/meta/qyoNKN5rd__dbHKv/cs1 > 1351605377 INFO Hypertable.RangeServer : > (/root/src/hypertable/src/cc/Hypertable/RangeServer/Range.cc:269) Loading > CellStore 2/10/default/qyoNKN5rd__dbHKv/cs0 > 1351605387 INFO Hypertable.RangeServer : > (/root/src/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:3038) > Entering get_statistics() > 1351605394 INFO Hypertable.RangeServer : > (/root/src/hypertable/src/cc/Hypertable/RangeServer/RSStats.h:83) > Maintenance stats scans=(7 31 1667 0.000002) updates=(276 2170351 > 2710090640 3.387613 0) > > > bin/ht shell > > Failed: > (f93e6298-0906-4e29-81ea-942667efc22d,anahtarkelime,1349738298383020001) - > HYPERTABLE request timeout > Failed: (f93e6298-0906-4e29-81ea-942667efc22d,headers,1349738298458034001) > - HYPERTABLE request timeout > Failed: (f93e6298-0906-4e29-81ea-942667efc22d,icerik,1349738298318529001) > - HYPERTABLE request timeout > Failed: (f93e70cd-a513-406b-86dd-53d993cd3616,url,1349333861927562001) - > HYPERTABLE request timeout > Failed: > (f93e70cd-a513-406b-86dd-53d993cd3616,sayfabasligi,1349333861960247001) - > HYPERTABLE request timeout > > *rs1(dfs3) last error "Hypertable no response" on the monitoring. * i > restart rs1 on dfs3. so RangeServer pid change. So all tables disappeared > on monitoring and i can not scan. when i scan : select * from <table> : > > 1351759073 ERROR hypertable : > (/root/src/hypertable/src/cc/AsyncComm/IOHandlerData.cc:632) connect() > completion error - Connection refused > 1351759073 INFO hypertable : > (/root/src/hypertable/src/cc/AsyncComm/ConnectionManager.cc:359) Event: > type=DISCONNECT from=dfs3:38060; Problem connecting to Root RangeServer, > will retry in 3000 milliseconds... > 1351759076 ERROR hypertable : > (/root/src/hypertable/src/cc/AsyncComm/IOHandlerData.cc:632) connect() > completion error - Connection refused > 1351759076 INFO hypertable : > (/root/src/hypertable/src/cc/AsyncComm/ConnectionManager.cc:359) Event: > type=DISCONNECT from=dfs3:38060; Problem connecting to Root RangeServer, > will retry in 3000 milliseconds... > > if i do the following: > cap cleandb > cap start, > i lost all of the data... > > > > > > 2012/10/30 Doug Judd <[email protected]> > >> First check to see if there are any errors in the DfsBroker.hadoop.log >> file on dfs3. If so, that might indicate that HDFS is having problems with >> the load being placed on it. If there are no errors in the >> DfsBroker.hadoop.log file on rs3, then try your test again and when the >> RangeServer deadlocks, do the following: >> >> 1. On dfs3, figure out what the process ID of the RangeServer is (e.g. >> with the 'ps' command) >> >> 2. Attach to the RangeServer process in gdb with the following command: >> >> /opt/hypertable/current/bin/ht gdb >> /opt/hypertable/current/bin/Hypertable.RangeServer <pid-of-range-server> >> >> 3. Issue the following gdb command and post the output: >> >> (gdb) thread apply all where >> >> >> That should give us more information about what's going on. >> >> - Doug >> >> >> On Tue, Oct 30, 2012 at 6:34 AM, Hatice Çobanoğlu < >> [email protected]> wrote: >> >>> hi, >>> my hypertable.cfg file is: >>> >>> # >>> # hypertable.cfg >>> # >>> >>> # HDFS Broker >>> HdfsBroker.Hadoop.ConfDir=/hadoop/conf >>> >>> # Ceph Broker >>> CephBroker.MonAddr=10.0.1.245:6789 >>> >>> # Local Broker >>> DfsBroker.Local.Root=fs/local >>> >>> # DFS Broker - for clients >>> DfsBroker.Port=38030 >>> >>> # Hyperspace >>> Hyperspace.Replica.Host=dfs1 >>> Hyperspace.Replica.Port=38040 >>> Hyperspace.Replica.Dir=hyperspace >>> Hyperspace.Replica.Workers=20 >>> >>> # Hypertable.Master >>> Hypertable.Master.Port=38050 >>> >>> # Hypertable.RangeServer >>> Hypertable.RangeServer.Port=38060 >>> >>> Hyperspace.KeepAlive.Interval=30000 >>> Hyperspace.Lease.Interval=1000000 >>> Hyperspace.GracePeriod=200000 >>> >>> Hypertable.RangeServer.Range.SplitSize=2GB >>> Hypertable.RangeServer.QueryCache.MaxMemory=4GB >>> Hypertable.RangeServer.CellStore.SkipNotFound=true >>> Hypertable.RangeServer.Maintenance.Interval=300000 >>> Hypertable.RangeServer.IgnoreClockSkewErrors=true >>> Hypertable.CommitLog.SkipErrors=true >>> Hypertable.RangeServer.Scanner.Ttl=7200000 >>> >>> # ThriftBroker >>> ThriftBroker.Port=38080 >>> >>> config file is: >>> >>> set :source_machine, "dfs1" >>> set :install_dir, "/opt/hypertable" >>> set :hypertable_version, "0.9.6.4" >>> set :default_pkg, "/tmp/hypertable-0.9.6.4-linux-x86_64.deb" >>> set :default_dfs, "hadoop" >>> set :default_config, "/opt/hypertable/current/conf/hypertable.cfg" >>> >>> role :source, "dfs1" >>> role :master, "dfs1" >>> role :hyperspace, "dfs1" >>> role :slave, "dfs3" >>> role :localhost, "dfs1" >>> role :spare >>> >>> >>> >>> 2012/10/30 Hatice Çobanoğlu <[email protected]> >>> >>>> hi, >>>> i m running on >>>> >>>> Ram: 32gb >>>> >>>> CPU Core: 8 >>>> >>>> İşletim Sistemi: Ubuntu 12.04 LTS >>>> >>>> Disk: 105+673gb >>>> >>>> >>>> and RangeServer running on >>>> >>>> >>>> Ram: 32gb >>>> >>>> CPU Core: 8 >>>> >>>> İşletim Sistemi: Ubuntu 12.04 LTS >>>> >>>> Disk: 105+673gb >>>> >>>> >>>> >>>> when RangeServer is deadlocked, ht system hangs. i restart RangeServer >>>> but not used. >>>> >>>> >>>> 2012/10/19 Doug Judd <[email protected]> >>>> >>>>> What kind of system are you running on (OS, architecture, amount of >>>>> RAM, # of disks)? If it looks like the RangeServer is deadlocked, it >>>>> would >>>>> be good to get a stack trace of it while it is deadlocked. If you're >>>>> running on a 64-bit CentOS or Redhat system you can use the tool 'pstack'. >>>>> Otherwise, you should be able to get a stack trace with the gdb command >>>>> 'thread apply all where'. >>>>> >>>>> - Doug >>>>> >>>>> On Fri, Oct 19, 2012 at 5:41 AM, Hatice Çobanoğlu < >>>>> [email protected]> wrote: >>>>> >>>>>> Hi, >>>>>> when running Hypertable on HDFS, load data infile hangs. To get >>>>>> around this problem, added a --no-log-sync option but this not solve >>>>>> my problem. >>>>>> it is deadlock in the same place. >>>>>> >>>>>> (i using hypertable 0.9.6.4 version) >>>>>> >>>>>> Loading 2,097,140,225 bytes of input data... >>>>>> >>>>>> 0% 10 20 30 40 50 60 70 80 90 100% >>>>>> >>>>>> |----|----|----|----|----|----|----|----|----|----|----|----|----|----|----| >>>>>> ****************************************** >>>>>> >>>>>> range server log: >>>>>> >>>>>> 1350649914 INFO Hypertable.RangeServer : >>>>>> (/root/src/hypertable/src/cc/Hypertable/RangeServer/MaintenanceScheduler.cc:276) >>>>>> Memory Allocation: BlockCache=0.00% BlockIndex=0.06% BloomFilter=1.03% >>>>>> CellCache=40.78% ShadowCache=0.00% QueryCache=58.13% >>>>>> 1350649914 INFO Hypertable.RangeServer : >>>>>> (/root/src/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:3777) >>>>>> Memory Usage: 724344597 bytes >>>>>> 1350649923 INFO Hypertable.RangeServer : >>>>>> (/root/src/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:1400) >>>>>> Successfully created scanner (id=0) on table '0/0', returning 2 k/v >>>>>> pairs, >>>>>> more=0 >>>>>> 1350649934 INFO Hypertable.RangeServer : >>>>>> (/root/src/hypertable/src/cc/Hypertable/RangeServer/RSStats.h:83) >>>>>> Maintenance stats scans=(18 44 2118 0.000007) updates=(16 0 0 0.000000 0) >>>>>> 1350649934 INFO Hypertable.RangeServer : >>>>>> (/root/src/hypertable/src/cc/Hypertable/RangeServer/MaintenanceScheduler.cc:271) >>>>>> Memory Statistics (MB): VM=3907.13, RSS=3069.55, tracked=690.79, >>>>>> computed=1649.82 limit=4795.20 >>>>>> 1350649934 INFO Hypertable.RangeServer : >>>>>> (/root/src/hypertable/src/cc/Hypertable/RangeServer/MaintenanceScheduler.cc:276) >>>>>> Memory Allocation: BlockCache=0.00% BlockIndex=0.06% BloomFilter=1.03% >>>>>> CellCache=40.78% ShadowCache=0.00% QueryCache=58.13% >>>>>> 1350649934 INFO Hypertable.RangeServer : >>>>>> (/root/src/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:3777) >>>>>> Memory Usage: 724344597 bytes >>>>>> 1350649934 INFO Hypertable.RangeServer : >>>>>> (/root/src/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:3038) >>>>>> Entering get_statistics() >>>>>> 1350649934 INFO Hypertable.RangeServer : >>>>>> (/root/src/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:3283) >>>>>> Exiting get_statistics() >>>>>> ...... >>>>>> .... >>>>>> ...... >>>>>> >>>>>> after... >>>>>> >>>>>> Failed: >>>>>> (ef6d5480-9216-424c-a9c9-4dc25ff9e2c1,headers,1349278165372653001) - >>>>>> HYPERTABLE request timeout >>>>>> Failed: >>>>>> (ef6d5480-9216-424c-a9c9-4dc25ff9e2c1,icerik,1349278165358924001) - >>>>>> HYPERTABLE request timeout >>>>>> Failed: >>>>>> (ef6d6758-8291-421c-974d-8d9d01b81e76,url,1349742562322557001) - >>>>>> HYPERTABLE >>>>>> request timeout >>>>>> >>>>>> >>>>>> -- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "Hypertable Development" group. >>>>>> To post to this group, send email to [email protected]. >>>>>> To unsubscribe from this group, send email to >>>>>> [email protected]. >>>>>> For more options, visit this group at >>>>>> http://groups.google.com/group/hypertable-dev?hl=en. >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> Doug Judd >>>>> CEO, Hypertable Inc. >>>>> >>>>> -- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "Hypertable Development" group. >>>>> To post to this group, send email to [email protected]. >>>>> To unsubscribe from this group, send email to >>>>> [email protected]. >>>>> For more options, visit this group at >>>>> http://groups.google.com/group/hypertable-dev?hl=en. >>>>> >>>> >>>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Hypertable Development" group. >>> To post to this group, send email to [email protected]. >>> To unsubscribe from this group, send email to >>> [email protected]. >>> For more options, visit this group at >>> http://groups.google.com/group/hypertable-dev?hl=en. >>> >> >> >> >> -- >> Doug Judd >> CEO, Hypertable Inc. >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Hypertable Development" group. >> To post to this group, send email to [email protected]. >> To unsubscribe from this group, send email to >> [email protected]. >> For more options, visit this group at >> http://groups.google.com/group/hypertable-dev?hl=en. >> > > -- > You received this message because you are subscribed to the Google Groups > "Hypertable Development" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]. > For more options, visit this group at > http://groups.google.com/group/hypertable-dev?hl=en. > -- Doug Judd CEO, Hypertable Inc. -- You received this message because you are subscribed to the Google Groups "Hypertable Development" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/hypertable-dev?hl=en.
