First check to see if there are any errors in the DfsBroker.hadoop.log file on dfs3. If so, that might indicate that HDFS is having problems with the load being placed on it. If there are no errors in the DfsBroker.hadoop.log file on rs3, then try your test again and when the RangeServer deadlocks, do the following:
1. On dfs3, figure out what the process ID of the RangeServer is (e.g. with the 'ps' command) 2. Attach to the RangeServer process in gdb with the following command: /opt/hypertable/current/bin/ht gdb /opt/hypertable/current/bin/Hypertable.RangeServer <pid-of-range-server> 3. Issue the following gdb command and post the output: (gdb) thread apply all where That should give us more information about what's going on. - Doug On Tue, Oct 30, 2012 at 6:34 AM, Hatice Çobanoğlu < [email protected]> wrote: > hi, > my hypertable.cfg file is: > > # > # hypertable.cfg > # > > # HDFS Broker > HdfsBroker.Hadoop.ConfDir=/hadoop/conf > > # Ceph Broker > CephBroker.MonAddr=10.0.1.245:6789 > > # Local Broker > DfsBroker.Local.Root=fs/local > > # DFS Broker - for clients > DfsBroker.Port=38030 > > # Hyperspace > Hyperspace.Replica.Host=dfs1 > Hyperspace.Replica.Port=38040 > Hyperspace.Replica.Dir=hyperspace > Hyperspace.Replica.Workers=20 > > # Hypertable.Master > Hypertable.Master.Port=38050 > > # Hypertable.RangeServer > Hypertable.RangeServer.Port=38060 > > Hyperspace.KeepAlive.Interval=30000 > Hyperspace.Lease.Interval=1000000 > Hyperspace.GracePeriod=200000 > > Hypertable.RangeServer.Range.SplitSize=2GB > Hypertable.RangeServer.QueryCache.MaxMemory=4GB > Hypertable.RangeServer.CellStore.SkipNotFound=true > Hypertable.RangeServer.Maintenance.Interval=300000 > Hypertable.RangeServer.IgnoreClockSkewErrors=true > Hypertable.CommitLog.SkipErrors=true > Hypertable.RangeServer.Scanner.Ttl=7200000 > > # ThriftBroker > ThriftBroker.Port=38080 > > config file is: > > set :source_machine, "dfs1" > set :install_dir, "/opt/hypertable" > set :hypertable_version, "0.9.6.4" > set :default_pkg, "/tmp/hypertable-0.9.6.4-linux-x86_64.deb" > set :default_dfs, "hadoop" > set :default_config, "/opt/hypertable/current/conf/hypertable.cfg" > > role :source, "dfs1" > role :master, "dfs1" > role :hyperspace, "dfs1" > role :slave, "dfs3" > role :localhost, "dfs1" > role :spare > > > > 2012/10/30 Hatice Çobanoğlu <[email protected]> > >> hi, >> i m running on >> >> Ram: 32gb >> >> CPU Core: 8 >> >> İşletim Sistemi: Ubuntu 12.04 LTS >> >> Disk: 105+673gb >> >> >> and RangeServer running on >> >> >> Ram: 32gb >> >> CPU Core: 8 >> >> İşletim Sistemi: Ubuntu 12.04 LTS >> >> Disk: 105+673gb >> >> >> >> when RangeServer is deadlocked, ht system hangs. i restart RangeServer >> but not used. >> >> >> 2012/10/19 Doug Judd <[email protected]> >> >>> What kind of system are you running on (OS, architecture, amount of RAM, >>> # of disks)? If it looks like the RangeServer is deadlocked, it would be >>> good to get a stack trace of it while it is deadlocked. If you're running >>> on a 64-bit CentOS or Redhat system you can use the tool 'pstack'. >>> Otherwise, you should be able to get a stack trace with the gdb command >>> 'thread apply all where'. >>> >>> - Doug >>> >>> On Fri, Oct 19, 2012 at 5:41 AM, Hatice Çobanoğlu < >>> [email protected]> wrote: >>> >>>> Hi, >>>> when running Hypertable on HDFS, load data infile hangs. To get around >>>> this problem, added a --no-log-sync option but this not solve my >>>> problem. >>>> it is deadlock in the same place. >>>> >>>> (i using hypertable 0.9.6.4 version) >>>> >>>> Loading 2,097,140,225 bytes of input data... >>>> >>>> 0% 10 20 30 40 50 60 70 80 90 100% >>>> >>>> |----|----|----|----|----|----|----|----|----|----|----|----|----|----|----| >>>> ****************************************** >>>> >>>> range server log: >>>> >>>> 1350649914 INFO Hypertable.RangeServer : >>>> (/root/src/hypertable/src/cc/Hypertable/RangeServer/MaintenanceScheduler.cc:276) >>>> Memory Allocation: BlockCache=0.00% BlockIndex=0.06% BloomFilter=1.03% >>>> CellCache=40.78% ShadowCache=0.00% QueryCache=58.13% >>>> 1350649914 INFO Hypertable.RangeServer : >>>> (/root/src/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:3777) >>>> Memory Usage: 724344597 bytes >>>> 1350649923 INFO Hypertable.RangeServer : >>>> (/root/src/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:1400) >>>> Successfully created scanner (id=0) on table '0/0', returning 2 k/v pairs, >>>> more=0 >>>> 1350649934 INFO Hypertable.RangeServer : >>>> (/root/src/hypertable/src/cc/Hypertable/RangeServer/RSStats.h:83) >>>> Maintenance stats scans=(18 44 2118 0.000007) updates=(16 0 0 0.000000 0) >>>> 1350649934 INFO Hypertable.RangeServer : >>>> (/root/src/hypertable/src/cc/Hypertable/RangeServer/MaintenanceScheduler.cc:271) >>>> Memory Statistics (MB): VM=3907.13, RSS=3069.55, tracked=690.79, >>>> computed=1649.82 limit=4795.20 >>>> 1350649934 INFO Hypertable.RangeServer : >>>> (/root/src/hypertable/src/cc/Hypertable/RangeServer/MaintenanceScheduler.cc:276) >>>> Memory Allocation: BlockCache=0.00% BlockIndex=0.06% BloomFilter=1.03% >>>> CellCache=40.78% ShadowCache=0.00% QueryCache=58.13% >>>> 1350649934 INFO Hypertable.RangeServer : >>>> (/root/src/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:3777) >>>> Memory Usage: 724344597 bytes >>>> 1350649934 INFO Hypertable.RangeServer : >>>> (/root/src/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:3038) >>>> Entering get_statistics() >>>> 1350649934 INFO Hypertable.RangeServer : >>>> (/root/src/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:3283) >>>> Exiting get_statistics() >>>> ...... >>>> .... >>>> ...... >>>> >>>> after... >>>> >>>> Failed: >>>> (ef6d5480-9216-424c-a9c9-4dc25ff9e2c1,headers,1349278165372653001) - >>>> HYPERTABLE request timeout >>>> Failed: >>>> (ef6d5480-9216-424c-a9c9-4dc25ff9e2c1,icerik,1349278165358924001) - >>>> HYPERTABLE request timeout >>>> Failed: (ef6d6758-8291-421c-974d-8d9d01b81e76,url,1349742562322557001) >>>> - HYPERTABLE request timeout >>>> >>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "Hypertable Development" group. >>>> To post to this group, send email to [email protected]. >>>> To unsubscribe from this group, send email to >>>> [email protected]. >>>> For more options, visit this group at >>>> http://groups.google.com/group/hypertable-dev?hl=en. >>>> >>> >>> >>> >>> -- >>> Doug Judd >>> CEO, Hypertable Inc. >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Hypertable Development" group. >>> To post to this group, send email to [email protected]. >>> To unsubscribe from this group, send email to >>> [email protected]. >>> For more options, visit this group at >>> http://groups.google.com/group/hypertable-dev?hl=en. >>> >> >> > -- > You received this message because you are subscribed to the Google Groups > "Hypertable Development" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]. > For more options, visit this group at > http://groups.google.com/group/hypertable-dev?hl=en. > -- Doug Judd CEO, Hypertable Inc. -- You received this message because you are subscribed to the Google Groups "Hypertable Development" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/hypertable-dev?hl=en.
