Yeah more handlers won't do it here since there's tons of calls waiting on a single synchronized method, I guess the IndexedRegion should use a pool of HTables instead of a single one in order to improve indexation throughput.
J-D On Fri, Apr 30, 2010 at 2:26 PM, Chris Tarnas <c...@email.com> wrote: > Here is the thread dump: > > I cranked up the handlers to 300 just in case and ran 40 mappers that loaded > data via thrift. Each node runs its own thrift server. I saw an average of 18 > rows/sec/mapper with no node using more than 10% CPU and no IO wait. It seems > no matter how many mappers I throw the total number of rows/sec doesn't go > much above 700 rows/second total, which seems very, very slow to me. > > Here is the thread dump from a node: > > http://pastebin.com/U3eLRdMV > > I do see quite a bit of waiting and some blocking in there, not sure how > exactly to interpret it all though. > > thanks for any help! > -chris > > On Apr 29, 2010, at 9:14 PM, Ryan Rawson wrote: > >> One thing to check is at the peak of your load, run jstack on one of >> the regionservers, and look at the handler threads - if all of them >> are doing something you might be running into handler contention. >> >> it is basically ultimately IO bound. >> >> -ryan >> >> On Thu, Apr 29, 2010 at 9:12 PM, Chris Tarnas <c...@email.com> wrote: >>> They are all at 100, but none of the regionservers are loaded - most are >>> less than 20% CPU. Is this all network latency? >>> >>> -chris >>> >>> On Apr 29, 2010, at 8:29 PM, Ryan Rawson <ryano...@gmail.com> wrote: >>> >>>> Every insert on an indexed would require at the very least an RPC to a >>>> different regionserver. If the regionservers are busy, your request >>>> could wait in the queue for a moment. >>>> >>>> One param to tune would be the handler thread count. Set it to 100 at >>>> least. >>>> >>>> On Thu, Apr 29, 2010 at 2:16 AM, Chris Tarnas <c...@email.com> wrote: >>>>> >>>>> I just finished some testing with JDK 1.6 u17 - so far no performance >>>>> improvements with just changing that. Disabling LZO compression did gain a >>>>> little bit (up to about 30/sec from 25/sec per thread). Turning of indexes >>>>> helped the most - that brought me up to 115/sec @ 2875 total rows a >>>>> second. >>>>> A single perl/thrift process can load at over 350 rows/sec so its not >>>>> scaling as well as I would have expected, even without the indexes. >>>>> >>>>> Are the transactional indexes that costly? What is the bottleneck there? >>>>> CPU utilization and network packets went up when I disabled the indexes, I >>>>> don't think those are the bottlenecks for the indexes. I was even able to >>>>> add another 15 insert process (total of 40) and only lost about 10% on a >>>>> per >>>>> process throughput. I probably could go even higher, none of the nodes are >>>>> above CPU 60% utilization and IO wait was at most 3.5%. >>>>> >>>>> Each rowkey is unique, so there should not be any blocking on the row >>>>> locks. I'll do more indexed tests tomorrow. >>>>> >>>>> thanks, >>>>> -chris >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> On Apr 29, 2010, at 12:18 AM, Todd Lipcon wrote: >>>>> >>>>>> Definitely smells like JDK 1.6.0_18. Downgrade that back to 16 or 17 and >>>>>> you >>>>>> should be good to go. _18 is a botched release if I ever saw one. >>>>>> >>>>>> -Todd >>>>>> >>>>>> On Wed, Apr 28, 2010 at 10:54 PM, Chris Tarnas <c...@email.com> wrote: >>>>>> >>>>>>> Hi Stack, >>>>>>> >>>>>>> Thanks for looking. I checked the ganglia charts, no server was at more >>>>>>> than ~20% CPU utilization at any time during the load test and swap was >>>>>>> never used. Network traffic was light - just running a count through >>>>>>> hbase >>>>>>> shell generates a much higher use. One the server hosting meta >>>>>>> specifically, >>>>>>> it was at about 15-20% CPU, and IO wait never went above 3%, was >>>>>>> usually >>>>>>> down at near 0. >>>>>>> >>>>>>> The load also died with a thrift timeout on every single node (each >>>>>>> node >>>>>>> connecting to localhost for its thrift server), it looks like a >>>>>>> datanode >>>>>>> just died and caused every thrift connection to timeout - I'll have to >>>>>>> up >>>>>>> that limit to handle a node death. >>>>>>> >>>>>>> Checking logs this appears in the logs of the region server hosting >>>>>>> meta, >>>>>>> looks like the dead datanode causing this error: >>>>>>> >>>>>>> 2010-04-29 01:01:38,948 WARN org.apache.hadoop.hdfs.DFSClient: >>>>>>> DFSOutputStream ResponseProcessor exception for block >>>>>>> blk_508630839844593817_11180java.io.IOException: Bad response 1 for >>>>>>> block >>>>>>> blk_508630839844593817_11180 from datanode 10.195.150.255:50010 >>>>>>> at >>>>>>> >>>>>>> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$ResponseProcessor.run(DFSClient.java:2423) >>>>>>> >>>>>>> The regionserver log on teh dead node, 10.195.150.255 has some more >>>>>>> errors >>>>>>> in it: >>>>>>> >>>>>>> http://pastebin.com/EFH9jz0w >>>>>>> >>>>>>> I found this in the .out file on the datanode: >>>>>>> >>>>>>> # Java VM: Java HotSpot(TM) 64-Bit Server VM (16.0-b13 mixed mode >>>>>>> linux-amd64 ) >>>>>>> # Problematic frame: >>>>>>> # V [libjvm.so+0x62263c] >>>>>>> # >>>>>>> # An error report file with more information is saved as: >>>>>>> # /usr/local/hadoop-0.20.1/hs_err_pid1364.log >>>>>>> # >>>>>>> # If you would like to submit a bug report, please visit: >>>>>>> # http://java.sun.com/webapps/bugreport/crash.jsp >>>>>>> # >>>>>>> >>>>>>> >>>>>>> There is not a single error in the datanode's log though. Also of note >>>>>>> - >>>>>>> this happened well into the test, so the node dying cause the load to >>>>>>> abort >>>>>>> but not the prior poor performance. Looking through the mailing list it >>>>>>> looks like java 1.6.0_18 has a bad rep so I'll update the AMI (although >>>>>>> I'm >>>>>>> using the same JVM on other servers in the office w/o issue and decent >>>>>>> single node performance and never dying...). >>>>>>> >>>>>>> Thanks for any help! >>>>>>> -chris >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Apr 28, 2010, at 10:10 PM, Stack wrote: >>>>>>> >>>>>>>> What is load on the server hosting meta like? Higher than others? >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Apr 28, 2010, at 8:42 PM, Chris Tarnas <c...@email.com> wrote: >>>>>>>> >>>>>>>>> Hi JG, >>>>>>>>> >>>>>>>>> Speed is now down to 18 rows/sec/table per process. >>>>>>>>> >>>>>>>>> Here is a regionserver log that is serving two of the regions: >>>>>>>>> >>>>>>>>> http://pastebin.com/Hx5se0hz >>>>>>>>> >>>>>>>>> Here is the GC Log from the same server: >>>>>>>>> >>>>>>>>> http://pastebin.com/ChrRvxCx >>>>>>>>> >>>>>>>>> Here is the master log: >>>>>>>>> >>>>>>>>> http://pastebin.com/L1Kn66qU >>>>>>>>> >>>>>>>>> The thrift server logs have nothing in them in the same time period. >>>>>>>>> >>>>>>>>> Thanks in advance! >>>>>>>>> >>>>>>>>> -chris >>>>>>>>> >>>>>>>>> On Apr 28, 2010, at 7:32 PM, Jonathan Gray wrote: >>>>>>>>> >>>>>>>>>> Hey Chris, >>>>>>>>>> >>>>>>>>>> That's a really significant slowdown. I can't think of anything >>>>>>> >>>>>>> obvious that would cause that in your setup. >>>>>>>>>> >>>>>>>>>> Any chance of some regionserver and master logs from the time it was >>>>>>> >>>>>>> going slow? Is there any activity in the logs of the regionservers >>>>>>> hosting >>>>>>> the regions of the table being written to? >>>>>>>>>> >>>>>>>>>> JG >>>>>>>>>> >>>>>>>>>>> -----Original Message----- >>>>>>>>>>> From: Christopher Tarnas [mailto:c...@tarnas.org] On Behalf Of Chris >>>>>>>>>>> Tarnas >>>>>>>>>>> Sent: Wednesday, April 28, 2010 6:27 PM >>>>>>>>>>> To: hbase-user@hadoop.apache.org >>>>>>>>>>> Subject: EC2 + Thrift inserts >>>>>>>>>>> >>>>>>>>>>> Hello all, >>>>>>>>>>> >>>>>>>>>>> First, thanks to all the HBase developers for producing this, it's >>>>>>>>>>> a >>>>>>>>>>> great project and I'm glad to be able to use it. >>>>>>>>>>> >>>>>>>>>>> I'm looking for some help and hints here with insert performance >>>>>>>>>>> help. >>>>>>>>>>> I'm doing some benchmarking, testing how I can scale up using >>>>>>>>>>> HBase, >>>>>>>>>>> not really looking at raw speed. The testing is happening on EC2, >>>>>>> >>>>>>> using >>>>>>>>>>> >>>>>>>>>>> Andrew's scripts (thanks - those were very helpful) to set them up >>>>>>>>>>> and >>>>>>>>>>> with a slightly customized version of the default AMIs (added my >>>>>>>>>>> application modules). I'm using HBase 20.3 and Hadoop 20.1. I've >>>>>>> >>>>>>> looked >>>>>>>>>>> >>>>>>>>>>> at the tips in the Wiki and it looks like Andrew's scripts are >>>>>>>>>>> already >>>>>>>>>>> setup that way. >>>>>>>>>>> >>>>>>>>>>> I'm inserting into HBase from a hadoop streaming job that runs perl >>>>>>> >>>>>>> and >>>>>>>>>>> >>>>>>>>>>> uses the thrift gateway. I'm also using the Transactional tables so >>>>>>>>>>> that alone could be the case, but from what I can tell I don't >>>>>>>>>>> think >>>>>>>>>>> so. LZO compression is also enabled for the column families (much >>>>>>>>>>> of >>>>>>>>>>> the data is highly compressible). My cluster has 7 nodes, 5 >>>>>>>>>>> regionservers, 1 master and 1 zookeeper. The regionservers and >>>>>>>>>>> master >>>>>>>>>>> are c1.xlarges. Each regionserver has the tasktrackers that runs >>>>>>>>>>> the >>>>>>>>>>> hadoop streaming jobs, and regionserver also runs its own thrift >>>>>>>>>>> server. Each mapper that does the load talks to the localhost's >>>>>>>>>>> thrift >>>>>>>>>>> server. >>>>>>>>>>> >>>>>>>>>>> The Row keys a fixed string + an incremental number then the order >>>>>>>>>>> of >>>>>>>>>>> the bytes are reversed, so runA123 becomes 321Anur. I though of >>>>>>>>>>> using >>>>>>>>>>> murmur hash but was worried about collisions. >>>>>>>>>>> >>>>>>>>>>> As I add more insert jobs, each jobs throughput goes down. Way >>>>>>>>>>> down. I >>>>>>>>>>> went from about 200 row/sec/table per job with one job to about 24 >>>>>>>>>>> rows/sec/table per job with 25 running jobs. The servers are mostly >>>>>>>>>>> idle. I'm loading into two tables, one has several indexes and I'm >>>>>>>>>>> loading into three column families, the other has no indexes and >>>>>>>>>>> one >>>>>>>>>>> column family. Both tables only currently have two region each. >>>>>>>>>>> >>>>>>>>>>> The regionserver that serves the indexed table's regions is using >>>>>>>>>>> the >>>>>>>>>>> most CPU but is 87% idle. The other servers are all at ~90% idle. >>>>>>> >>>>>>> There >>>>>>>>>>> >>>>>>>>>>> is no IO wait. the perl processes are barely ticking over. Java on >>>>>>>>>>> the >>>>>>>>>>> most "loaded" server is using about 50-60% of one CPU. >>>>>>>>>>> >>>>>>>>>>> Normally when I do load in a pseudo-distrbuted hbase (my >>>>>>>>>>> development >>>>>>>>>>> platform) perl's speed is the limiting factor and uses about 85% of >>>>>>>>>>> a >>>>>>>>>>> CPU. In this cluster they are using only 5-10% of a CPU as they are >>>>>>> >>>>>>> all >>>>>>>>>>> >>>>>>>>>>> waiting on thrift (hbase). When I run only 1 process on the >>>>>>>>>>> cluster, >>>>>>>>>>> perl uses much more of a CPU, maybe 70%. >>>>>>>>>>> >>>>>>>>>>> Any tips or help in getting the speed/scalability up would be >>>>>>>>>>> great. >>>>>>>>>>> Please let me know if you need any other info. >>>>>>>>>>> >>>>>>>>>>> As I send this - it looks like the main table has split again and >>>>>>>>>>> is >>>>>>>>>>> being served by three regionservers.. My performance is going up a >>>>>>>>>>> bit >>>>>>>>>>> (now 35 rows/sec/table per processes), but still seems like I'm not >>>>>>>>>>> using the full potential of even the limited EC2 system, no IO wait >>>>>>> >>>>>>> and >>>>>>>>>>> >>>>>>>>>>> lots of idle CPU. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> many thanks >>>>>>>>>>> -chris >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Todd Lipcon >>>>>> Software Engineer, Cloudera >>>>> >>>>> >>> > >
