Adam, You could also try to run 1 REST server per client machine or, if the clients are on the region servers, 1 REST server per RS. The Thrift and REST servers are basically wrappers around the Java client API.
J-D On Wed, Oct 7, 2009 at 1:52 AM, Adam Silberstein <[email protected]> wrote: > Hey, > Thanks for all the info... > > First, a few details to clarify my use case: > -I have 6 region servers. > -I loaded a total of 120GB in 1K records into my table, so 20GB per > server. I'm not sure how many regions that has created. > -My reported numbers are on workloads taking place once the 120GB is in > place, rather than while loading the 120GB. > -I've run with combinations of 50,100,200 clients hitting the REST > server. So that's e.g. 200 clients across all region servers, not per > region server. Each client just repeatedly a) generates a random record > known to exist, and b) reads or updates it. > -I'm interested in both throughput and latency. First, at medium > throughputs (i.e. not at maximum capacity) what are average read/write > latencies. And then, what is the maximum possible throughput, even as > that causes latencies to be very high. What is the throughput wall? > Plotting throughput vs. latency for different target throughputs reveals > both of these. > > When I have 50 clients across 6 region server, this is fairly close to > your read throughput experiment with 8 clients on 1 region server. Your > 2.4 k/sec throughput is obviously a lot better than what I'm seeing at > 300/sec. Since you had 10GB loaded, is it reasonable to assume that > ~50% of the reads were from memory? In my case, with 20GB loaded and > 6GB heapspace, I assume ~30% was served from memory. I haven't run > enough tests on different size tables to estimate the impact of having > data in memory, though intuitively, in the time it takes to read a > record from disk, you could read several from memory. And the more the > data is disk resident, the more the disk contention. > > Finally, I haven't tried LZO or increasing the logroll multiplier yet, > and I'm hoping to move to the java client soon. As you might recall, > we're working toward a benchmark for cloud serving stores. We're > testing the newest version of our tool now. Since it's in java, we'll > be able to use it with HBase. > > I'll report back when I find out how much these changes close the > performance gap, and how much seems inherent when much of the data is > disk resident. > > -Adam > > -----Original Message----- > From: [email protected] [mailto:[email protected]] On Behalf Of > stack > Sent: Tuesday, October 06, 2009 1:08 PM > To: [email protected] > Subject: Re: random read/write performance > > Hey Adam: > > Thanks for checking in. > > I just did some rough loadings on a small (old hardware) cluster using > less > memory per regionserver than you. Its described on this page: > http://wiki.apache.org/hadoop/Hbase/PerformanceEvaluation. Random > writing > 1k records with the PerformanceEvaluation script to a single > regionserver, I > can do about 8-10k/writes/second on average using the 0.20.1 release > candidate 1 with a single client. Sequential writes are about the same > speed usually. Random reads are about 650/second on average with single > client and about 2.4k/second on average with 8 concurrent clients. > > So it seems like you should be able to do better than > 300ops/persecond/permachine -- especially if you can do the java api. > > This single regionserver was carrying about 50 regions. Thats about > 10GB. > How many regions loaded in your case? > > If throughput is important to you, lzo should help (as per J-D). > Turning > off WAL will also help with write throughput but that might not be what > you > want. Random-read-wise, the best thing you can do is give it RAM (6G > should > be good). > > Is that 50-200 clients per regionserver or for the overall cluster? If > per > regionserver, I can try that over here. I can try with bigger regions > if > you'd like -- 1G regions -- to see if that'd help your use case (if you > enable lzo, this should up your throughput and shrink the number of > regions > any one server is hosting). > > St.Ack > > > > > > On Tue, Oct 6, 2009 at 8:59 AM, Adam Silberstein > <[email protected]>wrote: > >> Hi, >> >> Just wanted to give a quick update on our HBase benchmarking efforts > at >> Yahoo. The basic use case we're looking at is: >> >> 1K records >> >> 20GB of records per node (and 6GB of memory per node, so data is not >> memory resident) >> >> Workloads that do random reads/writes (e.g. 95% reads, 5% writes). >> >> Multiple clients doing the reads/writes (i.e. 50-200) >> >> Measure throughput vs. latency, and see how high we can push the >> throughput. >> >> Note that although we want to see where throughput maxes out, the >> workload is random, rather than scan-oriented. >> >> >> >> I've been tweaking our HBase installation based on advice I've >> read/gotten from a few people. Currently, I'm running 0.20.0, have > heap >> size set to 6GB per server, and have iCMS off. I'm still using the > REST >> server instead of the java client. We're about to move our > benchmarking >> tool to java, so at that point we can use the java API. At that > point, >> I want to turn off WAL as well. If anyone has more suggestions for > this >> workload (either things to try while still using REST, or things to > try >> once I have a java client), please let me know. >> >> >> >> Given all that, I'm currently seeing maximal throughput of about 300 >> ops/sec/server. Has anyone with a similar disk-resident and random >> workload seen drastically different numbers, or guesses for what I can >> expect with the java client? >> >> >> >> Thanks! >> >> Adam >> >> >
