Woof. Of course.
Harold, You appear to be running on about 10 disks total. Each disk should be capable of about 100 ops per second but they appear to be doing about 70. This is plausible overhead. Try attaching 5 or 10 small EBS partitions to each of your nodes and use them in HDFS. That may substantially increase your maximum IOP rate and thus your read rate. Is your actual load going to be completely uniformly random? Or will there be a Zipf distribution? Will there be burst of repeated accesses? Uniform random can be a reasonably good approximation if you are running behind a cache large enough to cache all repeated accesses. If you aren't behind a cache, uniform access might be very unrealistic (and pessimistic). Do you have logs that you can use to model your actual read behaviors? On Tue, May 31, 2011 at 10:00 PM, Harold Lim <rold...@yahoo.com> wrote: > Hi Andrew, > > I tried running on c1.xlarge instances and the performance improved a > little bit but the throughput is still low. I can now get throughput of 700+ > read operations per second (up from 400-500+). I was hoping to get > throughput in the order of thousands. > > I was wondering if there is something wrong with my set-up or is it normal > for HBase running on ec2 instances to get low throughput numbers? > > > -Harold > > > --- On Mon, 5/30/11, Andrew Purtell <apurt...@apache.org> wrote: > > > From: Andrew Purtell <apurt...@apache.org> > > Subject: Re: How to improve HBase throughput with YCSB? > > To: user@hbase.apache.org > > Date: Monday, May 30, 2011, 8:33 PM > > The hypervisor steals a lot of CPU > > time from m1.large instances. You should be using > > c1.xlarge instances. > > > > Are you using local storage or EBS? > > > > Be aware that I/O performance on EC2 for any system is > > lower than if you are using real hardware, significantly so > > if not using one of the instance types with I/O performance > > listed as "high". > > > > > 2011/5/30 Harold Lim <rold...@yahoo.com> > > > > > > > Hi All, > > > > > > > > I have an HBase cluster on ec2 m1.large instance > > (10 > > > > region servers). > > > > >
