That's kind of a bummer with respect to the small EC2 instances and throughput. What can the CouchDB guys share about their setup with CouchIO - I'm not sure if I have the name right, but I mean the service where you guys host the CouchDB on EC2.
-Dustin On Tue, May 5, 2009 at 12:59 PM, Tom Nichols <[email protected]> wrote: > Thanks Jason. > > I removed the EBS volume and things seemed a little faster. CouchDB > is actually saturating the CPU now, so it looks like I'm getting > "accurate" benchmarks now. i.e. I'm actually exercising CouchDB and > I'm not limited by some hardware/OS latency. > > So I had 10 concurrent ruby processes performing batch inserts into > two test databases. After 2 hours, I have 1.1M documents (1.5 GB) in > each database, for a total of 2GB and 2.2 million documents. Does > this still seem slow? > > Thanks. > -Tom > > > On Mon, May 4, 2009 at 5:29 PM, Jason Smith <[email protected]> > wrote: > > Tom Nichols wrote: > >> > >> Hi, I have some questions about insert performance. > >> > >> I have a single CouchDB 0.9.0 node running on small EC2 instance. I > >> attached a huge EBS volume to it and mounted it where CouchDB's data > >> files are stored. I fired up about ruby scripts running inserts and > >> after a weekend I only have about 30GB/ 12M rows of data... Which > >> seems small. 'top' tells me that my CPU is only about 30% utilized. > >> > >> Any idea what I might be doing wrong? I pretty much just followed > >> these instructions: > >> http://wiki.apache.org/couchdb/Getting_started_with_Amazon_EC2 > > > > Hi, Tom. I believe I read somewhere before that the smallest EC2 > instances > > have a slower and/or higher-latency connection to EBS, so you might want > to > > consider a large instance, or maybe even a high-memory small instance and > > see whether you get better "hardware" performance. > > > > Although strangely, when googling it, the first article I found says that > > their benchmarks found no difference between EBS or even the ephemeral > > filesystem. > > > > > http://www.paessler.com/blog/2009/04/07/prtg-7/monitoring-cloud-performance-with-prtg-comparing-disk-speed-for-instance-stores-and-ebs-volumes-on-amazon-ec2/ > > > > On the other hand, here is a forum posting and a random benchmark > indicating > > that more expensive instances get better throughput: > > > > > http://developer.amazonwebservices.com/connect/message.jspa?messageID=125197 > > > http://blog.getasysadmin.com/2009/02/mysql-benchmarks-using-amazon-ec2.html > > > > -- > > Jason Smith > > Proven Corporation > > Bangkok, Thailand > > http://www.proven-corporation.com > > >
