Just wanted to follow up on this.
We were never able to achieve throughput scaling in the cloud. We were able to
verify that many of our cluster nodes and test servers were collocated on the
same physical hardware (thanks Stu for the tip on the Rackspace REST API), and
that performance on
Just wanted to follow up on this.
We were never able to achieve throughput scaling in the cloud. We were able
to verify that many of our cluster nodes and test servers were collocated on
the same physical hardware (thanks Stu for the tip on the Rackspace REST
API), and that performance
I managed to run a few benchmarks.
Servers r/s
164.5k
259.5k
The configuration:
Client: Machine with four Quad Core Intel Xeon CPU E5520 @ 2.27Ghz
cpus (total 16 cores), 4530 bogomips per core. 12 GB ECC corrected
memory. Supermicro mainboard (not sure about exact type).
But what's then the point with adding nodes into the ring? Disk speed!
Well, it may also be cheaper to service an RPC request than service a
full read or write, even in terms of CPU.
But: Even taking into account that requests are distributed randomly,
the cluster should still scale. You will
On Tue, Jul 20, 2010 at 6:20 AM, Juho Mäkinen juho.maki...@gmail.com wrote:
I managed to run a few benchmarks.
Servers r/s
1 64.5k
2 59.5k
The configuration:
Client: Machine with four Quad Core Intel Xeon CPU E5520 @ 2.27Ghz
cpus (total 16 cores), 4530 bogomips per
(I'm hoping to have time to run my test on EC2 tonight; will see.)
Well, I needed three c1.xlarge EC2 instances running py_stress to even
saturate more than one core on the c1.xlarge instance running a single
cassandra node (at roughly 21k reqs/second)... Depending on how
reliable vmstat/top is
Yes, as the size of the data on disk increases and the OS cannot avoid disk
seeks the read performance degrades. You can see this in the results from the
original post where the number of keys in the test goes from 10M to 100M the
reads drop from 4,600/s to 200/s. 10M keys in the stress.py
To: user@cassandra.apache.org user@cassandra.apache.org
Subject: Re: Cassandra benchmarking on Rackspace Cloud
Certainly I'm using multiple cloud servers for the multiple client tests.
Whether or not they are resident on the same physical machine, I just don't
know.
-- Oren
On Jul 18
This may be too much work... but you might consider building an Amazon EC2
AMI of your nodes. This would let others quickly boot up your nodes and run
the stress test against it.
I know you mentioned that you're using Rackspace Cloud. I'm not super
familiar with the internals of RSCloud, but
Another thing: Is the py_stress traffic definitely non-determinstic
such that each client will generate a definitely unique series of
requests? If all clients are deterministically requesting the same
sequence of keys, it would otherwise be plausible that they end up in
effective lock-step, if the
...@rackspace.com wrote:
How many physical client machines are running stress.py?
-Original Message-
From: David Schoonover david.schoono...@gmail.com
Sent: Monday, July 19, 2010 12:11pm
To: user@cassandra.apache.org
Subject: Re: Cassandra benchmarking on Rackspace Cloud
Hello all, I'm
Another thing: Is the py_stress traffic definitely non-determinstic
such that each client will generate a definitely unique series of
requests?
The tests were run both with --random and --std 0.1; in both cases, the
key-sequence is non-deterministic.
Cheers,
Dave
On Jul 19, 2010, at
: Monday, July 19, 2010 10:30am
To: user@cassandra.apache.org user@cassandra.apache.org
Subject: Re: Cassandra benchmarking on Rackspace Cloud
Certainly I'm using multiple cloud servers for the multiple client tests.
Whether or not they are resident on the same physical machine, I just don't
One with 50 threads; it is remote from the cluster but within the same
DC in both cases. I also run the test with multiple clients and saw
similar results when summing the reqs/sec.
Multiple client processes, or multiple client machines?
In particular, note that the way CPython works, if
stress.py uses multiprocessing if it is present, circumventing the GIL; we ran
the tests with python 2.6.5.
David Schoonover
On Jul 19, 2010, at 1:51 PM, Peter Schuller wrote:
One with 50 threads; it is remote from the cluster but within the same
DC in both cases. I also run the test with
Multiple client processes, or multiple client machines?
I ran it with both one and two client machines making requests, and ensured the
sum of the request threads across the clients was 50. That was on the cloud. I
am re-running the multi-host test against the 4-node cluster on dedicated
additional gains: above that point, you should use
more machines, each running K processes.
-Original Message-
From: David Schoonover david.schoono...@gmail.com
Sent: Monday, July 19, 2010 1:02pm
To: user@cassandra.apache.org
Subject: Re: Cassandra benchmarking on Rackspace Cloud
Multiple
I'm reading what this thread and I am a little lost, what should the
expected behavioral be?
Should it maintain 53K regardless of nodes?
nodes reads/sec
1 53,000
2 37,000
4 37,000
I ran this test previously on the cloud, with similar results:
nodes reads/sec
1
On Mon, Jul 19, 2010 at 11:02 AM, David Schoonover
david.schoono...@gmail.com wrote:
Multiple client processes, or multiple client machines?
I ran it with both one and two client machines making requests, and ensured
the sum of the request threads across the clients was 50. That was on the
stress.py uses multiprocessing if it is present, circumventing the GIL; we
ran the tests with python 2.6.5.
Ah, sorry about that. I was mis-remembering because I had to use
threading with pystress because multiprocessing was broken/unavailabie
(can't remember which) on FreeBSD.
I agree with
Sorry, mixed signals in my response. I was partially replying to suggestions
that we were limited by the box's NIC or DC's bandwidth (which is gigabit, no
dice there). I also ran the tests with -t50 on multiple tester machines in the
cloud with no change in performance; I've now rerun those
I'll just add that CPU usage hovered around 50% during these tests.
On Jul 19, 2010, at 3:51 PM, David Schoonover wrote:
Sorry, mixed signals in my response. I was partially replying to suggestions
that we were limited by the box's NIC or DC's bandwidth (which is gigabit, no
dice there). I
I'm about to extend my two node cluster with four dedicated nodes and
removing one of the old nodes, leaving a five node cluster. The
cluster is in production, but I can spare it to do some stress testing
in the meantime as I'm also interested about my cluster performance. I
can't dedicate the
Thanks a ton, Juho.
The command was:
./stress.py -o read -t 50 -d $NODELIST -n 7500 -k -i 2
I made a few minor modifications to stress.py to count errors instead of
logging them, and avoid the pointless try-catch on missing keys. (There are
also unrelated edits to restart long
Did you see about equal CPU usage on the cassandra nodes during the
test? Is it possible that most or all of the keys generated by
stress.py simply fall on a single node?
CPU was approximately equal across the cluster; it was around 50%.
stress.py generates keys randomly or using a gaussian
Now keep adding clients until it stops making the numbers go up...
On Mon, Jul 19, 2010 at 2:51 PM, David Schoonover
david.schoono...@gmail.com wrote:
Sorry, mixed signals in my response. I was partially replying to suggestions
that we were limited by the box's NIC or DC's bandwidth (which is
CPU was approximately equal across the cluster; it was around 50%.
stress.py generates keys randomly or using a gaussian distribution, both
methods showed the same results.
Finally, we're using a random partitioner, so Cassandra will hash the keys
using md5 to map it to a position on the
Thanks for the info. Very helpful in validating what I've been seeing. As for
the scaling limit...
The above was single node testing. I'd expect to be able to add nodes and
scale throughput. Unfortunately, I seem to be running into a cap of 21,000
reads/s regardless of the number of
On Sun, Jul 18, 2010 at 8:45 PM, Oren Benjamin o...@clearspring.com wrote:
Thanks for the info. Very helpful in validating what I've been seeing. As
for the scaling limit...
The above was single node testing. I'd expect to be able to add nodes
and scale throughput. Unfortunately, I seem
On Fri, Jul 16, 2010 at 6:06 PM, Oren Benjamin o...@clearspring.com wrote:
The first goal was to reproduce the test described on spyced here:
http://spyced.blogspot.com/2010/01/cassandra-05.html
Using Cassandra 0.6.3, a 4GB/160GB cloud server
Hi Jonathan,
The 7k reads/s is very high, could you please make more explain about your
benchmark?
7000 reads/s makes average latency of each read operation only talks
0.143ms. Consider 2 disks in the benchmark, it may be 0.286ms.
But in most random read applications on very large dataset, OS
I fact, in my cassandra-0.6.2, I can only get about 40~50 reads/s with
disabled Key/Row cache.
On Sun, Jul 18, 2010 at 1:02 AM, Schubert Zhang zson...@gmail.com wrote:
Hi Jonathan,
The 7k reads/s is very high, could you please make more explain about your
benchmark?
7000 reads/s makes
On Sat, Jul 17, 2010 at 12:02 PM, Schubert Zhang zson...@gmail.com wrote:
Hi Jonathan,
The 7k reads/s is very high, could you please make more explain about your
benchmark?
7000 reads/s makes average latency of each read operation only talks
0.143ms. Consider 2 disks in the benchmark, it
On Sat, Jul 17, 2010 at 1:31 PM, Brandon Williams dri...@gmail.com wrote:
Most of the data was hot in either the row/key cache, or in the OS file
cache. The point was to benchmark Cassandra, not how fast the disk can
seek.
IIRC we were running w/ the defaults of 200k key cache and no row
I think your read throughput is very high, and it may be unauthentic.
For random read, the disk seek will always be the bottleneck (100% utils)
There will be about 3 random disk-seeks for a random read, and aout 10ms for
one seek. So, there will be 30ms for a random read.
If you have only one
35 matches
Mail list logo
