Re: Latency overhead on Cassandra cluster deployed on multiple AZs (AWS)

[Kevin O'Connor]

Are you in VPC or EC2 Classic? Are you using enhanced networking?

On Tue, Apr 12, 2016 at 9:52 AM, Alessandro Pieri  wrote:

> Hi Jack,
>
> As mentioned before I've used m3.xlarge instance types together with two
> ephemeral disks in raid 0 and, according to Amazon, they have "high"
> network performance.
>
> I ran many tests starting with a brand-new cluster every time and I got
> consistent results.
>
> I believe there's something that I cannot explain yet with the client used
> by cassandra-stress to connect to the nodes, I'd like to understand why
> there is such a big difference:
>
> Multi-AZ, CL=ONE, "--nodes node1,node2,node3,node4,node5,node6" -> 95th
> percentile: 38.14ms
> Multi-AZ, CL=ONE, "--nodes node1" -> 95th percentile: 5.9ms
>
> Hope you can help to figure it out.
>
> Cheers,
> Alessandro
>
>
>
>
> On Tue, Apr 12, 2016 at 5:43 PM, Jack Krupansky 
> wrote:
>
>> Which instance type are you using? Some may be throttled for EBS access,
>> so you could bump into a rate limit, and who knows what AWS will do at that
>> point.
>>
>> -- Jack Krupansky
>>
>> On Tue, Apr 12, 2016 at 6:02 AM, Alessandro Pieri <
>> alessan...@getstream.io> wrote:
>>
>>> Thanks Chris for your reply.
>>>
>>> I ran the tests 3 times for 20 minutes/each and I monitored the network
>>> latency in the meanwhile, it was very low (even the 99th percentile).
>>>
>>> I didn't notice any cpu spike caused by the GC but, as you pointed out,
>>> I will look into the GC log, just to be sure.
>>>
>>> In order to avoid the problem you mentioned with EBS and to keep the
>>> deviation under control I used two ephemeral disks in raid 0.
>>>
>>> I think the odd results come from the way cassandra-stress deals with
>>> multiple nodes. As soon as possible I will go through the Java code to get
>>> some more detail.
>>>
>>> If you have something else in your mind please let me know, your
>>> comments were really appreciated.
>>>
>>> Cheers,
>>> Alessandro
>>>
>>>
>>> On Mon, Apr 11, 2016 at 4:15 PM, Chris Lohfink 
>>> wrote:
>>>
 Where do you get the ~1ms latency between AZs? Comparing a short term
 average to a 99th percentile isn't very fair.

 "Over the last month, the median is 2.09 ms, 90th percentile is
 20ms, 99th percentile is 47ms." - per
 https://www.quora.com/What-are-typical-ping-times-between-different-EC2-availability-zones-within-the-same-region

 Are you using EBS? That would further impact latency on reads and GCs
 will always cause hiccups in the 99th+.

 Chris


 On Mon, Apr 11, 2016 at 7:57 AM, Alessandro Pieri 
 wrote:

> Hi everyone,
>
> Last week I ran some tests to estimate the latency overhead introduces
> in a Cassandra cluster by a multi availability zones setup on AWS EC2.
>
> I started a Cassandra cluster of 6 nodes deployed on 3 different AZs
> (2 nodes/AZ).
>
> Then, I used cassandra-stress to create an INSERT (write) test of 20M
> entries with a replication factor = 3, right after, I ran cassandra-stress
> again to READ 10M entries.
>
> Well, I got the following unexpected result:
>
> Single-AZ, CL=ONE -> median/95th percentile/99th percentile:
> 1.06ms/7.41ms/55.81ms
> Multi-AZ, CL=ONE -> median/95th percentile/99th percentile:
> 1.16ms/38.14ms/47.75ms
>
> Basically, switching to the multi-AZ setup the latency increased of
> ~30ms. That's too much considering the the average network latency between
> AZs on AWS is ~1ms.
>
> Since I couldn't find anything to explain those results, I decided to
> run the cassandra-stress specifying only a single node entry (i.e. 
> "--nodes
> node1" instead of "--nodes node1,node2,node3,node4,node5,node6") and
> surprisingly the latency went back to 5.9 ms.
>
> Trying to recap:
>
> Multi-AZ, CL=ONE, "--nodes node1,node2,node3,node4,node5,node6" ->
> 95th percentile: 38.14ms
> Multi-AZ, CL=ONE, "--nodes node1" -> 95th percentile: 5.9ms
>
> For the sake of completeness I've ran a further test using a
> consistency level = LOCAL_QUORUM and the test did not show any large
> variance with using a single node or multiple ones.
>
> Do you guys know what could be the reason?
>
> The test were executed on a m3.xlarge (network optimized) using the
> DataStax AMI 2.6.3 running Cassandra v2.0.15.
>
> Thank you in advance for your help.
>
> Cheers,
> Alessandro
>


>>>
>>>
>>> --
>>> *Alessandro Pieri*
>>> *Software Architect @ Stream.io Inc*
>>> e-Mail: alessan...@getstream.io - twitter: sirio7g
>>> 
>>>
>>>
>>
>

Re: Latency overhead on Cassandra cluster deployed on multiple AZs (AWS)

[Alessandro Pieri]

Hi Jack,

As mentioned before I've used m3.xlarge instance types together with two
ephemeral disks in raid 0 and, according to Amazon, they have "high"
network performance.

I ran many tests starting with a brand-new cluster every time and I got
consistent results.

I believe there's something that I cannot explain yet with the client used
by cassandra-stress to connect to the nodes, I'd like to understand why
there is such a big difference:

Multi-AZ, CL=ONE, "--nodes node1,node2,node3,node4,node5,node6" -> 95th
percentile: 38.14ms
Multi-AZ, CL=ONE, "--nodes node1" -> 95th percentile: 5.9ms

Hope you can help to figure it out.

Cheers,
Alessandro




On Tue, Apr 12, 2016 at 5:43 PM, Jack Krupansky 
wrote:

> Which instance type are you using? Some may be throttled for EBS access,
> so you could bump into a rate limit, and who knows what AWS will do at that
> point.
>
> -- Jack Krupansky
>
> On Tue, Apr 12, 2016 at 6:02 AM, Alessandro Pieri  > wrote:
>
>> Thanks Chris for your reply.
>>
>> I ran the tests 3 times for 20 minutes/each and I monitored the network
>> latency in the meanwhile, it was very low (even the 99th percentile).
>>
>> I didn't notice any cpu spike caused by the GC but, as you pointed out, I
>> will look into the GC log, just to be sure.
>>
>> In order to avoid the problem you mentioned with EBS and to keep the
>> deviation under control I used two ephemeral disks in raid 0.
>>
>> I think the odd results come from the way cassandra-stress deals with
>> multiple nodes. As soon as possible I will go through the Java code to get
>> some more detail.
>>
>> If you have something else in your mind please let me know, your comments
>> were really appreciated.
>>
>> Cheers,
>> Alessandro
>>
>>
>> On Mon, Apr 11, 2016 at 4:15 PM, Chris Lohfink 
>> wrote:
>>
>>> Where do you get the ~1ms latency between AZs? Comparing a short term
>>> average to a 99th percentile isn't very fair.
>>>
>>> "Over the last month, the median is 2.09 ms, 90th percentile is
>>> 20ms, 99th percentile is 47ms." - per
>>> https://www.quora.com/What-are-typical-ping-times-between-different-EC2-availability-zones-within-the-same-region
>>>
>>> Are you using EBS? That would further impact latency on reads and GCs
>>> will always cause hiccups in the 99th+.
>>>
>>> Chris
>>>
>>>
>>> On Mon, Apr 11, 2016 at 7:57 AM, Alessandro Pieri 
>>> wrote:
>>>
 Hi everyone,

 Last week I ran some tests to estimate the latency overhead introduces
 in a Cassandra cluster by a multi availability zones setup on AWS EC2.

 I started a Cassandra cluster of 6 nodes deployed on 3 different AZs (2
 nodes/AZ).

 Then, I used cassandra-stress to create an INSERT (write) test of 20M
 entries with a replication factor = 3, right after, I ran cassandra-stress
 again to READ 10M entries.

 Well, I got the following unexpected result:

 Single-AZ, CL=ONE -> median/95th percentile/99th percentile:
 1.06ms/7.41ms/55.81ms
 Multi-AZ, CL=ONE -> median/95th percentile/99th percentile:
 1.16ms/38.14ms/47.75ms

 Basically, switching to the multi-AZ setup the latency increased of
 ~30ms. That's too much considering the the average network latency between
 AZs on AWS is ~1ms.

 Since I couldn't find anything to explain those results, I decided to
 run the cassandra-stress specifying only a single node entry (i.e. "--nodes
 node1" instead of "--nodes node1,node2,node3,node4,node5,node6") and
 surprisingly the latency went back to 5.9 ms.

 Trying to recap:

 Multi-AZ, CL=ONE, "--nodes node1,node2,node3,node4,node5,node6" -> 95th
 percentile: 38.14ms
 Multi-AZ, CL=ONE, "--nodes node1" -> 95th percentile: 5.9ms

 For the sake of completeness I've ran a further test using a
 consistency level = LOCAL_QUORUM and the test did not show any large
 variance with using a single node or multiple ones.

 Do you guys know what could be the reason?

 The test were executed on a m3.xlarge (network optimized) using the
 DataStax AMI 2.6.3 running Cassandra v2.0.15.

 Thank you in advance for your help.

 Cheers,
 Alessandro

>>>
>>>
>>
>>
>> --
>> *Alessandro Pieri*
>> *Software Architect @ Stream.io Inc*
>> e-Mail: alessan...@getstream.io - twitter: sirio7g
>> 
>>
>>
>

Re: Latency overhead on Cassandra cluster deployed on multiple AZs (AWS)

[Jack Krupansky]

Which instance type are you using? Some may be throttled for EBS access, so
you could bump into a rate limit, and who knows what AWS will do at that
point.

-- Jack Krupansky

On Tue, Apr 12, 2016 at 6:02 AM, Alessandro Pieri 
wrote:

> Thanks Chris for your reply.
>
> I ran the tests 3 times for 20 minutes/each and I monitored the network
> latency in the meanwhile, it was very low (even the 99th percentile).
>
> I didn't notice any cpu spike caused by the GC but, as you pointed out, I
> will look into the GC log, just to be sure.
>
> In order to avoid the problem you mentioned with EBS and to keep the
> deviation under control I used two ephemeral disks in raid 0.
>
> I think the odd results come from the way cassandra-stress deals with
> multiple nodes. As soon as possible I will go through the Java code to get
> some more detail.
>
> If you have something else in your mind please let me know, your comments
> were really appreciated.
>
> Cheers,
> Alessandro
>
>
> On Mon, Apr 11, 2016 at 4:15 PM, Chris Lohfink 
> wrote:
>
>> Where do you get the ~1ms latency between AZs? Comparing a short term
>> average to a 99th percentile isn't very fair.
>>
>> "Over the last month, the median is 2.09 ms, 90th percentile is
>> 20ms, 99th percentile is 47ms." - per
>> https://www.quora.com/What-are-typical-ping-times-between-different-EC2-availability-zones-within-the-same-region
>>
>> Are you using EBS? That would further impact latency on reads and GCs
>> will always cause hiccups in the 99th+.
>>
>> Chris
>>
>>
>> On Mon, Apr 11, 2016 at 7:57 AM, Alessandro Pieri 
>> wrote:
>>
>>> Hi everyone,
>>>
>>> Last week I ran some tests to estimate the latency overhead introduces
>>> in a Cassandra cluster by a multi availability zones setup on AWS EC2.
>>>
>>> I started a Cassandra cluster of 6 nodes deployed on 3 different AZs (2
>>> nodes/AZ).
>>>
>>> Then, I used cassandra-stress to create an INSERT (write) test of 20M
>>> entries with a replication factor = 3, right after, I ran cassandra-stress
>>> again to READ 10M entries.
>>>
>>> Well, I got the following unexpected result:
>>>
>>> Single-AZ, CL=ONE -> median/95th percentile/99th percentile:
>>> 1.06ms/7.41ms/55.81ms
>>> Multi-AZ, CL=ONE -> median/95th percentile/99th percentile:
>>> 1.16ms/38.14ms/47.75ms
>>>
>>> Basically, switching to the multi-AZ setup the latency increased of
>>> ~30ms. That's too much considering the the average network latency between
>>> AZs on AWS is ~1ms.
>>>
>>> Since I couldn't find anything to explain those results, I decided to
>>> run the cassandra-stress specifying only a single node entry (i.e. "--nodes
>>> node1" instead of "--nodes node1,node2,node3,node4,node5,node6") and
>>> surprisingly the latency went back to 5.9 ms.
>>>
>>> Trying to recap:
>>>
>>> Multi-AZ, CL=ONE, "--nodes node1,node2,node3,node4,node5,node6" -> 95th
>>> percentile: 38.14ms
>>> Multi-AZ, CL=ONE, "--nodes node1" -> 95th percentile: 5.9ms
>>>
>>> For the sake of completeness I've ran a further test using a consistency
>>> level = LOCAL_QUORUM and the test did not show any large variance with
>>> using a single node or multiple ones.
>>>
>>> Do you guys know what could be the reason?
>>>
>>> The test were executed on a m3.xlarge (network optimized) using the
>>> DataStax AMI 2.6.3 running Cassandra v2.0.15.
>>>
>>> Thank you in advance for your help.
>>>
>>> Cheers,
>>> Alessandro
>>>
>>
>>
>
>
> --
> *Alessandro Pieri*
> *Software Architect @ Stream.io Inc*
> e-Mail: alessan...@getstream.io - twitter: sirio7g
> 
>
>

Re: Latency overhead on Cassandra cluster deployed on multiple AZs (AWS)

[Alessandro Pieri]

Thanks Chris for your reply.

I ran the tests 3 times for 20 minutes/each and I monitored the network
latency in the meanwhile, it was very low (even the 99th percentile).

I didn't notice any cpu spike caused by the GC but, as you pointed out, I
will look into the GC log, just to be sure.

In order to avoid the problem you mentioned with EBS and to keep the
deviation under control I used two ephemeral disks in raid 0.

I think the odd results come from the way cassandra-stress deals with
multiple nodes. As soon as possible I will go through the Java code to get
some more detail.

If you have something else in your mind please let me know, your comments
were really appreciated.

Cheers,
Alessandro


On Mon, Apr 11, 2016 at 4:15 PM, Chris Lohfink  wrote:

> Where do you get the ~1ms latency between AZs? Comparing a short term
> average to a 99th percentile isn't very fair.
>
> "Over the last month, the median is 2.09 ms, 90th percentile is
> 20ms, 99th percentile is 47ms." - per
> https://www.quora.com/What-are-typical-ping-times-between-different-EC2-availability-zones-within-the-same-region
>
> Are you using EBS? That would further impact latency on reads and GCs will
> always cause hiccups in the 99th+.
>
> Chris
>
>
> On Mon, Apr 11, 2016 at 7:57 AM, Alessandro Pieri 
> wrote:
>
>> Hi everyone,
>>
>> Last week I ran some tests to estimate the latency overhead introduces in
>> a Cassandra cluster by a multi availability zones setup on AWS EC2.
>>
>> I started a Cassandra cluster of 6 nodes deployed on 3 different AZs (2
>> nodes/AZ).
>>
>> Then, I used cassandra-stress to create an INSERT (write) test of 20M
>> entries with a replication factor = 3, right after, I ran cassandra-stress
>> again to READ 10M entries.
>>
>> Well, I got the following unexpected result:
>>
>> Single-AZ, CL=ONE -> median/95th percentile/99th percentile:
>> 1.06ms/7.41ms/55.81ms
>> Multi-AZ, CL=ONE -> median/95th percentile/99th percentile:
>> 1.16ms/38.14ms/47.75ms
>>
>> Basically, switching to the multi-AZ setup the latency increased of
>> ~30ms. That's too much considering the the average network latency between
>> AZs on AWS is ~1ms.
>>
>> Since I couldn't find anything to explain those results, I decided to run
>> the cassandra-stress specifying only a single node entry (i.e. "--nodes
>> node1" instead of "--nodes node1,node2,node3,node4,node5,node6") and
>> surprisingly the latency went back to 5.9 ms.
>>
>> Trying to recap:
>>
>> Multi-AZ, CL=ONE, "--nodes node1,node2,node3,node4,node5,node6" -> 95th
>> percentile: 38.14ms
>> Multi-AZ, CL=ONE, "--nodes node1" -> 95th percentile: 5.9ms
>>
>> For the sake of completeness I've ran a further test using a consistency
>> level = LOCAL_QUORUM and the test did not show any large variance with
>> using a single node or multiple ones.
>>
>> Do you guys know what could be the reason?
>>
>> The test were executed on a m3.xlarge (network optimized) using the
>> DataStax AMI 2.6.3 running Cassandra v2.0.15.
>>
>> Thank you in advance for your help.
>>
>> Cheers,
>> Alessandro
>>
>
>


-- 
*Alessandro Pieri*
*Software Architect @ Stream.io Inc*
e-Mail: alessan...@getstream.io - twitter: sirio7g

Re: Latency overhead on Cassandra cluster deployed on multiple AZs (AWS)

[Chris Lohfink]

Where do you get the ~1ms latency between AZs? Comparing a short term
average to a 99th percentile isn't very fair.

"Over the last month, the median is 2.09 ms, 90th percentile is 20ms,
99th percentile
is 47ms." - per
https://www.quora.com/What-are-typical-ping-times-between-different-EC2-availability-zones-within-the-same-region

Are you using EBS? That would further impact latency on reads and GCs will
always cause hiccups in the 99th+.

Chris


On Mon, Apr 11, 2016 at 7:57 AM, Alessandro Pieri  wrote:

> Hi everyone,
>
> Last week I ran some tests to estimate the latency overhead introduces in
> a Cassandra cluster by a multi availability zones setup on AWS EC2.
>
> I started a Cassandra cluster of 6 nodes deployed on 3 different AZs (2
> nodes/AZ).
>
> Then, I used cassandra-stress to create an INSERT (write) test of 20M
> entries with a replication factor = 3, right after, I ran cassandra-stress
> again to READ 10M entries.
>
> Well, I got the following unexpected result:
>
> Single-AZ, CL=ONE -> median/95th percentile/99th percentile:
> 1.06ms/7.41ms/55.81ms
> Multi-AZ, CL=ONE -> median/95th percentile/99th percentile:
> 1.16ms/38.14ms/47.75ms
>
> Basically, switching to the multi-AZ setup the latency increased of ~30ms.
> That's too much considering the the average network latency between AZs on
> AWS is ~1ms.
>
> Since I couldn't find anything to explain those results, I decided to run
> the cassandra-stress specifying only a single node entry (i.e. "--nodes
> node1" instead of "--nodes node1,node2,node3,node4,node5,node6") and
> surprisingly the latency went back to 5.9 ms.
>
> Trying to recap:
>
> Multi-AZ, CL=ONE, "--nodes node1,node2,node3,node4,node5,node6" -> 95th
> percentile: 38.14ms
> Multi-AZ, CL=ONE, "--nodes node1" -> 95th percentile: 5.9ms
>
> For the sake of completeness I've ran a further test using a consistency
> level = LOCAL_QUORUM and the test did not show any large variance with
> using a single node or multiple ones.
>
> Do you guys know what could be the reason?
>
> The test were executed on a m3.xlarge (network optimized) using the
> DataStax AMI 2.6.3 running Cassandra v2.0.15.
>
> Thank you in advance for your help.
>
> Cheers,
> Alessandro
>

Latency overhead on Cassandra cluster deployed on multiple AZs (AWS)

[Alessandro Pieri]

Hi everyone,

Last week I ran some tests to estimate the latency overhead introduces in a
Cassandra cluster by a multi availability zones setup on AWS EC2.

I started a Cassandra cluster of 6 nodes deployed on 3 different AZs (2
nodes/AZ).

Then, I used cassandra-stress to create an INSERT (write) test of 20M
entries with a replication factor = 3, right after, I ran cassandra-stress
again to READ 10M entries.

Well, I got the following unexpected result:

Single-AZ, CL=ONE -> median/95th percentile/99th percentile:
1.06ms/7.41ms/55.81ms
Multi-AZ, CL=ONE -> median/95th percentile/99th percentile:
1.16ms/38.14ms/47.75ms

Basically, switching to the multi-AZ setup the latency increased of ~30ms.
That's too much considering the the average network latency between AZs on
AWS is ~1ms.

Since I couldn't find anything to explain those results, I decided to run
the cassandra-stress specifying only a single node entry (i.e. "--nodes
node1" instead of "--nodes node1,node2,node3,node4,node5,node6") and
surprisingly the latency went back to 5.9 ms.

Trying to recap:

Multi-AZ, CL=ONE, "--nodes node1,node2,node3,node4,node5,node6" -> 95th
percentile: 38.14ms
Multi-AZ, CL=ONE, "--nodes node1" -> 95th percentile: 5.9ms

For the sake of completeness I've ran a further test using a consistency
level = LOCAL_QUORUM and the test did not show any large variance with
using a single node or multiple ones.

Do you guys know what could be the reason?

The test were executed on a m3.xlarge (network optimized) using the
DataStax AMI 2.6.3 running Cassandra v2.0.15.

Thank you in advance for your help.

Cheers,
Alessandro