Re: Why so slow?

[Rob Coli]

On Sun, Aug 19, 2012 at 11:09 AM, Peter Morris  wrote:
> Is the Windows community edition crippled for network use perhaps, or could
> the problem be something else?

It's not "crippled" but it underperforms Cassandra on Linux. Cassandra
contains various Linux specific optimizations which result in improved
performance if they can be used. I'm not sure anyone has shiny graphs
comparing the two, but I would expect Windows Cassandra to be
discernably less performant.

That said, this is not the issue in your OP. :)

=Rob

-- 
=Robert Coli
AIM>ALK - rc...@palominodb.com
YAHOO - rcoli.palominob
SKYPE - rcoli_palominodb

Re: Why so slow?

[aaron morton]

I did a talk on server side latency at Cassandra Summit 12 the other week 
http://www.datastax.com/events/cassandrasummit2012/presentations 

If you want to do some baseline tests think about: multiple clients, batch 
calls with maybe 10's of rows, connection pooling. 

There is a stress tool  in source distribution that maybe of help. 

Cheers


-
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com

On 21/08/2012, at 4:16 AM, "Hiller, Dean"  wrote:

> Be careful on bulk as cassandra takes a bit longer to process.  It was faster 
> not doing too many rows at a time multithreaded in our performance testing 
> and if I remember Aaron Morton might have told me that as well.
> 
> Definitely use the cassandra bulk testing tool as well.  I used that and 
> compared it to my tool until I got my tool in par with their tool and you can 
> post the numbers for the cassandra bulk testing tool and I know there was 
> someone on this list who told me the expected writes/ms(it was probably Aaron 
> as well).
> 
> Later,
> Dean
> 
> From: Carlos Carrasco 
> mailto:carlos.carra...@groupalia.com>>
> Reply-To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org>" 
> mailto:user@cassandra.apache.org>>
> Date: Monday, August 20, 2012 10:03 AM
> To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org>" 
> mailto:user@cassandra.apache.org>>
> Subject: Re: Why so slow?
> 
> Are you inserting in bulk? Try to increase the amount of mutations you send 
> in a single batch, otherwise you are just measuring the TCP roundtrip time.
> 
> On 20 August 2012 17:36, Peter Morris 
> mailto:mrpmor...@gmail.com>> wrote:
> My misunderstanding, thanks for correcting me!
> 
> 
> On Mon, Aug 20, 2012 at 4:32 PM, Hiller, Dean 
> mailto:dean.hil...@nrel.gov>> wrote:
> There is latency and throughput.  These are two totally different things even 
> for MySQL.  If you are single threaded, each request (even with MySql) has to 
> be delayed by 1ms or whatever your ping time is.  To fully utilize a 1Gps 
> bandwidth, you NEED to be multithreaded or you are wasting bandwidth…and even 
> then, you probably waste bandwidth as one CPU can't always keep up with 
> keeping the pipe filled.
> 
> 
> 

Re: Why so slow?

[Hiller, Dean]

Be careful on bulk as cassandra takes a bit longer to process.  It was faster 
not doing too many rows at a time multithreaded in our performance testing and 
if I remember Aaron Morton might have told me that as well.

Definitely use the cassandra bulk testing tool as well.  I used that and 
compared it to my tool until I got my tool in par with their tool and you can 
post the numbers for the cassandra bulk testing tool and I know there was 
someone on this list who told me the expected writes/ms(it was probably Aaron 
as well).

Later,
Dean

From: Carlos Carrasco 
mailto:carlos.carra...@groupalia.com>>
Reply-To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org>" 
mailto:user@cassandra.apache.org>>
Date: Monday, August 20, 2012 10:03 AM
To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org>" 
mailto:user@cassandra.apache.org>>
Subject: Re: Why so slow?

Are you inserting in bulk? Try to increase the amount of mutations you send in 
a single batch, otherwise you are just measuring the TCP roundtrip time.

On 20 August 2012 17:36, Peter Morris 
mailto:mrpmor...@gmail.com>> wrote:
My misunderstanding, thanks for correcting me!


On Mon, Aug 20, 2012 at 4:32 PM, Hiller, Dean 
mailto:dean.hil...@nrel.gov>> wrote:
There is latency and throughput.  These are two totally different things even 
for MySQL.  If you are single threaded, each request (even with MySql) has to 
be delayed by 1ms or whatever your ping time is.  To fully utilize a 1Gps 
bandwidth, you NEED to be multithreaded or you are wasting bandwidth…and even 
then, you probably waste bandwidth as one CPU can't always keep up with keeping 
the pipe filled.

Re: Why so slow?

[Carlos Carrasco]

Are you inserting in bulk? Try to increase the amount of mutations you send
in a single batch, otherwise you are just measuring the TCP roundtrip time.

On 20 August 2012 17:36, Peter Morris  wrote:

> My misunderstanding, thanks for correcting me!
>
>
> On Mon, Aug 20, 2012 at 4:32 PM, Hiller, Dean wrote:
>
>> There is latency and throughput.  These are two totally different things
>> even for MySQL.  If you are single threaded, each request (even with MySql)
>> has to be delayed by 1ms or whatever your ping time is.  To fully utilize a
>> 1Gps bandwidth, you NEED to be multithreaded or you are wasting
>> bandwidth…and even then, you probably waste bandwidth as one CPU can't
>> always keep up with keeping the pipe filled.
>>
>>

Re: Why so slow?

[Peter Morris]

My misunderstanding, thanks for correcting me!


On Mon, Aug 20, 2012 at 4:32 PM, Hiller, Dean  wrote:

> There is latency and throughput.  These are two totally different things
> even for MySQL.  If you are single threaded, each request (even with MySql)
> has to be delayed by 1ms or whatever your ping time is.  To fully utilize a
> 1Gps bandwidth, you NEED to be multithreaded or you are wasting
> bandwidth…and even then, you probably waste bandwidth as one CPU can't
> always keep up with keeping the pipe filled.
>
>

Re: Why so slow?

[Hiller, Dean]

There is latency and throughput.  These are two totally different things even 
for MySQL.  If you are single threaded, each request (even with MySql) has to 
be delayed by 1ms or whatever your ping time is.  To fully utilize a 1Gps 
bandwidth, you NEED to be multithreaded or you are wasting bandwidth…and even 
then, you probably waste bandwidth as one CPU can't always keep up with keeping 
the pipe filled.

Later,
Dean

From: Peter Morris mailto:mrpmor...@gmail.com>>
Reply-To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org>" 
mailto:user@cassandra.apache.org>>
Date: Monday, August 20, 2012 9:29 AM
To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org>" 
mailto:user@cassandra.apache.org>>
Subject: Re: Why so slow?

I'm assessing how quickly on average I can deal with a single request.  I 
cannot believe that connecting through a 1Gbps network cable is 14 times 
slower.  I think I get a higher insert rate for SQL Server.



On Mon, Aug 20, 2012 at 1:20 PM, Hiller, Dean 
mailto:dean.hil...@nrel.gov>> wrote:
 IF one has 1ms delay per request and the other has .001, 1000 requests will be 
a one second delay tacked on(which is huge).  This is why he suggested 
multi-threaded ;).  Maybe there is some other factors as well.

Dean

From: Peter Morris 
mailto:mrpmor...@gmail.com><mailto:mrpmor...@gmail.com<mailto:mrpmor...@gmail.com>>>
Reply-To: 
"user@cassandra.apache.org<mailto:user@cassandra.apache.org><mailto:user@cassandra.apache.org<mailto:user@cassandra.apache.org>>"
 
mailto:user@cassandra.apache.org><mailto:user@cassandra.apache.org<mailto:user@cassandra.apache.org>>>
Date: Monday, August 20, 2012 4:49 AM
To: 
"user@cassandra.apache.org<mailto:user@cassandra.apache.org><mailto:user@cassandra.apache.org<mailto:user@cassandra.apache.org>>"
 
mailto:user@cassandra.apache.org><mailto:user@cassandra.apache.org<mailto:user@cassandra.apache.org>>>
Subject: Re: Why so slow?

I've set NoDelay = true on the socket, and although it is much better it is 
still only giving me 500 record inserts per second over a 1Gbps crossover cable 
- (I now also get 200 record inserts per second over wireless.)

I would expect the cross over to have much better performance than this.  Any 
other ideas?

Re: Why so slow?

[Peter Morris]

I'm assessing how quickly on average I can deal with a single request.  I
cannot believe that connecting through a 1Gbps network cable is 14 times
slower.  I think I get a higher insert rate for SQL Server.



On Mon, Aug 20, 2012 at 1:20 PM, Hiller, Dean  wrote:

>  IF one has 1ms delay per request and the other has .001, 1000 requests
> will be a one second delay tacked on(which is huge).  This is why he
> suggested multi-threaded ;).  Maybe there is some other factors as well.
>
> Dean
>
> From: Peter Morris mailto:mrpmor...@gmail.com>>
> Reply-To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org>" <
> user@cassandra.apache.org<mailto:user@cassandra.apache.org>>
> Date: Monday, August 20, 2012 4:49 AM
> To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org>" <
> user@cassandra.apache.org<mailto:user@cassandra.apache.org>>
> Subject: Re: Why so slow?
>
> I've set NoDelay = true on the socket, and although it is much better it
> is still only giving me 500 record inserts per second over a 1Gbps
> crossover cable - (I now also get 200 record inserts per second over
> wireless.)
>
> I would expect the cross over to have much better performance than this.
>  Any other ideas?
>
>
>

Re: Why so slow?

[Hiller, Dean]

IF one has 1ms delay per request and the other has .001, 1000 requests will be 
a one second delay tacked on(which is huge).  This is why he suggested 
multi-threaded ;).  Maybe there is some other factors as well.

Dean

From: Peter Morris mailto:mrpmor...@gmail.com>>
Reply-To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org>" 
mailto:user@cassandra.apache.org>>
Date: Monday, August 20, 2012 4:49 AM
To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org>" 
mailto:user@cassandra.apache.org>>
Subject: Re: Why so slow?

I've set NoDelay = true on the socket, and although it is much better it is 
still only giving me 500 record inserts per second over a 1Gbps crossover cable 
- (I now also get 200 record inserts per second over wireless.)

I would expect the cross over to have much better performance than this.  Any 
other ideas?

Re: Why so slow?

[Peter Morris]

I've set NoDelay = true on the socket, and although it is much better it is
still only giving me 500 record inserts per second over a 1Gbps crossover
cable - (I now also get 200 record inserts per second over wireless.)

I would expect the cross over to have much better performance than this.
 Any other ideas?
<>

Re: Why so slow?

[Peter Morris]

Thanks, I shall get onto the developer of the library :)


On Sun, Aug 19, 2012 at 10:13 PM, Peter Schuller <
peter.schul...@infidyne.com> wrote:

> You're almost certainly using a client that doesn't set TCP_NODELAY on
> the thrift TCP socket. The nagle algorithm is enabled, leading to 200
> ms latency for each, and thus 5 requests/second.
>
> http://en.wikipedia.org/wiki/Nagle's_algorithm
>
>
> --
> / Peter Schuller (@scode, http://worldmodscode.wordpress.com)
>

Re: Why so slow?

[Peter Schuller]

You're almost certainly using a client that doesn't set TCP_NODELAY on
the thrift TCP socket. The nagle algorithm is enabled, leading to 200
ms latency for each, and thus 5 requests/second.

http://en.wikipedia.org/wiki/Nagle's_algorithm


-- 
/ Peter Schuller (@scode, http://worldmodscode.wordpress.com)

Re: Why so slow?

[Peter Morris]

No I am using a single thread. My aim at this point is to see how quickly I
can connect, post, then complete.  I just find it bizarre that it goes from
7000 per second down to 5!


On Sun, Aug 19, 2012 at 8:22 PM, Dave Brosius wrote:

>  Are you using multiple client threads?
>
> You might want to try the stress tool in the distribution.
>
>
>
>
> On 08/19/2012 02:09 PM, Peter Morris wrote:
>
> Hi all
>
>  I have a Windows 7 machine (64 bit) with DataStax community server
> installed.  Running a benchmark app on the server gives me 7000 inserts per
> second.  Running the same app on a networked client gives me only 5 inserts
> per second.  The two computers are connected directly via a cross over
> cable, and the network properties tell me that it is a 1Gbps connection.
>
>  Is the Windows community edition crippled for network use perhaps, or
> could the problem be something else?
>
>  Pete
>
>  Pinging 10.0.0.2 with 32 bytes of data:
> Reply from 10.0.0.2: bytes=32 time=1ms TTL=128
> Reply from 10.0.0.2: bytes=32 time<1ms TTL=128
> Reply from 10.0.0.2: bytes=32 time<1ms TTL=128
> Reply from 10.0.0.2: bytes=32 time<1ms TTL=128
>
>
>

Re: Why so slow?

[Dave Brosius]

Are you using multiple client threads?

You might want to try the stress tool in the distribution.



On 08/19/2012 02:09 PM, Peter Morris wrote:

Hi all

I have a Windows 7 machine (64 bit) with DataStax community server 
installed.  Running a benchmark app on the server gives me 7000 
inserts per second.  Running the same app on a networked client gives 
me only 5 inserts per second.  The two computers are connected 
directly via a cross over cable, and the network properties tell me 
that it is a 1Gbps connection.


Is the Windows community edition crippled for network use perhaps, or 
could the problem be something else?


Pete

Pinging 10.0.0.2 with 32 bytes of data:
Reply from 10.0.0.2 : bytes=32 time=1ms TTL=128
Reply from 10.0.0.2 : bytes=32 time<1ms TTL=128
Reply from 10.0.0.2 : bytes=32 time<1ms TTL=128
Reply from 10.0.0.2 : bytes=32 time<1ms TTL=128

Why so slow?

[Peter Morris]

Hi all

I have a Windows 7 machine (64 bit) with DataStax community server
installed.  Running a benchmark app on the server gives me 7000 inserts per
second.  Running the same app on a networked client gives me only 5 inserts
per second.  The two computers are connected directly via a cross over
cable, and the network properties tell me that it is a 1Gbps connection.

Is the Windows community edition crippled for network use perhaps, or could
the problem be something else?

Pete

Pinging 10.0.0.2 with 32 bytes of data:
Reply from 10.0.0.2: bytes=32 time=1ms TTL=128
Reply from 10.0.0.2: bytes=32 time<1ms TTL=128
Reply from 10.0.0.2: bytes=32 time<1ms TTL=128
Reply from 10.0.0.2: bytes=32 time<1ms TTL=128