My bad...I guess that makes it even more important to find out if this
is the problem on Windows? :)
-Jon
On Oct 6, 2009, at 6:26 PM, Jonathan Ellis wrote:
Just to clarify, Cassandra only uses UDP for gossip -- passing data
between nodes is done over TCP.
-Jonathan
On Tue, Oct 6, 2009 at 5:31 PM, Jonathan Mischo
<jmis...@quagility.com> wrote:
Back in the day, I was involved in performance testing JVMs,
Solaris on
Intel, and SWS at SunSoft, and our lab actually ran not just our own
numbers, but competitors' numbers as well, for comparison. One
thing we
discovered, when analyzing some performance issues we'd seen with
Windows
clients, was that Windows' network stacks don't behave as you'd
expect.
They very regularly transmitted packets out of sequence.
I'm not sure if this is still the case, as this was the late 90's,
but we
discovered this when we were working on SPECweb numbers (we were a
SPEC lab)
for SWS and, in looking at the JVM and other system settings to
understand
why we were seeing unexpectedly large TCP buffers. When we started
sniffing
packets directly from the Windows clients, we discovered the
packets were
being emitted out of sequence, which was causing the server to
require
larger per-connection buffers and was pushing TCP window size
boundaries.
Now, Cassandra uses UDP primarily, but we never tested to find out
whether
this was a TCP stack issue, IP stack issue, or Ethernet stack
issue, so it
may be a similar case. Since reassembly still has to happen, if
the packets
are being transmitted out of order, it makes sense that your write
latency
would be significantly higher.
I'd be interested to see the results if you dig deeper into this.
-Jon
On Oct 6, 2009, at 11:14 AM, Igor Katkov wrote:
I think I finally found what. It's implementation of Java NIOon
Windows (JVM
1.6.0.16, 64b on Windows 2003)
The very same code, same network but CentOS linux gives almost 4x
performance. (in cassan...@linux -> cassan...@windows setup)
I don't have another linux box to test (cassan...@linux ->
cassan...@linux)
performance, but expect it to be even better.
A lesson learnt: don't use windows.
P.S.
Here at Viigo we also learnt the hard way that async IO is also
broken in
.NET (C#). Now I start to wonder if there is some fundamental flaw
in async
IO on windows...
On Mon, Oct 5, 2009 at 3:23 PM, Jonathan Ellis <jbel...@gmail.com>
wrote:
On Mon, Oct 5, 2009 at 2:17 PM, Igor Katkov <ikat...@gmail.com>
wrote:
measured via JMX console i.e. does not include client-cassandra-
client
latency
20 client threads 176975b value StorageProxy.WriteLatency ~660ms
10 client threads 176975b value StorageProxy.WriteLatency ~350ms
05 client threads 176975b value StorageProxy.WriteLatency ~156ms
this is going up basically linearly with amount of (data x clients),
so clearly something is getting saturated.
