[jira] [Commented] (CASSANDRA-9558) Cassandra-stress regression in 2.2
[ https://issues.apache.org/jira/browse/CASSANDRA-9558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14618242#comment-14618242 ] Olivier Michallat commented on CASSANDRA-9558: -- Turns out that this doesn't explain the performance differences (I made a mistake in my early tests which led me to believe so). We're still profiling the driver, I'll report back when we find something. Cassandra-stress regression in 2.2 -- Key: CASSANDRA-9558 URL: https://issues.apache.org/jira/browse/CASSANDRA-9558 Project: Cassandra Issue Type: Bug Reporter: Alan Boudreault Assignee: Andy Tolbert Fix For: 2.2.0 rc2 Attachments: 2.1.log, 2.2.log, CASSANDRA-9558-2.patch, CASSANDRA-9558-ProtocolV2.patch, atolber-CASSANDRA-9558-stress.tgz, atolber-trunk-driver-coalescing-disabled.txt, stress-2.1-java-driver-2.0.9.2.log, stress-2.1-java-driver-2.2+PATCH.log, stress-2.1-java-driver-2.2.log, stress-2.2-java-driver-2.2+PATCH.log, stress-2.2-java-driver-2.2.log We are seeing some regression in performance when using cassandra-stress 2.2. You can see the difference at this url: http://riptano.github.io/cassandra_performance/graph_v5/graph.html?stats=stress_regression.jsonmetric=op_rateoperation=1_writesmoothing=1show_aggregates=truexmin=0xmax=108.57ymin=0ymax=168147.1 The cassandra version of the cluster doesn't seem to have any impact. //cc [~tjake] [~benedict] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9558) Cassandra-stress regression in 2.2
[ https://issues.apache.org/jira/browse/CASSANDRA-9558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14614813#comment-14614813 ] Benedict commented on CASSANDRA-9558: - bq. The problem does not exist with Cassandra because it's a server, both the producer and the consumer is the event loop. They aren't, but there is a bound on the number of concurrent connections we can be processing requests for on the server, and so the queue size must itself be bounded. I would also suggest imposing a user-configurable bound on the size of your queues in the driver (or the total number of not-yet-sent messages), as there can be a multitude of reasons for the message queues to back up, and that's independently bad for the health of the application process. That wouldn't solve this problem, but it would have likely helped a great deal, and is something to consider as well (especially as we may start blocking receipt of messages to cope with cluster overload, which would translate to a growing application send buffer). Either way, good catch. Looks like the fix should be simple (let's hope it brings throughput right back up). Cassandra-stress regression in 2.2 -- Key: CASSANDRA-9558 URL: https://issues.apache.org/jira/browse/CASSANDRA-9558 Project: Cassandra Issue Type: Bug Reporter: Alan Boudreault Assignee: Andy Tolbert Fix For: 2.2.0 rc2 Attachments: 2.1.log, 2.2.log, CASSANDRA-9558-2.patch, CASSANDRA-9558-ProtocolV2.patch, atolber-CASSANDRA-9558-stress.tgz, atolber-trunk-driver-coalescing-disabled.txt, stress-2.1-java-driver-2.0.9.2.log, stress-2.1-java-driver-2.2+PATCH.log, stress-2.1-java-driver-2.2.log, stress-2.2-java-driver-2.2+PATCH.log, stress-2.2-java-driver-2.2.log We are seeing some regression in performance when using cassandra-stress 2.2. You can see the difference at this url: http://riptano.github.io/cassandra_performance/graph_v5/graph.html?stats=stress_regression.jsonmetric=op_rateoperation=1_writesmoothing=1show_aggregates=truexmin=0xmax=108.57ymin=0ymax=168147.1 The cassandra version of the cluster doesn't seem to have any impact. //cc [~tjake] [~benedict] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9558) Cassandra-stress regression in 2.2
[
https://issues.apache.org/jira/browse/CASSANDRA-9558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14614774#comment-14614774
]
Olivier Michallat commented on CASSANDRA-9558:
--
[~pingtimeout] found the bottleneck in the driver's code. The culprit is this
line in the flusher code:
{code}
while (null != (flush = queued.poll())) {
{code}
In the driver, producers for this queue are application threads flushing their
queries; the consumer is the Netty event loop, which executes the flusher code.
What happens in stress tests is that we have many producers constantly
enqueuing new messages, so the consumer ends up spinning a lot in this loop,
which delays messages. This explains why it works better with more connections:
more connections = more event loops = more queues = less pressure on each queue.
The workaround is to add a limit to the maximum number of messages that can be
flushed in one go. We're experimenting with this right now, it will go into
2.1.7 and 2.0.11.
The problem does not exist with Cassandra because it's a server, both the
producer and the consumer is the event loop.
Cassandra-stress regression in 2.2
--
Key: CASSANDRA-9558
URL: https://issues.apache.org/jira/browse/CASSANDRA-9558
Project: Cassandra
Issue Type: Bug
Reporter: Alan Boudreault
Assignee: Andy Tolbert
Fix For: 2.2.0 rc2
Attachments: 2.1.log, 2.2.log, CASSANDRA-9558-2.patch,
CASSANDRA-9558-ProtocolV2.patch, atolber-CASSANDRA-9558-stress.tgz,
atolber-trunk-driver-coalescing-disabled.txt,
stress-2.1-java-driver-2.0.9.2.log, stress-2.1-java-driver-2.2+PATCH.log,
stress-2.1-java-driver-2.2.log, stress-2.2-java-driver-2.2+PATCH.log,
stress-2.2-java-driver-2.2.log
We are seeing some regression in performance when using cassandra-stress 2.2.
You can see the difference at this url:
http://riptano.github.io/cassandra_performance/graph_v5/graph.html?stats=stress_regression.jsonmetric=op_rateoperation=1_writesmoothing=1show_aggregates=truexmin=0xmax=108.57ymin=0ymax=168147.1
The cassandra version of the cluster doesn't seem to have any impact.
//cc [~tjake] [~benedict]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9558) Cassandra-stress regression in 2.2
[ https://issues.apache.org/jira/browse/CASSANDRA-9558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14602554#comment-14602554 ] Benedict commented on CASSANDRA-9558: - Can we answer my question before forging ahead and changing any default pooling settings? Like I say, it's not at all necessarily a *bug*. It is quite likely that this configuration improves throughput for many normal cluster configurations, and has negative implications only for very small clusters. We want the fewest connections we can get away with; perhaps, the client should automatically scale the connections based on throughput or cluster size. We haven't undertaken sufficient investigation to say with certainty, but it seems that what we are doing here is increasing the CPU _overhead_ per operation in order to _saturate_ the processing capacity of each box. However when there are more machines, or more simulated clients, this increased overhead is highly likely to reduce throughput due to the increased overhead. What we should probably do on our end is implement CASSANDRA-8466, since this is how a majority of users really use their clusters: many clients, not one client with many connections in the Java driver. Cassandra-stress regression in 2.2 -- Key: CASSANDRA-9558 URL: https://issues.apache.org/jira/browse/CASSANDRA-9558 Project: Cassandra Issue Type: Bug Reporter: Alan Boudreault Assignee: Andy Tolbert Fix For: 2.2.0 rc2 Attachments: 2.1.log, 2.2.log, CASSANDRA-9558-2.patch, CASSANDRA-9558-ProtocolV2.patch, atolber-CASSANDRA-9558-stress.tgz, atolber-trunk-driver-coalescing-disabled.txt, stress-2.1-java-driver-2.0.9.2.log, stress-2.1-java-driver-2.2+PATCH.log, stress-2.1-java-driver-2.2.log, stress-2.2-java-driver-2.2+PATCH.log, stress-2.2-java-driver-2.2.log We are seeing some regression in performance when using cassandra-stress 2.2. You can see the difference at this url: http://riptano.github.io/cassandra_performance/graph_v5/graph.html?stats=stress_regression.jsonmetric=op_rateoperation=1_writesmoothing=1show_aggregates=truexmin=0xmax=108.57ymin=0ymax=168147.1 The cassandra version of the cluster doesn't seem to have any impact. //cc [~tjake] [~benedict] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9558) Cassandra-stress regression in 2.2
[ https://issues.apache.org/jira/browse/CASSANDRA-9558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14600116#comment-14600116 ] T Jake Luciani commented on CASSANDRA-9558: --- So can we go with a patch that changes the defaults for stress to use v2 and 8 connections in the pool? Cassandra-stress regression in 2.2 -- Key: CASSANDRA-9558 URL: https://issues.apache.org/jira/browse/CASSANDRA-9558 Project: Cassandra Issue Type: Bug Reporter: Alan Boudreault Assignee: T Jake Luciani Fix For: 2.2.x Attachments: 2.1.log, 2.2.log, CASSANDRA-9558-2.patch, CASSANDRA-9558-ProtocolV2.patch, atolber-CASSANDRA-9558-stress.tgz, atolber-trunk-driver-coalescing-disabled.txt, stress-2.1-java-driver-2.0.9.2.log, stress-2.1-java-driver-2.2+PATCH.log, stress-2.1-java-driver-2.2.log, stress-2.2-java-driver-2.2+PATCH.log, stress-2.2-java-driver-2.2.log We are seeing some regression in performance when using cassandra-stress 2.2. You can see the difference at this url: http://riptano.github.io/cassandra_performance/graph_v5/graph.html?stats=stress_regression.jsonmetric=op_rateoperation=1_writesmoothing=1show_aggregates=truexmin=0xmax=108.57ymin=0ymax=168147.1 The cassandra version of the cluster doesn't seem to have any impact. //cc [~tjake] [~benedict] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9558) Cassandra-stress regression in 2.2
[ https://issues.apache.org/jira/browse/CASSANDRA-9558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14600135#comment-14600135 ] Benedict commented on CASSANDRA-9558: - The reason I ask about the size of the cluster this is being tested on, is this worsens performance as the cluster grows, as we coalesce fewer messages. So this could be improving our benchmark performance at the expense of real-world performance. Cassandra-stress regression in 2.2 -- Key: CASSANDRA-9558 URL: https://issues.apache.org/jira/browse/CASSANDRA-9558 Project: Cassandra Issue Type: Bug Reporter: Alan Boudreault Assignee: T Jake Luciani Fix For: 2.2.x Attachments: 2.1.log, 2.2.log, CASSANDRA-9558-2.patch, CASSANDRA-9558-ProtocolV2.patch, atolber-CASSANDRA-9558-stress.tgz, atolber-trunk-driver-coalescing-disabled.txt, stress-2.1-java-driver-2.0.9.2.log, stress-2.1-java-driver-2.2+PATCH.log, stress-2.1-java-driver-2.2.log, stress-2.2-java-driver-2.2+PATCH.log, stress-2.2-java-driver-2.2.log We are seeing some regression in performance when using cassandra-stress 2.2. You can see the difference at this url: http://riptano.github.io/cassandra_performance/graph_v5/graph.html?stats=stress_regression.jsonmetric=op_rateoperation=1_writesmoothing=1show_aggregates=truexmin=0xmax=108.57ymin=0ymax=168147.1 The cassandra version of the cluster doesn't seem to have any impact. //cc [~tjake] [~benedict] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9558) Cassandra-stress regression in 2.2
[ https://issues.apache.org/jira/browse/CASSANDRA-9558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14598451#comment-14598451 ] Olivier Michallat commented on CASSANDRA-9558: -- Coalescing is handled by [Connection.Flusher|https://github.com/datastax/java-driver/blob/2.0/driver-core/src/main/java/com/datastax/driver/core/Connection.java#L825] in the driver, it's scheduled on the event loop. Based on Benedict's feedback, I'm experimenting with [these changes|https://github.com/datastax/java-driver/commit/9ab18c454f67175904af98377454ba60e7fa9958] to avoid artificially delaying message, but I'm not observing a significant difference yet, at least in the context of cassandra-stress tests. Cassandra-stress regression in 2.2 -- Key: CASSANDRA-9558 URL: https://issues.apache.org/jira/browse/CASSANDRA-9558 Project: Cassandra Issue Type: Bug Reporter: Alan Boudreault Assignee: T Jake Luciani Fix For: 2.2.x Attachments: 2.1.log, 2.2.log, CASSANDRA-9558-2.patch, CASSANDRA-9558-ProtocolV2.patch, atolber-CASSANDRA-9558-stress.tgz, atolber-trunk-driver-coalescing-disabled.txt, stress-2.1-java-driver-2.0.9.2.log, stress-2.1-java-driver-2.2+PATCH.log, stress-2.1-java-driver-2.2.log, stress-2.2-java-driver-2.2+PATCH.log, stress-2.2-java-driver-2.2.log We are seeing some regression in performance when using cassandra-stress 2.2. You can see the difference at this url: http://riptano.github.io/cassandra_performance/graph_v5/graph.html?stats=stress_regression.jsonmetric=op_rateoperation=1_writesmoothing=1show_aggregates=truexmin=0xmax=108.57ymin=0ymax=168147.1 The cassandra version of the cluster doesn't seem to have any impact. //cc [~tjake] [~benedict] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9558) Cassandra-stress regression in 2.2
[ https://issues.apache.org/jira/browse/CASSANDRA-9558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14598421#comment-14598421 ] Olivier Michallat commented on CASSANDRA-9558: -- I ran tests on physical hardware that confirm that the number of connections is a major factor. As was explained before, 2.1.x versions of the driver are currently forcing a single connection per host when protocol v3 is in use. [JAVA-738|https://datastax-oss.atlassian.net/browse/JAVA-738] (scheduled for 2.1.7 and 2.2.0-rc2) will make the pool size dynamic again. Running cassandra-stress with driver 2.1.6 and protocol v3 (1 connection), I get a performance hit of about 25%. Switching to a 2.1.7 snapshot with protocol v3 and 8 connections per host, I'm back to the same as 2.0.x driver / protocol v2. Cassandra-stress regression in 2.2 -- Key: CASSANDRA-9558 URL: https://issues.apache.org/jira/browse/CASSANDRA-9558 Project: Cassandra Issue Type: Bug Reporter: Alan Boudreault Assignee: T Jake Luciani Fix For: 2.2.x Attachments: 2.1.log, 2.2.log, CASSANDRA-9558-2.patch, CASSANDRA-9558-ProtocolV2.patch, atolber-CASSANDRA-9558-stress.tgz, atolber-trunk-driver-coalescing-disabled.txt, stress-2.1-java-driver-2.0.9.2.log, stress-2.1-java-driver-2.2+PATCH.log, stress-2.1-java-driver-2.2.log, stress-2.2-java-driver-2.2+PATCH.log, stress-2.2-java-driver-2.2.log We are seeing some regression in performance when using cassandra-stress 2.2. You can see the difference at this url: http://riptano.github.io/cassandra_performance/graph_v5/graph.html?stats=stress_regression.jsonmetric=op_rateoperation=1_writesmoothing=1show_aggregates=truexmin=0xmax=108.57ymin=0ymax=168147.1 The cassandra version of the cluster doesn't seem to have any impact. //cc [~tjake] [~benedict] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9558) Cassandra-stress regression in 2.2
[ https://issues.apache.org/jira/browse/CASSANDRA-9558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14598389#comment-14598389 ] Olivier Michallat commented on CASSANDRA-9558: -- This is without Andy's patch, right? Cassandra-stress regression in 2.2 -- Key: CASSANDRA-9558 URL: https://issues.apache.org/jira/browse/CASSANDRA-9558 Project: Cassandra Issue Type: Bug Reporter: Alan Boudreault Assignee: T Jake Luciani Fix For: 2.2.x Attachments: 2.1.log, 2.2.log, CASSANDRA-9558-2.patch, CASSANDRA-9558-ProtocolV2.patch, atolber-CASSANDRA-9558-stress.tgz, atolber-trunk-driver-coalescing-disabled.txt, stress-2.1-java-driver-2.0.9.2.log, stress-2.1-java-driver-2.2+PATCH.log, stress-2.1-java-driver-2.2.log, stress-2.2-java-driver-2.2+PATCH.log, stress-2.2-java-driver-2.2.log We are seeing some regression in performance when using cassandra-stress 2.2. You can see the difference at this url: http://riptano.github.io/cassandra_performance/graph_v5/graph.html?stats=stress_regression.jsonmetric=op_rateoperation=1_writesmoothing=1show_aggregates=truexmin=0xmax=108.57ymin=0ymax=168147.1 The cassandra version of the cluster doesn't seem to have any impact. //cc [~tjake] [~benedict] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9558) Cassandra-stress regression in 2.2
[
https://issues.apache.org/jira/browse/CASSANDRA-9558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14598550#comment-14598550
]
Benedict commented on CASSANDRA-9558:
-
bq. That all said, some basic back-of-envelope maths suggest this cannot
sufficiently account for the problem in this case
It was a lengthy message but I already realised this could not explain it.
Should have highlighted it more, sorry.
I wonder if we could try playing with the {{ethtool -c}} settings on the C\*
server, to see if reducing the number of interrupts can help (by e.g. raising
the rx-usecs setting).
What size cluster are these tests being run against, for reference? If it's a
single node, I'm not actually very worried or perturbed by a reduction,
although we should probably try to mitigate the issue for benchmark purposes.
Cassandra-stress regression in 2.2
--
Key: CASSANDRA-9558
URL: https://issues.apache.org/jira/browse/CASSANDRA-9558
Project: Cassandra
Issue Type: Bug
Reporter: Alan Boudreault
Assignee: T Jake Luciani
Fix For: 2.2.x
Attachments: 2.1.log, 2.2.log, CASSANDRA-9558-2.patch,
CASSANDRA-9558-ProtocolV2.patch, atolber-CASSANDRA-9558-stress.tgz,
atolber-trunk-driver-coalescing-disabled.txt,
stress-2.1-java-driver-2.0.9.2.log, stress-2.1-java-driver-2.2+PATCH.log,
stress-2.1-java-driver-2.2.log, stress-2.2-java-driver-2.2+PATCH.log,
stress-2.2-java-driver-2.2.log
We are seeing some regression in performance when using cassandra-stress 2.2.
You can see the difference at this url:
http://riptano.github.io/cassandra_performance/graph_v5/graph.html?stats=stress_regression.jsonmetric=op_rateoperation=1_writesmoothing=1show_aggregates=truexmin=0xmax=108.57ymin=0ymax=168147.1
The cassandra version of the cluster doesn't seem to have any impact.
//cc [~tjake] [~benedict]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9558) Cassandra-stress regression in 2.2
[ https://issues.apache.org/jira/browse/CASSANDRA-9558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14593780#comment-14593780 ] Norman Maurer commented on CASSANDRA-9558: -- Sorry for been late to the party, but it somehow got lost in my inbox :( So from a netty standpoint your are right flushing from outside the EventLoop is pretty expensive as it will need to wakeup the selector if it is not already woken up and processing stuff. So the best thing you can do is either always write / flush etc from within the EventLoop or try to minimize the flushes from outside the EventLoop. That said if you point me to the place in your code where you do the flush and the other stuff I'm happy to have a look and see if I can give you some idea how to improve. Just let me know! Cassandra-stress regression in 2.2 -- Key: CASSANDRA-9558 URL: https://issues.apache.org/jira/browse/CASSANDRA-9558 Project: Cassandra Issue Type: Bug Reporter: Alan Boudreault Fix For: 2.2.0 rc2 Attachments: 2.1.log, 2.2.log, CASSANDRA-9558-2.patch, CASSANDRA-9558-ProtocolV2.patch, atolber-CASSANDRA-9558-stress.tgz, atolber-trunk-driver-coalescing-disabled.txt, stress-2.1-java-driver-2.0.9.2.log, stress-2.1-java-driver-2.2+PATCH.log, stress-2.1-java-driver-2.2.log, stress-2.2-java-driver-2.2+PATCH.log, stress-2.2-java-driver-2.2.log We are seeing some regression in performance when using cassandra-stress 2.2. You can see the difference at this url: http://riptano.github.io/cassandra_performance/graph_v5/graph.html?stats=stress_regression.jsonmetric=op_rateoperation=1_writesmoothing=1show_aggregates=truexmin=0xmax=108.57ymin=0ymax=168147.1 The cassandra version of the cluster doesn't seem to have any impact. //cc [~tjake] [~benedict] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9558) Cassandra-stress regression in 2.2
[ https://issues.apache.org/jira/browse/CASSANDRA-9558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14585397#comment-14585397 ] Jonathan Ellis commented on CASSANDRA-9558: --- I'm setting priority to Major because if push comes to shove we can release 2.2 and tell people use the 2.1 stress if you really need it. Cassandra-stress regression in 2.2 -- Key: CASSANDRA-9558 URL: https://issues.apache.org/jira/browse/CASSANDRA-9558 Project: Cassandra Issue Type: Bug Reporter: Alan Boudreault Fix For: 2.2.0 rc2 Attachments: 2.1.log, 2.2.log, CASSANDRA-9558-2.patch, CASSANDRA-9558-ProtocolV2.patch, atolber-CASSANDRA-9558-stress.tgz, atolber-trunk-driver-coalescing-disabled.txt, stress-2.1-java-driver-2.0.9.2.log, stress-2.1-java-driver-2.2+PATCH.log, stress-2.1-java-driver-2.2.log, stress-2.2-java-driver-2.2+PATCH.log, stress-2.2-java-driver-2.2.log We are seeing some regression in performance when using cassandra-stress 2.2. You can see the difference at this url: http://riptano.github.io/cassandra_performance/graph_v5/graph.html?stats=stress_regression.jsonmetric=op_rateoperation=1_writesmoothing=1show_aggregates=truexmin=0xmax=108.57ymin=0ymax=168147.1 The cassandra version of the cluster doesn't seem to have any impact. //cc [~tjake] [~benedict] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9558) Cassandra-stress regression in 2.2
[ https://issues.apache.org/jira/browse/CASSANDRA-9558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14582096#comment-14582096 ] Alan Boudreault commented on CASSANDRA-9558: On GCE, I'm seeing 80k op/s (cassandra-stress 2.1) versus 55k op/s (cassandra-stress 2.2). Locally I'm only seeing a difference of ~6k op/s (48k op/s for 2.1 versus 42k op/s for 2.2), but I am mostly CPU-limited on my laptop and cannot fully benefit of the 300 threads. Cassandra-stress regression in 2.2 -- Key: CASSANDRA-9558 URL: https://issues.apache.org/jira/browse/CASSANDRA-9558 Project: Cassandra Issue Type: Bug Reporter: Alan Boudreault Priority: Blocker Fix For: 2.2.0 rc2 Attachments: 2.1.log, 2.2.log, CASSANDRA-9558-2.patch, CASSANDRA-9558-ProtocolV2.patch, atolber-CASSANDRA-9558-stress.tgz, atolber-trunk-driver-coalescing-disabled.txt, stress-2.1-java-driver-2.0.9.2.log, stress-2.1-java-driver-2.2+PATCH.log, stress-2.1-java-driver-2.2.log, stress-2.2-java-driver-2.2+PATCH.log, stress-2.2-java-driver-2.2.log We are seeing some regression in performance when using cassandra-stress 2.2. You can see the difference at this url: http://riptano.github.io/cassandra_performance/graph_v5/graph.html?stats=stress_regression.jsonmetric=op_rateoperation=1_writesmoothing=1show_aggregates=truexmin=0xmax=108.57ymin=0ymax=168147.1 The cassandra version of the cluster doesn't seem to have any impact. //cc [~tjake] [~benedict] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9558) Cassandra-stress regression in 2.2
[ https://issues.apache.org/jira/browse/CASSANDRA-9558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14581086#comment-14581086 ] Aleksey Yeschenko commented on CASSANDRA-9558: -- Think of rc2 as 2.2.0. RCs are not betas. Cassandra-stress regression in 2.2 -- Key: CASSANDRA-9558 URL: https://issues.apache.org/jira/browse/CASSANDRA-9558 Project: Cassandra Issue Type: Bug Reporter: Alan Boudreault Priority: Blocker Fix For: 2.2.0 rc2 Attachments: 2.1.log, 2.2.log, CASSANDRA-9558-2.patch, CASSANDRA-9558-ProtocolV2.patch, atolber-CASSANDRA-9558-stress.tgz, atolber-trunk-driver-coalescing-disabled.txt, stress-2.1-java-driver-2.0.9.2.log, stress-2.1-java-driver-2.2+PATCH.log, stress-2.1-java-driver-2.2.log, stress-2.2-java-driver-2.2+PATCH.log, stress-2.2-java-driver-2.2.log We are seeing some regression in performance when using cassandra-stress 2.2. You can see the difference at this url: http://riptano.github.io/cassandra_performance/graph_v5/graph.html?stats=stress_regression.jsonmetric=op_rateoperation=1_writesmoothing=1show_aggregates=truexmin=0xmax=108.57ymin=0ymax=168147.1 The cassandra version of the cluster doesn't seem to have any impact. //cc [~tjake] [~benedict] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9558) Cassandra-stress regression in 2.2
[ https://issues.apache.org/jira/browse/CASSANDRA-9558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14581076#comment-14581076 ] Aleksey Yeschenko commented on CASSANDRA-9558: -- Strictly speaking, is it actually a blocker for rc2, or should it go into 2.2.x? Cassandra-stress regression in 2.2 -- Key: CASSANDRA-9558 URL: https://issues.apache.org/jira/browse/CASSANDRA-9558 Project: Cassandra Issue Type: Bug Reporter: Alan Boudreault Priority: Blocker Fix For: 2.2.0 rc2 Attachments: 2.1.log, 2.2.log, CASSANDRA-9558-2.patch, CASSANDRA-9558-ProtocolV2.patch, atolber-CASSANDRA-9558-stress.tgz, atolber-trunk-driver-coalescing-disabled.txt, stress-2.1-java-driver-2.0.9.2.log, stress-2.1-java-driver-2.2+PATCH.log, stress-2.1-java-driver-2.2.log, stress-2.2-java-driver-2.2+PATCH.log, stress-2.2-java-driver-2.2.log We are seeing some regression in performance when using cassandra-stress 2.2. You can see the difference at this url: http://riptano.github.io/cassandra_performance/graph_v5/graph.html?stats=stress_regression.jsonmetric=op_rateoperation=1_writesmoothing=1show_aggregates=truexmin=0xmax=108.57ymin=0ymax=168147.1 The cassandra version of the cluster doesn't seem to have any impact. //cc [~tjake] [~benedict] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9558) Cassandra-stress regression in 2.2
[ https://issues.apache.org/jira/browse/CASSANDRA-9558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14581085#comment-14581085 ] Alan Boudreault commented on CASSANDRA-9558: maybe not a blocker for rc2... but I think this one is important for 2.2.0. What others think? Cassandra-stress regression in 2.2 -- Key: CASSANDRA-9558 URL: https://issues.apache.org/jira/browse/CASSANDRA-9558 Project: Cassandra Issue Type: Bug Reporter: Alan Boudreault Priority: Blocker Fix For: 2.2.0 rc2 Attachments: 2.1.log, 2.2.log, CASSANDRA-9558-2.patch, CASSANDRA-9558-ProtocolV2.patch, atolber-CASSANDRA-9558-stress.tgz, atolber-trunk-driver-coalescing-disabled.txt, stress-2.1-java-driver-2.0.9.2.log, stress-2.1-java-driver-2.2+PATCH.log, stress-2.1-java-driver-2.2.log, stress-2.2-java-driver-2.2+PATCH.log, stress-2.2-java-driver-2.2.log We are seeing some regression in performance when using cassandra-stress 2.2. You can see the difference at this url: http://riptano.github.io/cassandra_performance/graph_v5/graph.html?stats=stress_regression.jsonmetric=op_rateoperation=1_writesmoothing=1show_aggregates=truexmin=0xmax=108.57ymin=0ymax=168147.1 The cassandra version of the cluster doesn't seem to have any impact. //cc [~tjake] [~benedict] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9558) Cassandra-stress regression in 2.2
[ https://issues.apache.org/jira/browse/CASSANDRA-9558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14581193#comment-14581193 ] Benedict commented on CASSANDRA-9558: - Is this issue only appearing on EC2? Or is it more widespread? If we're seeing a tanking of performance across the board, it probably is pretty urgent. If it's isolated to e.g. non-private EC2 networks , we can take our time over it. Cassandra-stress regression in 2.2 -- Key: CASSANDRA-9558 URL: https://issues.apache.org/jira/browse/CASSANDRA-9558 Project: Cassandra Issue Type: Bug Reporter: Alan Boudreault Priority: Blocker Fix For: 2.2.0 rc2 Attachments: 2.1.log, 2.2.log, CASSANDRA-9558-2.patch, CASSANDRA-9558-ProtocolV2.patch, atolber-CASSANDRA-9558-stress.tgz, atolber-trunk-driver-coalescing-disabled.txt, stress-2.1-java-driver-2.0.9.2.log, stress-2.1-java-driver-2.2+PATCH.log, stress-2.1-java-driver-2.2.log, stress-2.2-java-driver-2.2+PATCH.log, stress-2.2-java-driver-2.2.log We are seeing some regression in performance when using cassandra-stress 2.2. You can see the difference at this url: http://riptano.github.io/cassandra_performance/graph_v5/graph.html?stats=stress_regression.jsonmetric=op_rateoperation=1_writesmoothing=1show_aggregates=truexmin=0xmax=108.57ymin=0ymax=168147.1 The cassandra version of the cluster doesn't seem to have any impact. //cc [~tjake] [~benedict] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9558) Cassandra-stress regression in 2.2
[ https://issues.apache.org/jira/browse/CASSANDRA-9558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14581323#comment-14581323 ] Alan Boudreault commented on CASSANDRA-9558: I'll get back to you tomorrow about that. I will test on gce and locally. Cassandra-stress regression in 2.2 -- Key: CASSANDRA-9558 URL: https://issues.apache.org/jira/browse/CASSANDRA-9558 Project: Cassandra Issue Type: Bug Reporter: Alan Boudreault Priority: Blocker Fix For: 2.2.0 rc2 Attachments: 2.1.log, 2.2.log, CASSANDRA-9558-2.patch, CASSANDRA-9558-ProtocolV2.patch, atolber-CASSANDRA-9558-stress.tgz, atolber-trunk-driver-coalescing-disabled.txt, stress-2.1-java-driver-2.0.9.2.log, stress-2.1-java-driver-2.2+PATCH.log, stress-2.1-java-driver-2.2.log, stress-2.2-java-driver-2.2+PATCH.log, stress-2.2-java-driver-2.2.log We are seeing some regression in performance when using cassandra-stress 2.2. You can see the difference at this url: http://riptano.github.io/cassandra_performance/graph_v5/graph.html?stats=stress_regression.jsonmetric=op_rateoperation=1_writesmoothing=1show_aggregates=truexmin=0xmax=108.57ymin=0ymax=168147.1 The cassandra version of the cluster doesn't seem to have any impact. //cc [~tjake] [~benedict] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9558) Cassandra-stress regression in 2.2
[ https://issues.apache.org/jira/browse/CASSANDRA-9558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14577606#comment-14577606 ] Benedict commented on CASSANDRA-9558: - bq. Evidently this is exactly what writeAndFlush does which is what the driver is using when coalescing is disabled, but i'll keep exploring alternatives But it's not a choice between the two. There should absolutely be coalescing, and it should never be disabled. The question is if we should artificially delay our messages in order to coalesce more of them. On a client I cannot see it making sense to do so: on the server, we expect the server to have other useful work to do, to produce more responses that can be coalesced together. On a client, however, we should not make that assumption: if the client is synchronously waiting for a result, we're pointlessly delaying them (and cannot know if this is the case), whereas if they are asynchronously producing work, this will accumulate or not, completely independent of our delay, and after the first potentially more costly message the costs will reach a steady state, that the delay is unlikely to have any positive effect on. The main idea of it on the server is that it permits the server to exhaust its current burst of messages (if possible), so that all messages that would naturally be grouped given the chance can be. That all said, some basic back-of-envelope maths suggest this cannot sufficiently account for the problem in this case. That doesn't mean we shouldn't change it though, but it is unlikely to explain this ticket. We should really try to profile the client and server, to establish which is the bottleneck, and where. It should not be the case that we need multiple threads to deal with this workload: we're effectively batching up to 300 of these messages together, with a single point-to-point high-bandwidth TCP connection. The fact that this cannot cope with more than 7MB/s is crazy. There is maximal amortization of costs. It is possible we're hitting another weird issue with interrupt queues in AWS. Cassandra-stress regression in 2.2 -- Key: CASSANDRA-9558 URL: https://issues.apache.org/jira/browse/CASSANDRA-9558 Project: Cassandra Issue Type: Bug Reporter: Alan Boudreault Priority: Blocker Attachments: 2.1.log, 2.2.log, CASSANDRA-9558-2.patch, CASSANDRA-9558-ProtocolV2.patch, atolber-CASSANDRA-9558-stress.tgz, atolber-trunk-driver-coalescing-disabled.txt, stress-2.1-java-driver-2.0.9.2.log, stress-2.1-java-driver-2.2+PATCH.log, stress-2.1-java-driver-2.2.log, stress-2.2-java-driver-2.2+PATCH.log, stress-2.2-java-driver-2.2.log We are seeing some regression in performance when using cassandra-stress 2.2. You can see the difference at this url: http://riptano.github.io/cassandra_performance/graph_v5/graph.html?stats=stress_regression.jsonmetric=op_rateoperation=1_writesmoothing=1show_aggregates=truexmin=0xmax=108.57ymin=0ymax=168147.1 The cassandra version of the cluster doesn't seem to have any impact. //cc [~tjake] [~benedict] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9558) Cassandra-stress regression in 2.2
[ https://issues.apache.org/jira/browse/CASSANDRA-9558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14577316#comment-14577316 ] Alan Boudreault commented on CASSANDRA-9558: I confirm that with the last patch provided by Andy [^CASSANDRA-9558-2.patch], we are back at the same performance we had with cassandra-2.1 using java-driver 2.0. Thanks Andy! Cassandra-stress regression in 2.2 -- Key: CASSANDRA-9558 URL: https://issues.apache.org/jira/browse/CASSANDRA-9558 Project: Cassandra Issue Type: Bug Reporter: Alan Boudreault Priority: Blocker Attachments: 2.1.log, 2.2.log, CASSANDRA-9558-2.patch, CASSANDRA-9558-ProtocolV2.patch, atolber-CASSANDRA-9558-stress.tgz, atolber-trunk-driver-coalescing-disabled.txt, stress-2.1-java-driver-2.0.9.2.log, stress-2.1-java-driver-2.2+PATCH.log, stress-2.1-java-driver-2.2.log, stress-2.2-java-driver-2.2+PATCH.log, stress-2.2-java-driver-2.2.log We are seeing some regression in performance when using cassandra-stress 2.2. You can see the difference at this url: http://riptano.github.io/cassandra_performance/graph_v5/graph.html?stats=stress_regression.jsonmetric=op_rateoperation=1_writesmoothing=1show_aggregates=truexmin=0xmax=108.57ymin=0ymax=168147.1 The cassandra version of the cluster doesn't seem to have any impact. //cc [~tjake] [~benedict] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9558) Cassandra-stress regression in 2.2
[ https://issues.apache.org/jira/browse/CASSANDRA-9558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14575697#comment-14575697 ] Benedict commented on CASSANDRA-9558: - [this|https://github.com/datastax/java-driver/blob/2.1/driver-core/src/main/java/com/datastax/driver/core/Connection.java#L887] may be the problem This logic makes sense on the server-side, since there is likely more useful work for the server to do in the meantime, but on the client we're just delaying the server from getting started on processing our requests. We should be batching, but not delaying. i.e. flushing as soon as we have written all pending messages. Cassandra-stress regression in 2.2 -- Key: CASSANDRA-9558 URL: https://issues.apache.org/jira/browse/CASSANDRA-9558 Project: Cassandra Issue Type: Bug Reporter: Alan Boudreault Priority: Blocker Attachments: 2.1.log, 2.2.log, CASSANDRA-9558-2.patch, CASSANDRA-9558-ProtocolV2.patch, atolber-CASSANDRA-9558-stress.tgz, stress-2.1-java-driver-2.0.9.2.log, stress-2.1-java-driver-2.2+PATCH.log, stress-2.1-java-driver-2.2.log, stress-2.2-java-driver-2.2+PATCH.log, stress-2.2-java-driver-2.2.log We are seeing some regression in performance when using cassandra-stress 2.2. You can see the difference at this url: http://riptano.github.io/cassandra_performance/graph_v5/graph.html?stats=stress_regression.jsonmetric=op_rateoperation=1_writesmoothing=1show_aggregates=truexmin=0xmax=108.57ymin=0ymax=168147.1 The cassandra version of the cluster doesn't seem to have any impact. //cc [~tjake] [~benedict] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9558) Cassandra-stress regression in 2.2
[ https://issues.apache.org/jira/browse/CASSANDRA-9558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14575851#comment-14575851 ] Robert Stupp commented on CASSANDRA-9558: - /cc [~norman] (added Norman in the loop - maybe he has some insights to what's possible with Netty ;) ) Cassandra-stress regression in 2.2 -- Key: CASSANDRA-9558 URL: https://issues.apache.org/jira/browse/CASSANDRA-9558 Project: Cassandra Issue Type: Bug Reporter: Alan Boudreault Priority: Blocker Attachments: 2.1.log, 2.2.log, CASSANDRA-9558-2.patch, CASSANDRA-9558-ProtocolV2.patch, atolber-CASSANDRA-9558-stress.tgz, atolber-trunk-driver-coalescing-disabled.txt, stress-2.1-java-driver-2.0.9.2.log, stress-2.1-java-driver-2.2+PATCH.log, stress-2.1-java-driver-2.2.log, stress-2.2-java-driver-2.2+PATCH.log, stress-2.2-java-driver-2.2.log We are seeing some regression in performance when using cassandra-stress 2.2. You can see the difference at this url: http://riptano.github.io/cassandra_performance/graph_v5/graph.html?stats=stress_regression.jsonmetric=op_rateoperation=1_writesmoothing=1show_aggregates=truexmin=0xmax=108.57ymin=0ymax=168147.1 The cassandra version of the cluster doesn't seem to have any impact. //cc [~tjake] [~benedict] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9558) Cassandra-stress regression in 2.2
[ https://issues.apache.org/jira/browse/CASSANDRA-9558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14574945#comment-14574945 ] Alan Boudreault commented on CASSANDRA-9558: I am currently bisecting this issue. We reproduced it on a c3.8xlarge cluster and it is currently hard to reproduce locally. Cassandra-stress regression in 2.2 -- Key: CASSANDRA-9558 URL: https://issues.apache.org/jira/browse/CASSANDRA-9558 Project: Cassandra Issue Type: Bug Reporter: Alan Boudreault Priority: Blocker We are seeing some regression in performance when using cassandra-stress 2.2. You can see the difference at this url: http://riptano.github.io/cassandra_performance/graph_v5/graph.html?stats=stress_regression.jsonmetric=op_rateoperation=1_writesmoothing=1show_aggregates=truexmin=0xmax=108.57ymin=0ymax=168147.1 The cassandra version of the cluster doesn't seem to have any impact. //cc [~tjake] [~benedict] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9558) Cassandra-stress regression in 2.2
[ https://issues.apache.org/jira/browse/CASSANDRA-9558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14575162#comment-14575162 ] Alan Boudreault commented on CASSANDRA-9558: yeah, I am currently working with Andy to confirm the java-driver is the cause. Cassandra-stress regression in 2.2 -- Key: CASSANDRA-9558 URL: https://issues.apache.org/jira/browse/CASSANDRA-9558 Project: Cassandra Issue Type: Bug Reporter: Alan Boudreault Priority: Blocker Attachments: 2.1.log, 2.2.log We are seeing some regression in performance when using cassandra-stress 2.2. You can see the difference at this url: http://riptano.github.io/cassandra_performance/graph_v5/graph.html?stats=stress_regression.jsonmetric=op_rateoperation=1_writesmoothing=1show_aggregates=truexmin=0xmax=108.57ymin=0ymax=168147.1 The cassandra version of the cluster doesn't seem to have any impact. //cc [~tjake] [~benedict] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9558) Cassandra-stress regression in 2.2
[ https://issues.apache.org/jira/browse/CASSANDRA-9558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14575092#comment-14575092 ] Ryan McGuire commented on CASSANDRA-9558: - AFAIK the only use of thrfit in the 2.1 run was for the schema creation, which was recently modified in CASSANDRA-9374 Cassandra-stress regression in 2.2 -- Key: CASSANDRA-9558 URL: https://issues.apache.org/jira/browse/CASSANDRA-9558 Project: Cassandra Issue Type: Bug Reporter: Alan Boudreault Priority: Blocker Attachments: 2.1.log, 2.2.log We are seeing some regression in performance when using cassandra-stress 2.2. You can see the difference at this url: http://riptano.github.io/cassandra_performance/graph_v5/graph.html?stats=stress_regression.jsonmetric=op_rateoperation=1_writesmoothing=1show_aggregates=truexmin=0xmax=108.57ymin=0ymax=168147.1 The cassandra version of the cluster doesn't seem to have any impact. //cc [~tjake] [~benedict] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9558) Cassandra-stress regression in 2.2
[ https://issues.apache.org/jira/browse/CASSANDRA-9558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14575158#comment-14575158 ] T Jake Luciani commented on CASSANDRA-9558: --- My guess is it's CASSANDRA-9493 This is the java driver used by stress. the last time I tried it the performance was much lower in stress because the driver uses a single connection to send many messages, vs multiple connections. Cassandra-stress regression in 2.2 -- Key: CASSANDRA-9558 URL: https://issues.apache.org/jira/browse/CASSANDRA-9558 Project: Cassandra Issue Type: Bug Reporter: Alan Boudreault Priority: Blocker Attachments: 2.1.log, 2.2.log We are seeing some regression in performance when using cassandra-stress 2.2. You can see the difference at this url: http://riptano.github.io/cassandra_performance/graph_v5/graph.html?stats=stress_regression.jsonmetric=op_rateoperation=1_writesmoothing=1show_aggregates=truexmin=0xmax=108.57ymin=0ymax=168147.1 The cassandra version of the cluster doesn't seem to have any impact. //cc [~tjake] [~benedict] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9558) Cassandra-stress regression in 2.2
[ https://issues.apache.org/jira/browse/CASSANDRA-9558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14575241#comment-14575241 ] Andy Tolbert commented on CASSANDRA-9558: - Just confirming that this is very likely the java driver. I went back to cassandra-2.1 branch and built it with driver 2.2.0-rc1-SNAPSHOT and noticed the same performance degradation. [~tjake] is on point in that this is likely caused by the driver only using a single connection. To get around this we could force protocol version 2 in the driver in the stress tool, it's not ideal but worth a try. I'll attach some performance numbers shortly. Cassandra-stress regression in 2.2 -- Key: CASSANDRA-9558 URL: https://issues.apache.org/jira/browse/CASSANDRA-9558 Project: Cassandra Issue Type: Bug Reporter: Alan Boudreault Priority: Blocker Attachments: 2.1.log, 2.2.log We are seeing some regression in performance when using cassandra-stress 2.2. You can see the difference at this url: http://riptano.github.io/cassandra_performance/graph_v5/graph.html?stats=stress_regression.jsonmetric=op_rateoperation=1_writesmoothing=1show_aggregates=truexmin=0xmax=108.57ymin=0ymax=168147.1 The cassandra version of the cluster doesn't seem to have any impact. //cc [~tjake] [~benedict] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9558) Cassandra-stress regression in 2.2
[ https://issues.apache.org/jira/browse/CASSANDRA-9558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14575318#comment-14575318 ] Andy Tolbert commented on CASSANDRA-9558: - Thanks [~aboudreault]! For reference, [JAVA-738|https://datastax-oss.atlassian.net/browse/JAVA-738] was previously opened to evaluate the pool implementation for protocol version 3+. I've also opened up [JAVA-802|https://datastax-oss.atlassian.net/browse/JAVA-802] to ensure we incorporate testing java-driver RCs with cassandra-stress to ensure there is no performance regression as part of the test/release process. I'll continue doing a performance evaluation of the driver with cassandra-stress to see if anything obvious sticks out on the driver side. Cassandra-stress regression in 2.2 -- Key: CASSANDRA-9558 URL: https://issues.apache.org/jira/browse/CASSANDRA-9558 Project: Cassandra Issue Type: Bug Reporter: Alan Boudreault Priority: Blocker Attachments: 2.1.log, 2.2.log, CASSANDRA-9558-ProtocolV2.patch, stress-2.1-java-driver-2.0.9.2.log, stress-2.1-java-driver-2.2+PATCH.log, stress-2.1-java-driver-2.2.log, stress-2.2-java-driver-2.2+PATCH.log, stress-2.2-java-driver-2.2.log We are seeing some regression in performance when using cassandra-stress 2.2. You can see the difference at this url: http://riptano.github.io/cassandra_performance/graph_v5/graph.html?stats=stress_regression.jsonmetric=op_rateoperation=1_writesmoothing=1show_aggregates=truexmin=0xmax=108.57ymin=0ymax=168147.1 The cassandra version of the cluster doesn't seem to have any impact. //cc [~tjake] [~benedict] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9558) Cassandra-stress regression in 2.2
[ https://issues.apache.org/jira/browse/CASSANDRA-9558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14575467#comment-14575467 ] Andy Tolbert commented on CASSANDRA-9558: - It could be that all it would take to get the stress tool back to where it is in cassandra-2.1 is to simply place cassandra-driver-core-2.0.9.2.jar into tools/lib/ much like it is on the [cassandra-2.1|https://github.com/apache/cassandra/tree/cassandra-2.1/tools/lib] branch. I've tried this in my environment and see a ~15-20% improvement over 'cassandra-2.2 w/ the patch to force protocol v2'. [~aboudreault], when you have a chance could you try this and let me know how it goes? All it should take it placing cassandra-driver-core-2.0.9.2 in tools/lib/. I've done some investigation and it looks like the regression is present between java-driver 2.1.5 and 2.1.6 (also observed in 2.0.9.2 and 2.0.10 so it probably stems from there), I'll continue investigating. I observe very similar performance with java-driver 2.1.5 /w proto v2 and 2.0.9.2. Cassandra-stress regression in 2.2 -- Key: CASSANDRA-9558 URL: https://issues.apache.org/jira/browse/CASSANDRA-9558 Project: Cassandra Issue Type: Bug Reporter: Alan Boudreault Priority: Blocker Attachments: 2.1.log, 2.2.log, CASSANDRA-9558-ProtocolV2.patch, stress-2.1-java-driver-2.0.9.2.log, stress-2.1-java-driver-2.2+PATCH.log, stress-2.1-java-driver-2.2.log, stress-2.2-java-driver-2.2+PATCH.log, stress-2.2-java-driver-2.2.log We are seeing some regression in performance when using cassandra-stress 2.2. You can see the difference at this url: http://riptano.github.io/cassandra_performance/graph_v5/graph.html?stats=stress_regression.jsonmetric=op_rateoperation=1_writesmoothing=1show_aggregates=truexmin=0xmax=108.57ymin=0ymax=168147.1 The cassandra version of the cluster doesn't seem to have any impact. //cc [~tjake] [~benedict] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
