[jira] [Updated] (CASSANDRA-6187) Cassandra-2.0 node seems to be pausing periodically under normal operations
[
https://issues.apache.org/jira/browse/CASSANDRA-6187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Li Zou updated CASSANDRA-6187:
--
Description:
Under normal operation condition without any interruption, the traffic
generator will periodically see a couple of seconds of zero (or very little)
transactions, i.e. the throughput can be quite high for a while, then comes
down to zero (or very little) transactions for 10 ~ 20 seconds.
The Cassandra system log occasionally also logs message drops. But these
message drop events do not correlate in time to the observed transaction drop
events.
Example of message dropping log:
{noformat}
INFO [ScheduledTasks:1] 2013-10-11 16:36:12,216 MessagingService.java (line
812) 1191 MUTATION messages dropped in last 5000ms
INFO [ScheduledTasks:1] 2013-10-11 16:36:12,217 MessagingService.java (line
812) 502 READ_REPAIR messages dropped in last 5000ms
INFO [ScheduledTasks:1] 2013-10-11 16:36:12,217 StatusLogger.java (line 55)
Pool NameActive Pending Completed Blocked All
Time Blocked
INFO [ScheduledTasks:1] 2013-10-11 16:36:12,246 StatusLogger.java (line 70)
ReadStage 0 0 845326 0
0
INFO [ScheduledTasks:1] 2013-10-11 16:36:12,246 StatusLogger.java (line 70)
RequestResponseStage 0 01643358 0
0
INFO [ScheduledTasks:1] 2013-10-11 16:36:12,246 StatusLogger.java (line 70)
ReadRepairStage 0 0 61247 0
0
INFO [ScheduledTasks:1] 2013-10-11 16:36:12,247 StatusLogger.java (line 70)
MutationStage 0 01155502 0
0
INFO [ScheduledTasks:1] 2013-10-11 16:36:12,247 StatusLogger.java (line 70)
ReplicateOnWriteStage 0 0 0 0
0
INFO [ScheduledTasks:1] 2013-10-11 16:36:12,247 StatusLogger.java (line 70)
GossipStage 0 0 5391 0
0
INFO [ScheduledTasks:1] 2013-10-11 16:36:12,248 StatusLogger.java (line 70)
AntiEntropyStage 0 0 0 0
0
INFO [ScheduledTasks:1] 2013-10-11 16:36:12,248 StatusLogger.java (line 70)
MigrationStage0 0 14 0
0
INFO [ScheduledTasks:1] 2013-10-11 16:36:12,248 StatusLogger.java (line 70)
MemtablePostFlusher 0 0 99 0
0
INFO [ScheduledTasks:1] 2013-10-11 16:36:12,248 StatusLogger.java (line 70)
MemoryMeter 0 0 58 0
0
INFO [ScheduledTasks:1] 2013-10-11 16:36:12,249 StatusLogger.java (line 70)
FlushWriter 0 0 45 0
0
INFO [ScheduledTasks:1] 2013-10-11 16:36:12,249 StatusLogger.java (line 70)
MiscStage 0 0 0 0
0
INFO [ScheduledTasks:1] 2013-10-11 16:36:12,249 StatusLogger.java (line 70)
commitlog_archiver0 0 0 0
0
INFO [ScheduledTasks:1] 2013-10-11 16:36:12,250 StatusLogger.java (line 70)
InternalResponseStage 0 0 3 0
0
INFO [ScheduledTasks:1] 2013-10-11 16:36:12,250 StatusLogger.java (line 70)
HintedHandoff 0 0 7 0
0
INFO [ScheduledTasks:1] 2013-10-11 16:36:12,250 StatusLogger.java (line 79)
CompactionManager 0 0
INFO [ScheduledTasks:1] 2013-10-11 16:36:12,250 StatusLogger.java (line 81)
Commitlog n/a 0
INFO [ScheduledTasks:1] 2013-10-11 16:36:12,251 StatusLogger.java (line 93)
MessagingServicen/a 0,0
INFO [ScheduledTasks:1] 2013-10-11 16:36:12,251 StatusLogger.java (line 103)
Cache Type Size Capacity
KeysToSave
INFO [ScheduledTasks:1] 2013-10-11 16:36:12,251 StatusLogger.java (line 105)
KeyCache 74140104857600
all
INFO [ScheduledTasks:1] 2013-10-11 16:36:12,251 StatusLogger.java (line 111)
RowCache 00
all
INFO [ScheduledTasks:1] 2013-10-11 16:36:12,251 StatusLogger.java (line 118)
ColumnFamilyMemtable ops,data
INFO [ScheduledTasks:1] 2013-10-11 16:36:12,251 StatusLogger.java (line 121)
system.schema_triggers0,0
INFO [ScheduledTasks:1] 2013-10-11 16:36:12,252 StatusLogger.java (line 121)
system.local 18,520
INFO [ScheduledTasks:1] 2013-10-11 16:36:12,252 StatusLogger.java (line 121)
system.peers
[jira] [Created] (CASSANDRA-6187) Cassandra-2.0 node seems to be pausing periodically under normal operations
Li Zou created CASSANDRA-6187:
-
Summary: Cassandra-2.0 node seems to be pausing periodically under
normal operations
Key: CASSANDRA-6187
URL: https://issues.apache.org/jira/browse/CASSANDRA-6187
Project: Cassandra
Issue Type: Bug
Components: Core
Environment: One data center of 4 Cassandra-2.0 nodes with default
configuration parameters deployed on 4 separate machines.
A testing app (with either Astyanax or DataStax client driver) interacts with
the Cassandra data center.
A traffic generator is sending traffic to the testing app for testing purpose.
Reporter: Li Zou
Under normal operation condition without any interruption, the traffic
generator will periodically see a couple of seconds of zero (or very little)
transactions, i.e. the throughput can be quite high for a while, then comes
down to zero (or very little) transactions for 10 ~ 20 seconds.
The Cassandra system log occasionally also logs message drops. But these
message drop events do not correlate in time to the observed transaction drop
events.
Example of message dropping log:
{noformat}
INFO [ScheduledTasks:1] 2013-10-11 16:36:12,216 MessagingService.java (line
812) 1191 MUTATION messages dropped in last 5000ms
INFO [ScheduledTasks:1] 2013-10-11 16:36:12,217 MessagingService.java (line
812) 502 READ_REPAIR messages dropped in last 5000ms
INFO [ScheduledTasks:1] 2013-10-11 16:36:12,217 StatusLogger.java (line 55)
Pool NameActive Pending Completed Blocked All
Time Blocked
INFO [ScheduledTasks:1] 2013-10-11 16:36:12,246 StatusLogger.java (line 70)
ReadStage 0 0 845326 0
0
INFO [ScheduledTasks:1] 2013-10-11 16:36:12,246 StatusLogger.java (line 70)
RequestResponseStage 0 01643358 0
0
INFO [ScheduledTasks:1] 2013-10-11 16:36:12,246 StatusLogger.java (line 70)
ReadRepairStage 0 0 61247 0
0
INFO [ScheduledTasks:1] 2013-10-11 16:36:12,247 StatusLogger.java (line 70)
MutationStage 0 01155502 0
0
INFO [ScheduledTasks:1] 2013-10-11 16:36:12,247 StatusLogger.java (line 70)
ReplicateOnWriteStage 0 0 0 0
0
INFO [ScheduledTasks:1] 2013-10-11 16:36:12,247 StatusLogger.java (line 70)
GossipStage 0 0 5391 0
0
INFO [ScheduledTasks:1] 2013-10-11 16:36:12,248 StatusLogger.java (line 70)
AntiEntropyStage 0 0 0 0
0
INFO [ScheduledTasks:1] 2013-10-11 16:36:12,248 StatusLogger.java (line 70)
MigrationStage0 0 14 0
0
INFO [ScheduledTasks:1] 2013-10-11 16:36:12,248 StatusLogger.java (line 70)
MemtablePostFlusher 0 0 99 0
0
INFO [ScheduledTasks:1] 2013-10-11 16:36:12,248 StatusLogger.java (line 70)
MemoryMeter 0 0 58 0
0
INFO [ScheduledTasks:1] 2013-10-11 16:36:12,249 StatusLogger.java (line 70)
FlushWriter 0 0 45 0
0
INFO [ScheduledTasks:1] 2013-10-11 16:36:12,249 StatusLogger.java (line 70)
MiscStage 0 0 0 0
0
INFO [ScheduledTasks:1] 2013-10-11 16:36:12,249 StatusLogger.java (line 70)
commitlog_archiver0 0 0 0
0
INFO [ScheduledTasks:1] 2013-10-11 16:36:12,250 StatusLogger.java (line 70)
InternalResponseStage 0 0 3 0
0
INFO [ScheduledTasks:1] 2013-10-11 16:36:12,250 StatusLogger.java (line 70)
HintedHandoff 0 0 7 0
0
INFO [ScheduledTasks:1] 2013-10-11 16:36:12,250 StatusLogger.java (line 79)
CompactionManager 0 0
INFO [ScheduledTasks:1] 2013-10-11 16:36:12,250 StatusLogger.java (line 81)
Commitlog n/a 0
INFO [ScheduledTasks:1] 2013-10-11 16:36:12,251 StatusLogger.java (line 93)
MessagingServicen/a 0,0
INFO [ScheduledTasks:1] 2013-10-11 16:36:12,251 StatusLogger.java (line 103)
Cache Type Size Capacity
KeysToSave
INFO [ScheduledTasks:1] 2013-10-11 16:36:12,251 StatusLogger.java (line 105)
KeyCache 74140104857600
all
INFO [ScheduledTasks:1] 2013-10-11 16:36:12,251 StatusLogger.java (line 111)
RowCache 0
[jira] [Created] (CASSANDRA-6188) The JMX stats for Speculative Retry stops moving during a node failure outage period
Li Zou created CASSANDRA-6188: - Summary: The JMX stats for Speculative Retry stops moving during a node failure outage period Key: CASSANDRA-6188 URL: https://issues.apache.org/jira/browse/CASSANDRA-6188 Project: Cassandra Issue Type: Bug Components: Core Environment: One data center of 4 Cassandra-2.0 nodes with default configuration parameters deployed on 4 separate machines. A testing app (with either Astyanax or DataStax client driver) interacts with the Cassandra data center. A traffic generator is sending traffic to the testing app for testing purpose. Reporter: Li Zou Under normal testing traffic level with default Cassandra Speculative Retry configuration for each table (i.e. 99 percentile), JConsole shows that the JMX stats for Speculative Retry increments slowly. However, during the node failure outage period (i.e. immediately after the node was killed and before the gossip figures out that the node is down), JConsole shows that the JMX stats for Speculative Retry stops moving. That is, for around 20 seconds, the JMX stats for Speculative Retry does not move. This is true for all other Speculative Retry options. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (CASSANDRA-6187) Cassandra-2.0 node seems to be pausing periodically under normal operations
[
https://issues.apache.org/jira/browse/CASSANDRA-6187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13793141#comment-13793141
]
Li Zou commented on CASSANDRA-6187:
---
This issue is only observed fairly recently when testing the Cassandra-2.0.
We have been using the same tool, the same configuration and the same set of
machines for Cassandra-1.2.4 and it has never shown any sign of this issue.
Cassandra-2.0 node seems to be pausing periodically under normal operations
---
Key: CASSANDRA-6187
URL: https://issues.apache.org/jira/browse/CASSANDRA-6187
Project: Cassandra
Issue Type: Bug
Components: Core
Environment: One data center of 4 Cassandra-2.0 nodes with default
configuration parameters deployed on 4 separate machines.
A testing app (with either Astyanax or DataStax client driver) interacts with
the Cassandra data center.
A traffic generator is sending traffic to the testing app for testing purpose.
Reporter: Li Zou
Under normal operation condition without any interruption, the traffic
generator will periodically see a couple of seconds of zero (or very little)
transactions, i.e. the throughput can be quite high for a while, then comes
down to zero (or very little) transactions for 10 ~ 20 seconds.
The Cassandra system log occasionally also logs message drops. But these
message drop events do not correlate in time to the observed transaction drop
events.
Example of message dropping log:
{noformat}
INFO [ScheduledTasks:1] 2013-10-11 16:36:12,216 MessagingService.java (line
812) 1191 MUTATION messages dropped in last 5000ms
INFO [ScheduledTasks:1] 2013-10-11 16:36:12,217 MessagingService.java (line
812) 502 READ_REPAIR messages dropped in last 5000ms
INFO [ScheduledTasks:1] 2013-10-11 16:36:12,217 StatusLogger.java (line 55)
Pool NameActive Pending Completed Blocked All
Time Blocked
INFO [ScheduledTasks:1] 2013-10-11 16:36:12,246 StatusLogger.java (line 70)
ReadStage 0 0 845326 0
0
INFO [ScheduledTasks:1] 2013-10-11 16:36:12,246 StatusLogger.java (line 70)
RequestResponseStage 0 01643358 0
0
INFO [ScheduledTasks:1] 2013-10-11 16:36:12,246 StatusLogger.java (line 70)
ReadRepairStage 0 0 61247 0
0
INFO [ScheduledTasks:1] 2013-10-11 16:36:12,247 StatusLogger.java (line 70)
MutationStage 0 01155502 0
0
INFO [ScheduledTasks:1] 2013-10-11 16:36:12,247 StatusLogger.java (line 70)
ReplicateOnWriteStage 0 0 0 0
0
INFO [ScheduledTasks:1] 2013-10-11 16:36:12,247 StatusLogger.java (line 70)
GossipStage 0 0 5391 0
0
INFO [ScheduledTasks:1] 2013-10-11 16:36:12,248 StatusLogger.java (line 70)
AntiEntropyStage 0 0 0 0
0
INFO [ScheduledTasks:1] 2013-10-11 16:36:12,248 StatusLogger.java (line 70)
MigrationStage0 0 14 0
0
INFO [ScheduledTasks:1] 2013-10-11 16:36:12,248 StatusLogger.java (line 70)
MemtablePostFlusher 0 0 99 0
0
INFO [ScheduledTasks:1] 2013-10-11 16:36:12,248 StatusLogger.java (line 70)
MemoryMeter 0 0 58 0
0
INFO [ScheduledTasks:1] 2013-10-11 16:36:12,249 StatusLogger.java (line 70)
FlushWriter 0 0 45 0
0
INFO [ScheduledTasks:1] 2013-10-11 16:36:12,249 StatusLogger.java (line 70)
MiscStage 0 0 0 0
0
INFO [ScheduledTasks:1] 2013-10-11 16:36:12,249 StatusLogger.java (line 70)
commitlog_archiver0 0 0 0
0
INFO [ScheduledTasks:1] 2013-10-11 16:36:12,250 StatusLogger.java (line 70)
InternalResponseStage 0 0 3 0
0
INFO [ScheduledTasks:1] 2013-10-11 16:36:12,250 StatusLogger.java (line 70)
HintedHandoff 0 0 7 0
0
INFO [ScheduledTasks:1] 2013-10-11 16:36:12,250 StatusLogger.java (line 79)
CompactionManager 0 0
INFO [ScheduledTasks:1] 2013-10-11 16:36:12,250 StatusLogger.java (line 81)
Commitlog n/a 0
INFO [ScheduledTasks:1] 2013-10-11 16:36:12,251
[jira] [Commented] (CASSANDRA-6154) Inserts are blocked in 2.1
[
https://issues.apache.org/jira/browse/CASSANDRA-6154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13789737#comment-13789737
]
Li Zou commented on CASSANDRA-6154:
---
I just pulled the Cassandra-2.0 source code and did a quick testing. Though
slightly better than yesterday's trunk load. But there are still lots of
assertion errors for {{MessagingService.addCallback()}} without the argument
of consistency level for MessageOutRowMutation. E.g.
{noformat}
private void doDeliverHintsToEndpoint(InetAddress endpoint)
{
...
MessageOutRowMutation message = rm.createMessage();
...
MessagingService.instance().sendRR(message, endpoint,
responseHandler);
responseHandlers.add(responseHandler);
{noformat}
and
{noformat}
public static void sendToHintedEndpoints(final RowMutation rm,
IterableInetAddress targets,
AbstractWriteResponseHandler
responseHandler,
String localDataCenter,
ConsistencyLevel consistency_level)
throws OverloadedException
{
...
// belongs on a different server
if (message == null)
message = rm.createMessage();
String dc =
DatabaseDescriptor.getEndpointSnitch().getDatacenter(destination);
// direct writes to local DC or old Cassandra versions
// (1.1 knows how to forward old-style String message IDs;
updated to int in 2.0)
if (localDataCenter.equals(dc) ||
MessagingService.instance().getVersion(destination)
MessagingService.VERSION_20)
{
MessagingService.instance().sendRR(message,
destination, responseHandler);
}
...
{noformat}
In the above examples, the {{sendRR()}} will call {{addCallback()}} which
asserts as the message is of type RowMutation.
Inserts are blocked in 2.1
--
Key: CASSANDRA-6154
URL: https://issues.apache.org/jira/browse/CASSANDRA-6154
Project: Cassandra
Issue Type: Bug
Reporter: Ryan McGuire
Assignee: Jonathan Ellis
Priority: Critical
Attachments: 6154.txt, 6154-v2.txt
With cluster sizes 1 inserts are blocked indefinitely:
{code}
$ ccm create -v git:trunk test
Fetching Cassandra updates...
Current cluster is now: test
$ ccm populate -n 2
$ ccm start
$ ccm node1 cqlsh
Connected to test at 127.0.0.1:9160.
[cqlsh 4.0.1 | Cassandra 2.1-SNAPSHOT | CQL spec 3.1.1 | Thrift protocol
19.37.0]
Use HELP for help.
cqlsh CREATE KEYSPACE timeline WITH replication = {'class':
'SimpleStrategy', 'replication_factor': 1};
cqlsh USE timeline;
cqlsh:timeline CREATE TABLE user_events (userid text, event timestamp, value
text, PRIMARY KEY (userid, event));
cqlsh:timeline INSERT INTO user_events (userid, event , value ) VALUES (
'ryan', '2013-10-07', 'attempt');
{code}
The last INSERT statement never returns..
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Comment Edited] (CASSANDRA-6154) Inserts are blocked in 2.1
[
https://issues.apache.org/jira/browse/CASSANDRA-6154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13789737#comment-13789737
]
Li Zou edited comment on CASSANDRA-6154 at 10/8/13 9:52 PM:
[~jbellis] I just pulled the Cassandra-2.0 source code and did a quick testing.
Though slightly better than yesterday's trunk load. But there are still lots of
assertion errors for {{MessagingService.addCallback()}} without the argument
of consistency level for MessageOutRowMutation. E.g.
{noformat}
private void doDeliverHintsToEndpoint(InetAddress endpoint)
{
...
MessageOutRowMutation message = rm.createMessage();
...
MessagingService.instance().sendRR(message, endpoint,
responseHandler);
responseHandlers.add(responseHandler);
{noformat}
and
{noformat}
public static void sendToHintedEndpoints(final RowMutation rm,
IterableInetAddress targets,
AbstractWriteResponseHandler
responseHandler,
String localDataCenter,
ConsistencyLevel consistency_level)
throws OverloadedException
{
...
// belongs on a different server
if (message == null)
message = rm.createMessage();
String dc =
DatabaseDescriptor.getEndpointSnitch().getDatacenter(destination);
// direct writes to local DC or old Cassandra versions
// (1.1 knows how to forward old-style String message IDs;
updated to int in 2.0)
if (localDataCenter.equals(dc) ||
MessagingService.instance().getVersion(destination)
MessagingService.VERSION_20)
{
MessagingService.instance().sendRR(message,
destination, responseHandler);
}
...
{noformat}
In the above examples, the {{sendRR()}} will call {{addCallback()}} which
asserts as the message is of type RowMutation.
was (Author: lizou):
I just pulled the Cassandra-2.0 source code and did a quick testing. Though
slightly better than yesterday's trunk load. But there are still lots of
assertion errors for {{MessagingService.addCallback()}} without the argument
of consistency level for MessageOutRowMutation. E.g.
{noformat}
private void doDeliverHintsToEndpoint(InetAddress endpoint)
{
...
MessageOutRowMutation message = rm.createMessage();
...
MessagingService.instance().sendRR(message, endpoint,
responseHandler);
responseHandlers.add(responseHandler);
{noformat}
and
{noformat}
public static void sendToHintedEndpoints(final RowMutation rm,
IterableInetAddress targets,
AbstractWriteResponseHandler
responseHandler,
String localDataCenter,
ConsistencyLevel consistency_level)
throws OverloadedException
{
...
// belongs on a different server
if (message == null)
message = rm.createMessage();
String dc =
DatabaseDescriptor.getEndpointSnitch().getDatacenter(destination);
// direct writes to local DC or old Cassandra versions
// (1.1 knows how to forward old-style String message IDs;
updated to int in 2.0)
if (localDataCenter.equals(dc) ||
MessagingService.instance().getVersion(destination)
MessagingService.VERSION_20)
{
MessagingService.instance().sendRR(message,
destination, responseHandler);
}
...
{noformat}
In the above examples, the {{sendRR()}} will call {{addCallback()}} which
asserts as the message is of type RowMutation.
Inserts are blocked in 2.1
--
Key: CASSANDRA-6154
URL: https://issues.apache.org/jira/browse/CASSANDRA-6154
Project: Cassandra
Issue Type: Bug
Reporter: Ryan McGuire
Assignee: Jonathan Ellis
Priority: Critical
Attachments: 6154.txt, 6154-v2.txt
With cluster sizes 1 inserts are blocked indefinitely:
{code}
$ ccm create -v git:trunk test
Fetching Cassandra updates...
Current cluster is now: test
$ ccm populate -n 2
$ ccm start
$ ccm node1 cqlsh
Connected to test at 127.0.0.1:9160.
[cqlsh 4.0.1 | Cassandra 2.1-SNAPSHOT | CQL spec 3.1.1 | Thrift protocol
19.37.0]
Use HELP for help.
cqlsh CREATE KEYSPACE timeline WITH replication = {'class':
'SimpleStrategy', 'replication_factor': 1};
cqlsh USE timeline;
[jira] [Commented] (CASSANDRA-6154) Inserts are blocked in 2.1
[
https://issues.apache.org/jira/browse/CASSANDRA-6154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13789768#comment-13789768
]
Li Zou commented on CASSANDRA-6154:
---
Yes, this has been observed in my test this afternoon after one node was killed.
I think that we should systematically check for this issue, as some other
places can also have this {{MessagingService.addCallback()}} assertion issue.
Inserts are blocked in 2.1
--
Key: CASSANDRA-6154
URL: https://issues.apache.org/jira/browse/CASSANDRA-6154
Project: Cassandra
Issue Type: Bug
Reporter: Ryan McGuire
Assignee: Jonathan Ellis
Priority: Critical
Attachments: 6154.txt, 6154-v2.txt
With cluster sizes 1 inserts are blocked indefinitely:
{code}
$ ccm create -v git:trunk test
Fetching Cassandra updates...
Current cluster is now: test
$ ccm populate -n 2
$ ccm start
$ ccm node1 cqlsh
Connected to test at 127.0.0.1:9160.
[cqlsh 4.0.1 | Cassandra 2.1-SNAPSHOT | CQL spec 3.1.1 | Thrift protocol
19.37.0]
Use HELP for help.
cqlsh CREATE KEYSPACE timeline WITH replication = {'class':
'SimpleStrategy', 'replication_factor': 1};
cqlsh USE timeline;
cqlsh:timeline CREATE TABLE user_events (userid text, event timestamp, value
text, PRIMARY KEY (userid, event));
cqlsh:timeline INSERT INTO user_events (userid, event , value ) VALUES (
'ryan', '2013-10-07', 'attempt');
{code}
The last INSERT statement never returns..
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (CASSANDRA-5932) Speculative read performance data show unexpected results
[
https://issues.apache.org/jira/browse/CASSANDRA-5932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13788437#comment-13788437
]
Li Zou commented on CASSANDRA-5932:
---
This morning's trunk load has a slightly different symptom, and is even more
serious than last Friday's load, as this time just commenting out the assert
statement in the {{MessagingService.addCallback()}} will not help.
I copy the {{/var/log/cassandra/system.log}} exception errors below.
{noformat}
ERROR [Thrift:12] 2013-10-07 14:42:39,396 Caller+0 at
org.apache.cassandra.service.CassandraDaemon$2.uncaughtException(CassandraDaemon.java:134)
- Exception in thread Thread[Thrift:12,5,main]
java.lang.AssertionError: null
at
org.apache.cassandra.net.MessagingService.addCallback(MessagingService.java:543)
~[apache-cassandra-2.1-SNAPSHOT.jar:2.1-SNAPSHOT]
at
org.apache.cassandra.net.MessagingService.sendRR(MessagingService.java:591)
~[apache-cassandra-2.1-SNAPSHOT.jar:2.1-SNAPSHOT]
at
org.apache.cassandra.net.MessagingService.sendRR(MessagingService.java:571)
~[apache-cassandra-2.1-SNAPSHOT.jar:2.1-SNAPSHOT]
at
org.apache.cassandra.service.StorageProxy.sendToHintedEndpoints(StorageProxy.java:869)
~[apache-cassandra-2.1-SNAPSHOT.jar:2.1-SNAPSHOT]
at
org.apache.cassandra.service.StorageProxy$2.apply(StorageProxy.java:123)
~[apache-cassandra-2.1-SNAPSHOT.jar:2.1-SNAPSHOT]
at
org.apache.cassandra.service.StorageProxy.performWrite(StorageProxy.java:739)
~[apache-cassandra-2.1-SNAPSHOT.jar:2.1-SNAPSHOT]
at
org.apache.cassandra.service.StorageProxy.mutate(StorageProxy.java:511)
~[apache-cassandra-2.1-SNAPSHOT.jar:2.1-SNAPSHOT]
at
org.apache.cassandra.service.StorageProxy.mutateWithTriggers(StorageProxy.java:581)
~[apache-cassandra-2.1-SNAPSHOT.jar:2.1-SNAPSHOT]
at
org.apache.cassandra.cql3.statements.ModificationStatement.executeWithoutCondition(ModificationStatement.java:379)
~[apache-cassandra-2.1-SNAPSHOT.jar:2.1-SNAPSHOT]
at
org.apache.cassandra.cql3.statements.ModificationStatement.execute(ModificationStatement.java:363)
~[apache-cassandra-2.1-SNAPSHOT.jar:2.1-SNAPSHOT]
at
org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:126)
~[apache-cassandra-2.1-SNAPSHOT.jar:2.1-SNAPSHOT]
at
org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:267)
~[apache-cassandra-2.1-SNAPSHOT.jar:2.1-SNAPSHOT]
at
org.apache.cassandra.thrift.CassandraServer.execute_prepared_cql3_query(CassandraServer.java:2061)
~[apache-cassandra-2.1-SNAPSHOT.jar:2.1-SNAPSHOT]
at
org.apache.cassandra.thrift.Cassandra$Processor$execute_prepared_cql3_query.getResult(Cassandra.java:4502)
~[apache-cassandra-thrift-2.1-SNAPSHOT.jar:2.1-SNAPSHOT]
at
org.apache.cassandra.thrift.Cassandra$Processor$execute_prepared_cql3_query.getResult(Cassandra.java:4486)
~[apache-cassandra-thrift-2.1-SNAPSHOT.jar:2.1-SNAPSHOT]
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
~[libthrift-0.9.1.jar:0.9.1]
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
~[libthrift-0.9.1.jar:0.9.1]
at
org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:194)
~[apache-cassandra-2.1-SNAPSHOT.jar:2.1-SNAPSHOT]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
~[na:1.7.0_25]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
~[na:1.7.0_25]
at java.lang.Thread.run(Thread.java:724) ~[na:1.7.0_25]
{noformat}
Speculative read performance data show unexpected results
-
Key: CASSANDRA-5932
URL: https://issues.apache.org/jira/browse/CASSANDRA-5932
Project: Cassandra
Issue Type: Bug
Reporter: Ryan McGuire
Assignee: Aleksey Yeschenko
Fix For: 2.0.2
Attachments: 5932.6692c50412ef7d.compaction.png,
5932-6692c50412ef7d.png, 5932.6692c50412ef7d.rr0.png,
5932.6692c50412ef7d.rr1.png, 5932.ded39c7e1c2fa.logs.tar.gz, 5932.txt,
5933-128_and_200rc1.png, 5933-7a87fc11.png, 5933-logs.tar.gz,
5933-randomized-dsnitch-replica.2.png, 5933-randomized-dsnitch-replica.3.png,
5933-randomized-dsnitch-replica.png, compaction-makes-slow.png,
compaction-makes-slow-stats.png, eager-read-looks-promising.png,
eager-read-looks-promising-stats.png, eager-read-not-consistent.png,
eager-read-not-consistent-stats.png, node-down-increase-performance.png
I've done a series of stress tests with eager retries enabled that show
undesirable behavior. I'm grouping these behaviours into one ticket as they
are most likely related.
1) Killing off a node in a 4 node cluster actually increases performance.
[jira] [Comment Edited] (CASSANDRA-5932) Speculative read performance data show unexpected results
[
https://issues.apache.org/jira/browse/CASSANDRA-5932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13788437#comment-13788437
]
Li Zou edited comment on CASSANDRA-5932 at 10/7/13 7:17 PM:
[~jbellis], this morning's trunk load has a slightly different symptom, and is
even more serious than last Friday's load, as this time just commenting out the
assert statement in the {{MessagingService.addCallback()}} will not help.
I copy the {{/var/log/cassandra/system.log}} exception errors below.
{noformat}
ERROR [Thrift:12] 2013-10-07 14:42:39,396 Caller+0 at
org.apache.cassandra.service.CassandraDaemon$2.uncaughtException(CassandraDaemon.java:134)
- Exception in thread Thread[Thrift:12,5,main]
java.lang.AssertionError: null
at
org.apache.cassandra.net.MessagingService.addCallback(MessagingService.java:543)
~[apache-cassandra-2.1-SNAPSHOT.jar:2.1-SNAPSHOT]
at
org.apache.cassandra.net.MessagingService.sendRR(MessagingService.java:591)
~[apache-cassandra-2.1-SNAPSHOT.jar:2.1-SNAPSHOT]
at
org.apache.cassandra.net.MessagingService.sendRR(MessagingService.java:571)
~[apache-cassandra-2.1-SNAPSHOT.jar:2.1-SNAPSHOT]
at
org.apache.cassandra.service.StorageProxy.sendToHintedEndpoints(StorageProxy.java:869)
~[apache-cassandra-2.1-SNAPSHOT.jar:2.1-SNAPSHOT]
at
org.apache.cassandra.service.StorageProxy$2.apply(StorageProxy.java:123)
~[apache-cassandra-2.1-SNAPSHOT.jar:2.1-SNAPSHOT]
at
org.apache.cassandra.service.StorageProxy.performWrite(StorageProxy.java:739)
~[apache-cassandra-2.1-SNAPSHOT.jar:2.1-SNAPSHOT]
at
org.apache.cassandra.service.StorageProxy.mutate(StorageProxy.java:511)
~[apache-cassandra-2.1-SNAPSHOT.jar:2.1-SNAPSHOT]
at
org.apache.cassandra.service.StorageProxy.mutateWithTriggers(StorageProxy.java:581)
~[apache-cassandra-2.1-SNAPSHOT.jar:2.1-SNAPSHOT]
at
org.apache.cassandra.cql3.statements.ModificationStatement.executeWithoutCondition(ModificationStatement.java:379)
~[apache-cassandra-2.1-SNAPSHOT.jar:2.1-SNAPSHOT]
at
org.apache.cassandra.cql3.statements.ModificationStatement.execute(ModificationStatement.java:363)
~[apache-cassandra-2.1-SNAPSHOT.jar:2.1-SNAPSHOT]
at
org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:126)
~[apache-cassandra-2.1-SNAPSHOT.jar:2.1-SNAPSHOT]
at
org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:267)
~[apache-cassandra-2.1-SNAPSHOT.jar:2.1-SNAPSHOT]
at
org.apache.cassandra.thrift.CassandraServer.execute_prepared_cql3_query(CassandraServer.java:2061)
~[apache-cassandra-2.1-SNAPSHOT.jar:2.1-SNAPSHOT]
at
org.apache.cassandra.thrift.Cassandra$Processor$execute_prepared_cql3_query.getResult(Cassandra.java:4502)
~[apache-cassandra-thrift-2.1-SNAPSHOT.jar:2.1-SNAPSHOT]
at
org.apache.cassandra.thrift.Cassandra$Processor$execute_prepared_cql3_query.getResult(Cassandra.java:4486)
~[apache-cassandra-thrift-2.1-SNAPSHOT.jar:2.1-SNAPSHOT]
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
~[libthrift-0.9.1.jar:0.9.1]
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
~[libthrift-0.9.1.jar:0.9.1]
at
org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:194)
~[apache-cassandra-2.1-SNAPSHOT.jar:2.1-SNAPSHOT]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
~[na:1.7.0_25]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
~[na:1.7.0_25]
at java.lang.Thread.run(Thread.java:724) ~[na:1.7.0_25]
{noformat}
was (Author: lizou):
This morning's trunk load has a slightly different symptom, and is even more
serious than last Friday's load, as this time just commenting out the assert
statement in the {{MessagingService.addCallback()}} will not help.
I copy the {{/var/log/cassandra/system.log}} exception errors below.
{noformat}
ERROR [Thrift:12] 2013-10-07 14:42:39,396 Caller+0 at
org.apache.cassandra.service.CassandraDaemon$2.uncaughtException(CassandraDaemon.java:134)
- Exception in thread Thread[Thrift:12,5,main]
java.lang.AssertionError: null
at
org.apache.cassandra.net.MessagingService.addCallback(MessagingService.java:543)
~[apache-cassandra-2.1-SNAPSHOT.jar:2.1-SNAPSHOT]
at
org.apache.cassandra.net.MessagingService.sendRR(MessagingService.java:591)
~[apache-cassandra-2.1-SNAPSHOT.jar:2.1-SNAPSHOT]
at
org.apache.cassandra.net.MessagingService.sendRR(MessagingService.java:571)
~[apache-cassandra-2.1-SNAPSHOT.jar:2.1-SNAPSHOT]
at
org.apache.cassandra.service.StorageProxy.sendToHintedEndpoints(StorageProxy.java:869)
~[apache-cassandra-2.1-SNAPSHOT.jar:2.1-SNAPSHOT]
at
[jira] [Commented] (CASSANDRA-5932) Speculative read performance data show unexpected results
[ https://issues.apache.org/jira/browse/CASSANDRA-5932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13788613#comment-13788613 ] Li Zou commented on CASSANDRA-5932: --- Hello, As this ticket is already fixed in 2.0.2, where can I get the 2.0.2 source code? Currently, my git tag only shows up to 2.0.1. Speculative read performance data show unexpected results - Key: CASSANDRA-5932 URL: https://issues.apache.org/jira/browse/CASSANDRA-5932 Project: Cassandra Issue Type: Bug Reporter: Ryan McGuire Assignee: Aleksey Yeschenko Fix For: 2.0.2 Attachments: 5932.6692c50412ef7d.compaction.png, 5932-6692c50412ef7d.png, 5932.6692c50412ef7d.rr0.png, 5932.6692c50412ef7d.rr1.png, 5932.ded39c7e1c2fa.logs.tar.gz, 5932.txt, 5933-128_and_200rc1.png, 5933-7a87fc11.png, 5933-logs.tar.gz, 5933-randomized-dsnitch-replica.2.png, 5933-randomized-dsnitch-replica.3.png, 5933-randomized-dsnitch-replica.png, compaction-makes-slow.png, compaction-makes-slow-stats.png, eager-read-looks-promising.png, eager-read-looks-promising-stats.png, eager-read-not-consistent.png, eager-read-not-consistent-stats.png, node-down-increase-performance.png I've done a series of stress tests with eager retries enabled that show undesirable behavior. I'm grouping these behaviours into one ticket as they are most likely related. 1) Killing off a node in a 4 node cluster actually increases performance. 2) Compactions make nodes slow, even after the compaction is done. 3) Eager Reads tend to lessen the *immediate* performance impact of a node going down, but not consistently. My Environment: 1 stress machine: node0 4 C* nodes: node4, node5, node6, node7 My script: node0 writes some data: stress -d node4 -F 3000 -n 3000 -i 5 -l 2 -K 20 node0 reads some data: stress -d node4 -n 3000 -o read -i 5 -K 20 h3. Examples: h5. A node going down increases performance: !node-down-increase-performance.png! [Data for this test here|http://ryanmcguire.info/ds/graph/graph.html?stats=stats.eager_retry.node_killed.just_20.jsonmetric=interval_op_rateoperation=stress-readsmoothing=1] At 450s, I kill -9 one of the nodes. There is a brief decrease in performance as the snitch adapts, but then it recovers... to even higher performance than before. h5. Compactions make nodes permanently slow: !compaction-makes-slow.png! !compaction-makes-slow-stats.png! The green and orange lines represent trials with eager retry enabled, they never recover their op-rate from before the compaction as the red and blue lines do. [Data for this test here|http://ryanmcguire.info/ds/graph/graph.html?stats=stats.eager_retry.compaction.2.jsonmetric=interval_op_rateoperation=stress-readsmoothing=1] h5. Speculative Read tends to lessen the *immediate* impact: !eager-read-looks-promising.png! !eager-read-looks-promising-stats.png! This graph looked the most promising to me, the two trials with eager retry, the green and orange line, at 450s showed the smallest dip in performance. [Data for this test here|http://ryanmcguire.info/ds/graph/graph.html?stats=stats.eager_retry.node_killed.jsonmetric=interval_op_rateoperation=stress-readsmoothing=1] h5. But not always: !eager-read-not-consistent.png! !eager-read-not-consistent-stats.png! This is a retrial with the same settings as above, yet the 95percentile eager retry (red line) did poorly this time at 450s. [Data for this test here|http://ryanmcguire.info/ds/graph/graph.html?stats=stats.eager_retry.node_killed.just_20.rc1.try2.jsonmetric=interval_op_rateoperation=stress-readsmoothing=1] -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (CASSANDRA-5932) Speculative read performance data show unexpected results
[
https://issues.apache.org/jira/browse/CASSANDRA-5932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13788654#comment-13788654
]
Li Zou commented on CASSANDRA-5932:
---
I even cannot see the cassandra-2.0 branch.
My git tag gives a list including following branches.
{noformat}
$ git tag
1.2.8
1.2.8-tentative
cassandra-0.3.0-final
cassandra-0.3.0-rc1
cassandra-0.3.0-rc2
...
cassandra-1.2.4
cassandra-1.2.5
cassandra-1.2.6
cassandra-1.2.7
cassandra-1.2.8
cassandra-1.2.9
cassandra-2.0.0
cassandra-2.0.0-beta1
cassandra-2.0.0-beta2
cassandra-2.0.0-rc1
cassandra-2.0.0-rc2
cassandra-2.0.1
drivers
list
{noformat}
There is no cassandra-2.0 branch. Where can I find it?
Speculative read performance data show unexpected results
-
Key: CASSANDRA-5932
URL: https://issues.apache.org/jira/browse/CASSANDRA-5932
Project: Cassandra
Issue Type: Bug
Reporter: Ryan McGuire
Assignee: Aleksey Yeschenko
Fix For: 2.0.2
Attachments: 5932.6692c50412ef7d.compaction.png,
5932-6692c50412ef7d.png, 5932.6692c50412ef7d.rr0.png,
5932.6692c50412ef7d.rr1.png, 5932.ded39c7e1c2fa.logs.tar.gz, 5932.txt,
5933-128_and_200rc1.png, 5933-7a87fc11.png, 5933-logs.tar.gz,
5933-randomized-dsnitch-replica.2.png, 5933-randomized-dsnitch-replica.3.png,
5933-randomized-dsnitch-replica.png, compaction-makes-slow.png,
compaction-makes-slow-stats.png, eager-read-looks-promising.png,
eager-read-looks-promising-stats.png, eager-read-not-consistent.png,
eager-read-not-consistent-stats.png, node-down-increase-performance.png
I've done a series of stress tests with eager retries enabled that show
undesirable behavior. I'm grouping these behaviours into one ticket as they
are most likely related.
1) Killing off a node in a 4 node cluster actually increases performance.
2) Compactions make nodes slow, even after the compaction is done.
3) Eager Reads tend to lessen the *immediate* performance impact of a node
going down, but not consistently.
My Environment:
1 stress machine: node0
4 C* nodes: node4, node5, node6, node7
My script:
node0 writes some data: stress -d node4 -F 3000 -n 3000 -i 5 -l 2 -K
20
node0 reads some data: stress -d node4 -n 3000 -o read -i 5 -K 20
h3. Examples:
h5. A node going down increases performance:
!node-down-increase-performance.png!
[Data for this test
here|http://ryanmcguire.info/ds/graph/graph.html?stats=stats.eager_retry.node_killed.just_20.jsonmetric=interval_op_rateoperation=stress-readsmoothing=1]
At 450s, I kill -9 one of the nodes. There is a brief decrease in performance
as the snitch adapts, but then it recovers... to even higher performance than
before.
h5. Compactions make nodes permanently slow:
!compaction-makes-slow.png!
!compaction-makes-slow-stats.png!
The green and orange lines represent trials with eager retry enabled, they
never recover their op-rate from before the compaction as the red and blue
lines do.
[Data for this test
here|http://ryanmcguire.info/ds/graph/graph.html?stats=stats.eager_retry.compaction.2.jsonmetric=interval_op_rateoperation=stress-readsmoothing=1]
h5. Speculative Read tends to lessen the *immediate* impact:
!eager-read-looks-promising.png!
!eager-read-looks-promising-stats.png!
This graph looked the most promising to me, the two trials with eager retry,
the green and orange line, at 450s showed the smallest dip in performance.
[Data for this test
here|http://ryanmcguire.info/ds/graph/graph.html?stats=stats.eager_retry.node_killed.jsonmetric=interval_op_rateoperation=stress-readsmoothing=1]
h5. But not always:
!eager-read-not-consistent.png!
!eager-read-not-consistent-stats.png!
This is a retrial with the same settings as above, yet the 95percentile eager
retry (red line) did poorly this time at 450s.
[Data for this test
here|http://ryanmcguire.info/ds/graph/graph.html?stats=stats.eager_retry.node_killed.just_20.rc1.try2.jsonmetric=interval_op_rateoperation=stress-readsmoothing=1]
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (CASSANDRA-5932) Speculative read performance data show unexpected results
[
https://issues.apache.org/jira/browse/CASSANDRA-5932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13786585#comment-13786585
]
Li Zou commented on CASSANDRA-5932:
---
Have done some testing using today's trunk. Have observed following issues.
*Issue 1* -- The first method {{MessagingService.addCallback()}} (i.e. without
the ConsistencyLevel argument) asserts.
Commenting out the assert statement seems to work. But the Cassandra servers
themselves will produce 10-second outage (i.e. zero transactions from the
client point of view) periodically.
*Issue 2* -- The Speculative Retry seems stop retrying during the outage window.
During the outage window triggered either by killing one of Cassandra nodes or
produced by Cassandra servers themselves, the JConsole shows that the JMX
stats, SpeculativeRetry counter stops incrementing until the gossip figures out
the outage issue.
What is the reason for this? The Speculative Retry is meant to help during the
outage period. This observed behavior is consistent with Cassandra 2.0.0-rc2.
Speculative read performance data show unexpected results
-
Key: CASSANDRA-5932
URL: https://issues.apache.org/jira/browse/CASSANDRA-5932
Project: Cassandra
Issue Type: Bug
Reporter: Ryan McGuire
Assignee: Aleksey Yeschenko
Fix For: 2.0.2
Attachments: 5932.6692c50412ef7d.compaction.png,
5932-6692c50412ef7d.png, 5932.6692c50412ef7d.rr0.png,
5932.6692c50412ef7d.rr1.png, 5932.ded39c7e1c2fa.logs.tar.gz, 5932.txt,
5933-128_and_200rc1.png, 5933-7a87fc11.png, 5933-logs.tar.gz,
5933-randomized-dsnitch-replica.2.png, 5933-randomized-dsnitch-replica.3.png,
5933-randomized-dsnitch-replica.png, compaction-makes-slow.png,
compaction-makes-slow-stats.png, eager-read-looks-promising.png,
eager-read-looks-promising-stats.png, eager-read-not-consistent.png,
eager-read-not-consistent-stats.png, node-down-increase-performance.png
I've done a series of stress tests with eager retries enabled that show
undesirable behavior. I'm grouping these behaviours into one ticket as they
are most likely related.
1) Killing off a node in a 4 node cluster actually increases performance.
2) Compactions make nodes slow, even after the compaction is done.
3) Eager Reads tend to lessen the *immediate* performance impact of a node
going down, but not consistently.
My Environment:
1 stress machine: node0
4 C* nodes: node4, node5, node6, node7
My script:
node0 writes some data: stress -d node4 -F 3000 -n 3000 -i 5 -l 2 -K
20
node0 reads some data: stress -d node4 -n 3000 -o read -i 5 -K 20
h3. Examples:
h5. A node going down increases performance:
!node-down-increase-performance.png!
[Data for this test
here|http://ryanmcguire.info/ds/graph/graph.html?stats=stats.eager_retry.node_killed.just_20.jsonmetric=interval_op_rateoperation=stress-readsmoothing=1]
At 450s, I kill -9 one of the nodes. There is a brief decrease in performance
as the snitch adapts, but then it recovers... to even higher performance than
before.
h5. Compactions make nodes permanently slow:
!compaction-makes-slow.png!
!compaction-makes-slow-stats.png!
The green and orange lines represent trials with eager retry enabled, they
never recover their op-rate from before the compaction as the red and blue
lines do.
[Data for this test
here|http://ryanmcguire.info/ds/graph/graph.html?stats=stats.eager_retry.compaction.2.jsonmetric=interval_op_rateoperation=stress-readsmoothing=1]
h5. Speculative Read tends to lessen the *immediate* impact:
!eager-read-looks-promising.png!
!eager-read-looks-promising-stats.png!
This graph looked the most promising to me, the two trials with eager retry,
the green and orange line, at 450s showed the smallest dip in performance.
[Data for this test
here|http://ryanmcguire.info/ds/graph/graph.html?stats=stats.eager_retry.node_killed.jsonmetric=interval_op_rateoperation=stress-readsmoothing=1]
h5. But not always:
!eager-read-not-consistent.png!
!eager-read-not-consistent-stats.png!
This is a retrial with the same settings as above, yet the 95percentile eager
retry (red line) did poorly this time at 450s.
[Data for this test
here|http://ryanmcguire.info/ds/graph/graph.html?stats=stats.eager_retry.node_killed.just_20.rc1.try2.jsonmetric=interval_op_rateoperation=stress-readsmoothing=1]
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (CASSANDRA-5932) Speculative read performance data show unexpected results
[ https://issues.apache.org/jira/browse/CASSANDRA-5932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13786613#comment-13786613 ] Li Zou commented on CASSANDRA-5932: --- I just did a git pull after your comment (i.e. a couple of seconds ago) for trunk. The git pull indicates that Already up-to-date. I checked the MessagingService.java, there are two addCallback() methods. * The one without ConsistencyLevel is called by sendRR() * The one with ConsistencyLevel is called by sendMessageToNonLocalDC() Speculative read performance data show unexpected results - Key: CASSANDRA-5932 URL: https://issues.apache.org/jira/browse/CASSANDRA-5932 Project: Cassandra Issue Type: Bug Reporter: Ryan McGuire Assignee: Aleksey Yeschenko Fix For: 2.0.2 Attachments: 5932.6692c50412ef7d.compaction.png, 5932-6692c50412ef7d.png, 5932.6692c50412ef7d.rr0.png, 5932.6692c50412ef7d.rr1.png, 5932.ded39c7e1c2fa.logs.tar.gz, 5932.txt, 5933-128_and_200rc1.png, 5933-7a87fc11.png, 5933-logs.tar.gz, 5933-randomized-dsnitch-replica.2.png, 5933-randomized-dsnitch-replica.3.png, 5933-randomized-dsnitch-replica.png, compaction-makes-slow.png, compaction-makes-slow-stats.png, eager-read-looks-promising.png, eager-read-looks-promising-stats.png, eager-read-not-consistent.png, eager-read-not-consistent-stats.png, node-down-increase-performance.png I've done a series of stress tests with eager retries enabled that show undesirable behavior. I'm grouping these behaviours into one ticket as they are most likely related. 1) Killing off a node in a 4 node cluster actually increases performance. 2) Compactions make nodes slow, even after the compaction is done. 3) Eager Reads tend to lessen the *immediate* performance impact of a node going down, but not consistently. My Environment: 1 stress machine: node0 4 C* nodes: node4, node5, node6, node7 My script: node0 writes some data: stress -d node4 -F 3000 -n 3000 -i 5 -l 2 -K 20 node0 reads some data: stress -d node4 -n 3000 -o read -i 5 -K 20 h3. Examples: h5. A node going down increases performance: !node-down-increase-performance.png! [Data for this test here|http://ryanmcguire.info/ds/graph/graph.html?stats=stats.eager_retry.node_killed.just_20.jsonmetric=interval_op_rateoperation=stress-readsmoothing=1] At 450s, I kill -9 one of the nodes. There is a brief decrease in performance as the snitch adapts, but then it recovers... to even higher performance than before. h5. Compactions make nodes permanently slow: !compaction-makes-slow.png! !compaction-makes-slow-stats.png! The green and orange lines represent trials with eager retry enabled, they never recover their op-rate from before the compaction as the red and blue lines do. [Data for this test here|http://ryanmcguire.info/ds/graph/graph.html?stats=stats.eager_retry.compaction.2.jsonmetric=interval_op_rateoperation=stress-readsmoothing=1] h5. Speculative Read tends to lessen the *immediate* impact: !eager-read-looks-promising.png! !eager-read-looks-promising-stats.png! This graph looked the most promising to me, the two trials with eager retry, the green and orange line, at 450s showed the smallest dip in performance. [Data for this test here|http://ryanmcguire.info/ds/graph/graph.html?stats=stats.eager_retry.node_killed.jsonmetric=interval_op_rateoperation=stress-readsmoothing=1] h5. But not always: !eager-read-not-consistent.png! !eager-read-not-consistent-stats.png! This is a retrial with the same settings as above, yet the 95percentile eager retry (red line) did poorly this time at 450s. [Data for this test here|http://ryanmcguire.info/ds/graph/graph.html?stats=stats.eager_retry.node_killed.just_20.rc1.try2.jsonmetric=interval_op_rateoperation=stress-readsmoothing=1] -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Comment Edited] (CASSANDRA-5932) Speculative read performance data show unexpected results
[ https://issues.apache.org/jira/browse/CASSANDRA-5932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13786613#comment-13786613 ] Li Zou edited comment on CASSANDRA-5932 at 10/4/13 9:02 PM: I just did a git pull after your comment (i.e. a couple of seconds ago) for trunk. The git pull indicates that Already up-to-date. I checked the MessagingService.java, there are two addCallback() methods. * The one without ConsistencyLevel is called by sendRR() * The one with ConsistencyLevel is called by sendMessageToNonLocalDC() The trunk load I used for testing was pulled this noon. It has two addCallback() methods. One of them (i.e. without the ConsistencyLevel) asserts. was (Author: lizou): I just did a git pull after your comment (i.e. a couple of seconds ago) for trunk. The git pull indicates that Already up-to-date. I checked the MessagingService.java, there are two addCallback() methods. * The one without ConsistencyLevel is called by sendRR() * The one with ConsistencyLevel is called by sendMessageToNonLocalDC() Speculative read performance data show unexpected results - Key: CASSANDRA-5932 URL: https://issues.apache.org/jira/browse/CASSANDRA-5932 Project: Cassandra Issue Type: Bug Reporter: Ryan McGuire Assignee: Aleksey Yeschenko Fix For: 2.0.2 Attachments: 5932.6692c50412ef7d.compaction.png, 5932-6692c50412ef7d.png, 5932.6692c50412ef7d.rr0.png, 5932.6692c50412ef7d.rr1.png, 5932.ded39c7e1c2fa.logs.tar.gz, 5932.txt, 5933-128_and_200rc1.png, 5933-7a87fc11.png, 5933-logs.tar.gz, 5933-randomized-dsnitch-replica.2.png, 5933-randomized-dsnitch-replica.3.png, 5933-randomized-dsnitch-replica.png, compaction-makes-slow.png, compaction-makes-slow-stats.png, eager-read-looks-promising.png, eager-read-looks-promising-stats.png, eager-read-not-consistent.png, eager-read-not-consistent-stats.png, node-down-increase-performance.png I've done a series of stress tests with eager retries enabled that show undesirable behavior. I'm grouping these behaviours into one ticket as they are most likely related. 1) Killing off a node in a 4 node cluster actually increases performance. 2) Compactions make nodes slow, even after the compaction is done. 3) Eager Reads tend to lessen the *immediate* performance impact of a node going down, but not consistently. My Environment: 1 stress machine: node0 4 C* nodes: node4, node5, node6, node7 My script: node0 writes some data: stress -d node4 -F 3000 -n 3000 -i 5 -l 2 -K 20 node0 reads some data: stress -d node4 -n 3000 -o read -i 5 -K 20 h3. Examples: h5. A node going down increases performance: !node-down-increase-performance.png! [Data for this test here|http://ryanmcguire.info/ds/graph/graph.html?stats=stats.eager_retry.node_killed.just_20.jsonmetric=interval_op_rateoperation=stress-readsmoothing=1] At 450s, I kill -9 one of the nodes. There is a brief decrease in performance as the snitch adapts, but then it recovers... to even higher performance than before. h5. Compactions make nodes permanently slow: !compaction-makes-slow.png! !compaction-makes-slow-stats.png! The green and orange lines represent trials with eager retry enabled, they never recover their op-rate from before the compaction as the red and blue lines do. [Data for this test here|http://ryanmcguire.info/ds/graph/graph.html?stats=stats.eager_retry.compaction.2.jsonmetric=interval_op_rateoperation=stress-readsmoothing=1] h5. Speculative Read tends to lessen the *immediate* impact: !eager-read-looks-promising.png! !eager-read-looks-promising-stats.png! This graph looked the most promising to me, the two trials with eager retry, the green and orange line, at 450s showed the smallest dip in performance. [Data for this test here|http://ryanmcguire.info/ds/graph/graph.html?stats=stats.eager_retry.node_killed.jsonmetric=interval_op_rateoperation=stress-readsmoothing=1] h5. But not always: !eager-read-not-consistent.png! !eager-read-not-consistent-stats.png! This is a retrial with the same settings as above, yet the 95percentile eager retry (red line) did poorly this time at 450s. [Data for this test here|http://ryanmcguire.info/ds/graph/graph.html?stats=stats.eager_retry.node_killed.just_20.rc1.try2.jsonmetric=interval_op_rateoperation=stress-readsmoothing=1] -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (CASSANDRA-5932) Speculative read performance data show unexpected results
[ https://issues.apache.org/jira/browse/CASSANDRA-5932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13786622#comment-13786622 ] Li Zou commented on CASSANDRA-5932: --- As for yesterday's trunk load, there were two addCallback() methods. But the one with ConsistencyLevel was not called by anyone. The one without ConsistencyLevel asserts. Speculative read performance data show unexpected results - Key: CASSANDRA-5932 URL: https://issues.apache.org/jira/browse/CASSANDRA-5932 Project: Cassandra Issue Type: Bug Reporter: Ryan McGuire Assignee: Aleksey Yeschenko Fix For: 2.0.2 Attachments: 5932.6692c50412ef7d.compaction.png, 5932-6692c50412ef7d.png, 5932.6692c50412ef7d.rr0.png, 5932.6692c50412ef7d.rr1.png, 5932.ded39c7e1c2fa.logs.tar.gz, 5932.txt, 5933-128_and_200rc1.png, 5933-7a87fc11.png, 5933-logs.tar.gz, 5933-randomized-dsnitch-replica.2.png, 5933-randomized-dsnitch-replica.3.png, 5933-randomized-dsnitch-replica.png, compaction-makes-slow.png, compaction-makes-slow-stats.png, eager-read-looks-promising.png, eager-read-looks-promising-stats.png, eager-read-not-consistent.png, eager-read-not-consistent-stats.png, node-down-increase-performance.png I've done a series of stress tests with eager retries enabled that show undesirable behavior. I'm grouping these behaviours into one ticket as they are most likely related. 1) Killing off a node in a 4 node cluster actually increases performance. 2) Compactions make nodes slow, even after the compaction is done. 3) Eager Reads tend to lessen the *immediate* performance impact of a node going down, but not consistently. My Environment: 1 stress machine: node0 4 C* nodes: node4, node5, node6, node7 My script: node0 writes some data: stress -d node4 -F 3000 -n 3000 -i 5 -l 2 -K 20 node0 reads some data: stress -d node4 -n 3000 -o read -i 5 -K 20 h3. Examples: h5. A node going down increases performance: !node-down-increase-performance.png! [Data for this test here|http://ryanmcguire.info/ds/graph/graph.html?stats=stats.eager_retry.node_killed.just_20.jsonmetric=interval_op_rateoperation=stress-readsmoothing=1] At 450s, I kill -9 one of the nodes. There is a brief decrease in performance as the snitch adapts, but then it recovers... to even higher performance than before. h5. Compactions make nodes permanently slow: !compaction-makes-slow.png! !compaction-makes-slow-stats.png! The green and orange lines represent trials with eager retry enabled, they never recover their op-rate from before the compaction as the red and blue lines do. [Data for this test here|http://ryanmcguire.info/ds/graph/graph.html?stats=stats.eager_retry.compaction.2.jsonmetric=interval_op_rateoperation=stress-readsmoothing=1] h5. Speculative Read tends to lessen the *immediate* impact: !eager-read-looks-promising.png! !eager-read-looks-promising-stats.png! This graph looked the most promising to me, the two trials with eager retry, the green and orange line, at 450s showed the smallest dip in performance. [Data for this test here|http://ryanmcguire.info/ds/graph/graph.html?stats=stats.eager_retry.node_killed.jsonmetric=interval_op_rateoperation=stress-readsmoothing=1] h5. But not always: !eager-read-not-consistent.png! !eager-read-not-consistent-stats.png! This is a retrial with the same settings as above, yet the 95percentile eager retry (red line) did poorly this time at 450s. [Data for this test here|http://ryanmcguire.info/ds/graph/graph.html?stats=stats.eager_retry.node_killed.just_20.rc1.try2.jsonmetric=interval_op_rateoperation=stress-readsmoothing=1] -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Comment Edited] (CASSANDRA-5932) Speculative read performance data show unexpected results
[ https://issues.apache.org/jira/browse/CASSANDRA-5932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13786613#comment-13786613 ] Li Zou edited comment on CASSANDRA-5932 at 10/4/13 9:10 PM: The trunk load I used for testing was pulled this noon. It has two addCallback() methods. One of them (i.e. without the ConsistencyLevel) asserts. I checked the MessagingService.java, there are two addCallback() methods. * The one without ConsistencyLevel is called by sendRR() * The one with ConsistencyLevel is called by sendMessageToNonLocalDC() was (Author: lizou): I just did a git pull after your comment (i.e. a couple of seconds ago) for trunk. The git pull indicates that Already up-to-date. I checked the MessagingService.java, there are two addCallback() methods. * The one without ConsistencyLevel is called by sendRR() * The one with ConsistencyLevel is called by sendMessageToNonLocalDC() The trunk load I used for testing was pulled this noon. It has two addCallback() methods. One of them (i.e. without the ConsistencyLevel) asserts. Speculative read performance data show unexpected results - Key: CASSANDRA-5932 URL: https://issues.apache.org/jira/browse/CASSANDRA-5932 Project: Cassandra Issue Type: Bug Reporter: Ryan McGuire Assignee: Aleksey Yeschenko Fix For: 2.0.2 Attachments: 5932.6692c50412ef7d.compaction.png, 5932-6692c50412ef7d.png, 5932.6692c50412ef7d.rr0.png, 5932.6692c50412ef7d.rr1.png, 5932.ded39c7e1c2fa.logs.tar.gz, 5932.txt, 5933-128_and_200rc1.png, 5933-7a87fc11.png, 5933-logs.tar.gz, 5933-randomized-dsnitch-replica.2.png, 5933-randomized-dsnitch-replica.3.png, 5933-randomized-dsnitch-replica.png, compaction-makes-slow.png, compaction-makes-slow-stats.png, eager-read-looks-promising.png, eager-read-looks-promising-stats.png, eager-read-not-consistent.png, eager-read-not-consistent-stats.png, node-down-increase-performance.png I've done a series of stress tests with eager retries enabled that show undesirable behavior. I'm grouping these behaviours into one ticket as they are most likely related. 1) Killing off a node in a 4 node cluster actually increases performance. 2) Compactions make nodes slow, even after the compaction is done. 3) Eager Reads tend to lessen the *immediate* performance impact of a node going down, but not consistently. My Environment: 1 stress machine: node0 4 C* nodes: node4, node5, node6, node7 My script: node0 writes some data: stress -d node4 -F 3000 -n 3000 -i 5 -l 2 -K 20 node0 reads some data: stress -d node4 -n 3000 -o read -i 5 -K 20 h3. Examples: h5. A node going down increases performance: !node-down-increase-performance.png! [Data for this test here|http://ryanmcguire.info/ds/graph/graph.html?stats=stats.eager_retry.node_killed.just_20.jsonmetric=interval_op_rateoperation=stress-readsmoothing=1] At 450s, I kill -9 one of the nodes. There is a brief decrease in performance as the snitch adapts, but then it recovers... to even higher performance than before. h5. Compactions make nodes permanently slow: !compaction-makes-slow.png! !compaction-makes-slow-stats.png! The green and orange lines represent trials with eager retry enabled, they never recover their op-rate from before the compaction as the red and blue lines do. [Data for this test here|http://ryanmcguire.info/ds/graph/graph.html?stats=stats.eager_retry.compaction.2.jsonmetric=interval_op_rateoperation=stress-readsmoothing=1] h5. Speculative Read tends to lessen the *immediate* impact: !eager-read-looks-promising.png! !eager-read-looks-promising-stats.png! This graph looked the most promising to me, the two trials with eager retry, the green and orange line, at 450s showed the smallest dip in performance. [Data for this test here|http://ryanmcguire.info/ds/graph/graph.html?stats=stats.eager_retry.node_killed.jsonmetric=interval_op_rateoperation=stress-readsmoothing=1] h5. But not always: !eager-read-not-consistent.png! !eager-read-not-consistent-stats.png! This is a retrial with the same settings as above, yet the 95percentile eager retry (red line) did poorly this time at 450s. [Data for this test here|http://ryanmcguire.info/ds/graph/graph.html?stats=stats.eager_retry.node_killed.just_20.rc1.try2.jsonmetric=interval_op_rateoperation=stress-readsmoothing=1] -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (CASSANDRA-5932) Speculative read performance data show unexpected results
[
https://issues.apache.org/jira/browse/CASSANDRA-5932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13784089#comment-13784089
]
Li Zou commented on CASSANDRA-5932:
---
This testing result is reasonable and what is expected.
For PERCENTILE / CUSTOM configuration, the larger the
{{cfs.sampleLatencyNanos}} the smaller the throughput impact for normal
operations before the outage. However, during the outage period, the situation
is reversed, i.e. the smaller {{cfs.sampleLatencyNanos}}, the smaller the
throughput impact will be, as it times out quicker and triggers the speculative
retries.
For the ALWAYS configuration, as it always sends out one speculative in
addition to the usual read requests, the throughput performance should be lower
than those of PERCENTILE / CUSTOM for normal operations before the outage.
Since it always sends out the speculative retries, the throughput impact during
the outage period should be the smallest. The testing result indicates that
this is true.
Speculative read performance data show unexpected results
-
Key: CASSANDRA-5932
URL: https://issues.apache.org/jira/browse/CASSANDRA-5932
Project: Cassandra
Issue Type: Bug
Reporter: Ryan McGuire
Assignee: Aleksey Yeschenko
Fix For: 2.0.2
Attachments: 5932.6692c50412ef7d.compaction.png,
5932-6692c50412ef7d.png, 5932.6692c50412ef7d.rr0.png,
5932.ded39c7e1c2fa.logs.tar.gz, 5932.txt, 5933-128_and_200rc1.png,
5933-7a87fc11.png, 5933-logs.tar.gz, 5933-randomized-dsnitch-replica.2.png,
5933-randomized-dsnitch-replica.3.png, 5933-randomized-dsnitch-replica.png,
compaction-makes-slow.png, compaction-makes-slow-stats.png,
eager-read-looks-promising.png, eager-read-looks-promising-stats.png,
eager-read-not-consistent.png, eager-read-not-consistent-stats.png,
node-down-increase-performance.png
I've done a series of stress tests with eager retries enabled that show
undesirable behavior. I'm grouping these behaviours into one ticket as they
are most likely related.
1) Killing off a node in a 4 node cluster actually increases performance.
2) Compactions make nodes slow, even after the compaction is done.
3) Eager Reads tend to lessen the *immediate* performance impact of a node
going down, but not consistently.
My Environment:
1 stress machine: node0
4 C* nodes: node4, node5, node6, node7
My script:
node0 writes some data: stress -d node4 -F 3000 -n 3000 -i 5 -l 2 -K
20
node0 reads some data: stress -d node4 -n 3000 -o read -i 5 -K 20
h3. Examples:
h5. A node going down increases performance:
!node-down-increase-performance.png!
[Data for this test
here|http://ryanmcguire.info/ds/graph/graph.html?stats=stats.eager_retry.node_killed.just_20.jsonmetric=interval_op_rateoperation=stress-readsmoothing=1]
At 450s, I kill -9 one of the nodes. There is a brief decrease in performance
as the snitch adapts, but then it recovers... to even higher performance than
before.
h5. Compactions make nodes permanently slow:
!compaction-makes-slow.png!
!compaction-makes-slow-stats.png!
The green and orange lines represent trials with eager retry enabled, they
never recover their op-rate from before the compaction as the red and blue
lines do.
[Data for this test
here|http://ryanmcguire.info/ds/graph/graph.html?stats=stats.eager_retry.compaction.2.jsonmetric=interval_op_rateoperation=stress-readsmoothing=1]
h5. Speculative Read tends to lessen the *immediate* impact:
!eager-read-looks-promising.png!
!eager-read-looks-promising-stats.png!
This graph looked the most promising to me, the two trials with eager retry,
the green and orange line, at 450s showed the smallest dip in performance.
[Data for this test
here|http://ryanmcguire.info/ds/graph/graph.html?stats=stats.eager_retry.node_killed.jsonmetric=interval_op_rateoperation=stress-readsmoothing=1]
h5. But not always:
!eager-read-not-consistent.png!
!eager-read-not-consistent-stats.png!
This is a retrial with the same settings as above, yet the 95percentile eager
retry (red line) did poorly this time at 450s.
[Data for this test
here|http://ryanmcguire.info/ds/graph/graph.html?stats=stats.eager_retry.node_killed.just_20.rc1.try2.jsonmetric=interval_op_rateoperation=stress-readsmoothing=1]
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (CASSANDRA-5932) Speculative read performance data show unexpected results
[
https://issues.apache.org/jira/browse/CASSANDRA-5932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13782027#comment-13782027
]
Li Zou commented on CASSANDRA-5932:
---
Hello [~iamaleksey] and [~jbellis],
It appears to me that the testing results have suggested that the _data read +
speculative retry_ path work as expected. This _data read + speculative
retry_ path has greatly minimized the throughput impact caused by the failure
of one of Cassandra server nodes.
The observed small degradation of throughput performance when _speculative
retry_ is enabled is very likely to be caused by the *_read repair_* path. I
did the code reading of this path last Friday and noticed some design / coding
issues. I would like to discuss them with you.
Please note that my code base is still the Cassandra 2.0.0 tarball, not updated
with the latest code changes.
*Issue 1* -- When handling {{DigestMismatchException}} in
{{StorageProxy.fetchRows()}}, all _data read requests_ are sent out using
{{sendRR}} without distinguishing remote nodes from the local node.
Will this cause an issue, as {{MessagingService.instance().sendRR()}} will send
out enqueued messages for a specified remote node via its pre-established TCP
socket connection. For local node, this should be done via
{{LocalReadRunnable}}, i.e. {{StageManager.getStage(Stage.READ).execute(new
LocalReadRunnable(command, handler))}}.
If this may cause an issue, the following wait may block.
{noformat}
// read the results for the digest mismatch retries
if (repairResponseHandlers != null)
{
for (int i = 0; i repairCommands.size(); i++)
{
ReadCommand command = repairCommands.get(i);
ReadCallbackReadResponse, Row handler =
repairResponseHandlers.get(i);
Row row;
try
{
row = handler.get();
}
{noformat}
For two reasons.
* The data read request for local node may never sent out
* As one of the nodes is down (which triggered the Speculative Retry) will
cause one missing response.
*If missing two responses, this will block for 10 seconds*.
*Issue 2* -- For _data repair_, {{RowDataResolver.resolve()}} has a similar
issue as it calls {{scheduleRepairs()}} to send out messages using sendRR()
without distinguishing remote nodes from the local node.
*Issue 3* -- When handling _data repair_, {{StorageProxy.fetchRows()}} blocks
waiting for acks to all of {{data repair}} requests sent out using sendRR().
This may cause the thread to block.
For _data repair_ path, *data requests* are sent out and then compare / merge
the received responses; send out the merged / diff version and then block for
acks.
How do we handle the case for _local node_? Does the sendRR() and the
corresponding receive part can handle the case for local node? If not, then
this may block for 10 seconds.
{noformat}
if (repairResponseHandlers != null)
{
for (int i = 0; i repairCommands.size(); i++)
{
ReadCommand command = repairCommands.get(i);
ReadCallbackReadResponse, Row handler =
repairResponseHandlers.get(i);
Row row;
try
{
row = handler.get();
}
catch (DigestMismatchException e)
...
RowDataResolver resolver =
(RowDataResolver)handler.resolver;
try
{
// wait for the repair writes to be acknowledged, to
minimize impact on any replica that's
// behind on writes in case the out-of-sync row is read
multiple times in quick succession
FBUtilities.waitOnFutures(resolver.repairResults,
DatabaseDescriptor.getWriteRpcTimeout());
}
catch (TimeoutException e)
{
Tracing.trace(Timed out on digest mismatch retries);
int blockFor =
consistency_level.blockFor(Keyspace.open(command.getKeyspace()));
throw new ReadTimeoutException(consistency_level,
blockFor, blockFor, true);
}
{noformat}
*Question for waiting for the ack* -- Do we really need to wait for the ack?
We should assume the best effort approach, i.e. do the data repair and then
return. No need to block waiting for the acks for confirmation.
*Question for the Randomized approach* -- Since the end points are randomized,
the first node in the list is no likely the local node. This may cause a higher
possibility of data repair.
In the *Randomized Approach*, the end points are reshuffled. Then, the first
node in the list
[jira] [Comment Edited] (CASSANDRA-5932) Speculative read performance data show unexpected results
[
https://issues.apache.org/jira/browse/CASSANDRA-5932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13782027#comment-13782027
]
Li Zou edited comment on CASSANDRA-5932 at 9/30/13 6:04 PM:
Hello [~iamaleksey] and [~jbellis],
It appears to me that the testing results have suggested that the _data read +
speculative retry_ path work as expected. This _data read + speculative
retry_ path has greatly minimized the throughput impact caused by the failure
of one of Cassandra server nodes.
The observed small degradation of throughput performance when _speculative
retry_ is enabled is very likely to be caused by the *_read repair_* path. I
did the code reading of this path last Friday and noticed some design / coding
issues. I would like to discuss them with you.
Please note that my code base is still the Cassandra 2.0.0 tarball, not updated
with the latest code changes.
*Issue 1* -- When handling {{DigestMismatchException}} in
{{StorageProxy.fetchRows()}}, all _data read requests_ are sent out using
{{sendRR}} without distinguishing remote nodes from the local node.
Will this cause an issue, as {{MessagingService.instance().sendRR()}} will send
out enqueued messages for a specified remote node via its pre-established TCP
socket connection. For local node, this should be done via
{{LocalReadRunnable}}, i.e. {{StageManager.getStage(Stage.READ).execute(new
LocalReadRunnable(command, handler))}}.
If this may cause an issue, the following wait may block.
{noformat}
// read the results for the digest mismatch retries
if (repairResponseHandlers != null)
{
for (int i = 0; i repairCommands.size(); i++)
{
ReadCommand command = repairCommands.get(i);
ReadCallbackReadResponse, Row handler =
repairResponseHandlers.get(i);
Row row;
try
{
row = handler.get();
}
{noformat}
For two reasons.
* The data read request for local node may never sent out
* As one of the nodes is down (which triggered the Speculative Retry) will
cause one missing response.
*If missing two responses, this will block for 10 seconds*.
*Issue 2* -- For _data repair_, {{RowDataResolver.resolve()}} has a similar
issue as it calls {{scheduleRepairs()}} to send out messages using sendRR()
without distinguishing remote nodes from the local node.
*Issue 3* -- When handling _data repair_, {{StorageProxy.fetchRows()}} blocks
waiting for acks to all of {{data repair}} requests sent out using sendRR().
This may cause the thread to block.
For _data repair_ path, *data requests* are sent out and then compare / merge
the received responses; send out the merged / diff version and then block for
acks.
How do we handle the case for _local node_? Does the sendRR() and the
corresponding receive part can handle the case for local node? If not, then
this may block for 10 seconds.
{noformat}
if (repairResponseHandlers != null)
{
for (int i = 0; i repairCommands.size(); i++)
{
ReadCommand command = repairCommands.get(i);
ReadCallbackReadResponse, Row handler =
repairResponseHandlers.get(i);
Row row;
try
{
row = handler.get();
}
catch (DigestMismatchException e)
...
RowDataResolver resolver =
(RowDataResolver)handler.resolver;
try
{
// wait for the repair writes to be acknowledged, to
minimize impact on any replica that's
// behind on writes in case the out-of-sync row is read
multiple times in quick succession
FBUtilities.waitOnFutures(resolver.repairResults,
DatabaseDescriptor.getWriteRpcTimeout());
}
catch (TimeoutException e)
{
Tracing.trace(Timed out on digest mismatch retries);
int blockFor =
consistency_level.blockFor(Keyspace.open(command.getKeyspace()));
throw new ReadTimeoutException(consistency_level,
blockFor, blockFor, true);
}
{noformat}
*Question for waiting for the ack* -- Do we really need to wait for the ack?
We should assume the best effort approach, i.e. do the data repair and then
return. No need to block waiting for the acks for confirmation.
*Question for the Randomized approach* -- Since the end points are randomized,
the first node in the list is no likely the local node. This may cause a higher
possibility of data repair.
In the *Randomized Approach*, the end points
[jira] [Commented] (CASSANDRA-5932) Speculative read performance data show unexpected results
[ https://issues.apache.org/jira/browse/CASSANDRA-5932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13782280#comment-13782280 ] Li Zou commented on CASSANDRA-5932: --- Thanks for the clarification of the sendRR issue. Since the Randomized approach is not checked in, let us skip over it. For the _data repair_, do we need to block waiting for the acks? Speculative read performance data show unexpected results - Key: CASSANDRA-5932 URL: https://issues.apache.org/jira/browse/CASSANDRA-5932 Project: Cassandra Issue Type: Bug Reporter: Ryan McGuire Assignee: Aleksey Yeschenko Fix For: 2.0.2 Attachments: 5932-6692c50412ef7d.png, 5932.ded39c7e1c2fa.logs.tar.gz, 5932.txt, 5933-128_and_200rc1.png, 5933-7a87fc11.png, 5933-logs.tar.gz, 5933-randomized-dsnitch-replica.2.png, 5933-randomized-dsnitch-replica.3.png, 5933-randomized-dsnitch-replica.png, compaction-makes-slow.png, compaction-makes-slow-stats.png, eager-read-looks-promising.png, eager-read-looks-promising-stats.png, eager-read-not-consistent.png, eager-read-not-consistent-stats.png, node-down-increase-performance.png I've done a series of stress tests with eager retries enabled that show undesirable behavior. I'm grouping these behaviours into one ticket as they are most likely related. 1) Killing off a node in a 4 node cluster actually increases performance. 2) Compactions make nodes slow, even after the compaction is done. 3) Eager Reads tend to lessen the *immediate* performance impact of a node going down, but not consistently. My Environment: 1 stress machine: node0 4 C* nodes: node4, node5, node6, node7 My script: node0 writes some data: stress -d node4 -F 3000 -n 3000 -i 5 -l 2 -K 20 node0 reads some data: stress -d node4 -n 3000 -o read -i 5 -K 20 h3. Examples: h5. A node going down increases performance: !node-down-increase-performance.png! [Data for this test here|http://ryanmcguire.info/ds/graph/graph.html?stats=stats.eager_retry.node_killed.just_20.jsonmetric=interval_op_rateoperation=stress-readsmoothing=1] At 450s, I kill -9 one of the nodes. There is a brief decrease in performance as the snitch adapts, but then it recovers... to even higher performance than before. h5. Compactions make nodes permanently slow: !compaction-makes-slow.png! !compaction-makes-slow-stats.png! The green and orange lines represent trials with eager retry enabled, they never recover their op-rate from before the compaction as the red and blue lines do. [Data for this test here|http://ryanmcguire.info/ds/graph/graph.html?stats=stats.eager_retry.compaction.2.jsonmetric=interval_op_rateoperation=stress-readsmoothing=1] h5. Speculative Read tends to lessen the *immediate* impact: !eager-read-looks-promising.png! !eager-read-looks-promising-stats.png! This graph looked the most promising to me, the two trials with eager retry, the green and orange line, at 450s showed the smallest dip in performance. [Data for this test here|http://ryanmcguire.info/ds/graph/graph.html?stats=stats.eager_retry.node_killed.jsonmetric=interval_op_rateoperation=stress-readsmoothing=1] h5. But not always: !eager-read-not-consistent.png! !eager-read-not-consistent-stats.png! This is a retrial with the same settings as above, yet the 95percentile eager retry (red line) did poorly this time at 450s. [Data for this test here|http://ryanmcguire.info/ds/graph/graph.html?stats=stats.eager_retry.node_killed.just_20.rc1.try2.jsonmetric=interval_op_rateoperation=stress-readsmoothing=1] -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (CASSANDRA-5932) Speculative read performance data show unexpected results
[
https://issues.apache.org/jira/browse/CASSANDRA-5932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13779202#comment-13779202
]
Li Zou commented on CASSANDRA-5932:
---
Hello [~iamaleksey] and [~jbellis],
I took a quick look at the code changes. The new code looks very good to me.
But I saw one potential issue in
{{AlwaysSpeculatingReadExecutor.executeAsync()}}, in which it makes at least
*two* data / digest requests. This will cause problems for a data center with
only one Cassandra server node (e.g. bring up an embedded Cassandra node in JVM
for JUnit test) or a deployed production data center of two Cassandra server
nodes with one node shut down for maintenance. In the above mentioned two
cases, {{AbstractReadExecutor.getReadExecutor()}} will return the
{{AlwaysSpeculatingReadExecutor}} as condition {{(targetReplicas.size() ==
allReplicas.size())}} is met, though the tables may / may not be configured
with ??Speculative ALWAYS??.
It is true for our legacy products we are considering to deploy each data
center with only two Cassandra server nodes with RF = 2 and CL = 1.
Speculative read performance data show unexpected results
-
Key: CASSANDRA-5932
URL: https://issues.apache.org/jira/browse/CASSANDRA-5932
Project: Cassandra
Issue Type: Bug
Reporter: Ryan McGuire
Assignee: Aleksey Yeschenko
Fix For: 2.0.2
Attachments: 5932.txt, compaction-makes-slow.png,
compaction-makes-slow-stats.png, eager-read-looks-promising.png,
eager-read-looks-promising-stats.png, eager-read-not-consistent.png,
eager-read-not-consistent-stats.png, node-down-increase-performance.png
I've done a series of stress tests with eager retries enabled that show
undesirable behavior. I'm grouping these behaviours into one ticket as they
are most likely related.
1) Killing off a node in a 4 node cluster actually increases performance.
2) Compactions make nodes slow, even after the compaction is done.
3) Eager Reads tend to lessen the *immediate* performance impact of a node
going down, but not consistently.
My Environment:
1 stress machine: node0
4 C* nodes: node4, node5, node6, node7
My script:
node0 writes some data: stress -d node4 -F 3000 -n 3000 -i 5 -l 2 -K
20
node0 reads some data: stress -d node4 -n 3000 -o read -i 5 -K 20
h3. Examples:
h5. A node going down increases performance:
!node-down-increase-performance.png!
[Data for this test
here|http://ryanmcguire.info/ds/graph/graph.html?stats=stats.eager_retry.node_killed.just_20.jsonmetric=interval_op_rateoperation=stress-readsmoothing=1]
At 450s, I kill -9 one of the nodes. There is a brief decrease in performance
as the snitch adapts, but then it recovers... to even higher performance than
before.
h5. Compactions make nodes permanently slow:
!compaction-makes-slow.png!
!compaction-makes-slow-stats.png!
The green and orange lines represent trials with eager retry enabled, they
never recover their op-rate from before the compaction as the red and blue
lines do.
[Data for this test
here|http://ryanmcguire.info/ds/graph/graph.html?stats=stats.eager_retry.compaction.2.jsonmetric=interval_op_rateoperation=stress-readsmoothing=1]
h5. Speculative Read tends to lessen the *immediate* impact:
!eager-read-looks-promising.png!
!eager-read-looks-promising-stats.png!
This graph looked the most promising to me, the two trials with eager retry,
the green and orange line, at 450s showed the smallest dip in performance.
[Data for this test
here|http://ryanmcguire.info/ds/graph/graph.html?stats=stats.eager_retry.node_killed.jsonmetric=interval_op_rateoperation=stress-readsmoothing=1]
h5. But not always:
!eager-read-not-consistent.png!
!eager-read-not-consistent-stats.png!
This is a retrial with the same settings as above, yet the 95percentile eager
retry (red line) did poorly this time at 450s.
[Data for this test
here|http://ryanmcguire.info/ds/graph/graph.html?stats=stats.eager_retry.node_killed.just_20.rc1.try2.jsonmetric=interval_op_rateoperation=stress-readsmoothing=1]
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-5932) Speculative read performance data show unexpected results
[
https://issues.apache.org/jira/browse/CASSANDRA-5932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13779233#comment-13779233
]
Li Zou commented on CASSANDRA-5932:
---
The logic for {{AlwaysSpeculatingReadExecutor}} is good. What I meant in my
previous comment is that when {{targetReplicas.size() == allReplicas.size()}}
and {{targetReplicas.size() == 1}}, then
{{AlwaysSpeculatingReadExecutor.executeAsync()}} will throw an exception as
there is only one endpoint in {{targetReplicas}}, but it tries to access two
endpoints in {{targetReplicas}}.
Speculative read performance data show unexpected results
-
Key: CASSANDRA-5932
URL: https://issues.apache.org/jira/browse/CASSANDRA-5932
Project: Cassandra
Issue Type: Bug
Reporter: Ryan McGuire
Assignee: Aleksey Yeschenko
Fix For: 2.0.2
Attachments: 5932.txt, compaction-makes-slow.png,
compaction-makes-slow-stats.png, eager-read-looks-promising.png,
eager-read-looks-promising-stats.png, eager-read-not-consistent.png,
eager-read-not-consistent-stats.png, node-down-increase-performance.png
I've done a series of stress tests with eager retries enabled that show
undesirable behavior. I'm grouping these behaviours into one ticket as they
are most likely related.
1) Killing off a node in a 4 node cluster actually increases performance.
2) Compactions make nodes slow, even after the compaction is done.
3) Eager Reads tend to lessen the *immediate* performance impact of a node
going down, but not consistently.
My Environment:
1 stress machine: node0
4 C* nodes: node4, node5, node6, node7
My script:
node0 writes some data: stress -d node4 -F 3000 -n 3000 -i 5 -l 2 -K
20
node0 reads some data: stress -d node4 -n 3000 -o read -i 5 -K 20
h3. Examples:
h5. A node going down increases performance:
!node-down-increase-performance.png!
[Data for this test
here|http://ryanmcguire.info/ds/graph/graph.html?stats=stats.eager_retry.node_killed.just_20.jsonmetric=interval_op_rateoperation=stress-readsmoothing=1]
At 450s, I kill -9 one of the nodes. There is a brief decrease in performance
as the snitch adapts, but then it recovers... to even higher performance than
before.
h5. Compactions make nodes permanently slow:
!compaction-makes-slow.png!
!compaction-makes-slow-stats.png!
The green and orange lines represent trials with eager retry enabled, they
never recover their op-rate from before the compaction as the red and blue
lines do.
[Data for this test
here|http://ryanmcguire.info/ds/graph/graph.html?stats=stats.eager_retry.compaction.2.jsonmetric=interval_op_rateoperation=stress-readsmoothing=1]
h5. Speculative Read tends to lessen the *immediate* impact:
!eager-read-looks-promising.png!
!eager-read-looks-promising-stats.png!
This graph looked the most promising to me, the two trials with eager retry,
the green and orange line, at 450s showed the smallest dip in performance.
[Data for this test
here|http://ryanmcguire.info/ds/graph/graph.html?stats=stats.eager_retry.node_killed.jsonmetric=interval_op_rateoperation=stress-readsmoothing=1]
h5. But not always:
!eager-read-not-consistent.png!
!eager-read-not-consistent-stats.png!
This is a retrial with the same settings as above, yet the 95percentile eager
retry (red line) did poorly this time at 450s.
[Data for this test
here|http://ryanmcguire.info/ds/graph/graph.html?stats=stats.eager_retry.node_killed.just_20.rc1.try2.jsonmetric=interval_op_rateoperation=stress-readsmoothing=1]
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4705) Speculative execution for reads / eager retries
[ https://issues.apache.org/jira/browse/CASSANDRA-4705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13776674#comment-13776674 ] Li Zou commented on CASSANDRA-4705: --- [~iamaleksey] please see my comments at [CASSANDRA-5932|https://issues.apache.org/jira/browse/CASSANDRA-5932] Speculative execution for reads / eager retries --- Key: CASSANDRA-4705 URL: https://issues.apache.org/jira/browse/CASSANDRA-4705 Project: Cassandra Issue Type: Improvement Reporter: Vijay Assignee: Vijay Fix For: 2.0 beta 1 Attachments: 0001-CASSANDRA-4705.patch, 0001-CASSANDRA-4705-v2.patch, 0001-CASSANDRA-4705-v3.patch, 0001-Refactor-to-introduce-AbstractReadExecutor.patch, 0002-Add-Speculative-execution-for-Row-Reads.patch When read_repair is not 1.0, we send the request to one node for some of the requests. When a node goes down or when a node is too busy the client has to wait for the timeout before it can retry. It would be nice to watch for latency and execute an additional request to a different node, if the response is not received within average/99% of the response times recorded in the past. CASSANDRA-2540 might be able to solve the variance when read_repair is set to 1.0 1) May be we need to use metrics-core to record various Percentiles 2) Modify ReadCallback.get to execute additional request speculatively. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4705) Speculative execution for reads / eager retries
[ https://issues.apache.org/jira/browse/CASSANDRA-4705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13764863#comment-13764863 ] Li Zou commented on CASSANDRA-4705: --- Hello [~vijay2...@yahoo.com] and [~jbellis], My company is adopting the Cassandra db technology and we are very much interested in this new feature of Speculative Execution for Reads, as it could potentially help reduce the overall outage window to the sub-second range upon the failure of any one of Cassandra nodes in a data center. I have recently tested the Speculative Execution for Reads using Cassandra 2.0.0-rc2 and I have not seen the expected results. In my tests, I have already excluded the possible effects from the client (connection pool) side. Not sure what is still missed out in my tests. Here is my setup for the tests. One data center of four Cassandra nodes is configured on machine A, B, C and D with each physical machine has one Cassandra node running on it. My testing app (Cassandra client) is running on machine A and it is configured to connect to Cassandra nodes A, B, and C only. In other words, my testing app will never connect to Cassandra node D. * Replication Factor (RF) is set to 3 * Client requested Consistency Level (CL) is set to CL_TWO For Cassandra 1.2.4, upon the failure of Cassandra node D (via kill -9 pid or kill -s SIGKILL pid), there will be an outage window for around 20 seconds with zero transactions. The outage ends as soon as the gossip protocol detects the failure of Cassandra node D and marks it down. For Cassandra 2.0.0-rc2, database tables are configured with speculative_retry proper values (such as 'ALWAYS', '10 ms', '100 ms', '80 percentile', 'NONE'). No expected results are observed in the tests. The testing results are quite similar to those observed for Cassandra 1.2.4. That is, upon the failure of Cassandra node D, there will be an outage window for around 20 seconds with zero transactions. The outage ends as soon as the gossip protocol detects the failure of Cassandra node D and marks it down. The tested values for _speculative_retry_ against database table are listed below: * ALWAYS * 10 ms * 100 ms * 80 percentile * NONE In my tests, I also checked the JMX stats of Cassandra node A for the speculative_retry for each table. What I observed really surprised me. For an instance, the speculative_retry (for each table) is set to '20 ms'. During normal operations with all of four Cassandra nodes up, the SpeculativeRetry count went up occasionally. However, during the 20-second outage window, the SpeculativeRetry count somehow never went up for three tests in a row. On the contrary, I would expect to see lots of speculative retries were executed (say on Cassandra node A) during the 20-second outage window. Some notes for the kill signals. I tested if the Cassandra node D is killed using SIGTERM (for Cassandra 1.2.4) and SIGINT / SIGTERM (for Cassandra 2.0.0-rc2), there will be no observed outage in second range, as the Cassandra node D does the orderly shutdown and the gossip announces the node down. From my testing app point of view, everything is going very well. But if the Cassandra node D is killed using SIGKILL, there will be an outage with zero transactions. I guess this might be a TCP socket related issue. I often observed TCP socket CLOSE_WAIT on the passive close side (i.e. Cassandra node A, B, and C) and TCP socket TIME_WAIT on the active close side (Cassandra node D). Speculative execution for reads / eager retries --- Key: CASSANDRA-4705 URL: https://issues.apache.org/jira/browse/CASSANDRA-4705 Project: Cassandra Issue Type: Improvement Reporter: Vijay Assignee: Vijay Fix For: 2.0 beta 1 Attachments: 0001-CASSANDRA-4705.patch, 0001-CASSANDRA-4705-v2.patch, 0001-CASSANDRA-4705-v3.patch, 0001-Refactor-to-introduce-AbstractReadExecutor.patch, 0002-Add-Speculative-execution-for-Row-Reads.patch When read_repair is not 1.0, we send the request to one node for some of the requests. When a node goes down or when a node is too busy the client has to wait for the timeout before it can retry. It would be nice to watch for latency and execute an additional request to a different node, if the response is not received within average/99% of the response times recorded in the past. CASSANDRA-2540 might be able to solve the variance when read_repair is set to 1.0 1) May be we need to use metrics-core to record various Percentiles 2) Modify ReadCallback.get to execute additional request speculatively. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see:
