from:"\"Li Zou \\\(JIRA\\\)\""

[jira] [Updated] (CASSANDRA-6187) Cassandra-2.0 node seems to be pausing periodically under normal operations

2013-11-06 Thread Li Zou (JIRA)


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-6187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Li Zou updated CASSANDRA-6187:
--

Description: 
Under normal operation condition without any interruption, the traffic 
generator  will periodically see a couple of seconds of zero (or very little) 
transactions, i.e. the throughput can be quite high for a while, then comes 
down to zero (or very little) transactions for 10 ~ 20 seconds.

The Cassandra system log occasionally also logs message drops. But these 
message drop events do not correlate in time to the observed transaction drop 
events.

Example of message dropping log:

{noformat}
 INFO [ScheduledTasks:1] 2013-10-11 16:36:12,216 MessagingService.java (line 
812) 1191 MUTATION messages dropped in last 5000ms
 INFO [ScheduledTasks:1] 2013-10-11 16:36:12,217 MessagingService.java (line 
812) 502 READ_REPAIR messages dropped in last 5000ms
 INFO [ScheduledTasks:1] 2013-10-11 16:36:12,217 StatusLogger.java (line 55) 
Pool NameActive   Pending  Completed   Blocked  All 
Time Blocked
 INFO [ScheduledTasks:1] 2013-10-11 16:36:12,246 StatusLogger.java (line 70) 
ReadStage 0 0 845326 0  
   0
 INFO [ScheduledTasks:1] 2013-10-11 16:36:12,246 StatusLogger.java (line 70) 
RequestResponseStage  0 01643358 0  
   0
 INFO [ScheduledTasks:1] 2013-10-11 16:36:12,246 StatusLogger.java (line 70) 
ReadRepairStage   0 0  61247 0  
   0
 INFO [ScheduledTasks:1] 2013-10-11 16:36:12,247 StatusLogger.java (line 70) 
MutationStage 0 01155502 0  
   0
 INFO [ScheduledTasks:1] 2013-10-11 16:36:12,247 StatusLogger.java (line 70) 
ReplicateOnWriteStage 0 0  0 0  
   0
 INFO [ScheduledTasks:1] 2013-10-11 16:36:12,247 StatusLogger.java (line 70) 
GossipStage   0 0   5391 0  
   0
 INFO [ScheduledTasks:1] 2013-10-11 16:36:12,248 StatusLogger.java (line 70) 
AntiEntropyStage  0 0  0 0  
   0
 INFO [ScheduledTasks:1] 2013-10-11 16:36:12,248 StatusLogger.java (line 70) 
MigrationStage0 0 14 0  
   0
 INFO [ScheduledTasks:1] 2013-10-11 16:36:12,248 StatusLogger.java (line 70) 
MemtablePostFlusher   0 0 99 0  
   0
 INFO [ScheduledTasks:1] 2013-10-11 16:36:12,248 StatusLogger.java (line 70) 
MemoryMeter   0 0 58 0  
   0
 INFO [ScheduledTasks:1] 2013-10-11 16:36:12,249 StatusLogger.java (line 70) 
FlushWriter   0 0 45 0  
   0
 INFO [ScheduledTasks:1] 2013-10-11 16:36:12,249 StatusLogger.java (line 70) 
MiscStage 0 0  0 0  
   0
 INFO [ScheduledTasks:1] 2013-10-11 16:36:12,249 StatusLogger.java (line 70) 
commitlog_archiver0 0  0 0  
   0
 INFO [ScheduledTasks:1] 2013-10-11 16:36:12,250 StatusLogger.java (line 70) 
InternalResponseStage 0 0  3 0  
   0
 INFO [ScheduledTasks:1] 2013-10-11 16:36:12,250 StatusLogger.java (line 70) 
HintedHandoff 0 0  7 0  
   0
 INFO [ScheduledTasks:1] 2013-10-11 16:36:12,250 StatusLogger.java (line 79) 
CompactionManager 0 0
 INFO [ScheduledTasks:1] 2013-10-11 16:36:12,250 StatusLogger.java (line 81) 
Commitlog   n/a 0
 INFO [ScheduledTasks:1] 2013-10-11 16:36:12,251 StatusLogger.java (line 93) 
MessagingServicen/a   0,0
 INFO [ScheduledTasks:1] 2013-10-11 16:36:12,251 StatusLogger.java (line 103) 
Cache Type Size Capacity   
KeysToSave
 INFO [ScheduledTasks:1] 2013-10-11 16:36:12,251 StatusLogger.java (line 105) 
KeyCache  74140104857600
  all
 INFO [ScheduledTasks:1] 2013-10-11 16:36:12,251 StatusLogger.java (line 111) 
RowCache  00
  all
 INFO [ScheduledTasks:1] 2013-10-11 16:36:12,251 StatusLogger.java (line 118) 
ColumnFamilyMemtable ops,data
 INFO [ScheduledTasks:1] 2013-10-11 16:36:12,251 StatusLogger.java (line 121) 
system.schema_triggers0,0
 INFO [ScheduledTasks:1] 2013-10-11 16:36:12,252 StatusLogger.java (line 121) 
system.local   18,520
 INFO [ScheduledTasks:1] 2013-10-11 16:36:12,252 StatusLogger.java (line 121) 
system.peers

[jira] [Commented] (CASSANDRA-6187) Cassandra-2.0 node seems to be pausing periodically under normal operations

2013-10-11 Thread Li Zou (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-6187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13793141#comment-13793141
 ] 

Li Zou commented on CASSANDRA-6187:
---

This issue is only observed fairly recently when testing the Cassandra-2.0.

We have been using the same tool, the same configuration and the same set of 
machines for Cassandra-1.2.4 and it has never shown any sign of this issue.

> Cassandra-2.0 node seems to be pausing periodically under normal operations
> ---
>
> Key: CASSANDRA-6187
> URL: https://issues.apache.org/jira/browse/CASSANDRA-6187
> Project: Cassandra
>  Issue Type: Bug
>  Components: Core
> Environment: One data center of 4 Cassandra-2.0 nodes with default 
> configuration parameters deployed on 4 separate machines.
> A testing app (with either Astyanax or DataStax client driver) interacts with 
> the Cassandra data center.
> A traffic generator is sending traffic to the testing app for testing purpose.
>Reporter: Li Zou
>
> Under normal operation condition without any interruption, the traffic 
> generator  will periodically see a couple of seconds of zero (or very little) 
> transactions, i.e. the throughput can be quite high for a while, then comes 
> down to zero (or very little) transactions for 10 ~ 20 seconds.
> The Cassandra system log occasionally also logs message drops. But these 
> message drop events do not correlate in time to the observed transaction drop 
> events.
> Example of message dropping log:
> {noformat}
>  INFO [ScheduledTasks:1] 2013-10-11 16:36:12,216 MessagingService.java (line 
> 812) 1191 MUTATION messages dropped in last 5000ms
>  INFO [ScheduledTasks:1] 2013-10-11 16:36:12,217 MessagingService.java (line 
> 812) 502 READ_REPAIR messages dropped in last 5000ms
>  INFO [ScheduledTasks:1] 2013-10-11 16:36:12,217 StatusLogger.java (line 55) 
> Pool NameActive   Pending  Completed   Blocked  All 
> Time Blocked
>  INFO [ScheduledTasks:1] 2013-10-11 16:36:12,246 StatusLogger.java (line 70) 
> ReadStage 0 0 845326 0
>  0
>  INFO [ScheduledTasks:1] 2013-10-11 16:36:12,246 StatusLogger.java (line 70) 
> RequestResponseStage  0 01643358 0
>  0
>  INFO [ScheduledTasks:1] 2013-10-11 16:36:12,246 StatusLogger.java (line 70) 
> ReadRepairStage   0 0  61247 0
>  0
>  INFO [ScheduledTasks:1] 2013-10-11 16:36:12,247 StatusLogger.java (line 70) 
> MutationStage 0 01155502 0
>  0
>  INFO [ScheduledTasks:1] 2013-10-11 16:36:12,247 StatusLogger.java (line 70) 
> ReplicateOnWriteStage 0 0  0 0
>  0
>  INFO [ScheduledTasks:1] 2013-10-11 16:36:12,247 StatusLogger.java (line 70) 
> GossipStage   0 0   5391 0
>  0
>  INFO [ScheduledTasks:1] 2013-10-11 16:36:12,248 StatusLogger.java (line 70) 
> AntiEntropyStage  0 0  0 0
>  0
>  INFO [ScheduledTasks:1] 2013-10-11 16:36:12,248 StatusLogger.java (line 70) 
> MigrationStage0 0 14 0
>  0
>  INFO [ScheduledTasks:1] 2013-10-11 16:36:12,248 StatusLogger.java (line 70) 
> MemtablePostFlusher   0 0 99 0
>  0
>  INFO [ScheduledTasks:1] 2013-10-11 16:36:12,248 StatusLogger.java (line 70) 
> MemoryMeter   0 0 58 0
>  0
>  INFO [ScheduledTasks:1] 2013-10-11 16:36:12,249 StatusLogger.java (line 70) 
> FlushWriter   0 0 45 0
>  0
>  INFO [ScheduledTasks:1] 2013-10-11 16:36:12,249 StatusLogger.java (line 70) 
> MiscStage 0 0  0 0
>  0
>  INFO [ScheduledTasks:1] 2013-10-11 16:36:12,249 StatusLogger.java (line 70) 
> commitlog_archiver0 0  0 0
>  0
>  INFO [ScheduledTasks:1] 2013-10-11 16:36:12,250 StatusLogger.java (line 70) 
> InternalResponseStage 0 0  3 0
>  0
>  INFO [ScheduledTasks:1] 2013-10-11 16:36:12,250 StatusLogger.java (line 70) 
> HintedHandoff 0 0  7 0
>  0
>  INFO [ScheduledTasks:1] 2013-10-11 16:36:12,250 StatusLogger.java (line 79) 
> CompactionManager 0 0
>  INFO [ScheduledTasks:1] 2013-10-11 16:36:12,250 StatusLogger.java (line 81) 
> Commitlog

[jira] [Created] (CASSANDRA-6188) The JMX stats for Speculative Retry stops moving during a node failure outage period

2013-10-11 Thread Li Zou (JIRA)

Li Zou created CASSANDRA-6188:
-

 Summary: The JMX stats for Speculative Retry stops moving during a 
node failure outage period
 Key: CASSANDRA-6188
 URL: https://issues.apache.org/jira/browse/CASSANDRA-6188
 Project: Cassandra
  Issue Type: Bug
  Components: Core
 Environment: One data center of 4 Cassandra-2.0 nodes with default 
configuration parameters deployed on 4 separate machines. A testing app (with 
either Astyanax or DataStax client driver) interacts with the Cassandra data 
center. A traffic generator is sending traffic to the testing app for testing 
purpose.
Reporter: Li Zou


Under normal testing traffic level with default Cassandra Speculative Retry 
configuration for each table (i.e. 99 percentile), JConsole shows that the JMX 
stats for Speculative Retry increments slowly. However, during the node failure 
outage period (i.e. immediately after the node was killed and before the gossip 
figures out that the node is down), JConsole shows that the JMX stats for 
Speculative Retry stops moving. That is, for around 20 seconds, the JMX stats 
for Speculative Retry does not move.

This is true for all other Speculative Retry options. 




--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Created] (CASSANDRA-6187) Cassandra-2.0 node seems to be pausing periodically under normal operations

2013-10-11 Thread Li Zou (JIRA)

Li Zou created CASSANDRA-6187:
-

 Summary: Cassandra-2.0 node seems to be pausing periodically under 
normal operations
 Key: CASSANDRA-6187
 URL: https://issues.apache.org/jira/browse/CASSANDRA-6187
 Project: Cassandra
  Issue Type: Bug
  Components: Core
 Environment: One data center of 4 Cassandra-2.0 nodes with default 
configuration parameters deployed on 4 separate machines.

A testing app (with either Astyanax or DataStax client driver) interacts with 
the Cassandra data center.

A traffic generator is sending traffic to the testing app for testing purpose.
Reporter: Li Zou


Under normal operation condition without any interruption, the traffic 
generator  will periodically see a couple of seconds of zero (or very little) 
transactions, i.e. the throughput can be quite high for a while, then comes 
down to zero (or very little) transactions for 10 ~ 20 seconds.

The Cassandra system log occasionally also logs message drops. But these 
message drop events do not correlate in time to the observed transaction drop 
events.

Example of message dropping log:

{noformat}
 INFO [ScheduledTasks:1] 2013-10-11 16:36:12,216 MessagingService.java (line 
812) 1191 MUTATION messages dropped in last 5000ms
 INFO [ScheduledTasks:1] 2013-10-11 16:36:12,217 MessagingService.java (line 
812) 502 READ_REPAIR messages dropped in last 5000ms
 INFO [ScheduledTasks:1] 2013-10-11 16:36:12,217 StatusLogger.java (line 55) 
Pool NameActive   Pending  Completed   Blocked  All 
Time Blocked
 INFO [ScheduledTasks:1] 2013-10-11 16:36:12,246 StatusLogger.java (line 70) 
ReadStage 0 0 845326 0  
   0
 INFO [ScheduledTasks:1] 2013-10-11 16:36:12,246 StatusLogger.java (line 70) 
RequestResponseStage  0 01643358 0  
   0
 INFO [ScheduledTasks:1] 2013-10-11 16:36:12,246 StatusLogger.java (line 70) 
ReadRepairStage   0 0  61247 0  
   0
 INFO [ScheduledTasks:1] 2013-10-11 16:36:12,247 StatusLogger.java (line 70) 
MutationStage 0 01155502 0  
   0
 INFO [ScheduledTasks:1] 2013-10-11 16:36:12,247 StatusLogger.java (line 70) 
ReplicateOnWriteStage 0 0  0 0  
   0
 INFO [ScheduledTasks:1] 2013-10-11 16:36:12,247 StatusLogger.java (line 70) 
GossipStage   0 0   5391 0  
   0
 INFO [ScheduledTasks:1] 2013-10-11 16:36:12,248 StatusLogger.java (line 70) 
AntiEntropyStage  0 0  0 0  
   0
 INFO [ScheduledTasks:1] 2013-10-11 16:36:12,248 StatusLogger.java (line 70) 
MigrationStage0 0 14 0  
   0
 INFO [ScheduledTasks:1] 2013-10-11 16:36:12,248 StatusLogger.java (line 70) 
MemtablePostFlusher   0 0 99 0  
   0
 INFO [ScheduledTasks:1] 2013-10-11 16:36:12,248 StatusLogger.java (line 70) 
MemoryMeter   0 0 58 0  
   0
 INFO [ScheduledTasks:1] 2013-10-11 16:36:12,249 StatusLogger.java (line 70) 
FlushWriter   0 0 45 0  
   0
 INFO [ScheduledTasks:1] 2013-10-11 16:36:12,249 StatusLogger.java (line 70) 
MiscStage 0 0  0 0  
   0
 INFO [ScheduledTasks:1] 2013-10-11 16:36:12,249 StatusLogger.java (line 70) 
commitlog_archiver0 0  0 0  
   0
 INFO [ScheduledTasks:1] 2013-10-11 16:36:12,250 StatusLogger.java (line 70) 
InternalResponseStage 0 0  3 0  
   0
 INFO [ScheduledTasks:1] 2013-10-11 16:36:12,250 StatusLogger.java (line 70) 
HintedHandoff 0 0  7 0  
   0
 INFO [ScheduledTasks:1] 2013-10-11 16:36:12,250 StatusLogger.java (line 79) 
CompactionManager 0 0
 INFO [ScheduledTasks:1] 2013-10-11 16:36:12,250 StatusLogger.java (line 81) 
Commitlog   n/a 0
 INFO [ScheduledTasks:1] 2013-10-11 16:36:12,251 StatusLogger.java (line 93) 
MessagingServicen/a   0,0
 INFO [ScheduledTasks:1] 2013-10-11 16:36:12,251 StatusLogger.java (line 103) 
Cache Type Size Capacity   
KeysToSave
 INFO [ScheduledTasks:1] 2013-10-11 16:36:12,251 StatusLogger.java (line 105) 
KeyCache  74140104857600
  all
 INFO [ScheduledTasks:1] 2013-10-11 16:36:12,251 StatusLogger.java (line 111) 
RowCache  0

[jira] [Commented] (CASSANDRA-6154) Inserts are blocked in 2.1

2013-10-08 Thread Li Zou (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-6154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13789768#comment-13789768
 ] 

Li Zou commented on CASSANDRA-6154:
---

Yes, this has been observed in my test this afternoon after one node was killed.

I think that we should systematically check for this issue, as some other 
places can also have this {{MessagingService.addCallback()}} assertion issue.

> Inserts are blocked in 2.1
> --
>
> Key: CASSANDRA-6154
> URL: https://issues.apache.org/jira/browse/CASSANDRA-6154
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Ryan McGuire
>Assignee: Jonathan Ellis
>Priority: Critical
> Attachments: 6154.txt, 6154-v2.txt
>
>
> With cluster sizes >1 inserts are blocked indefinitely:
> {code}
> $ ccm create -v git:trunk test
> Fetching Cassandra updates...
> Current cluster is now: test
> $ ccm populate -n 2
> $ ccm start
> $ ccm node1 cqlsh
> Connected to test at 127.0.0.1:9160.
> [cqlsh 4.0.1 | Cassandra 2.1-SNAPSHOT | CQL spec 3.1.1 | Thrift protocol 
> 19.37.0]
> Use HELP for help.
> cqlsh> CREATE KEYSPACE timeline WITH replication = {'class': 
> 'SimpleStrategy', 'replication_factor': 1};
> cqlsh> USE timeline;
> cqlsh:timeline> CREATE TABLE user_events (userid text, event timestamp, value 
> text, PRIMARY KEY (userid, event));
> cqlsh:timeline> INSERT INTO user_events (userid, event , value ) VALUES ( 
> 'ryan', '2013-10-07', 'attempt');
> {code}
> The last INSERT statement never returns..



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Comment Edited] (CASSANDRA-6154) Inserts are blocked in 2.1

2013-10-08 Thread Li Zou (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-6154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13789737#comment-13789737
 ] 

Li Zou edited comment on CASSANDRA-6154 at 10/8/13 9:52 PM:


[~jbellis] I just pulled the Cassandra-2.0 source code and did a quick testing. 
Though slightly better than yesterday's trunk load. But there are still lots of 
assertion errors for  {{MessagingService.addCallback()}} without the argument 
of consistency level for MessageOut. E.g.

{noformat}
   private void doDeliverHintsToEndpoint(InetAddress endpoint)
{
 ...
MessageOut message = rm.createMessage();
 ...
MessagingService.instance().sendRR(message, endpoint, 
responseHandler);
responseHandlers.add(responseHandler);
{noformat}

and
{noformat}
public static void sendToHintedEndpoints(final RowMutation rm,
 Iterable targets,
 AbstractWriteResponseHandler 
responseHandler,
 String localDataCenter,
 ConsistencyLevel consistency_level)
throws OverloadedException
{
...
// belongs on a different server
if (message == null)
message = rm.createMessage();
String dc = 
DatabaseDescriptor.getEndpointSnitch().getDatacenter(destination);
// direct writes to local DC or old Cassandra versions
// (1.1 knows how to forward old-style String message IDs; 
updated to int in 2.0)
if (localDataCenter.equals(dc) || 
MessagingService.instance().getVersion(destination) < 
MessagingService.VERSION_20)
{
MessagingService.instance().sendRR(message, 
destination, responseHandler);
}
...
{noformat}

In the above examples, the {{sendRR()}} will call {{addCallback()}} which 
asserts as the message is of  type RowMutation.



was (Author: lizou):
I just pulled the Cassandra-2.0 source code and did a quick testing. Though 
slightly better than yesterday's trunk load. But there are still lots of 
assertion errors for  {{MessagingService.addCallback()}} without the argument 
of consistency level for MessageOut. E.g.

{noformat}
   private void doDeliverHintsToEndpoint(InetAddress endpoint)
{
 ...
MessageOut message = rm.createMessage();
 ...
MessagingService.instance().sendRR(message, endpoint, 
responseHandler);
responseHandlers.add(responseHandler);
{noformat}

and
{noformat}
public static void sendToHintedEndpoints(final RowMutation rm,
 Iterable targets,
 AbstractWriteResponseHandler 
responseHandler,
 String localDataCenter,
 ConsistencyLevel consistency_level)
throws OverloadedException
{
...
// belongs on a different server
if (message == null)
message = rm.createMessage();
String dc = 
DatabaseDescriptor.getEndpointSnitch().getDatacenter(destination);
// direct writes to local DC or old Cassandra versions
// (1.1 knows how to forward old-style String message IDs; 
updated to int in 2.0)
if (localDataCenter.equals(dc) || 
MessagingService.instance().getVersion(destination) < 
MessagingService.VERSION_20)
{
MessagingService.instance().sendRR(message, 
destination, responseHandler);
}
...
{noformat}

In the above examples, the {{sendRR()}} will call {{addCallback()}} which 
asserts as the message is of  type RowMutation.


> Inserts are blocked in 2.1
> --
>
> Key: CASSANDRA-6154
> URL: https://issues.apache.org/jira/browse/CASSANDRA-6154
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Ryan McGuire
>Assignee: Jonathan Ellis
>Priority: Critical
> Attachments: 6154.txt, 6154-v2.txt
>
>
> With cluster sizes >1 inserts are blocked indefinitely:
> {code}
> $ ccm create -v git:trunk test
> Fetching Cassandra updates...
> Current cluster is now: test
> $ ccm populate -n 2
> $ ccm start
> $ ccm node1 cqlsh
> Connected to test at 127.0.0.1:9160.
> [cqlsh 4.0.1 | Cassandra 2.1-SNAPSHOT | CQL spec 3.1.1 | Thrift protocol 
> 19.37.0]
> Use HELP for help.
> cqlsh> CREATE KEYSPACE timeline WITH replication = {'class': 
> 'SimpleStrategy', 'replication_factor': 1};
> cqlsh> USE timeline;
> cqlsh:timeline> CREATE TABLE user_events (u

[jira] [Commented] (CASSANDRA-6154) Inserts are blocked in 2.1

2013-10-08 Thread Li Zou (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-6154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13789737#comment-13789737
 ] 

Li Zou commented on CASSANDRA-6154:
---

I just pulled the Cassandra-2.0 source code and did a quick testing. Though 
slightly better than yesterday's trunk load. But there are still lots of 
assertion errors for  {{MessagingService.addCallback()}} without the argument 
of consistency level for MessageOut. E.g.

{noformat}
   private void doDeliverHintsToEndpoint(InetAddress endpoint)
{
 ...
MessageOut message = rm.createMessage();
 ...
MessagingService.instance().sendRR(message, endpoint, 
responseHandler);
responseHandlers.add(responseHandler);
{noformat}

and
{noformat}
public static void sendToHintedEndpoints(final RowMutation rm,
 Iterable targets,
 AbstractWriteResponseHandler 
responseHandler,
 String localDataCenter,
 ConsistencyLevel consistency_level)
throws OverloadedException
{
...
// belongs on a different server
if (message == null)
message = rm.createMessage();
String dc = 
DatabaseDescriptor.getEndpointSnitch().getDatacenter(destination);
// direct writes to local DC or old Cassandra versions
// (1.1 knows how to forward old-style String message IDs; 
updated to int in 2.0)
if (localDataCenter.equals(dc) || 
MessagingService.instance().getVersion(destination) < 
MessagingService.VERSION_20)
{
MessagingService.instance().sendRR(message, 
destination, responseHandler);
}
...
{noformat}

In the above examples, the {{sendRR()}} will call {{addCallback()}} which 
asserts as the message is of  type RowMutation.


> Inserts are blocked in 2.1
> --
>
> Key: CASSANDRA-6154
> URL: https://issues.apache.org/jira/browse/CASSANDRA-6154
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Ryan McGuire
>Assignee: Jonathan Ellis
>Priority: Critical
> Attachments: 6154.txt, 6154-v2.txt
>
>
> With cluster sizes >1 inserts are blocked indefinitely:
> {code}
> $ ccm create -v git:trunk test
> Fetching Cassandra updates...
> Current cluster is now: test
> $ ccm populate -n 2
> $ ccm start
> $ ccm node1 cqlsh
> Connected to test at 127.0.0.1:9160.
> [cqlsh 4.0.1 | Cassandra 2.1-SNAPSHOT | CQL spec 3.1.1 | Thrift protocol 
> 19.37.0]
> Use HELP for help.
> cqlsh> CREATE KEYSPACE timeline WITH replication = {'class': 
> 'SimpleStrategy', 'replication_factor': 1};
> cqlsh> USE timeline;
> cqlsh:timeline> CREATE TABLE user_events (userid text, event timestamp, value 
> text, PRIMARY KEY (userid, event));
> cqlsh:timeline> INSERT INTO user_events (userid, event , value ) VALUES ( 
> 'ryan', '2013-10-07', 'attempt');
> {code}
> The last INSERT statement never returns..



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (CASSANDRA-5932) Speculative read performance data show unexpected results

2013-10-07 Thread Li Zou (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-5932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13788654#comment-13788654
 ] 

Li Zou commented on CASSANDRA-5932:
---

I even cannot see the cassandra-2.0 branch.
My "git tag" gives a list including following branches.

{noformat}
$ git tag
1.2.8
1.2.8-tentative
cassandra-0.3.0-final
cassandra-0.3.0-rc1
cassandra-0.3.0-rc2
...
cassandra-1.2.4
cassandra-1.2.5
cassandra-1.2.6
cassandra-1.2.7
cassandra-1.2.8
cassandra-1.2.9
cassandra-2.0.0
cassandra-2.0.0-beta1
cassandra-2.0.0-beta2
cassandra-2.0.0-rc1
cassandra-2.0.0-rc2
cassandra-2.0.1
drivers
list
{noformat}

There is no cassandra-2.0 branch. Where can I find it?


> Speculative read performance data show unexpected results
> -
>
> Key: CASSANDRA-5932
> URL: https://issues.apache.org/jira/browse/CASSANDRA-5932
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Ryan McGuire
>Assignee: Aleksey Yeschenko
> Fix For: 2.0.2
>
> Attachments: 5932.6692c50412ef7d.compaction.png, 
> 5932-6692c50412ef7d.png, 5932.6692c50412ef7d.rr0.png, 
> 5932.6692c50412ef7d.rr1.png, 5932.ded39c7e1c2fa.logs.tar.gz, 5932.txt, 
> 5933-128_and_200rc1.png, 5933-7a87fc11.png, 5933-logs.tar.gz, 
> 5933-randomized-dsnitch-replica.2.png, 5933-randomized-dsnitch-replica.3.png, 
> 5933-randomized-dsnitch-replica.png, compaction-makes-slow.png, 
> compaction-makes-slow-stats.png, eager-read-looks-promising.png, 
> eager-read-looks-promising-stats.png, eager-read-not-consistent.png, 
> eager-read-not-consistent-stats.png, node-down-increase-performance.png
>
>
> I've done a series of stress tests with eager retries enabled that show 
> undesirable behavior. I'm grouping these behaviours into one ticket as they 
> are most likely related.
> 1) Killing off a node in a 4 node cluster actually increases performance.
> 2) Compactions make nodes slow, even after the compaction is done.
> 3) Eager Reads tend to lessen the *immediate* performance impact of a node 
> going down, but not consistently.
> My Environment:
> 1 stress machine: node0
> 4 C* nodes: node4, node5, node6, node7
> My script:
> node0 writes some data: stress -d node4 -F 3000 -n 3000 -i 5 -l 2 -K 
> 20
> node0 reads some data: stress -d node4 -n 3000 -o read -i 5 -K 20
> h3. Examples:
> h5. A node going down increases performance:
> !node-down-increase-performance.png!
> [Data for this test 
> here|http://ryanmcguire.info/ds/graph/graph.html?stats=stats.eager_retry.node_killed.just_20.json&metric=interval_op_rate&operation=stress-read&smoothing=1]
> At 450s, I kill -9 one of the nodes. There is a brief decrease in performance 
> as the snitch adapts, but then it recovers... to even higher performance than 
> before.
> h5. Compactions make nodes permanently slow:
> !compaction-makes-slow.png!
> !compaction-makes-slow-stats.png!
> The green and orange lines represent trials with eager retry enabled, they 
> never recover their op-rate from before the compaction as the red and blue 
> lines do.
> [Data for this test 
> here|http://ryanmcguire.info/ds/graph/graph.html?stats=stats.eager_retry.compaction.2.json&metric=interval_op_rate&operation=stress-read&smoothing=1]
> h5. Speculative Read tends to lessen the *immediate* impact:
> !eager-read-looks-promising.png!
> !eager-read-looks-promising-stats.png!
> This graph looked the most promising to me, the two trials with eager retry, 
> the green and orange line, at 450s showed the smallest dip in performance. 
> [Data for this test 
> here|http://ryanmcguire.info/ds/graph/graph.html?stats=stats.eager_retry.node_killed.json&metric=interval_op_rate&operation=stress-read&smoothing=1]
> h5. But not always:
> !eager-read-not-consistent.png!
> !eager-read-not-consistent-stats.png!
> This is a retrial with the same settings as above, yet the 95percentile eager 
> retry (red line) did poorly this time at 450s.
> [Data for this test 
> here|http://ryanmcguire.info/ds/graph/graph.html?stats=stats.eager_retry.node_killed.just_20.rc1.try2.json&metric=interval_op_rate&operation=stress-read&smoothing=1]



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (CASSANDRA-5932) Speculative read performance data show unexpected results

2013-10-07 Thread Li Zou (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-5932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13788613#comment-13788613
 ] 

Li Zou commented on CASSANDRA-5932:
---

Hello,

As this ticket is already fixed in 2.0.2, where can I get the 2.0.2 source code?

Currently, my "git tag" only shows up to 2.0.1.


> Speculative read performance data show unexpected results
> -
>
> Key: CASSANDRA-5932
> URL: https://issues.apache.org/jira/browse/CASSANDRA-5932
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Ryan McGuire
>Assignee: Aleksey Yeschenko
> Fix For: 2.0.2
>
> Attachments: 5932.6692c50412ef7d.compaction.png, 
> 5932-6692c50412ef7d.png, 5932.6692c50412ef7d.rr0.png, 
> 5932.6692c50412ef7d.rr1.png, 5932.ded39c7e1c2fa.logs.tar.gz, 5932.txt, 
> 5933-128_and_200rc1.png, 5933-7a87fc11.png, 5933-logs.tar.gz, 
> 5933-randomized-dsnitch-replica.2.png, 5933-randomized-dsnitch-replica.3.png, 
> 5933-randomized-dsnitch-replica.png, compaction-makes-slow.png, 
> compaction-makes-slow-stats.png, eager-read-looks-promising.png, 
> eager-read-looks-promising-stats.png, eager-read-not-consistent.png, 
> eager-read-not-consistent-stats.png, node-down-increase-performance.png
>
>
> I've done a series of stress tests with eager retries enabled that show 
> undesirable behavior. I'm grouping these behaviours into one ticket as they 
> are most likely related.
> 1) Killing off a node in a 4 node cluster actually increases performance.
> 2) Compactions make nodes slow, even after the compaction is done.
> 3) Eager Reads tend to lessen the *immediate* performance impact of a node 
> going down, but not consistently.
> My Environment:
> 1 stress machine: node0
> 4 C* nodes: node4, node5, node6, node7
> My script:
> node0 writes some data: stress -d node4 -F 3000 -n 3000 -i 5 -l 2 -K 
> 20
> node0 reads some data: stress -d node4 -n 3000 -o read -i 5 -K 20
> h3. Examples:
> h5. A node going down increases performance:
> !node-down-increase-performance.png!
> [Data for this test 
> here|http://ryanmcguire.info/ds/graph/graph.html?stats=stats.eager_retry.node_killed.just_20.json&metric=interval_op_rate&operation=stress-read&smoothing=1]
> At 450s, I kill -9 one of the nodes. There is a brief decrease in performance 
> as the snitch adapts, but then it recovers... to even higher performance than 
> before.
> h5. Compactions make nodes permanently slow:
> !compaction-makes-slow.png!
> !compaction-makes-slow-stats.png!
> The green and orange lines represent trials with eager retry enabled, they 
> never recover their op-rate from before the compaction as the red and blue 
> lines do.
> [Data for this test 
> here|http://ryanmcguire.info/ds/graph/graph.html?stats=stats.eager_retry.compaction.2.json&metric=interval_op_rate&operation=stress-read&smoothing=1]
> h5. Speculative Read tends to lessen the *immediate* impact:
> !eager-read-looks-promising.png!
> !eager-read-looks-promising-stats.png!
> This graph looked the most promising to me, the two trials with eager retry, 
> the green and orange line, at 450s showed the smallest dip in performance. 
> [Data for this test 
> here|http://ryanmcguire.info/ds/graph/graph.html?stats=stats.eager_retry.node_killed.json&metric=interval_op_rate&operation=stress-read&smoothing=1]
> h5. But not always:
> !eager-read-not-consistent.png!
> !eager-read-not-consistent-stats.png!
> This is a retrial with the same settings as above, yet the 95percentile eager 
> retry (red line) did poorly this time at 450s.
> [Data for this test 
> here|http://ryanmcguire.info/ds/graph/graph.html?stats=stats.eager_retry.node_killed.just_20.rc1.try2.json&metric=interval_op_rate&operation=stress-read&smoothing=1]



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Comment Edited] (CASSANDRA-5932) Speculative read performance data show unexpected results

2013-10-07 Thread Li Zou (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-5932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13788437#comment-13788437
 ] 

Li Zou edited comment on CASSANDRA-5932 at 10/7/13 7:17 PM:


[~jbellis], this morning's trunk load has a slightly different symptom, and is 
even more serious than last Friday's load, as this time just commenting out the 
assert statement in the {{MessagingService.addCallback()}} will not help.

I copy the {{/var/log/cassandra/system.log}} exception errors below.

{noformat}
ERROR [Thrift:12] 2013-10-07 14:42:39,396 Caller+0   at 
org.apache.cassandra.service.CassandraDaemon$2.uncaughtException(CassandraDaemon.java:134)
 - Exception in thread Thread[Thrift:12,5,main]
java.lang.AssertionError: null
at 
org.apache.cassandra.net.MessagingService.addCallback(MessagingService.java:543)
 ~[apache-cassandra-2.1-SNAPSHOT.jar:2.1-SNAPSHOT]
at 
org.apache.cassandra.net.MessagingService.sendRR(MessagingService.java:591) 
~[apache-cassandra-2.1-SNAPSHOT.jar:2.1-SNAPSHOT]
at 
org.apache.cassandra.net.MessagingService.sendRR(MessagingService.java:571) 
~[apache-cassandra-2.1-SNAPSHOT.jar:2.1-SNAPSHOT]
at 
org.apache.cassandra.service.StorageProxy.sendToHintedEndpoints(StorageProxy.java:869)
 ~[apache-cassandra-2.1-SNAPSHOT.jar:2.1-SNAPSHOT]
at 
org.apache.cassandra.service.StorageProxy$2.apply(StorageProxy.java:123) 
~[apache-cassandra-2.1-SNAPSHOT.jar:2.1-SNAPSHOT]
at 
org.apache.cassandra.service.StorageProxy.performWrite(StorageProxy.java:739) 
~[apache-cassandra-2.1-SNAPSHOT.jar:2.1-SNAPSHOT]
at 
org.apache.cassandra.service.StorageProxy.mutate(StorageProxy.java:511) 
~[apache-cassandra-2.1-SNAPSHOT.jar:2.1-SNAPSHOT]
at 
org.apache.cassandra.service.StorageProxy.mutateWithTriggers(StorageProxy.java:581)
 ~[apache-cassandra-2.1-SNAPSHOT.jar:2.1-SNAPSHOT]
at 
org.apache.cassandra.cql3.statements.ModificationStatement.executeWithoutCondition(ModificationStatement.java:379)
 ~[apache-cassandra-2.1-SNAPSHOT.jar:2.1-SNAPSHOT]
at 
org.apache.cassandra.cql3.statements.ModificationStatement.execute(ModificationStatement.java:363)
 ~[apache-cassandra-2.1-SNAPSHOT.jar:2.1-SNAPSHOT]
at 
org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:126)
 ~[apache-cassandra-2.1-SNAPSHOT.jar:2.1-SNAPSHOT]
at 
org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:267)
 ~[apache-cassandra-2.1-SNAPSHOT.jar:2.1-SNAPSHOT]
at 
org.apache.cassandra.thrift.CassandraServer.execute_prepared_cql3_query(CassandraServer.java:2061)
 ~[apache-cassandra-2.1-SNAPSHOT.jar:2.1-SNAPSHOT]
at 
org.apache.cassandra.thrift.Cassandra$Processor$execute_prepared_cql3_query.getResult(Cassandra.java:4502)
 ~[apache-cassandra-thrift-2.1-SNAPSHOT.jar:2.1-SNAPSHOT]
at 
org.apache.cassandra.thrift.Cassandra$Processor$execute_prepared_cql3_query.getResult(Cassandra.java:4486)
 ~[apache-cassandra-thrift-2.1-SNAPSHOT.jar:2.1-SNAPSHOT]
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) 
~[libthrift-0.9.1.jar:0.9.1]
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) 
~[libthrift-0.9.1.jar:0.9.1]
at 
org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:194)
 ~[apache-cassandra-2.1-SNAPSHOT.jar:2.1-SNAPSHOT]
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) 
~[na:1.7.0_25]
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) 
~[na:1.7.0_25]
at java.lang.Thread.run(Thread.java:724) ~[na:1.7.0_25]

{noformat}




was (Author: lizou):
This morning's trunk load has a slightly different symptom, and is even more 
serious than last Friday's load, as this time just commenting out the assert 
statement in the {{MessagingService.addCallback()}} will not help.

I copy the {{/var/log/cassandra/system.log}} exception errors below.

{noformat}
ERROR [Thrift:12] 2013-10-07 14:42:39,396 Caller+0   at 
org.apache.cassandra.service.CassandraDaemon$2.uncaughtException(CassandraDaemon.java:134)
 - Exception in thread Thread[Thrift:12,5,main]
java.lang.AssertionError: null
at 
org.apache.cassandra.net.MessagingService.addCallback(MessagingService.java:543)
 ~[apache-cassandra-2.1-SNAPSHOT.jar:2.1-SNAPSHOT]
at 
org.apache.cassandra.net.MessagingService.sendRR(MessagingService.java:591) 
~[apache-cassandra-2.1-SNAPSHOT.jar:2.1-SNAPSHOT]
at 
org.apache.cassandra.net.MessagingService.sendRR(MessagingService.java:571) 
~[apache-cassandra-2.1-SNAPSHOT.jar:2.1-SNAPSHOT]
at 
org.apache.cassandra.service.StorageProxy.sendToHintedEndpoints(StorageProxy.java:869)
 ~[apache-cassandra-2.1-SNAPSHOT.jar:2.1-SNAPSHOT]
at 
org.apache.cassandra.service.StorageProxy$2.apply(Stor

[jira] [Commented] (CASSANDRA-5932) Speculative read performance data show unexpected results

2013-10-07 Thread Li Zou (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-5932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13788437#comment-13788437
 ] 

Li Zou commented on CASSANDRA-5932:
---

This morning's trunk load has a slightly different symptom, and is even more 
serious than last Friday's load, as this time just commenting out the assert 
statement in the {{MessagingService.addCallback()}} will not help.

I copy the {{/var/log/cassandra/system.log}} exception errors below.

{noformat}
ERROR [Thrift:12] 2013-10-07 14:42:39,396 Caller+0   at 
org.apache.cassandra.service.CassandraDaemon$2.uncaughtException(CassandraDaemon.java:134)
 - Exception in thread Thread[Thrift:12,5,main]
java.lang.AssertionError: null
at 
org.apache.cassandra.net.MessagingService.addCallback(MessagingService.java:543)
 ~[apache-cassandra-2.1-SNAPSHOT.jar:2.1-SNAPSHOT]
at 
org.apache.cassandra.net.MessagingService.sendRR(MessagingService.java:591) 
~[apache-cassandra-2.1-SNAPSHOT.jar:2.1-SNAPSHOT]
at 
org.apache.cassandra.net.MessagingService.sendRR(MessagingService.java:571) 
~[apache-cassandra-2.1-SNAPSHOT.jar:2.1-SNAPSHOT]
at 
org.apache.cassandra.service.StorageProxy.sendToHintedEndpoints(StorageProxy.java:869)
 ~[apache-cassandra-2.1-SNAPSHOT.jar:2.1-SNAPSHOT]
at 
org.apache.cassandra.service.StorageProxy$2.apply(StorageProxy.java:123) 
~[apache-cassandra-2.1-SNAPSHOT.jar:2.1-SNAPSHOT]
at 
org.apache.cassandra.service.StorageProxy.performWrite(StorageProxy.java:739) 
~[apache-cassandra-2.1-SNAPSHOT.jar:2.1-SNAPSHOT]
at 
org.apache.cassandra.service.StorageProxy.mutate(StorageProxy.java:511) 
~[apache-cassandra-2.1-SNAPSHOT.jar:2.1-SNAPSHOT]
at 
org.apache.cassandra.service.StorageProxy.mutateWithTriggers(StorageProxy.java:581)
 ~[apache-cassandra-2.1-SNAPSHOT.jar:2.1-SNAPSHOT]
at 
org.apache.cassandra.cql3.statements.ModificationStatement.executeWithoutCondition(ModificationStatement.java:379)
 ~[apache-cassandra-2.1-SNAPSHOT.jar:2.1-SNAPSHOT]
at 
org.apache.cassandra.cql3.statements.ModificationStatement.execute(ModificationStatement.java:363)
 ~[apache-cassandra-2.1-SNAPSHOT.jar:2.1-SNAPSHOT]
at 
org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:126)
 ~[apache-cassandra-2.1-SNAPSHOT.jar:2.1-SNAPSHOT]
at 
org.apache.cassandra.cql3.QueryProcessor.processPrepared(QueryProcessor.java:267)
 ~[apache-cassandra-2.1-SNAPSHOT.jar:2.1-SNAPSHOT]
at 
org.apache.cassandra.thrift.CassandraServer.execute_prepared_cql3_query(CassandraServer.java:2061)
 ~[apache-cassandra-2.1-SNAPSHOT.jar:2.1-SNAPSHOT]
at 
org.apache.cassandra.thrift.Cassandra$Processor$execute_prepared_cql3_query.getResult(Cassandra.java:4502)
 ~[apache-cassandra-thrift-2.1-SNAPSHOT.jar:2.1-SNAPSHOT]
at 
org.apache.cassandra.thrift.Cassandra$Processor$execute_prepared_cql3_query.getResult(Cassandra.java:4486)
 ~[apache-cassandra-thrift-2.1-SNAPSHOT.jar:2.1-SNAPSHOT]
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) 
~[libthrift-0.9.1.jar:0.9.1]
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) 
~[libthrift-0.9.1.jar:0.9.1]
at 
org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:194)
 ~[apache-cassandra-2.1-SNAPSHOT.jar:2.1-SNAPSHOT]
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) 
~[na:1.7.0_25]
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) 
~[na:1.7.0_25]
at java.lang.Thread.run(Thread.java:724) ~[na:1.7.0_25]

{noformat}



> Speculative read performance data show unexpected results
> -
>
> Key: CASSANDRA-5932
> URL: https://issues.apache.org/jira/browse/CASSANDRA-5932
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Ryan McGuire
>Assignee: Aleksey Yeschenko
> Fix For: 2.0.2
>
> Attachments: 5932.6692c50412ef7d.compaction.png, 
> 5932-6692c50412ef7d.png, 5932.6692c50412ef7d.rr0.png, 
> 5932.6692c50412ef7d.rr1.png, 5932.ded39c7e1c2fa.logs.tar.gz, 5932.txt, 
> 5933-128_and_200rc1.png, 5933-7a87fc11.png, 5933-logs.tar.gz, 
> 5933-randomized-dsnitch-replica.2.png, 5933-randomized-dsnitch-replica.3.png, 
> 5933-randomized-dsnitch-replica.png, compaction-makes-slow.png, 
> compaction-makes-slow-stats.png, eager-read-looks-promising.png, 
> eager-read-looks-promising-stats.png, eager-read-not-consistent.png, 
> eager-read-not-consistent-stats.png, node-down-increase-performance.png
>
>
> I've done a series of stress tests with eager retries enabled that show 
> undesirable behavior. I'm grouping these behaviours into one ticket as they 
> are most likely related.
> 1) Killing off a node in a 4 node cluster actual

[jira] [Comment Edited] (CASSANDRA-5932) Speculative read performance data show unexpected results

2013-10-04 Thread Li Zou (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-5932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13786613#comment-13786613
 ] 

Li Zou edited comment on CASSANDRA-5932 at 10/4/13 9:10 PM:


The trunk load I used for testing was pulled this noon. It has two 
addCallback() methods. One of them (i.e. without the ConsistencyLevel) asserts. 

I checked the MessagingService.java, there are two addCallback() methods.
* The one without ConsistencyLevel is called by sendRR()
* The one with ConsistencyLevel is called by sendMessageToNonLocalDC()



was (Author: lizou):
I just did a "git pull" after your comment (i.e. a couple of seconds ago) for 
trunk. The "git pull" indicates that "Already up-to-date".

I checked the MessagingService.java, there are two addCallback() methods.
* The one without ConsistencyLevel is called by sendRR()
* The one with ConsistencyLevel is called by sendMessageToNonLocalDC()

The trunk load I used for testing was pulled this noon. It has two 
addCallback() methods. One of them (i.e. without the ConsistencyLevel) asserts. 

> Speculative read performance data show unexpected results
> -
>
> Key: CASSANDRA-5932
> URL: https://issues.apache.org/jira/browse/CASSANDRA-5932
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Ryan McGuire
>Assignee: Aleksey Yeschenko
> Fix For: 2.0.2
>
> Attachments: 5932.6692c50412ef7d.compaction.png, 
> 5932-6692c50412ef7d.png, 5932.6692c50412ef7d.rr0.png, 
> 5932.6692c50412ef7d.rr1.png, 5932.ded39c7e1c2fa.logs.tar.gz, 5932.txt, 
> 5933-128_and_200rc1.png, 5933-7a87fc11.png, 5933-logs.tar.gz, 
> 5933-randomized-dsnitch-replica.2.png, 5933-randomized-dsnitch-replica.3.png, 
> 5933-randomized-dsnitch-replica.png, compaction-makes-slow.png, 
> compaction-makes-slow-stats.png, eager-read-looks-promising.png, 
> eager-read-looks-promising-stats.png, eager-read-not-consistent.png, 
> eager-read-not-consistent-stats.png, node-down-increase-performance.png
>
>
> I've done a series of stress tests with eager retries enabled that show 
> undesirable behavior. I'm grouping these behaviours into one ticket as they 
> are most likely related.
> 1) Killing off a node in a 4 node cluster actually increases performance.
> 2) Compactions make nodes slow, even after the compaction is done.
> 3) Eager Reads tend to lessen the *immediate* performance impact of a node 
> going down, but not consistently.
> My Environment:
> 1 stress machine: node0
> 4 C* nodes: node4, node5, node6, node7
> My script:
> node0 writes some data: stress -d node4 -F 3000 -n 3000 -i 5 -l 2 -K 
> 20
> node0 reads some data: stress -d node4 -n 3000 -o read -i 5 -K 20
> h3. Examples:
> h5. A node going down increases performance:
> !node-down-increase-performance.png!
> [Data for this test 
> here|http://ryanmcguire.info/ds/graph/graph.html?stats=stats.eager_retry.node_killed.just_20.json&metric=interval_op_rate&operation=stress-read&smoothing=1]
> At 450s, I kill -9 one of the nodes. There is a brief decrease in performance 
> as the snitch adapts, but then it recovers... to even higher performance than 
> before.
> h5. Compactions make nodes permanently slow:
> !compaction-makes-slow.png!
> !compaction-makes-slow-stats.png!
> The green and orange lines represent trials with eager retry enabled, they 
> never recover their op-rate from before the compaction as the red and blue 
> lines do.
> [Data for this test 
> here|http://ryanmcguire.info/ds/graph/graph.html?stats=stats.eager_retry.compaction.2.json&metric=interval_op_rate&operation=stress-read&smoothing=1]
> h5. Speculative Read tends to lessen the *immediate* impact:
> !eager-read-looks-promising.png!
> !eager-read-looks-promising-stats.png!
> This graph looked the most promising to me, the two trials with eager retry, 
> the green and orange line, at 450s showed the smallest dip in performance. 
> [Data for this test 
> here|http://ryanmcguire.info/ds/graph/graph.html?stats=stats.eager_retry.node_killed.json&metric=interval_op_rate&operation=stress-read&smoothing=1]
> h5. But not always:
> !eager-read-not-consistent.png!
> !eager-read-not-consistent-stats.png!
> This is a retrial with the same settings as above, yet the 95percentile eager 
> retry (red line) did poorly this time at 450s.
> [Data for this test 
> here|http://ryanmcguire.info/ds/graph/graph.html?stats=stats.eager_retry.node_killed.just_20.rc1.try2.json&metric=interval_op_rate&operation=stress-read&smoothing=1]



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (CASSANDRA-5932) Speculative read performance data show unexpected results

2013-10-04 Thread Li Zou (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-5932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13786622#comment-13786622
 ] 

Li Zou commented on CASSANDRA-5932:
---

As for yesterday's trunk load, there were two addCallback() methods. But the 
one with ConsistencyLevel was not called by anyone. The one without 
ConsistencyLevel asserts.

> Speculative read performance data show unexpected results
> -
>
> Key: CASSANDRA-5932
> URL: https://issues.apache.org/jira/browse/CASSANDRA-5932
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Ryan McGuire
>Assignee: Aleksey Yeschenko
> Fix For: 2.0.2
>
> Attachments: 5932.6692c50412ef7d.compaction.png, 
> 5932-6692c50412ef7d.png, 5932.6692c50412ef7d.rr0.png, 
> 5932.6692c50412ef7d.rr1.png, 5932.ded39c7e1c2fa.logs.tar.gz, 5932.txt, 
> 5933-128_and_200rc1.png, 5933-7a87fc11.png, 5933-logs.tar.gz, 
> 5933-randomized-dsnitch-replica.2.png, 5933-randomized-dsnitch-replica.3.png, 
> 5933-randomized-dsnitch-replica.png, compaction-makes-slow.png, 
> compaction-makes-slow-stats.png, eager-read-looks-promising.png, 
> eager-read-looks-promising-stats.png, eager-read-not-consistent.png, 
> eager-read-not-consistent-stats.png, node-down-increase-performance.png
>
>
> I've done a series of stress tests with eager retries enabled that show 
> undesirable behavior. I'm grouping these behaviours into one ticket as they 
> are most likely related.
> 1) Killing off a node in a 4 node cluster actually increases performance.
> 2) Compactions make nodes slow, even after the compaction is done.
> 3) Eager Reads tend to lessen the *immediate* performance impact of a node 
> going down, but not consistently.
> My Environment:
> 1 stress machine: node0
> 4 C* nodes: node4, node5, node6, node7
> My script:
> node0 writes some data: stress -d node4 -F 3000 -n 3000 -i 5 -l 2 -K 
> 20
> node0 reads some data: stress -d node4 -n 3000 -o read -i 5 -K 20
> h3. Examples:
> h5. A node going down increases performance:
> !node-down-increase-performance.png!
> [Data for this test 
> here|http://ryanmcguire.info/ds/graph/graph.html?stats=stats.eager_retry.node_killed.just_20.json&metric=interval_op_rate&operation=stress-read&smoothing=1]
> At 450s, I kill -9 one of the nodes. There is a brief decrease in performance 
> as the snitch adapts, but then it recovers... to even higher performance than 
> before.
> h5. Compactions make nodes permanently slow:
> !compaction-makes-slow.png!
> !compaction-makes-slow-stats.png!
> The green and orange lines represent trials with eager retry enabled, they 
> never recover their op-rate from before the compaction as the red and blue 
> lines do.
> [Data for this test 
> here|http://ryanmcguire.info/ds/graph/graph.html?stats=stats.eager_retry.compaction.2.json&metric=interval_op_rate&operation=stress-read&smoothing=1]
> h5. Speculative Read tends to lessen the *immediate* impact:
> !eager-read-looks-promising.png!
> !eager-read-looks-promising-stats.png!
> This graph looked the most promising to me, the two trials with eager retry, 
> the green and orange line, at 450s showed the smallest dip in performance. 
> [Data for this test 
> here|http://ryanmcguire.info/ds/graph/graph.html?stats=stats.eager_retry.node_killed.json&metric=interval_op_rate&operation=stress-read&smoothing=1]
> h5. But not always:
> !eager-read-not-consistent.png!
> !eager-read-not-consistent-stats.png!
> This is a retrial with the same settings as above, yet the 95percentile eager 
> retry (red line) did poorly this time at 450s.
> [Data for this test 
> here|http://ryanmcguire.info/ds/graph/graph.html?stats=stats.eager_retry.node_killed.just_20.rc1.try2.json&metric=interval_op_rate&operation=stress-read&smoothing=1]



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Comment Edited] (CASSANDRA-5932) Speculative read performance data show unexpected results

2013-10-04 Thread Li Zou (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-5932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13786613#comment-13786613
 ] 

Li Zou edited comment on CASSANDRA-5932 at 10/4/13 9:02 PM:


I just did a "git pull" after your comment (i.e. a couple of seconds ago) for 
trunk. The "git pull" indicates that "Already up-to-date".

I checked the MessagingService.java, there are two addCallback() methods.
* The one without ConsistencyLevel is called by sendRR()
* The one with ConsistencyLevel is called by sendMessageToNonLocalDC()

The trunk load I used for testing was pulled this noon. It has two 
addCallback() methods. One of them (i.e. without the ConsistencyLevel) asserts. 


was (Author: lizou):
I just did a "git pull" after your comment (i.e. a couple of seconds ago) for 
trunk. The "git pull" indicates that "Already up-to-date".

I checked the MessagingService.java, there are two addCallback() methods.
* The one without ConsistencyLevel is called by sendRR()
* The one with ConsistencyLevel is called by sendMessageToNonLocalDC()


> Speculative read performance data show unexpected results
> -
>
> Key: CASSANDRA-5932
> URL: https://issues.apache.org/jira/browse/CASSANDRA-5932
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Ryan McGuire
>Assignee: Aleksey Yeschenko
> Fix For: 2.0.2
>
> Attachments: 5932.6692c50412ef7d.compaction.png, 
> 5932-6692c50412ef7d.png, 5932.6692c50412ef7d.rr0.png, 
> 5932.6692c50412ef7d.rr1.png, 5932.ded39c7e1c2fa.logs.tar.gz, 5932.txt, 
> 5933-128_and_200rc1.png, 5933-7a87fc11.png, 5933-logs.tar.gz, 
> 5933-randomized-dsnitch-replica.2.png, 5933-randomized-dsnitch-replica.3.png, 
> 5933-randomized-dsnitch-replica.png, compaction-makes-slow.png, 
> compaction-makes-slow-stats.png, eager-read-looks-promising.png, 
> eager-read-looks-promising-stats.png, eager-read-not-consistent.png, 
> eager-read-not-consistent-stats.png, node-down-increase-performance.png
>
>
> I've done a series of stress tests with eager retries enabled that show 
> undesirable behavior. I'm grouping these behaviours into one ticket as they 
> are most likely related.
> 1) Killing off a node in a 4 node cluster actually increases performance.
> 2) Compactions make nodes slow, even after the compaction is done.
> 3) Eager Reads tend to lessen the *immediate* performance impact of a node 
> going down, but not consistently.
> My Environment:
> 1 stress machine: node0
> 4 C* nodes: node4, node5, node6, node7
> My script:
> node0 writes some data: stress -d node4 -F 3000 -n 3000 -i 5 -l 2 -K 
> 20
> node0 reads some data: stress -d node4 -n 3000 -o read -i 5 -K 20
> h3. Examples:
> h5. A node going down increases performance:
> !node-down-increase-performance.png!
> [Data for this test 
> here|http://ryanmcguire.info/ds/graph/graph.html?stats=stats.eager_retry.node_killed.just_20.json&metric=interval_op_rate&operation=stress-read&smoothing=1]
> At 450s, I kill -9 one of the nodes. There is a brief decrease in performance 
> as the snitch adapts, but then it recovers... to even higher performance than 
> before.
> h5. Compactions make nodes permanently slow:
> !compaction-makes-slow.png!
> !compaction-makes-slow-stats.png!
> The green and orange lines represent trials with eager retry enabled, they 
> never recover their op-rate from before the compaction as the red and blue 
> lines do.
> [Data for this test 
> here|http://ryanmcguire.info/ds/graph/graph.html?stats=stats.eager_retry.compaction.2.json&metric=interval_op_rate&operation=stress-read&smoothing=1]
> h5. Speculative Read tends to lessen the *immediate* impact:
> !eager-read-looks-promising.png!
> !eager-read-looks-promising-stats.png!
> This graph looked the most promising to me, the two trials with eager retry, 
> the green and orange line, at 450s showed the smallest dip in performance. 
> [Data for this test 
> here|http://ryanmcguire.info/ds/graph/graph.html?stats=stats.eager_retry.node_killed.json&metric=interval_op_rate&operation=stress-read&smoothing=1]
> h5. But not always:
> !eager-read-not-consistent.png!
> !eager-read-not-consistent-stats.png!
> This is a retrial with the same settings as above, yet the 95percentile eager 
> retry (red line) did poorly this time at 450s.
> [Data for this test 
> here|http://ryanmcguire.info/ds/graph/graph.html?stats=stats.eager_retry.node_killed.just_20.rc1.try2.json&metric=interval_op_rate&operation=stress-read&smoothing=1]



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (CASSANDRA-5932) Speculative read performance data show unexpected results

2013-10-04 Thread Li Zou (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-5932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13786613#comment-13786613
 ] 

Li Zou commented on CASSANDRA-5932:
---

I just did a "git pull" after your comment (i.e. a couple of seconds ago) for 
trunk. The "git pull" indicates that "Already up-to-date".

I checked the MessagingService.java, there are two addCallback() methods.
* The one without ConsistencyLevel is called by sendRR()
* The one with ConsistencyLevel is called by sendMessageToNonLocalDC()


> Speculative read performance data show unexpected results
> -
>
> Key: CASSANDRA-5932
> URL: https://issues.apache.org/jira/browse/CASSANDRA-5932
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Ryan McGuire
>Assignee: Aleksey Yeschenko
> Fix For: 2.0.2
>
> Attachments: 5932.6692c50412ef7d.compaction.png, 
> 5932-6692c50412ef7d.png, 5932.6692c50412ef7d.rr0.png, 
> 5932.6692c50412ef7d.rr1.png, 5932.ded39c7e1c2fa.logs.tar.gz, 5932.txt, 
> 5933-128_and_200rc1.png, 5933-7a87fc11.png, 5933-logs.tar.gz, 
> 5933-randomized-dsnitch-replica.2.png, 5933-randomized-dsnitch-replica.3.png, 
> 5933-randomized-dsnitch-replica.png, compaction-makes-slow.png, 
> compaction-makes-slow-stats.png, eager-read-looks-promising.png, 
> eager-read-looks-promising-stats.png, eager-read-not-consistent.png, 
> eager-read-not-consistent-stats.png, node-down-increase-performance.png
>
>
> I've done a series of stress tests with eager retries enabled that show 
> undesirable behavior. I'm grouping these behaviours into one ticket as they 
> are most likely related.
> 1) Killing off a node in a 4 node cluster actually increases performance.
> 2) Compactions make nodes slow, even after the compaction is done.
> 3) Eager Reads tend to lessen the *immediate* performance impact of a node 
> going down, but not consistently.
> My Environment:
> 1 stress machine: node0
> 4 C* nodes: node4, node5, node6, node7
> My script:
> node0 writes some data: stress -d node4 -F 3000 -n 3000 -i 5 -l 2 -K 
> 20
> node0 reads some data: stress -d node4 -n 3000 -o read -i 5 -K 20
> h3. Examples:
> h5. A node going down increases performance:
> !node-down-increase-performance.png!
> [Data for this test 
> here|http://ryanmcguire.info/ds/graph/graph.html?stats=stats.eager_retry.node_killed.just_20.json&metric=interval_op_rate&operation=stress-read&smoothing=1]
> At 450s, I kill -9 one of the nodes. There is a brief decrease in performance 
> as the snitch adapts, but then it recovers... to even higher performance than 
> before.
> h5. Compactions make nodes permanently slow:
> !compaction-makes-slow.png!
> !compaction-makes-slow-stats.png!
> The green and orange lines represent trials with eager retry enabled, they 
> never recover their op-rate from before the compaction as the red and blue 
> lines do.
> [Data for this test 
> here|http://ryanmcguire.info/ds/graph/graph.html?stats=stats.eager_retry.compaction.2.json&metric=interval_op_rate&operation=stress-read&smoothing=1]
> h5. Speculative Read tends to lessen the *immediate* impact:
> !eager-read-looks-promising.png!
> !eager-read-looks-promising-stats.png!
> This graph looked the most promising to me, the two trials with eager retry, 
> the green and orange line, at 450s showed the smallest dip in performance. 
> [Data for this test 
> here|http://ryanmcguire.info/ds/graph/graph.html?stats=stats.eager_retry.node_killed.json&metric=interval_op_rate&operation=stress-read&smoothing=1]
> h5. But not always:
> !eager-read-not-consistent.png!
> !eager-read-not-consistent-stats.png!
> This is a retrial with the same settings as above, yet the 95percentile eager 
> retry (red line) did poorly this time at 450s.
> [Data for this test 
> here|http://ryanmcguire.info/ds/graph/graph.html?stats=stats.eager_retry.node_killed.just_20.rc1.try2.json&metric=interval_op_rate&operation=stress-read&smoothing=1]



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (CASSANDRA-5932) Speculative read performance data show unexpected results

2013-10-04 Thread Li Zou (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-5932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13786585#comment-13786585
 ] 

Li Zou commented on CASSANDRA-5932:
---

Have done some testing using today's trunk. Have observed following issues.

*Issue 1* -- The first method {{MessagingService.addCallback()}} (i.e. without 
the ConsistencyLevel argument) asserts.

Commenting out the assert statement seems to work. But the Cassandra servers 
themselves will produce 10-second outage (i.e. zero transactions from the 
client point of view) periodically.

*Issue 2* -- The Speculative Retry seems stop retrying during the outage window.

During the outage window triggered either by killing one of Cassandra nodes or 
produced by Cassandra servers themselves, the JConsole shows that the JMX 
stats, SpeculativeRetry counter stops incrementing until the gossip figures out 
the outage issue.

What is the reason for this? The Speculative Retry is meant to help during the 
outage period. This observed behavior is consistent with Cassandra 2.0.0-rc2.


> Speculative read performance data show unexpected results
> -
>
> Key: CASSANDRA-5932
> URL: https://issues.apache.org/jira/browse/CASSANDRA-5932
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Ryan McGuire
>Assignee: Aleksey Yeschenko
> Fix For: 2.0.2
>
> Attachments: 5932.6692c50412ef7d.compaction.png, 
> 5932-6692c50412ef7d.png, 5932.6692c50412ef7d.rr0.png, 
> 5932.6692c50412ef7d.rr1.png, 5932.ded39c7e1c2fa.logs.tar.gz, 5932.txt, 
> 5933-128_and_200rc1.png, 5933-7a87fc11.png, 5933-logs.tar.gz, 
> 5933-randomized-dsnitch-replica.2.png, 5933-randomized-dsnitch-replica.3.png, 
> 5933-randomized-dsnitch-replica.png, compaction-makes-slow.png, 
> compaction-makes-slow-stats.png, eager-read-looks-promising.png, 
> eager-read-looks-promising-stats.png, eager-read-not-consistent.png, 
> eager-read-not-consistent-stats.png, node-down-increase-performance.png
>
>
> I've done a series of stress tests with eager retries enabled that show 
> undesirable behavior. I'm grouping these behaviours into one ticket as they 
> are most likely related.
> 1) Killing off a node in a 4 node cluster actually increases performance.
> 2) Compactions make nodes slow, even after the compaction is done.
> 3) Eager Reads tend to lessen the *immediate* performance impact of a node 
> going down, but not consistently.
> My Environment:
> 1 stress machine: node0
> 4 C* nodes: node4, node5, node6, node7
> My script:
> node0 writes some data: stress -d node4 -F 3000 -n 3000 -i 5 -l 2 -K 
> 20
> node0 reads some data: stress -d node4 -n 3000 -o read -i 5 -K 20
> h3. Examples:
> h5. A node going down increases performance:
> !node-down-increase-performance.png!
> [Data for this test 
> here|http://ryanmcguire.info/ds/graph/graph.html?stats=stats.eager_retry.node_killed.just_20.json&metric=interval_op_rate&operation=stress-read&smoothing=1]
> At 450s, I kill -9 one of the nodes. There is a brief decrease in performance 
> as the snitch adapts, but then it recovers... to even higher performance than 
> before.
> h5. Compactions make nodes permanently slow:
> !compaction-makes-slow.png!
> !compaction-makes-slow-stats.png!
> The green and orange lines represent trials with eager retry enabled, they 
> never recover their op-rate from before the compaction as the red and blue 
> lines do.
> [Data for this test 
> here|http://ryanmcguire.info/ds/graph/graph.html?stats=stats.eager_retry.compaction.2.json&metric=interval_op_rate&operation=stress-read&smoothing=1]
> h5. Speculative Read tends to lessen the *immediate* impact:
> !eager-read-looks-promising.png!
> !eager-read-looks-promising-stats.png!
> This graph looked the most promising to me, the two trials with eager retry, 
> the green and orange line, at 450s showed the smallest dip in performance. 
> [Data for this test 
> here|http://ryanmcguire.info/ds/graph/graph.html?stats=stats.eager_retry.node_killed.json&metric=interval_op_rate&operation=stress-read&smoothing=1]
> h5. But not always:
> !eager-read-not-consistent.png!
> !eager-read-not-consistent-stats.png!
> This is a retrial with the same settings as above, yet the 95percentile eager 
> retry (red line) did poorly this time at 450s.
> [Data for this test 
> here|http://ryanmcguire.info/ds/graph/graph.html?stats=stats.eager_retry.node_killed.just_20.rc1.try2.json&metric=interval_op_rate&operation=stress-read&smoothing=1]



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (CASSANDRA-5932) Speculative read performance data show unexpected results

2013-10-02 Thread Li Zou (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-5932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13784089#comment-13784089
 ] 

Li Zou commented on CASSANDRA-5932:
---

This testing result is reasonable and what is expected.

For PERCENTILE / CUSTOM configuration, the larger the 
{{cfs.sampleLatencyNanos}} the smaller the throughput impact for normal 
operations before the outage. However, during the outage period, the situation 
is reversed, i.e. the smaller {{cfs.sampleLatencyNanos}}, the smaller the 
throughput impact will be, as it times out quicker and triggers the speculative 
retries.

For the ALWAYS configuration, as it always sends out one speculative in 
addition to the usual read requests, the throughput performance should be lower 
than those of PERCENTILE / CUSTOM for normal operations before the outage. 
Since it always sends out the speculative retries, the throughput impact during 
the outage period should be the smallest. The testing result indicates that 
this is true.


> Speculative read performance data show unexpected results
> -
>
> Key: CASSANDRA-5932
> URL: https://issues.apache.org/jira/browse/CASSANDRA-5932
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Ryan McGuire
>Assignee: Aleksey Yeschenko
> Fix For: 2.0.2
>
> Attachments: 5932.6692c50412ef7d.compaction.png, 
> 5932-6692c50412ef7d.png, 5932.6692c50412ef7d.rr0.png, 
> 5932.ded39c7e1c2fa.logs.tar.gz, 5932.txt, 5933-128_and_200rc1.png, 
> 5933-7a87fc11.png, 5933-logs.tar.gz, 5933-randomized-dsnitch-replica.2.png, 
> 5933-randomized-dsnitch-replica.3.png, 5933-randomized-dsnitch-replica.png, 
> compaction-makes-slow.png, compaction-makes-slow-stats.png, 
> eager-read-looks-promising.png, eager-read-looks-promising-stats.png, 
> eager-read-not-consistent.png, eager-read-not-consistent-stats.png, 
> node-down-increase-performance.png
>
>
> I've done a series of stress tests with eager retries enabled that show 
> undesirable behavior. I'm grouping these behaviours into one ticket as they 
> are most likely related.
> 1) Killing off a node in a 4 node cluster actually increases performance.
> 2) Compactions make nodes slow, even after the compaction is done.
> 3) Eager Reads tend to lessen the *immediate* performance impact of a node 
> going down, but not consistently.
> My Environment:
> 1 stress machine: node0
> 4 C* nodes: node4, node5, node6, node7
> My script:
> node0 writes some data: stress -d node4 -F 3000 -n 3000 -i 5 -l 2 -K 
> 20
> node0 reads some data: stress -d node4 -n 3000 -o read -i 5 -K 20
> h3. Examples:
> h5. A node going down increases performance:
> !node-down-increase-performance.png!
> [Data for this test 
> here|http://ryanmcguire.info/ds/graph/graph.html?stats=stats.eager_retry.node_killed.just_20.json&metric=interval_op_rate&operation=stress-read&smoothing=1]
> At 450s, I kill -9 one of the nodes. There is a brief decrease in performance 
> as the snitch adapts, but then it recovers... to even higher performance than 
> before.
> h5. Compactions make nodes permanently slow:
> !compaction-makes-slow.png!
> !compaction-makes-slow-stats.png!
> The green and orange lines represent trials with eager retry enabled, they 
> never recover their op-rate from before the compaction as the red and blue 
> lines do.
> [Data for this test 
> here|http://ryanmcguire.info/ds/graph/graph.html?stats=stats.eager_retry.compaction.2.json&metric=interval_op_rate&operation=stress-read&smoothing=1]
> h5. Speculative Read tends to lessen the *immediate* impact:
> !eager-read-looks-promising.png!
> !eager-read-looks-promising-stats.png!
> This graph looked the most promising to me, the two trials with eager retry, 
> the green and orange line, at 450s showed the smallest dip in performance. 
> [Data for this test 
> here|http://ryanmcguire.info/ds/graph/graph.html?stats=stats.eager_retry.node_killed.json&metric=interval_op_rate&operation=stress-read&smoothing=1]
> h5. But not always:
> !eager-read-not-consistent.png!
> !eager-read-not-consistent-stats.png!
> This is a retrial with the same settings as above, yet the 95percentile eager 
> retry (red line) did poorly this time at 450s.
> [Data for this test 
> here|http://ryanmcguire.info/ds/graph/graph.html?stats=stats.eager_retry.node_killed.just_20.rc1.try2.json&metric=interval_op_rate&operation=stress-read&smoothing=1]



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (CASSANDRA-5932) Speculative read performance data show unexpected results

2013-09-30 Thread Li Zou (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-5932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13782280#comment-13782280
 ] 

Li Zou commented on CASSANDRA-5932:
---

Thanks for the clarification of the sendRR issue.

Since the Randomized approach is not checked in, let us skip over it.

For the _data repair_, do we need to block waiting for the acks?

> Speculative read performance data show unexpected results
> -
>
> Key: CASSANDRA-5932
> URL: https://issues.apache.org/jira/browse/CASSANDRA-5932
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Ryan McGuire
>Assignee: Aleksey Yeschenko
> Fix For: 2.0.2
>
> Attachments: 5932-6692c50412ef7d.png, 5932.ded39c7e1c2fa.logs.tar.gz, 
> 5932.txt, 5933-128_and_200rc1.png, 5933-7a87fc11.png, 5933-logs.tar.gz, 
> 5933-randomized-dsnitch-replica.2.png, 5933-randomized-dsnitch-replica.3.png, 
> 5933-randomized-dsnitch-replica.png, compaction-makes-slow.png, 
> compaction-makes-slow-stats.png, eager-read-looks-promising.png, 
> eager-read-looks-promising-stats.png, eager-read-not-consistent.png, 
> eager-read-not-consistent-stats.png, node-down-increase-performance.png
>
>
> I've done a series of stress tests with eager retries enabled that show 
> undesirable behavior. I'm grouping these behaviours into one ticket as they 
> are most likely related.
> 1) Killing off a node in a 4 node cluster actually increases performance.
> 2) Compactions make nodes slow, even after the compaction is done.
> 3) Eager Reads tend to lessen the *immediate* performance impact of a node 
> going down, but not consistently.
> My Environment:
> 1 stress machine: node0
> 4 C* nodes: node4, node5, node6, node7
> My script:
> node0 writes some data: stress -d node4 -F 3000 -n 3000 -i 5 -l 2 -K 
> 20
> node0 reads some data: stress -d node4 -n 3000 -o read -i 5 -K 20
> h3. Examples:
> h5. A node going down increases performance:
> !node-down-increase-performance.png!
> [Data for this test 
> here|http://ryanmcguire.info/ds/graph/graph.html?stats=stats.eager_retry.node_killed.just_20.json&metric=interval_op_rate&operation=stress-read&smoothing=1]
> At 450s, I kill -9 one of the nodes. There is a brief decrease in performance 
> as the snitch adapts, but then it recovers... to even higher performance than 
> before.
> h5. Compactions make nodes permanently slow:
> !compaction-makes-slow.png!
> !compaction-makes-slow-stats.png!
> The green and orange lines represent trials with eager retry enabled, they 
> never recover their op-rate from before the compaction as the red and blue 
> lines do.
> [Data for this test 
> here|http://ryanmcguire.info/ds/graph/graph.html?stats=stats.eager_retry.compaction.2.json&metric=interval_op_rate&operation=stress-read&smoothing=1]
> h5. Speculative Read tends to lessen the *immediate* impact:
> !eager-read-looks-promising.png!
> !eager-read-looks-promising-stats.png!
> This graph looked the most promising to me, the two trials with eager retry, 
> the green and orange line, at 450s showed the smallest dip in performance. 
> [Data for this test 
> here|http://ryanmcguire.info/ds/graph/graph.html?stats=stats.eager_retry.node_killed.json&metric=interval_op_rate&operation=stress-read&smoothing=1]
> h5. But not always:
> !eager-read-not-consistent.png!
> !eager-read-not-consistent-stats.png!
> This is a retrial with the same settings as above, yet the 95percentile eager 
> retry (red line) did poorly this time at 450s.
> [Data for this test 
> here|http://ryanmcguire.info/ds/graph/graph.html?stats=stats.eager_retry.node_killed.just_20.rc1.try2.json&metric=interval_op_rate&operation=stress-read&smoothing=1]



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Comment Edited] (CASSANDRA-5932) Speculative read performance data show unexpected results

2013-09-30 Thread Li Zou (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-5932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13782027#comment-13782027
 ] 

Li Zou edited comment on CASSANDRA-5932 at 9/30/13 6:04 PM:


Hello [~iamaleksey] and [~jbellis],

It appears to me that the testing results have suggested that the "_data read + 
speculative retry_" path work as expected. This "_data read + speculative 
retry_" path has greatly minimized the throughput impact caused by the failure 
of one of Cassandra server nodes.

The observed small degradation of throughput performance when _speculative 
retry_ is enabled is very likely to be caused by the "*_read repair_*" path. I 
did the code reading of this path last Friday and noticed some design / coding 
issues. I would like to discuss them with you.

Please note that my code base is still the Cassandra 2.0.0 tarball, not updated 
with the latest code changes.

*Issue 1* -- When handling {{DigestMismatchException}} in 
{{StorageProxy.fetchRows()}}, all _data read requests_ are sent out using 
{{sendRR}} without distinguishing remote nodes from the local node.

Will this cause an issue, as {{MessagingService.instance().sendRR()}} will send 
out enqueued messages for a specified remote node via its pre-established TCP 
socket connection. For local node, this should be done via 
{{LocalReadRunnable}}, i.e. {{StageManager.getStage(Stage.READ).execute(new 
LocalReadRunnable(command, handler))}}.

If this may cause an issue, the following wait may block.

{noformat}
// read the results for the digest mismatch retries
if (repairResponseHandlers != null)
{
for (int i = 0; i < repairCommands.size(); i++)
{
ReadCommand command = repairCommands.get(i);
ReadCallback handler = 
repairResponseHandlers.get(i);

Row row;
try
{
row = handler.get();
}
{noformat}

For two reasons.
* The data read request for local node may never sent out
* As one of the nodes is down (which triggered the Speculative Retry) will 
cause one missing response.

*If missing two responses, this will block for 10 seconds*. 

*Issue 2* -- For _data repair_, {{RowDataResolver.resolve()}} has a similar 
issue as it calls  {{scheduleRepairs()}} to send out  messages using sendRR() 
without distinguishing remote nodes from the local node.

*Issue 3* -- When handling _data repair_, {{StorageProxy.fetchRows()}} blocks 
waiting for acks to all of {{data repair}} requests sent out using sendRR(). 
This may cause the thread to block.

For _data repair_ path, *data requests* are sent out and then compare / merge 
the received responses; send out the merged / diff version and then block for 
acks.

How do we handle the case for _local node_? Does the sendRR() and the 
corresponding receive part can handle the case for local node? If not, then 
this may block for 10 seconds.

{noformat}
if (repairResponseHandlers != null)
{
for (int i = 0; i < repairCommands.size(); i++)
{
ReadCommand command = repairCommands.get(i);
ReadCallback handler = 
repairResponseHandlers.get(i);

Row row;
try
{
row = handler.get();
}
catch (DigestMismatchException e)
...
RowDataResolver resolver = 
(RowDataResolver)handler.resolver;
try
{
// wait for the repair writes to be acknowledged, to 
minimize impact on any replica that's
// behind on writes in case the out-of-sync row is read 
multiple times in quick succession
FBUtilities.waitOnFutures(resolver.repairResults, 
DatabaseDescriptor.getWriteRpcTimeout());
}
catch (TimeoutException e)
{
Tracing.trace("Timed out on digest mismatch retries");
int blockFor = 
consistency_level.blockFor(Keyspace.open(command.getKeyspace()));
throw new ReadTimeoutException(consistency_level, 
blockFor, blockFor, true);
}
{noformat}

*Question for waiting for the ack* -- Do we really need to wait for the ack?

We should assume the best effort approach, i.e. do the data repair and then 
return. No need to block waiting for the acks for confirmation.

*Question for the Randomized approach* -- Since the end points are randomized, 
the first node in the list is no likely the local node. This may cause a higher 
possibility of data repair.

In the *Randomized Approach*, the end points are reshuffled. Then, th

[jira] [Commented] (CASSANDRA-5932) Speculative read performance data show unexpected results

2013-09-30 Thread Li Zou (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-5932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13782027#comment-13782027
 ] 

Li Zou commented on CASSANDRA-5932:
---

Hello [~iamaleksey] and [~jbellis],

It appears to me that the testing results have suggested that the "_data read + 
speculative retry_" path work as expected. This "_data read + speculative 
retry_" path has greatly minimized the throughput impact caused by the failure 
of one of Cassandra server nodes.

The observed small degradation of throughput performance when _speculative 
retry_ is enabled is very likely to be caused by the "*_read repair_*" path. I 
did the code reading of this path last Friday and noticed some design / coding 
issues. I would like to discuss them with you.

Please note that my code base is still the Cassandra 2.0.0 tarball, not updated 
with the latest code changes.

*Issue 1* -- When handling {{DigestMismatchException}} in 
{{StorageProxy.fetchRows()}}, all _data read requests_ are sent out using 
{{sendRR}} without distinguishing remote nodes from the local node.

Will this cause an issue, as {{MessagingService.instance().sendRR()}} will send 
out enqueued messages for a specified remote node via its pre-established TCP 
socket connection. For local node, this should be done via 
{{LocalReadRunnable}}, i.e. {{StageManager.getStage(Stage.READ).execute(new 
LocalReadRunnable(command, handler))}}.

If this may cause an issue, the following wait may block.

{noformat}
// read the results for the digest mismatch retries
if (repairResponseHandlers != null)
{
for (int i = 0; i < repairCommands.size(); i++)
{
ReadCommand command = repairCommands.get(i);
ReadCallback handler = 
repairResponseHandlers.get(i);

Row row;
try
{
row = handler.get();
}
{noformat}

For two reasons.
* The data read request for local node may never sent out
* As one of the nodes is down (which triggered the Speculative Retry) will 
cause one missing response.

*If missing two responses, this will block for 10 seconds*. 

*Issue 2* -- For _data repair_, {{RowDataResolver.resolve()}} has a similar 
issue as it calls  {{scheduleRepairs()}} to send out  messages using sendRR() 
without distinguishing remote nodes from the local node.

*Issue 3* -- When handling _data repair_, {{StorageProxy.fetchRows()}} blocks 
waiting for acks to all of {{data repair}} requests sent out using sendRR(). 
This may cause the thread to block.

For _data repair_ path, *data requests* are sent out and then compare / merge 
the received responses; send out the merged / diff version and then block for 
acks.

How do we handle the case for _local node_? Does the sendRR() and the 
corresponding receive part can handle the case for local node? If not, then 
this may block for 10 seconds.

{noformat}
if (repairResponseHandlers != null)
{
for (int i = 0; i < repairCommands.size(); i++)
{
ReadCommand command = repairCommands.get(i);
ReadCallback handler = 
repairResponseHandlers.get(i);

Row row;
try
{
row = handler.get();
}
catch (DigestMismatchException e)
...
RowDataResolver resolver = 
(RowDataResolver)handler.resolver;
try
{
// wait for the repair writes to be acknowledged, to 
minimize impact on any replica that's
// behind on writes in case the out-of-sync row is read 
multiple times in quick succession
FBUtilities.waitOnFutures(resolver.repairResults, 
DatabaseDescriptor.getWriteRpcTimeout());
}
catch (TimeoutException e)
{
Tracing.trace("Timed out on digest mismatch retries");
int blockFor = 
consistency_level.blockFor(Keyspace.open(command.getKeyspace()));
throw new ReadTimeoutException(consistency_level, 
blockFor, blockFor, true);
}
{noformat}

*Question for waiting for the ack* -- Do we really need to wait for the ack?

We should assume the best effort approach, i.e. do the data repair and then 
return. No need to block waiting for the acks for confirmation.

*Question for the Randomized approach* -- Since the end points are randomized, 
the first node in the list is no likely the local node. This may cause a higher 
possibility of data repair.

In the *Randomized Approach*, the end points are reshuffled. Then, the first 
node in the list used for _data read req

[jira] [Commented] (CASSANDRA-5932) Speculative read performance data show unexpected results

2013-09-27 Thread Li Zou (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-5932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13780473#comment-13780473
 ] 

Li Zou commented on CASSANDRA-5932:
---

I did some more code reading and noticed some potential issues and possible 
improvement. I've got to run now. I will get back to you guys Monday morning.

My guess is that the ??Speculative NONE?? is hit by the initial request reading 
path which is successfully resolved by the *Speculative Retry*. The observed 
throughput performance hit when Speculative Retry is enabled is caused by the 
ReadRepair path which has some coding / design issues. I will talk to you next 
Monday.

> Speculative read performance data show unexpected results
> -
>
> Key: CASSANDRA-5932
> URL: https://issues.apache.org/jira/browse/CASSANDRA-5932
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Ryan McGuire
>Assignee: Aleksey Yeschenko
> Fix For: 2.0.2
>
> Attachments: 5932.txt, 5933-128_and_200rc1.png, 5933-7a87fc11.png, 
> 5933-logs.tar.gz, 5933-randomized-dsnitch-replica.2.png, 
> 5933-randomized-dsnitch-replica.3.png, 5933-randomized-dsnitch-replica.png, 
> compaction-makes-slow.png, compaction-makes-slow-stats.png, 
> eager-read-looks-promising.png, eager-read-looks-promising-stats.png, 
> eager-read-not-consistent.png, eager-read-not-consistent-stats.png, 
> node-down-increase-performance.png
>
>
> I've done a series of stress tests with eager retries enabled that show 
> undesirable behavior. I'm grouping these behaviours into one ticket as they 
> are most likely related.
> 1) Killing off a node in a 4 node cluster actually increases performance.
> 2) Compactions make nodes slow, even after the compaction is done.
> 3) Eager Reads tend to lessen the *immediate* performance impact of a node 
> going down, but not consistently.
> My Environment:
> 1 stress machine: node0
> 4 C* nodes: node4, node5, node6, node7
> My script:
> node0 writes some data: stress -d node4 -F 3000 -n 3000 -i 5 -l 2 -K 
> 20
> node0 reads some data: stress -d node4 -n 3000 -o read -i 5 -K 20
> h3. Examples:
> h5. A node going down increases performance:
> !node-down-increase-performance.png!
> [Data for this test 
> here|http://ryanmcguire.info/ds/graph/graph.html?stats=stats.eager_retry.node_killed.just_20.json&metric=interval_op_rate&operation=stress-read&smoothing=1]
> At 450s, I kill -9 one of the nodes. There is a brief decrease in performance 
> as the snitch adapts, but then it recovers... to even higher performance than 
> before.
> h5. Compactions make nodes permanently slow:
> !compaction-makes-slow.png!
> !compaction-makes-slow-stats.png!
> The green and orange lines represent trials with eager retry enabled, they 
> never recover their op-rate from before the compaction as the red and blue 
> lines do.
> [Data for this test 
> here|http://ryanmcguire.info/ds/graph/graph.html?stats=stats.eager_retry.compaction.2.json&metric=interval_op_rate&operation=stress-read&smoothing=1]
> h5. Speculative Read tends to lessen the *immediate* impact:
> !eager-read-looks-promising.png!
> !eager-read-looks-promising-stats.png!
> This graph looked the most promising to me, the two trials with eager retry, 
> the green and orange line, at 450s showed the smallest dip in performance. 
> [Data for this test 
> here|http://ryanmcguire.info/ds/graph/graph.html?stats=stats.eager_retry.node_killed.json&metric=interval_op_rate&operation=stress-read&smoothing=1]
> h5. But not always:
> !eager-read-not-consistent.png!
> !eager-read-not-consistent-stats.png!
> This is a retrial with the same settings as above, yet the 95percentile eager 
> retry (red line) did poorly this time at 450s.
> [Data for this test 
> here|http://ryanmcguire.info/ds/graph/graph.html?stats=stats.eager_retry.node_killed.just_20.rc1.try2.json&metric=interval_op_rate&operation=stress-read&smoothing=1]

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (CASSANDRA-5932) Speculative read performance data show unexpected results

2013-09-26 Thread Li Zou (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-5932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13779233#comment-13779233
 ] 

Li Zou commented on CASSANDRA-5932:
---

The logic for {{AlwaysSpeculatingReadExecutor}} is good. What I meant in my 
previous comment is that when {{targetReplicas.size() == allReplicas.size()}} 
and {{targetReplicas.size() == 1}}, then 
{{AlwaysSpeculatingReadExecutor.executeAsync()}} will throw an exception as 
there is only one endpoint in {{targetReplicas}}, but it tries to access two 
endpoints in {{targetReplicas}}.

> Speculative read performance data show unexpected results
> -
>
> Key: CASSANDRA-5932
> URL: https://issues.apache.org/jira/browse/CASSANDRA-5932
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Ryan McGuire
>Assignee: Aleksey Yeschenko
> Fix For: 2.0.2
>
> Attachments: 5932.txt, compaction-makes-slow.png, 
> compaction-makes-slow-stats.png, eager-read-looks-promising.png, 
> eager-read-looks-promising-stats.png, eager-read-not-consistent.png, 
> eager-read-not-consistent-stats.png, node-down-increase-performance.png
>
>
> I've done a series of stress tests with eager retries enabled that show 
> undesirable behavior. I'm grouping these behaviours into one ticket as they 
> are most likely related.
> 1) Killing off a node in a 4 node cluster actually increases performance.
> 2) Compactions make nodes slow, even after the compaction is done.
> 3) Eager Reads tend to lessen the *immediate* performance impact of a node 
> going down, but not consistently.
> My Environment:
> 1 stress machine: node0
> 4 C* nodes: node4, node5, node6, node7
> My script:
> node0 writes some data: stress -d node4 -F 3000 -n 3000 -i 5 -l 2 -K 
> 20
> node0 reads some data: stress -d node4 -n 3000 -o read -i 5 -K 20
> h3. Examples:
> h5. A node going down increases performance:
> !node-down-increase-performance.png!
> [Data for this test 
> here|http://ryanmcguire.info/ds/graph/graph.html?stats=stats.eager_retry.node_killed.just_20.json&metric=interval_op_rate&operation=stress-read&smoothing=1]
> At 450s, I kill -9 one of the nodes. There is a brief decrease in performance 
> as the snitch adapts, but then it recovers... to even higher performance than 
> before.
> h5. Compactions make nodes permanently slow:
> !compaction-makes-slow.png!
> !compaction-makes-slow-stats.png!
> The green and orange lines represent trials with eager retry enabled, they 
> never recover their op-rate from before the compaction as the red and blue 
> lines do.
> [Data for this test 
> here|http://ryanmcguire.info/ds/graph/graph.html?stats=stats.eager_retry.compaction.2.json&metric=interval_op_rate&operation=stress-read&smoothing=1]
> h5. Speculative Read tends to lessen the *immediate* impact:
> !eager-read-looks-promising.png!
> !eager-read-looks-promising-stats.png!
> This graph looked the most promising to me, the two trials with eager retry, 
> the green and orange line, at 450s showed the smallest dip in performance. 
> [Data for this test 
> here|http://ryanmcguire.info/ds/graph/graph.html?stats=stats.eager_retry.node_killed.json&metric=interval_op_rate&operation=stress-read&smoothing=1]
> h5. But not always:
> !eager-read-not-consistent.png!
> !eager-read-not-consistent-stats.png!
> This is a retrial with the same settings as above, yet the 95percentile eager 
> retry (red line) did poorly this time at 450s.
> [Data for this test 
> here|http://ryanmcguire.info/ds/graph/graph.html?stats=stats.eager_retry.node_killed.just_20.rc1.try2.json&metric=interval_op_rate&operation=stress-read&smoothing=1]

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (CASSANDRA-5932) Speculative read performance data show unexpected results

2013-09-26 Thread Li Zou (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-5932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13779202#comment-13779202
 ] 

Li Zou commented on CASSANDRA-5932:
---

Hello [~iamaleksey] and [~jbellis],

I took a quick look at the code changes. The new code looks very good to me. 
But I saw one potential issue in 
{{AlwaysSpeculatingReadExecutor.executeAsync()}}, in which it makes at least 
*two* data / digest requests. This will cause problems for a data center with 
only one Cassandra server node (e.g. bring up an embedded Cassandra node in JVM 
for JUnit test) or a deployed production data center of two Cassandra server 
nodes with one node shut down for maintenance. In the above mentioned two 
cases, {{AbstractReadExecutor.getReadExecutor()}} will return the 
{{AlwaysSpeculatingReadExecutor}} as condition {{(targetReplicas.size() == 
allReplicas.size())}} is met, though the tables may / may not be configured 
with ??Speculative ALWAYS??.

It is true for our legacy products we are considering to deploy each data 
center with only two Cassandra server nodes with RF = 2 and CL = 1.


> Speculative read performance data show unexpected results
> -
>
> Key: CASSANDRA-5932
> URL: https://issues.apache.org/jira/browse/CASSANDRA-5932
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Ryan McGuire
>Assignee: Aleksey Yeschenko
> Fix For: 2.0.2
>
> Attachments: 5932.txt, compaction-makes-slow.png, 
> compaction-makes-slow-stats.png, eager-read-looks-promising.png, 
> eager-read-looks-promising-stats.png, eager-read-not-consistent.png, 
> eager-read-not-consistent-stats.png, node-down-increase-performance.png
>
>
> I've done a series of stress tests with eager retries enabled that show 
> undesirable behavior. I'm grouping these behaviours into one ticket as they 
> are most likely related.
> 1) Killing off a node in a 4 node cluster actually increases performance.
> 2) Compactions make nodes slow, even after the compaction is done.
> 3) Eager Reads tend to lessen the *immediate* performance impact of a node 
> going down, but not consistently.
> My Environment:
> 1 stress machine: node0
> 4 C* nodes: node4, node5, node6, node7
> My script:
> node0 writes some data: stress -d node4 -F 3000 -n 3000 -i 5 -l 2 -K 
> 20
> node0 reads some data: stress -d node4 -n 3000 -o read -i 5 -K 20
> h3. Examples:
> h5. A node going down increases performance:
> !node-down-increase-performance.png!
> [Data for this test 
> here|http://ryanmcguire.info/ds/graph/graph.html?stats=stats.eager_retry.node_killed.just_20.json&metric=interval_op_rate&operation=stress-read&smoothing=1]
> At 450s, I kill -9 one of the nodes. There is a brief decrease in performance 
> as the snitch adapts, but then it recovers... to even higher performance than 
> before.
> h5. Compactions make nodes permanently slow:
> !compaction-makes-slow.png!
> !compaction-makes-slow-stats.png!
> The green and orange lines represent trials with eager retry enabled, they 
> never recover their op-rate from before the compaction as the red and blue 
> lines do.
> [Data for this test 
> here|http://ryanmcguire.info/ds/graph/graph.html?stats=stats.eager_retry.compaction.2.json&metric=interval_op_rate&operation=stress-read&smoothing=1]
> h5. Speculative Read tends to lessen the *immediate* impact:
> !eager-read-looks-promising.png!
> !eager-read-looks-promising-stats.png!
> This graph looked the most promising to me, the two trials with eager retry, 
> the green and orange line, at 450s showed the smallest dip in performance. 
> [Data for this test 
> here|http://ryanmcguire.info/ds/graph/graph.html?stats=stats.eager_retry.node_killed.json&metric=interval_op_rate&operation=stress-read&smoothing=1]
> h5. But not always:
> !eager-read-not-consistent.png!
> !eager-read-not-consistent-stats.png!
> This is a retrial with the same settings as above, yet the 95percentile eager 
> retry (red line) did poorly this time at 450s.
> [Data for this test 
> here|http://ryanmcguire.info/ds/graph/graph.html?stats=stats.eager_retry.node_killed.just_20.rc1.try2.json&metric=interval_op_rate&operation=stress-read&smoothing=1]

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (CASSANDRA-5932) Speculative read performance data show unexpected results

2013-09-24 Thread Li Zou (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-5932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13776805#comment-13776805
 ] 

Li Zou commented on CASSANDRA-5932:
---

Hello [~iamaleksey],

Thanks for the link to this jira and for your very detailed testing results. It 
confirms what we have seen in our lab testing for the Cassandra 2.0.0-rc2 
"Speculative Execution for Reads".

We have a very simple data center setup consisting of four Cassandra nodes 
running on four server machines. A testing application (Cassandra client) is 
interacting with Cassandra nodes 1, 2 and 3. That is, the testing app does not 
directly connected to the Cassandra node 4.

The keyspace Replication Factor is set to 3 and the client requested 
Consistency Level is set to CL_TWO.

I have tested all of three configurations of the Speculative Execution for 
Reads ('ALWAYS', '85 PERCENTILE', '50 MS' / '100 MS'). It seems that none of 
them works as expected. From the test app log file point of view, they all give 
a 20-second window of outage immediately after the 4th node was killed. This 
behavior is consistent to Cassandra 1.2.4.

I have done a quick code reading of the Cassandra Server implementation 
(Cassandra 2.0.0 tarball) and I have noticed some design issues. I would like 
to discuss them with you.

*Issue 1* - StorageProxy.fetchRows() may still block for as long as 
conf.read_request_timeout_in_ms, though the speculative retry did fire 
correctly after the Cassandra node 4 was killed.

Take the speculative configuration of 'PERCENTILE' / 'CUSTOM' as example, after 
the Cassandra node 4 was killed, SpeculativeReadExecutor.speculate() would 
block for responses. If timed out, it would send out one more read request to 
an alternative node (from {{unfiltered}}) and increment the speculativeRetry 
counter. This part should work.

However, killing the 4th node would very likely cause inconsistency in the 
database and this will trigger the DigestMismatchException. In the fetchRows(), 
when handling DigestMismatchException, it uses handler.endpoints to send out 
digest mismatch retries and then block for responses. As we know that one of 
the endpoints was already killed, the handler.get() will block until it is 
timed out, which is 10 seconds.


{noformat}
catch (DigestMismatchException ex)
{
Tracing.trace("Digest mismatch: {}", ex);

...

MessageOut message = 
exec.command.createMessage();
for (InetAddress endpoint : exec.handler.endpoints)
{
Tracing.trace("Enqueuing full data read to {}", 
endpoint);
MessagingService.instance().sendRR(message, endpoint, 
repairHandler);
}
}
}

...

// read the results for the digest mismatch retries
if (repairResponseHandlers != null)
{
for (int i = 0; i < repairCommands.size(); i++)
{
ReadCommand command = repairCommands.get(i);
ReadCallback handler = 
repairResponseHandlers.get(i);

Row row;
try
{
row = handler.get();
}
{noformat}



*Issue 2* - The speculative 'ALWAYS' does NOT send out any more read requests. 
Thus, in face of the failure of node 4, it will not help at all.

The SpeculateAlwaysExecutor.executeAsync() only sends out 
handler.endpoints.size() number of read requests and it blocks for the 
responses to come back. If one of the nodes is killed, say node 4, this 
speculative retry 'ALWAYS' will work the same way as Cassandra 1.2.4, i.e. it 
will block until timed out, which is 10 seconds.

??My understanding of this speculative retry 'ALWAYS' should ALWAYS send out 
"handler.endpoints.size() + 1" number of read requests and block for 
handler.endpoints.size() number of responses??.

*Issue 3* - Since the ReadRepairDecison is determined by a Random() number, 
this speculative retry may not work as the ReadRepairDecision may be 
??ReadRepairDecision.GLOBAL??

*Issue 4* - For the ReadExecutor(s), the {{this.unfiltered}} and 
{{this.endpoints}} may not consistent. Thus, using {{this.unfiltered}} and 
{{this.endpoints}} for speculative retry may cause unexpected results. This is 
especially true when the Consistency Level is {{LOCAL_QUARUM}} and the 
ReadRepairDecision is {{DC_LOCAL}}.





> Speculative read performance data show unexpected results
> -
>
> Key: CASSANDRA-5932
> URL: https://issues.apache.org/jira/browse/CASSANDRA-5932
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Ryan McGuire
>Assignee:

[jira] [Commented] (CASSANDRA-4705) Speculative execution for reads / eager retries

2013-09-24 Thread Li Zou (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-4705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13776674#comment-13776674
 ] 

Li Zou commented on CASSANDRA-4705:
---

[~iamaleksey] please see my comments at 
[CASSANDRA-5932|https://issues.apache.org/jira/browse/CASSANDRA-5932]

> Speculative execution for reads / eager retries
> ---
>
> Key: CASSANDRA-4705
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4705
> Project: Cassandra
>  Issue Type: Improvement
>Reporter: Vijay
>Assignee: Vijay
> Fix For: 2.0 beta 1
>
> Attachments: 0001-CASSANDRA-4705.patch, 0001-CASSANDRA-4705-v2.patch, 
> 0001-CASSANDRA-4705-v3.patch, 
> 0001-Refactor-to-introduce-AbstractReadExecutor.patch, 
> 0002-Add-Speculative-execution-for-Row-Reads.patch
>
>
> When read_repair is not 1.0, we send the request to one node for some of the 
> requests. When a node goes down or when a node is too busy the client has to 
> wait for the timeout before it can retry. 
> It would be nice to watch for latency and execute an additional request to a 
> different node, if the response is not received within average/99% of the 
> response times recorded in the past.
> CASSANDRA-2540 might be able to solve the variance when read_repair is set to 
> 1.0
> 1) May be we need to use metrics-core to record various Percentiles
> 2) Modify ReadCallback.get to execute additional request speculatively.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (CASSANDRA-4705) Speculative execution for reads / eager retries

2013-09-11 Thread Li Zou (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-4705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13764863#comment-13764863
 ] 

Li Zou commented on CASSANDRA-4705:
---

Hello [~vijay2...@yahoo.com] and [~jbellis],

My company is adopting the Cassandra db technology and we are very much 
interested in this new feature of "Speculative Execution for Reads", as it 
could potentially help reduce the overall outage window to the sub-second range 
upon the failure of any one of Cassandra nodes in a data center.

I have recently tested the "Speculative Execution for Reads" using Cassandra 
2.0.0-rc2 and I have not seen the expected results. In my tests, I have already 
excluded the possible effects from the client (connection pool) side. Not sure 
what is still missed out in my tests.

Here is my setup for the tests. One data center of four Cassandra nodes is 
configured on machine A, B, C and D with each physical machine has one 
Cassandra node running on it. My testing app (Cassandra client) is running on 
machine A and it is configured to connect to Cassandra nodes A, B, and C only. 
In other words, my testing app will never connect to Cassandra node D.

* Replication Factor (RF) is set to 3
* Client requested Consistency Level (CL) is set to CL_TWO

For Cassandra 1.2.4, upon the failure of Cassandra node D (via "kill -9 " 
or "kill -s SIGKILL "), there will be an outage window for around 20 
seconds with zero transactions. The outage ends as soon as the gossip protocol 
detects the failure of Cassandra node D and marks it down.

For Cassandra 2.0.0-rc2, database tables are configured with 
"speculative_retry" proper values (such as 'ALWAYS', '10 ms', '100 ms', '80 
percentile', 'NONE'). No expected results are observed in the tests. The 
testing results are quite similar to those observed for Cassandra 1.2.4. That 
is, upon the failure of Cassandra node D, there will be an outage window for 
around 20 seconds with zero transactions. The outage ends as soon as the gossip 
protocol detects the failure of Cassandra node D and marks it down. The tested 
values for _speculative_retry_ against database table are listed below:

* ALWAYS
* 10 ms
* 100 ms
* 80 percentile
* NONE

In my tests, I also checked the JMX stats of Cassandra node A for the 
speculative_retry for each table. What I observed really surprised me. For an 
instance, the speculative_retry (for each table) is set to '20 ms'. During 
normal operations with all of four Cassandra nodes up, the "SpeculativeRetry" 
count went up occasionally. However, during the 20-second outage window, the 
"SpeculativeRetry" count somehow never went up for three tests in a row. On the 
contrary, I would expect to see lots of speculative retries were executed (say 
on Cassandra node A) during the 20-second outage window.

Some notes for the "kill" signals. I tested if the Cassandra node D is killed 
using SIGTERM (for Cassandra 1.2.4) and SIGINT / SIGTERM (for Cassandra 
2.0.0-rc2), there will be no observed outage in second range, as the Cassandra 
node D does the orderly shutdown and the gossip announces the node "down". From 
my testing app point of view, everything is going very well.

But if the Cassandra node D is killed using SIGKILL, there will be an outage 
with zero transactions. I guess this might be a TCP socket related issue. I 
often observed TCP socket "CLOSE_WAIT" on the passive close side (i.e. 
Cassandra node A, B, and C) and TCP socket TIME_WAIT on the active close side 
(Cassandra node D).


> Speculative execution for reads / eager retries
> ---
>
> Key: CASSANDRA-4705
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4705
> Project: Cassandra
>  Issue Type: Improvement
>Reporter: Vijay
>Assignee: Vijay
> Fix For: 2.0 beta 1
>
> Attachments: 0001-CASSANDRA-4705.patch, 0001-CASSANDRA-4705-v2.patch, 
> 0001-CASSANDRA-4705-v3.patch, 
> 0001-Refactor-to-introduce-AbstractReadExecutor.patch, 
> 0002-Add-Speculative-execution-for-Row-Reads.patch
>
>
> When read_repair is not 1.0, we send the request to one node for some of the 
> requests. When a node goes down or when a node is too busy the client has to 
> wait for the timeout before it can retry. 
> It would be nice to watch for latency and execute an additional request to a 
> different node, if the response is not received within average/99% of the 
> response times recorded in the past.
> CASSANDRA-2540 might be able to solve the variance when read_repair is set to 
> 1.0
> 1) May be we need to use metrics-core to record various Percentiles
> 2) Modify ReadCallback.get to execute additional request speculatively.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information

[jira] [Updated] (CASSANDRA-6187) Cassandra-2.0 node seems to be pausing periodically under normal operations

[jira] [Commented] (CASSANDRA-6187) Cassandra-2.0 node seems to be pausing periodically under normal operations

[jira] [Created] (CASSANDRA-6188) The JMX stats for Speculative Retry stops moving during a node failure outage period

[jira] [Created] (CASSANDRA-6187) Cassandra-2.0 node seems to be pausing periodically under normal operations

[jira] [Commented] (CASSANDRA-6154) Inserts are blocked in 2.1

[jira] [Comment Edited] (CASSANDRA-6154) Inserts are blocked in 2.1

[jira] [Commented] (CASSANDRA-6154) Inserts are blocked in 2.1

[jira] [Commented] (CASSANDRA-5932) Speculative read performance data show unexpected results

[jira] [Commented] (CASSANDRA-5932) Speculative read performance data show unexpected results

[jira] [Comment Edited] (CASSANDRA-5932) Speculative read performance data show unexpected results

[jira] [Commented] (CASSANDRA-5932) Speculative read performance data show unexpected results

[jira] [Comment Edited] (CASSANDRA-5932) Speculative read performance data show unexpected results

[jira] [Commented] (CASSANDRA-5932) Speculative read performance data show unexpected results

[jira] [Comment Edited] (CASSANDRA-5932) Speculative read performance data show unexpected results

[jira] [Commented] (CASSANDRA-5932) Speculative read performance data show unexpected results

[jira] [Commented] (CASSANDRA-5932) Speculative read performance data show unexpected results

[jira] [Commented] (CASSANDRA-5932) Speculative read performance data show unexpected results

[jira] [Commented] (CASSANDRA-5932) Speculative read performance data show unexpected results

[jira] [Comment Edited] (CASSANDRA-5932) Speculative read performance data show unexpected results

[jira] [Commented] (CASSANDRA-5932) Speculative read performance data show unexpected results

[jira] [Commented] (CASSANDRA-5932) Speculative read performance data show unexpected results

[jira] [Commented] (CASSANDRA-5932) Speculative read performance data show unexpected results

[jira] [Commented] (CASSANDRA-5932) Speculative read performance data show unexpected results

[jira] [Commented] (CASSANDRA-5932) Speculative read performance data show unexpected results

[jira] [Commented] (CASSANDRA-4705) Speculative execution for reads / eager retries

[jira] [Commented] (CASSANDRA-4705) Speculative execution for reads / eager retries

26 matches

Site Navigation

Mail list logo

Footer information