[jira] [Commented] (CASSANDRA-2889) Avoids having replicate on write tasks stacking up at CL.ONE
[ https://issues.apache.org/jira/browse/CASSANDRA-2889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13263765#comment-13263765 ] Jonathan Ellis commented on CASSANDRA-2889: --- +1 > Avoids having replicate on write tasks stacking up at CL.ONE > > > Key: CASSANDRA-2889 > URL: https://issues.apache.org/jira/browse/CASSANDRA-2889 > Project: Cassandra > Issue Type: Improvement > Components: Core >Affects Versions: 0.8.0 >Reporter: Sylvain Lebresne >Assignee: Sylvain Lebresne > Labels: counters > Fix For: 1.1.1 > > Attachments: 2889.txt > > > The counter design involves a read on the first replica during a write. At > CL.ONE, this read is not involved in the latency of the operation (the write > is acknowledged before). This means it is fairly easy to insert too quickly > at CL.ONE and have the replicate on write tasks falling behind. The goal of > this ticket is to protect against that. > An option could be to bound the replicate on write task queue so that write > start to block once we have too much of those in the queue. Another option > could be to drop the oldest tasks when they are too old, but it's probably a > more unsafe option. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2889) Avoids having replicate on write tasks stacking up at CL.ONE
[ https://issues.apache.org/jira/browse/CASSANDRA-2889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13263761#comment-13263761 ] Sylvain Lebresne commented on CASSANDRA-2889: - No, the write stage (it's the one pushing the replicate task) > Avoids having replicate on write tasks stacking up at CL.ONE > > > Key: CASSANDRA-2889 > URL: https://issues.apache.org/jira/browse/CASSANDRA-2889 > Project: Cassandra > Issue Type: Improvement > Components: Core >Affects Versions: 0.8.0 >Reporter: Sylvain Lebresne >Assignee: Sylvain Lebresne > Labels: counters > Fix For: 1.1.1 > > Attachments: 2889.txt > > > The counter design involves a read on the first replica during a write. At > CL.ONE, this read is not involved in the latency of the operation (the write > is acknowledged before). This means it is fairly easy to insert too quickly > at CL.ONE and have the replicate on write tasks falling behind. The goal of > this ticket is to protect against that. > An option could be to bound the replicate on write task queue so that write > start to block once we have too much of those in the queue. Another option > could be to drop the oldest tasks when they are too old, but it's probably a > more unsafe option. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2889) Avoids having replicate on write tasks stacking up at CL.ONE
[ https://issues.apache.org/jira/browse/CASSANDRA-2889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13263758#comment-13263758 ] Jonathan Ellis commented on CASSANDRA-2889: --- Who is going to block if ROW queue fills up? Read stage? > Avoids having replicate on write tasks stacking up at CL.ONE > > > Key: CASSANDRA-2889 > URL: https://issues.apache.org/jira/browse/CASSANDRA-2889 > Project: Cassandra > Issue Type: Improvement > Components: Core >Affects Versions: 0.8.0 >Reporter: Sylvain Lebresne >Assignee: Sylvain Lebresne > Labels: counters > Fix For: 1.1.1 > > Attachments: 2889.txt > > > The counter design involves a read on the first replica during a write. At > CL.ONE, this read is not involved in the latency of the operation (the write > is acknowledged before). This means it is fairly easy to insert too quickly > at CL.ONE and have the replicate on write tasks falling behind. The goal of > this ticket is to protect against that. > An option could be to bound the replicate on write task queue so that write > start to block once we have too much of those in the queue. Another option > could be to drop the oldest tasks when they are too old, but it's probably a > more unsafe option. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2889) Avoids having replicate on write tasks stacking up at CL.ONE
[ https://issues.apache.org/jira/browse/CASSANDRA-2889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13083830#comment-13083830 ] Stu Hood commented on CASSANDRA-2889: - bq. But maybe the best solution here would be to make CL.ONE wait for the read to have happened to ack the client. Or perhaps some kind of proportional backpressure for writes based on the length of the replicate-on-write queue? No backpressure below X, and then linear backpressure above X? > Avoids having replicate on write tasks stacking up at CL.ONE > > > Key: CASSANDRA-2889 > URL: https://issues.apache.org/jira/browse/CASSANDRA-2889 > Project: Cassandra > Issue Type: Improvement > Components: Core >Affects Versions: 0.8.0 >Reporter: Sylvain Lebresne >Assignee: Sylvain Lebresne > Labels: counters > > The counter design involves a read on the first replica during a write. At > CL.ONE, this read is not involved in the latency of the operation (the write > is acknowledged before). This means it is fairly easy to insert too quickly > at CL.ONE and have the replicate on write tasks falling behind. The goal of > this ticket is to protect against that. > An option could be to bound the replicate on write task queue so that write > start to block once we have too much of those in the queue. Another option > could be to drop the oldest tasks when they are too old, but it's probably a > more unsafe option. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2889) Avoids having replicate on write tasks stacking up at CL.ONE
[ https://issues.apache.org/jira/browse/CASSANDRA-2889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13083261#comment-13083261 ] Sylvain Lebresne commented on CASSANDRA-2889: - bq. Is this the same as CASSANDRA-2892? No, CASSANDRA-2892 was really just saying "if we have nothing to replicate to, let's not push a replication task that will do nothing anyway". It was really just a super easy optimization for the RF=1 case. This one is because at CL.ONE, we ack the client as soon as we have written the local mutation. But the replication involves a read. So if you write very quickly at CL.ONE, you're "read to replicate" task may stack up because you're not able to do them fast enough. But maybe the best solution here would be to make CL.ONE wait for the read to have happened to ack the client. The current make for a better latency at CL.ONE, but this is kind of a lie, because the hardest part of the work (the read) till happens in the background, and it is thus easy to flood the node. > Avoids having replicate on write tasks stacking up at CL.ONE > > > Key: CASSANDRA-2889 > URL: https://issues.apache.org/jira/browse/CASSANDRA-2889 > Project: Cassandra > Issue Type: Improvement > Components: Core >Affects Versions: 0.8.0 >Reporter: Sylvain Lebresne >Assignee: Sylvain Lebresne > Labels: counters > > The counter design involves a read on the first replica during a write. At > CL.ONE, this read is not involved in the latency of the operation (the write > is acknowledged before). This means it is fairly easy to insert too quickly > at CL.ONE and have the replicate on write tasks falling behind. The goal of > this ticket is to protect against that. > An option could be to bound the replicate on write task queue so that write > start to block once we have too much of those in the queue. Another option > could be to drop the oldest tasks when they are too old, but it's probably a > more unsafe option. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2889) Avoids having replicate on write tasks stacking up at CL.ONE
[ https://issues.apache.org/jira/browse/CASSANDRA-2889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13083228#comment-13083228 ] Jonathan Ellis commented on CASSANDRA-2889: --- Is this the same as CASSANDRA-2892? > Avoids having replicate on write tasks stacking up at CL.ONE > > > Key: CASSANDRA-2889 > URL: https://issues.apache.org/jira/browse/CASSANDRA-2889 > Project: Cassandra > Issue Type: Improvement > Components: Core >Affects Versions: 0.8.0 >Reporter: Sylvain Lebresne >Assignee: Sylvain Lebresne > Labels: counters > > The counter design involves a read on the first replica during a write. At > CL.ONE, this read is not involved in the latency of the operation (the write > is acknowledged before). This means it is fairly easy to insert too quickly > at CL.ONE and have the replicate on write tasks falling behind. The goal of > this ticket is to protect against that. > An option could be to bound the replicate on write task queue so that write > start to block once we have too much of those in the queue. Another option > could be to drop the oldest tasks when they are too old, but it's probably a > more unsafe option. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2889) Avoids having replicate on write tasks stacking up at CL.ONE
[
https://issues.apache.org/jira/browse/CASSANDRA-2889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13071873#comment-13071873
]
David Phillips commented on CASSANDRA-2889:
---
You can run this temporary fix before starting the server to bound the queue
size:
{code}
private static void monkeyPatchCassandra()
{
// hack to fix CASSANDRA-2889
try {
Field field = StageManager.class.getDeclaredField("stages");
field.setAccessible(true);
EnumMap stages = (EnumMap) field.get(StageManager.class);
Stage stage = Stage.REPLICATE_ON_WRITE;
stages.get(stage).shutdown();
stages.put(stage, new JMXConfigurableThreadPoolExecutor(
DatabaseDescriptor.getConcurrentReplicators(),
StageManager.KEEPALIVE,
TimeUnit.SECONDS,
new LinkedBlockingQueue(10),
new NamedThreadFactory(stage.getJmxName()),
stage.getJmxType()));
}
catch (Exception e) {
throw new AssertionError(e);
}
}
{code}
> Avoids having replicate on write tasks stacking up at CL.ONE
>
>
> Key: CASSANDRA-2889
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2889
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
>Affects Versions: 0.8.0
>Reporter: Sylvain Lebresne
>Assignee: Sylvain Lebresne
> Labels: counters
>
> The counter design involves a read on the first replica during a write. At
> CL.ONE, this read is not involved in the latency of the operation (the write
> is acknowledged before). This means it is fairly easy to insert too quickly
> at CL.ONE and have the replicate on write tasks falling behind. The goal of
> this ticket is to protect against that.
> An option could be to bound the replicate on write task queue so that write
> start to block once we have too much of those in the queue. Another option
> could be to drop the oldest tasks when they are too old, but it's probably a
> more unsafe option.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
