[jira] [Commented] (CASSANDRA-6764) Using Batch commitlog_sync is slow and doesn't actually batch writes
[ https://issues.apache.org/jira/browse/CASSANDRA-6764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13976782#comment-13976782 ] Jonathan Ellis commented on CASSANDRA-6764: --- committed > Using Batch commitlog_sync is slow and doesn't actually batch writes > > > Key: CASSANDRA-6764 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6764 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: John Carrino >Assignee: John Carrino > Fix For: 2.1 beta2 > > Attachments: 6764.fix.txt, 6764.fix2.txt, cassandra_6764_v2.patch, > cassandra_6764_v3.patch > > > The assumption behind batch commit mode is that the client does it's own > batching and wants to wait until the write is durable before returning. The > problem is that the queue that cassandra uses under the covers only allows > for a single ROW (RowMutation) per thread (concurrent_writes). This means > that commitlog_sync_batch_window_in_ms should really be called sleep_between > each_concurrent_writes_rows_in_ms. > I assume the reason this slipped by for so long is that no one uses batch > mode, probably because people say "it's slow". We need durability so this > isn't an option. > However it doesn't need to be this slow. > Also, if you write a row that is larger than the commit log size it silently > (warn) fails to put it in the commit log. This is not ideal for batch mode. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (CASSANDRA-6764) Using Batch commitlog_sync is slow and doesn't actually batch writes
[ https://issues.apache.org/jira/browse/CASSANDRA-6764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13955126#comment-13955126 ] Benedict commented on CASSANDRA-6764: - Well this _does_ come under the heading of 'guarantees' in my book :) Okay, in that case I'm a smidgen concerned we may end up breaking some people unexpectedly in 2.1, but they can just bump their CL size so it shouldn't be too bad. Perhaps we should mention this in the error message at least? I'll update the test. > Using Batch commitlog_sync is slow and doesn't actually batch writes > > > Key: CASSANDRA-6764 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6764 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: John Carrino >Assignee: John Carrino > Fix For: 2.1 beta2 > > Attachments: 6764.fix.txt, cassandra_6764_v2.patch, > cassandra_6764_v3.patch > > > The assumption behind batch commit mode is that the client does it's own > batching and wants to wait until the write is durable before returning. The > problem is that the queue that cassandra uses under the covers only allows > for a single ROW (RowMutation) per thread (concurrent_writes). This means > that commitlog_sync_batch_window_in_ms should really be called sleep_between > each_concurrent_writes_rows_in_ms. > I assume the reason this slipped by for so long is that no one uses batch > mode, probably because people say "it's slow". We need durability so this > isn't an option. > However it doesn't need to be this slow. > Also, if you write a row that is larger than the commit log size it silently > (warn) fails to put it in the commit log. This is not ideal for batch mode. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (CASSANDRA-6764) Using Batch commitlog_sync is slow and doesn't actually batch writes
[ https://issues.apache.org/jira/browse/CASSANDRA-6764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13955122#comment-13955122 ] Jonathan Ellis commented on CASSANDRA-6764: --- batch and periodic have always differed only by the guarantees provided. I think it makes sense to preserve that. > Using Batch commitlog_sync is slow and doesn't actually batch writes > > > Key: CASSANDRA-6764 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6764 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: John Carrino >Assignee: John Carrino > Fix For: 2.1 beta2 > > Attachments: 6764.fix.txt, cassandra_6764_v2.patch, > cassandra_6764_v3.patch > > > The assumption behind batch commit mode is that the client does it's own > batching and wants to wait until the write is durable before returning. The > problem is that the queue that cassandra uses under the covers only allows > for a single ROW (RowMutation) per thread (concurrent_writes). This means > that commitlog_sync_batch_window_in_ms should really be called sleep_between > each_concurrent_writes_rows_in_ms. > I assume the reason this slipped by for so long is that no one uses batch > mode, probably because people say "it's slow". We need durability so this > isn't an option. > However it doesn't need to be this slow. > Also, if you write a row that is larger than the commit log size it silently > (warn) fails to put it in the commit log. This is not ideal for batch mode. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (CASSANDRA-6764) Using Batch commitlog_sync is slow and doesn't actually batch writes
[ https://issues.apache.org/jira/browse/CASSANDRA-6764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13951525#comment-13951525 ] Benedict commented on CASSANDRA-6764: - bq. That was definitely the intention, and that's what it did unless the math was off. Right, agreed after re-reading, but the name testExceedSegmentSize... understandably threw me :) bq. How does this differ batch vs periodic? I thought we had agreed to only change batch behaviour here? I must admit I didn't check the code, I was going by the conversation. > Using Batch commitlog_sync is slow and doesn't actually batch writes > > > Key: CASSANDRA-6764 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6764 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: John Carrino >Assignee: John Carrino > Fix For: 2.1 beta2 > > Attachments: 6764.fix.txt, cassandra_6764_v2.patch, > cassandra_6764_v3.patch > > > The assumption behind batch commit mode is that the client does it's own > batching and wants to wait until the write is durable before returning. The > problem is that the queue that cassandra uses under the covers only allows > for a single ROW (RowMutation) per thread (concurrent_writes). This means > that commitlog_sync_batch_window_in_ms should really be called sleep_between > each_concurrent_writes_rows_in_ms. > I assume the reason this slipped by for so long is that no one uses batch > mode, probably because people say "it's slow". We need durability so this > isn't an option. > However it doesn't need to be this slow. > Also, if you write a row that is larger than the commit log size it silently > (warn) fails to put it in the commit log. This is not ideal for batch mode. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (CASSANDRA-6764) Using Batch commitlog_sync is slow and doesn't actually batch writes
[
https://issues.apache.org/jira/browse/CASSANDRA-6764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13951520#comment-13951520
]
Jonathan Ellis commented on CASSANDRA-6764:
---
That was definitely the intention, and that's what it did unless the math was
off.
How does this differ batch vs periodic?
{code}
long totalSize = size + ENTRY_OVERHEAD_SIZE;
if (totalSize > MAX_MUTATION_SIZE)
{
throw new IllegalArgumentException(String.format("Mutation of %s
bytes is too large for the maxiumum size of %s",
totalSize,
MAX_MUTATION_SIZE));
}
{code}
> Using Batch commitlog_sync is slow and doesn't actually batch writes
>
>
> Key: CASSANDRA-6764
> URL: https://issues.apache.org/jira/browse/CASSANDRA-6764
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
>Reporter: John Carrino
>Assignee: John Carrino
> Fix For: 2.1 beta2
>
> Attachments: 6764.fix.txt, cassandra_6764_v2.patch,
> cassandra_6764_v3.patch
>
>
> The assumption behind batch commit mode is that the client does it's own
> batching and wants to wait until the write is durable before returning. The
> problem is that the queue that cassandra uses under the covers only allows
> for a single ROW (RowMutation) per thread (concurrent_writes). This means
> that commitlog_sync_batch_window_in_ms should really be called sleep_between
> each_concurrent_writes_rows_in_ms.
> I assume the reason this slipped by for so long is that no one uses batch
> mode, probably because people say "it's slow". We need durability so this
> isn't an option.
> However it doesn't need to be this slow.
> Also, if you write a row that is larger than the commit log size it silently
> (warn) fails to put it in the commit log. This is not ideal for batch mode.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (CASSANDRA-6764) Using Batch commitlog_sync is slow and doesn't actually batch writes
[ https://issues.apache.org/jira/browse/CASSANDRA-6764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13951510#comment-13951510 ] Benedict commented on CASSANDRA-6764: - But reading it again it looks like you are right. I think both are sensible tests, however, so I'll introduce a second test that checks the maximum size also. > Using Batch commitlog_sync is slow and doesn't actually batch writes > > > Key: CASSANDRA-6764 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6764 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: John Carrino >Assignee: John Carrino > Fix For: 2.1 beta2 > > Attachments: 6764.fix.txt, cassandra_6764_v2.patch, > cassandra_6764_v3.patch > > > The assumption behind batch commit mode is that the client does it's own > batching and wants to wait until the write is durable before returning. The > problem is that the queue that cassandra uses under the covers only allows > for a single ROW (RowMutation) per thread (concurrent_writes). This means > that commitlog_sync_batch_window_in_ms should really be called sleep_between > each_concurrent_writes_rows_in_ms. > I assume the reason this slipped by for so long is that no one uses batch > mode, probably because people say "it's slow". We need durability so this > isn't an option. > However it doesn't need to be this slow. > Also, if you write a row that is larger than the commit log size it silently > (warn) fails to put it in the commit log. This is not ideal for batch mode. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (CASSANDRA-6764) Using Batch commitlog_sync is slow and doesn't actually batch writes
[ https://issues.apache.org/jira/browse/CASSANDRA-6764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13951507#comment-13951507 ] Benedict commented on CASSANDRA-6764: - I assumed from the name of the test (and the ticket) it was ensuring it could write a message that was _too large_. And in this case the behaviour differs now between batch and periodic. > Using Batch commitlog_sync is slow and doesn't actually batch writes > > > Key: CASSANDRA-6764 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6764 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: John Carrino >Assignee: John Carrino > Fix For: 2.1 beta2 > > Attachments: 6764.fix.txt, cassandra_6764_v2.patch, > cassandra_6764_v3.patch > > > The assumption behind batch commit mode is that the client does it's own > batching and wants to wait until the write is durable before returning. The > problem is that the queue that cassandra uses under the covers only allows > for a single ROW (RowMutation) per thread (concurrent_writes). This means > that commitlog_sync_batch_window_in_ms should really be called sleep_between > each_concurrent_writes_rows_in_ms. > I assume the reason this slipped by for so long is that no one uses batch > mode, probably because people say "it's slow". We need durability so this > isn't an option. > However it doesn't need to be this slow. > Also, if you write a row that is larger than the commit log size it silently > (warn) fails to put it in the commit log. This is not ideal for batch mode. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (CASSANDRA-6764) Using Batch commitlog_sync is slow and doesn't actually batch writes
[ https://issues.apache.org/jira/browse/CASSANDRA-6764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13951504#comment-13951504 ] Jonathan Ellis commented on CASSANDRA-6764: --- I think what this test is trying to do is, "make sure we can write the largest acceptable mutation." (CASSANDRA-3615) > Using Batch commitlog_sync is slow and doesn't actually batch writes > > > Key: CASSANDRA-6764 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6764 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: John Carrino >Assignee: John Carrino > Fix For: 2.1 beta2 > > Attachments: 6764.fix.txt, cassandra_6764_v2.patch, > cassandra_6764_v3.patch > > > The assumption behind batch commit mode is that the client does it's own > batching and wants to wait until the write is durable before returning. The > problem is that the queue that cassandra uses under the covers only allows > for a single ROW (RowMutation) per thread (concurrent_writes). This means > that commitlog_sync_batch_window_in_ms should really be called sleep_between > each_concurrent_writes_rows_in_ms. > I assume the reason this slipped by for so long is that no one uses batch > mode, probably because people say "it's slow". We need durability so this > isn't an option. > However it doesn't need to be this slow. > Also, if you write a row that is larger than the commit log size it silently > (warn) fails to put it in the commit log. This is not ideal for batch mode. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (CASSANDRA-6764) Using Batch commitlog_sync is slow and doesn't actually batch writes
[ https://issues.apache.org/jira/browse/CASSANDRA-6764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13951500#comment-13951500 ] Jonathan Ellis commented on CASSANDRA-6764: --- Is this right, that periodic and batch have different max mutation sizes? That's not how CL.add looks to me. > Using Batch commitlog_sync is slow and doesn't actually batch writes > > > Key: CASSANDRA-6764 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6764 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: John Carrino >Assignee: John Carrino > Fix For: 2.1 beta2 > > Attachments: 6764.fix.txt, cassandra_6764_v2.patch, > cassandra_6764_v3.patch > > > The assumption behind batch commit mode is that the client does it's own > batching and wants to wait until the write is durable before returning. The > problem is that the queue that cassandra uses under the covers only allows > for a single ROW (RowMutation) per thread (concurrent_writes). This means > that commitlog_sync_batch_window_in_ms should really be called sleep_between > each_concurrent_writes_rows_in_ms. > I assume the reason this slipped by for so long is that no one uses batch > mode, probably because people say "it's slow". We need durability so this > isn't an option. > However it doesn't need to be this slow. > Also, if you write a row that is larger than the commit log size it silently > (warn) fails to put it in the commit log. This is not ideal for batch mode. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (CASSANDRA-6764) Using Batch commitlog_sync is slow and doesn't actually batch writes
[ https://issues.apache.org/jira/browse/CASSANDRA-6764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13913547#comment-13913547 ] John Carrino commented on CASSANDRA-6764: - If you have batch turned on the assumption is that the writes are durable if you respond successfully. Anything less is a violation of the trust clients have in the data store. At that point the system has become a logging framework and that's not what people want or expect. If people are running with quorum reads and writes and batch enabled there really should be an fsync protecting the writes. People expect reads following write success to return their data. If this is not the case I must know because I'm kinda betting my customer's data on this being the case. > Using Batch commitlog_sync is slow and doesn't actually batch writes > > > Key: CASSANDRA-6764 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6764 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: John Carrino > Fix For: 1.2.16 > > Attachments: cassandra_6764_v2.patch, cassandra_6764_v3.patch > > > The assumption behind batch commit mode is that the client does it's own > batching and wants to wait until the write is durable before returning. The > problem is that the queue that cassandra uses under the covers only allows > for a single ROW (RowMutation) per thread (concurrent_writes). This means > that commitlog_sync_batch_window_in_ms should really be called sleep_between > each_concurrent_writes_rows_in_ms. > I assume the reason this slipped by for so long is that no one uses batch > mode, probably because people say "it's slow". We need durability so this > isn't an option. > However it doesn't need to be this slow. > Also, if you write a row that is larger than the commit log size it silently > (warn) fails to put it in the commit log. This is not ideal for batch mode. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (CASSANDRA-6764) Using Batch commitlog_sync is slow and doesn't actually batch writes
[ https://issues.apache.org/jira/browse/CASSANDRA-6764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13913545#comment-13913545 ] Benedict commented on CASSANDRA-6764: - I think there's something to be said for refusing a mutation that couldn't be guaranteed to be durable. People running batch CL are obviously more paranoid about that already, and a warning is a bit late to fix it, especially if it's on the server logs (which we know all good boys and girls read, but not everyone is a good boy or girl. I wouldn't want to break people who are using the old behaviour, but I'm also ambivalent about more config parameters. I'd say if we had to choose, I'd lean in favour of durability, since this is our durability mode. People who aren't losing sleep over this are running periodic already. If we can find an unobtrusive/uncluttered way to make it optional, let's do that, but otherwise I'd make the switch with a major release when people are expected to be doing some degree of due diligence. There is one other possibility: we could create a whole CLS (of sufficient size) for the monster mutation, and guarantee never to drop any from CL. This would probably want to be configurable anyway, though, as could be a large penalty for such huge durable writes. > Using Batch commitlog_sync is slow and doesn't actually batch writes > > > Key: CASSANDRA-6764 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6764 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: John Carrino > Fix For: 1.2.16 > > Attachments: cassandra_6764_v2.patch, cassandra_6764_v3.patch > > > The assumption behind batch commit mode is that the client does it's own > batching and wants to wait until the write is durable before returning. The > problem is that the queue that cassandra uses under the covers only allows > for a single ROW (RowMutation) per thread (concurrent_writes). This means > that commitlog_sync_batch_window_in_ms should really be called sleep_between > each_concurrent_writes_rows_in_ms. > I assume the reason this slipped by for so long is that no one uses batch > mode, probably because people say "it's slow". We need durability so this > isn't an option. > However it doesn't need to be this slow. > Also, if you write a row that is larger than the commit log size it silently > (warn) fails to put it in the commit log. This is not ideal for batch mode. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (CASSANDRA-6764) Using Batch commitlog_sync is slow and doesn't actually batch writes
[ https://issues.apache.org/jira/browse/CASSANDRA-6764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13913523#comment-13913523 ] Jonathan Ellis commented on CASSANDRA-6764: --- I'm inclined to think it's more user-friendly to warn, and let them address it while the app stays up, than to throw errors immediately. The latter is guaranteed to cause problems; the former will only do so if you lose power before fixing your code. WDYT [~benedict]? > Using Batch commitlog_sync is slow and doesn't actually batch writes > > > Key: CASSANDRA-6764 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6764 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: John Carrino > Fix For: 1.2.16 > > Attachments: cassandra_6764_v2.patch, cassandra_6764_v3.patch > > > The assumption behind batch commit mode is that the client does it's own > batching and wants to wait until the write is durable before returning. The > problem is that the queue that cassandra uses under the covers only allows > for a single ROW (RowMutation) per thread (concurrent_writes). This means > that commitlog_sync_batch_window_in_ms should really be called sleep_between > each_concurrent_writes_rows_in_ms. > I assume the reason this slipped by for so long is that no one uses batch > mode, probably because people say "it's slow". We need durability so this > isn't an option. > However it doesn't need to be this slow. > Also, if you write a row that is larger than the commit log size it silently > (warn) fails to put it in the commit log. This is not ideal for batch mode. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (CASSANDRA-6764) Using Batch commitlog_sync is slow and doesn't actually batch writes
[ https://issues.apache.org/jira/browse/CASSANDRA-6764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13911955#comment-13911955 ] John Carrino commented on CASSANDRA-6764: - Ok, last patch skips over all the batching stuff and just ensures that mutations hit the commit log and don't get dropped. > Using Batch commitlog_sync is slow and doesn't actually batch writes > > > Key: CASSANDRA-6764 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6764 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: John Carrino > Fix For: 1.2.16 > > Attachments: cassandra_6764_v2.patch, cassandra_6764_v3.patch > > > The assumption behind batch commit mode is that the client does it's own > batching and wants to wait until the write is durable before returning. The > problem is that the queue that cassandra uses under the covers only allows > for a single ROW (RowMutation) per thread (concurrent_writes). This means > that commitlog_sync_batch_window_in_ms should really be called sleep_between > each_concurrent_writes_rows_in_ms. > I assume the reason this slipped by for so long is that no one uses batch > mode, probably because people say "it's slow". We need durability so this > isn't an option. > However it doesn't need to be this slow. > Also, if you write a row that is larger than the commit log size it silently > (warn) fails to put it in the commit log. This is not ideal for batch mode. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (CASSANDRA-6764) Using Batch commitlog_sync is slow and doesn't actually batch writes
[ https://issues.apache.org/jira/browse/CASSANDRA-6764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13911629#comment-13911629 ] Jonathan Ellis commented on CASSANDRA-6764: --- +1 > Using Batch commitlog_sync is slow and doesn't actually batch writes > > > Key: CASSANDRA-6764 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6764 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: John Carrino > Fix For: 1.2.16 > > Attachments: cassandra_6764_v2.patch > > > The assumption behind batch commit mode is that the client does it's own > batching and wants to wait until the write is durable before returning. The > problem is that the queue that cassandra uses under the covers only allows > for a single ROW (RowMutation) per thread (concurrent_writes). This means > that commitlog_sync_batch_window_in_ms should really be called sleep_between > each_concurrent_writes_rows_in_ms. > I assume the reason this slipped by for so long is that no one uses batch > mode, probably because people say "it's slow". We need durability so this > isn't an option. > However it doesn't need to be this slow. > Also, if you write a row that is larger than the commit log size it silently > (warn) fails to put it in the commit log. This is not ideal for batch mode. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (CASSANDRA-6764) Using Batch commitlog_sync is slow and doesn't actually batch writes
[ https://issues.apache.org/jira/browse/CASSANDRA-6764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13911432#comment-13911432 ] Benedict commented on CASSANDRA-6764: - bq. It's good to hear this was reworked in 2.1. I hope that batching is in there. This particular problem won't be fixed there still. Note that the issue isn't really with the CL, but with the process flow in the write stage. This is a known issue, but since the workaround is so simple (increase thread count) and relatively cost free on most platforms, it hasn't really been considered a major problem to address. The correct solution to this would be to perform the CL append, then place the record on a queue to be processed once it is synced and to continue receiving/processing other records until that time (rather than immediately blocking on the sync). > Using Batch commitlog_sync is slow and doesn't actually batch writes > > > Key: CASSANDRA-6764 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6764 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: John Carrino > Fix For: 1.2.16 > > Attachments: cassandra_6764_v2.patch > > > The assumption behind batch commit mode is that the client does it's own > batching and wants to wait until the write is durable before returning. The > problem is that the queue that cassandra uses under the covers only allows > for a single ROW (RowMutation) per thread (concurrent_writes). This means > that commitlog_sync_batch_window_in_ms should really be called sleep_between > each_concurrent_writes_rows_in_ms. > I assume the reason this slipped by for so long is that no one uses batch > mode, probably because people say "it's slow". We need durability so this > isn't an option. > However it doesn't need to be this slow. > Also, if you write a row that is larger than the commit log size it silently > (warn) fails to put it in the commit log. This is not ideal for batch mode. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (CASSANDRA-6764) Using Batch commitlog_sync is slow and doesn't actually batch writes
[ https://issues.apache.org/jira/browse/CASSANDRA-6764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13911207#comment-13911207 ] John Carrino commented on CASSANDRA-6764: - Good point. Ok. We may just run this with 2000 writer threads and call it a day. I would still like to add the throw if the mutation is being dropped. For periodic this throw will just be eaten by the FutureTask but for batch it will fail the write. > Using Batch commitlog_sync is slow and doesn't actually batch writes > > > Key: CASSANDRA-6764 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6764 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: John Carrino > Fix For: 1.2.16 > > Attachments: cassandra_6764_v2.patch > > > The assumption behind batch commit mode is that the client does it's own > batching and wants to wait until the write is durable before returning. The > problem is that the queue that cassandra uses under the covers only allows > for a single ROW (RowMutation) per thread (concurrent_writes). This means > that commitlog_sync_batch_window_in_ms should really be called sleep_between > each_concurrent_writes_rows_in_ms. > I assume the reason this slipped by for so long is that no one uses batch > mode, probably because people say "it's slow". We need durability so this > isn't an option. > However it doesn't need to be this slow. > Also, if you write a row that is larger than the commit log size it silently > (warn) fails to put it in the commit log. This is not ideal for batch mode. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (CASSANDRA-6764) Using Batch commitlog_sync is slow and doesn't actually batch writes
[ https://issues.apache.org/jira/browse/CASSANDRA-6764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13911203#comment-13911203 ] Michael Harris commented on CASSANDRA-6764: --- Carrino, does this change have the potential to cause any problems during rolling upgrades? It looks like you're modifying an existing codepath to use a new type that gets serialized to other nodes. That could be very...explosive. Am I missing something? Rolling upgrades are pretty common for minor version upgrades (correct me if I'm wrong Jonathan), so this is actually kind of scary. > Using Batch commitlog_sync is slow and doesn't actually batch writes > > > Key: CASSANDRA-6764 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6764 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: John Carrino > Fix For: 1.2.16 > > Attachments: cassandra_6764_v2.patch > > > The assumption behind batch commit mode is that the client does it's own > batching and wants to wait until the write is durable before returning. The > problem is that the queue that cassandra uses under the covers only allows > for a single ROW (RowMutation) per thread (concurrent_writes). This means > that commitlog_sync_batch_window_in_ms should really be called sleep_between > each_concurrent_writes_rows_in_ms. > I assume the reason this slipped by for so long is that no one uses batch > mode, probably because people say "it's slow". We need durability so this > isn't an option. > However it doesn't need to be this slow. > Also, if you write a row that is larger than the commit log size it silently > (warn) fails to put it in the commit log. This is not ideal for batch mode. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (CASSANDRA-6764) Using Batch commitlog_sync is slow and doesn't actually batch writes
[ https://issues.apache.org/jira/browse/CASSANDRA-6764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13911198#comment-13911198 ] Jonathan Ellis commented on CASSANDRA-6764: --- I'm sure you can also find a blog post about how processing one stage "in bulk" can starve other stages of work and reduce performance. That's the dominant effect in Cassandra. > Using Batch commitlog_sync is slow and doesn't actually batch writes > > > Key: CASSANDRA-6764 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6764 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: John Carrino > Fix For: 1.2.16 > > Attachments: cassandra_6764_v2.patch > > > The assumption behind batch commit mode is that the client does it's own > batching and wants to wait until the write is durable before returning. The > problem is that the queue that cassandra uses under the covers only allows > for a single ROW (RowMutation) per thread (concurrent_writes). This means > that commitlog_sync_batch_window_in_ms should really be called sleep_between > each_concurrent_writes_rows_in_ms. > I assume the reason this slipped by for so long is that no one uses batch > mode, probably because people say "it's slow". We need durability so this > isn't an option. > However it doesn't need to be this slow. > Also, if you write a row that is larger than the commit log size it silently > (warn) fails to put it in the commit log. This is not ideal for batch mode. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (CASSANDRA-6764) Using Batch commitlog_sync is slow and doesn't actually batch writes
[ https://issues.apache.org/jira/browse/CASSANDRA-6764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13911190#comment-13911190 ] John Carrino commented on CASSANDRA-6764: - All is not lost as we will use this in production as a patch on 1.2. I disagree using smart batching is an anti pattern. http://mechanical-sympathy.blogspot.com/2011/10/smart-batching.html?m=1 I think going 4000 threads wide just to achieve batched commits is excessive especially considering what a common workflow it is. Do people run with concurrent writes in the thousands? It's good to hear this was reworked in 2.1. I hope that batching is in there. > Using Batch commitlog_sync is slow and doesn't actually batch writes > > > Key: CASSANDRA-6764 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6764 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: John Carrino > Fix For: 1.2.16 > > Attachments: cassandra_6764_v2.patch > > > The assumption behind batch commit mode is that the client does it's own > batching and wants to wait until the write is durable before returning. The > problem is that the queue that cassandra uses under the covers only allows > for a single ROW (RowMutation) per thread (concurrent_writes). This means > that commitlog_sync_batch_window_in_ms should really be called sleep_between > each_concurrent_writes_rows_in_ms. > I assume the reason this slipped by for so long is that no one uses batch > mode, probably because people say "it's slow". We need durability so this > isn't an option. > However it doesn't need to be this slow. > Also, if you write a row that is larger than the commit log size it silently > (warn) fails to put it in the commit log. This is not ideal for batch mode. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (CASSANDRA-6764) Using Batch commitlog_sync is slow and doesn't actually batch writes
[
https://issues.apache.org/jira/browse/CASSANDRA-6764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13911166#comment-13911166
]
Jonathan Ellis commented on CASSANDRA-6764:
---
I hate to be That Guy after you've put this much effort into a patch, but I
have some concerns. First and most obviously,
# 1.2 and 2.0 are for bug fixes only at this point, not optimizations, and
especially not optimizations touching something as crucial to data safety as
CommitLog. Not at all comfortable slipping this into a minor update and
crossing my fingers.
# CommitLog has been modified substantially for 2.1 (CASSANDRA-3578)
Now, it could still make sense to aggregate rows from the same (request) batch
into the same (fsync) batch, however,
- using batching for unrelated rows (i.e. rows that don't need to be updated
atomically) is a performance antipattern, so optimizing for that seems wrong,
and
- I'm skeptical that you can't achieve your goals without the extra complexity
by simply increasing {{concurrent_writes}}
> Using Batch commitlog_sync is slow and doesn't actually batch writes
>
>
> Key: CASSANDRA-6764
> URL: https://issues.apache.org/jira/browse/CASSANDRA-6764
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
>Reporter: John Carrino
> Fix For: 1.2.16
>
> Attachments: cassandra_6764_v2.patch
>
>
> The assumption behind batch commit mode is that the client does it's own
> batching and wants to wait until the write is durable before returning. The
> problem is that the queue that cassandra uses under the covers only allows
> for a single ROW (RowMutation) per thread (concurrent_writes). This means
> that commitlog_sync_batch_window_in_ms should really be called sleep_between
> each_concurrent_writes_rows_in_ms.
> I assume the reason this slipped by for so long is that no one uses batch
> mode, probably because people say "it's slow". We need durability so this
> isn't an option.
> However it doesn't need to be this slow.
> Also, if you write a row that is larger than the commit log size it silently
> (warn) fails to put it in the commit log. This is not ideal for batch mode.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (CASSANDRA-6764) Using Batch commitlog_sync is slow and doesn't actually batch writes
[ https://issues.apache.org/jira/browse/CASSANDRA-6764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13911145#comment-13911145 ] John Carrino commented on CASSANDRA-6764: - I tried to stick as close as I could to the coding guidelines so let me know if I missed some stuff. (my native java style is open curly on the end of the line). Also let me know if there are any other config params I should add. > Using Batch commitlog_sync is slow and doesn't actually batch writes > > > Key: CASSANDRA-6764 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6764 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: John Carrino > Fix For: 1.2.16 > > Attachments: cassandra_6764_v2.patch > > > The assumption behind batch commit mode is that the client does it's own > batching and wants to wait until the write is durable before returning. The > problem is that the queue that cassandra uses under the covers only allows > for a single ROW (RowMutation) per thread (concurrent_writes). This means > that commitlog_sync_batch_window_in_ms should really be called sleep_between > each_concurrent_writes_rows_in_ms. > I assume the reason this slipped by for so long is that no one uses batch > mode, probably because people say "it's slow". We need durability so this > isn't an option. > However it doesn't need to be this slow. > Also, if you write a row that is larger than the commit log size it silently > (warn) fails to put it in the commit log. This is not ideal for batch mode. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
