[
https://issues.apache.org/jira/browse/CASSANDRA-12783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16256404#comment-16256404
]
Kurt Greaves commented on CASSANDRA-12783:
------------------------------------------
So an update here. Seeing as this is going into 4 it's only getting a limited
amount of my attention at the moment so I thought I'd put up my most recent
version here and where it's up to (so I don't forget, and if anyone has any
insights).
|[trunk|https://github.com/apache/cassandra/compare/trunk...kgreav:12783-trunk]|[dtest|https://github.com/apache/cassandra-dtest/compare/master...kgreav:12783]|
So at the moment it more or less works however the dtest fails occasionally due
to the corrupt value length, even though it shouldn't happen. You can even keep
the test dir, go run the query again, and get the correct results/no corrupt
values.
There's still a few TODO's, and needs some performance testing for comparison +
to see if it's worth chunking mutations in {{store}}. LegacyBatchlogMigrator
needs a few more fixes to support the new version (it's a copy of the old 3.0
version).
Also I think ideally the {{BatchlogManager.markBatchesActive()}} should
probably use the batchlog to mark the batches active so that we can guarantee
if they are marked they will eventually all be marked, but at the moment that's
not possible because the batchlog uses a GCGS of 0. It's probably safe to set a
GCGS higher than 0, it will just end with us reading some tombstones when we
want to replay the batchlog - but really it's not a great solution. Special
casing local batchlog writes is also another option (as there's no point
consulting GCGS for a local batchlog write) however we'd need to also store the
table the batch was destined for as part of the batchlog - which seems a bit
over the top just for the sake of MV's.
I also considered using an unlogged batch, because for a local write it should
be enough, however I couldn't figure out any nice and clean way to issue an
UNLOGGED batch, and according to CASSANDRA-9283 a LOGGED batch should
essentially be unlogged if it's for a single partition, but I haven't yet found
these "optimisations" that guarantee this and even still using a logged batch
doesn't work because GCGS=0.
> Break up large MV mutations to prevent OOMs
> -------------------------------------------
>
> Key: CASSANDRA-12783
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12783
> Project: Cassandra
> Issue Type: Bug
> Components: Local Write-Read Paths, Materialized Views
> Reporter: Carl Yeksigian
> Assignee: Kurt Greaves
> Fix For: 4.x
>
>
> We only use the code path added in CASSANDRA-12268 for the view builder
> because otherwise we would break the contract of the batchlog, where some
> mutations may be written and pushed out before the whole batch log has been
> saved.
> We would need to ensure that all of the updates make it to the batchlog
> before allowing the batchlog manager to try to replay them, but also before
> we start pushing out updates to the paired replicas.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
