[
https://issues.apache.org/jira/browse/CASSANDRA-12783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16144131#comment-16144131
]
ZhaoYang edited comment on CASSANDRA-12783 at 8/29/17 2:47 AM:
---------------------------------------------------------------
I think this design is the right direction to solve the OOM issue with MV.
1. The default TimeUUID(version 1).compareTo() doesn't compare Timestamp first
then machine part.
You could try {{TimeUUIDType.compare}}
{code}
UUID V1, Most Significant Bit:
0xFFFFFFFF00000000 time_low
0x00000000FFFF0000 time_mid
0x000000000000F000 version
0x0000000000000FFF time_hi
{code}
bq. Not really sure that it will prevent OOMs, but description isn't really
clear what that is about.
I think if we could achieve following 2 points, we could say base mutation
causing OOM is resolved
2. When partition/range deletion happen, it'd be good to avoid holding entire
view updates of the same base partition in memory and OOM.
Maybe we could store part by part into new batchlog, throttle by
max-mutation-size
3. Depends on the data modelling, one view mutation could exceed
max-mutation-size
{code}
eg. base: (pk, ck) v1 v2
view: (pk, v1, ck) v2
partition deletion on base, same amount of view updates are generated.
{code}
We may need some iterator mechansim to split a huge mutation into small chunks,
as long as new batchlog could provide atomicity guarantee.
was (Author: jasonstack):
I think this design is the right direction to solve the OOM issue with MV.
1. The default TimeUUID(version 1).compareTo() doesn't compare Timestamp first
then machine part.
You could try {{TimeUUIDType.compare}}
{code}
UUID V1, Most Significant Bit:
0xFFFFFFFF00000000 time_low
0x00000000FFFF0000 time_mid
0x000000000000F000 version
0x0000000000000FFF time_hi
{code}
bq. Not really sure that it will prevent OOMs, but description isn't really
clear what that is about.
I think if we could achieve following 2 points, we could say base mutation
causing OOM is resolved
2. When partition/range deletion happen, it'd be good to avoid holding entire
view updates in memory and OOM.
Maybe we could store part by part into new batchlog, throttle by
max-mutation-size
3. Depends on the data modelling, one view mutation could exceed
max-mutation-size
{code}
eg. base: (pk, ck) v1 v2
view: (pk, v1, ck) v2
partition deletion on base, same amount of view updates are generated.
{code}
We may need some iterator mechansim to split a huge mutation into small chunks,
as long as new batchlog could provide atomicity guarantee.
> Break up large MV mutations to prevent OOMs
> -------------------------------------------
>
> Key: CASSANDRA-12783
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12783
> Project: Cassandra
> Issue Type: Bug
> Components: Local Write-Read Paths, Materialized Views
> Reporter: Carl Yeksigian
> Assignee: Kurt Greaves
> Fix For: 4.x
>
>
> We only use the code path added in CASSANDRA-12268 for the view builder
> because otherwise we would break the contract of the batchlog, where some
> mutations may be written and pushed out before the whole batch log has been
> saved.
> We would need to ensure that all of the updates make it to the batchlog
> before allowing the batchlog manager to try to replay them, but also before
> we start pushing out updates to the paired replicas.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
