[jira] [Comment Edited] (CASSANDRA-14834) Avoid keeping StreamingTombstoneHistogramBuilder.Spool in memory during the whole compaction

2021-01-12 Thread Michael Semb Wever (Jira)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-14834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17261379#comment-17261379
 ] 

Michael Semb Wever edited comment on CASSANDRA-14834 at 1/12/21, 11:23 AM:
---

ci-cassandra CI – 
https://ci-cassandra.apache.org/blue/organizations/jenkins/Cassandra-devbranch/detail/Cassandra-devbranch/281/pipeline



was (Author: michaelsembwever):
ci-cassandra CI – 
https://ci-cassandra.apache.org/blue/organizations/jenkins/Cassandra-devbranch/detail/Cassandra-devbranch/280/pipeline

> Avoid keeping StreamingTombstoneHistogramBuilder.Spool in memory during the 
> whole compaction
> 
>
> Key: CASSANDRA-14834
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14834
> Project: Cassandra
>  Issue Type: Bug
>  Components: Local/Compaction
>Reporter: Marcus Eriksson
>Assignee: Adam Holmberg
>Priority: Low
> Fix For: 4.0-beta
>
>
> Since CASSANDRA-13444 {{StreamingTombstoneHistogramBuilder.Spool}} is 
> allocated to keep around an array with 131072 * 2 * 2 integers *per written 
> sstable* during the whole compaction. With LCS at times creating 1000s of 
> sstables during a compaction it kills the node.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org



[jira] [Comment Edited] (CASSANDRA-14834) Avoid keeping StreamingTombstoneHistogramBuilder.Spool in memory during the whole compaction

2020-12-03 Thread Adam Holmberg (Jira)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-14834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17243272#comment-17243272
 ] 

Adam Holmberg edited comment on CASSANDRA-14834 at 12/3/20, 4:02 PM:
-

I looked at it for a few minutes yesterday and at first blush it looked like it 
should be a simple solution to release the large buffer when metadata is 
"finalized". However, this 
[comment|https://github.com/apache/cassandra/blob/00fb6d76d0a97af06ba27c1180d6dcddfa337fea/src/java/org/apache/cassandra/utils/streamhist/StreamingTombstoneHistogramBuilder.java#L138-L140]
 gives me pause.

{quote}
 * Creates a 'finished' snapshot of the current state of the historgram, 
but leaves this builder instance
 * open for subsequent additions to the histograms. Basically, this allows 
us to have some degree of sanity
 * wrt sstable early open.
{quote}

It seems to suggest that we have instances being updated after finalization. 
Will have to dig into that a bit more. It might still be possible to update the 
histogram without the large buffers, but I need to understand how impactful 
that would be (or indeed, if the comment is still valid).

Not moving this ticket yet due to a handful of other things in-flight.


was (Author: aholmber):
I looked at it for a few minutes yesterday and at first blush it looked like it 
should be a simple solution to release the large buffer when metadata is 
"finalized". However, this 
[comment|https://github.com/apache/cassandra/blob/00fb6d76d0a97af06ba27c1180d6dcddfa337fea/src/java/org/apache/cassandra/utils/streamhist/StreamingTombstoneHistogramBuilder.java#L138-L140]
 gives me pause.

{quote}
 * Creates a 'finished' snapshot of the current state of the historgram, 
but leaves this builder instance
 * open for subsequent additions to the histograms. Basically, this allows 
us to have some degree of sanity
 * wrt sstable early open.
{quote}

It seems to suggest that we have instances being updated after finalization. 
Will have to dig into that a bit more. Not moving this ticket yet due to a 
handful of other things in-flight.

> Avoid keeping StreamingTombstoneHistogramBuilder.Spool in memory during the 
> whole compaction
> 
>
> Key: CASSANDRA-14834
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14834
> Project: Cassandra
>  Issue Type: Bug
>  Components: Local/Compaction
>Reporter: Marcus Eriksson
>Priority: Low
> Fix For: 4.0, 4.0-beta
>
>
> Since CASSANDRA-13444 {{StreamingTombstoneHistogramBuilder.Spool}} is 
> allocated to keep around an array with 131072 * 2 * 2 integers *per written 
> sstable* during the whole compaction. With LCS at times creating 1000s of 
> sstables during a compaction it kills the node.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org