[jira] [Commented] (CASSANDRA-13561) Purge TTL on expiration
[ https://issues.apache.org/jira/browse/CASSANDRA-13561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16093463#comment-16093463 ] Jeff Jirsa commented on CASSANDRA-13561: I'm just trying to page this into my mind and also try to correlate with other recent tickets I've seen. This seems pretty close to what [~bdeggleston] touched on with CASSANDRA-13643 , though this is more aggressive (and more invasive, in the sense that it needs a new table property). Does Blake's addition to call purge on the tombstone generated have a similar effect here? > Purge TTL on expiration > --- > > Key: CASSANDRA-13561 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13561 > Project: Cassandra > Issue Type: New Feature >Reporter: Andrew Whang >Assignee: Andrew Whang >Priority: Minor > Fix For: 4.0 > > > Tables with mostly TTL columns tend to suffer from high droppable tombstone > ratio, which results in higher read latency, cpu utilization, and disk usage. > Expired TTL data become tombstones, and the nature of purging tombstones > during compaction (due to checking for overlapping SSTables) make them > susceptible to surviving much longer than expected. A table option to purge > TTL on expiration would address this issue, by preventing them from becoming > tombstones. A boolean purge_ttl_on_expiration table setting would allow users > to easily turn the feature on or off. > Being more aggressive with gc_grace could also address the problem of long > lasting tombstones, but that would affect tombstones from deletes as well. > Even if a purged [expired] cell is revived via repair from a node that hasn't > yet compacted away the cell, it would be revived as an expiring cell with the > same localDeletionTime, so reads should properly handle them. As well, it > would be purged in the next compaction. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13561) Purge TTL on expiration
[
https://issues.apache.org/jira/browse/CASSANDRA-13561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16094112#comment-16094112
]
Kurt Greaves commented on CASSANDRA-13561:
--
I disregarded HH because HH cannot be relied on to provide consistency
guarantees. The scenario is still bad if either the node is down for longer
than HH, or if HH fails for some reason.
Also note that GCGS=0 disables hinted handoff for the table (at least last time
I checked this hadn't changed). So it's not exactly the same.
I see the point that this could be useful for cases where a default TTL is set,
however even with a default TTL you can still update/remove the TTL of columns.
This means the risk is only really mitigated where you set a default TTL, and
you never do anything to alter that TTL.
The best you can currently in this case is that you could set GCGS=hinted
handoff window and you don't sacrifice consistency, and you only keep the
expired cells around for min 3 hours. This case is really perfectly fine when
you are using TWCS/DTCS, as the SSTables should expire efficiently and it's
unlikely you'd be querying expired data anyway.
The only case I can think of where you would really get a benefit without any
risk is when using the same write strategy (default TTL/always TTL) but on a
table that doesn't work with TWCS/DTCS, so you use STCS/LCS, and you also have
to keep GCGS high because you also do manual deletes.
Your proposal would make it so that you can remove data a little bit faster if
it compacts within GCGS. I'm a bit skeptical if that's actually necessary,
especially with the introduction of {{provide_overlapping_tombstones}} in 3.10,
which should allow much more efficient removal of tombstones.
Really if you're generating that many tombstones within GCGS+time to compaction
per partition/PK that it is causing significant latency I'd be very surprised.
I'd be interested to see some metrics surrounding this, and confirm that other
options don't perform well enough first. Maybe you could give us an example of
your use case where this gave the benefits?
I feel that this is a sufficiently dangerous option with hard to understand
implications that we should need pretty good justification before making it
readily available for configure on a table level. That just screams
impending-shoot-yourself-in-the-foot. Maybe we could put some other safety net
around it (e.g a property passed into C*) that doesn't allow changing it unless
you start C* with that option set, but yeah, let's figure out some concrete
benefits first.
> Purge TTL on expiration
> ---
>
> Key: CASSANDRA-13561
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13561
> Project: Cassandra
> Issue Type: New Feature
>Reporter: Andrew Whang
>Assignee: Andrew Whang
>Priority: Minor
> Fix For: 4.0
>
>
> Tables with mostly TTL columns tend to suffer from high droppable tombstone
> ratio, which results in higher read latency, cpu utilization, and disk usage.
> Expired TTL data become tombstones, and the nature of purging tombstones
> during compaction (due to checking for overlapping SSTables) make them
> susceptible to surviving much longer than expected. A table option to purge
> TTL on expiration would address this issue, by preventing them from becoming
> tombstones. A boolean purge_ttl_on_expiration table setting would allow users
> to easily turn the feature on or off.
> Being more aggressive with gc_grace could also address the problem of long
> lasting tombstones, but that would affect tombstones from deletes as well.
> Even if a purged [expired] cell is revived via repair from a node that hasn't
> yet compacted away the cell, it would be revived as an expiring cell with the
> same localDeletionTime, so reads should properly handle them. As well, it
> would be purged in the next compaction.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13561) Purge TTL on expiration
[ https://issues.apache.org/jira/browse/CASSANDRA-13561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16143191#comment-16143191 ] Jeff Jirsa commented on CASSANDRA-13561: Sorry, I actually think I confused issues previously: CASSANDRA-13643 - converting expired ttl cells to tombstones causing unnecessary digest mismatches > Purge TTL on expiration > --- > > Key: CASSANDRA-13561 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13561 > Project: Cassandra > Issue Type: New Feature >Reporter: Andrew Whang >Assignee: Andrew Whang >Priority: Minor > Fix For: 4.0 > > > Tables with mostly TTL columns tend to suffer from high droppable tombstone > ratio, which results in higher read latency, cpu utilization, and disk usage. > Expired TTL data become tombstones, and the nature of purging tombstones > during compaction (due to checking for overlapping SSTables) make them > susceptible to surviving much longer than expected. A table option to purge > TTL on expiration would address this issue, by preventing them from becoming > tombstones. A boolean purge_ttl_on_expiration table setting would allow users > to easily turn the feature on or off. > Being more aggressive with gc_grace could also address the problem of long > lasting tombstones, but that would affect tombstones from deletes as well. > Even if a purged [expired] cell is revived via repair from a node that hasn't > yet compacted away the cell, it would be revived as an expiring cell with the > same localDeletionTime, so reads should properly handle them. As well, it > would be purged in the next compaction. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13561) Purge TTL on expiration
[ https://issues.apache.org/jira/browse/CASSANDRA-13561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16058582#comment-16058582 ] Jeff Jirsa commented on CASSANDRA-13561: This is being addressed in an alternate way - by adding a table option to instruct cassandra to ignore overlapping sstables ( see CASSANDRA-13418 ) - would you be OK with that approach? If so, can I close this as a duplicate? > Purge TTL on expiration > --- > > Key: CASSANDRA-13561 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13561 > Project: Cassandra > Issue Type: New Feature >Reporter: Andrew Whang >Assignee: Andrew Whang >Priority: Minor > Fix For: 4.0 > > > Tables with mostly TTL columns tend to suffer from high droppable tombstone > ratio, which results in higher read latency, cpu utilization, and disk usage. > Expired TTL data become tombstones, and the nature of purging tombstones > during compaction (due to checking for overlapping SSTables) make them > susceptible to surviving much longer than expected. A table option to purge > TTL on expiration would address this issue, by preventing them from becoming > tombstones. A boolean purge_ttl_on_expiration table setting would allow users > to easily turn the feature on or off. > Being more aggressive with gc_grace could also address the problem of long > lasting tombstones, but that would affect tombstones from deletes as well. > Even if a purged [expired] cell is revived via repair from a node that hasn't > yet compacted away the cell, it would be revived as an expiring cell with the > same localDeletionTime, so reads should properly handle them. As well, it > would be purged in the next compaction. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13561) Purge TTL on expiration
[ https://issues.apache.org/jira/browse/CASSANDRA-13561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16077291#comment-16077291 ] Andrew Whang commented on CASSANDRA-13561: -- Correct me if I'm wrong, but the limitations I see with CASSANDRA-13418 are: 1. It's tied to a compaction strategy, and currently only supported in TWCS. 2. It helps cleanup sstables, but only when all items in the sstable have expired. The approach I'm proposing would support dropping individual (expired) cells within the sstable, without having to wait for all other items to expire. > Purge TTL on expiration > --- > > Key: CASSANDRA-13561 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13561 > Project: Cassandra > Issue Type: New Feature >Reporter: Andrew Whang >Assignee: Andrew Whang >Priority: Minor > Fix For: 4.0 > > > Tables with mostly TTL columns tend to suffer from high droppable tombstone > ratio, which results in higher read latency, cpu utilization, and disk usage. > Expired TTL data become tombstones, and the nature of purging tombstones > during compaction (due to checking for overlapping SSTables) make them > susceptible to surviving much longer than expected. A table option to purge > TTL on expiration would address this issue, by preventing them from becoming > tombstones. A boolean purge_ttl_on_expiration table setting would allow users > to easily turn the feature on or off. > Being more aggressive with gc_grace could also address the problem of long > lasting tombstones, but that would affect tombstones from deletes as well. > Even if a purged [expired] cell is revived via repair from a node that hasn't > yet compacted away the cell, it would be revived as an expiring cell with the > same localDeletionTime, so reads should properly handle them. As well, it > would be purged in the next compaction. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13561) Purge TTL on expiration
[ https://issues.apache.org/jira/browse/CASSANDRA-13561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16091165#comment-16091165 ] Kurt Greaves commented on CASSANDRA-13561: -- You are right. CASSANDRA-13418 is a different issue to what you are talking about. It is a specific option for allowing expiration of only completely expired SSTables where there are overlaps with another SSTable. My understanding is that you are saying we should purge TTL'd cells as soon as they expire, rather than waiting for GCGS to pass, but obviously we would still require a compaction to make that happen. Is that correct? In theory it would be nice but I think once you take into account that an update that sets a TTL may not make it to all nodes, you have a problem. For example: 3 nodes A B and C, RF=3 An INSERT without any TTL's set for PK "a" makes it to all 3 nodes B goes down An update setting a single column "b" to TTL after x seconds for PK "a" is sent. B comes back up (disregarding HH) A and C are now in a state with "b" having a TTL while B has no TTL for the same cell. "b" expires on A and C, a compaction is triggered and the cell gets removed. "b" returns from the dead from B. > Purge TTL on expiration > --- > > Key: CASSANDRA-13561 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13561 > Project: Cassandra > Issue Type: New Feature >Reporter: Andrew Whang >Assignee: Andrew Whang >Priority: Minor > Fix For: 4.0 > > > Tables with mostly TTL columns tend to suffer from high droppable tombstone > ratio, which results in higher read latency, cpu utilization, and disk usage. > Expired TTL data become tombstones, and the nature of purging tombstones > during compaction (due to checking for overlapping SSTables) make them > susceptible to surviving much longer than expected. A table option to purge > TTL on expiration would address this issue, by preventing them from becoming > tombstones. A boolean purge_ttl_on_expiration table setting would allow users > to easily turn the feature on or off. > Being more aggressive with gc_grace could also address the problem of long > lasting tombstones, but that would affect tombstones from deletes as well. > Even if a purged [expired] cell is revived via repair from a node that hasn't > yet compacted away the cell, it would be revived as an expiring cell with the > same localDeletionTime, so reads should properly handle them. As well, it > would be purged in the next compaction. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13561) Purge TTL on expiration
[ https://issues.apache.org/jira/browse/CASSANDRA-13561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16093343#comment-16093343 ] Andrew Whang commented on CASSANDRA-13561: -- Correct, this implementation would still require a compaction to purge the expired cells. Also correct, in the scenario you described there is a risk of the expired cell returning from the dead. The scenario does disregard HH. If hints are properly delivered, the risk is mitigated. To be clear, the scenario is similar to setting GCGS=0. The user has to understand the risk of using these settings. The risk is also mitigated in use cases where there is a default TTL on the table or client mutations use a default TTL. These are the scenarios for which we use this feature in our environment. In these use cases, we noticed the table suffered from high droppable tombstone ratio and high read latency. Using this feature to purge TTL as they expired addressed both droppable tombstone ratio and read latency. > Purge TTL on expiration > --- > > Key: CASSANDRA-13561 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13561 > Project: Cassandra > Issue Type: New Feature >Reporter: Andrew Whang >Assignee: Andrew Whang >Priority: Minor > Fix For: 4.0 > > > Tables with mostly TTL columns tend to suffer from high droppable tombstone > ratio, which results in higher read latency, cpu utilization, and disk usage. > Expired TTL data become tombstones, and the nature of purging tombstones > during compaction (due to checking for overlapping SSTables) make them > susceptible to surviving much longer than expected. A table option to purge > TTL on expiration would address this issue, by preventing them from becoming > tombstones. A boolean purge_ttl_on_expiration table setting would allow users > to easily turn the feature on or off. > Being more aggressive with gc_grace could also address the problem of long > lasting tombstones, but that would affect tombstones from deletes as well. > Even if a purged [expired] cell is revived via repair from a node that hasn't > yet compacted away the cell, it would be revived as an expiring cell with the > same localDeletionTime, so reads should properly handle them. As well, it > would be purged in the next compaction. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13561) Purge TTL on expiration
[ https://issues.apache.org/jira/browse/CASSANDRA-13561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16246911#comment-16246911 ] Andrew Whang commented on CASSANDRA-13561: -- CASSANDRA-13643 seems to do something similar, but safer. Closing this out. > Purge TTL on expiration > --- > > Key: CASSANDRA-13561 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13561 > Project: Cassandra > Issue Type: New Feature >Reporter: Andrew Whang >Assignee: Andrew Whang >Priority: Minor > Fix For: 4.0 > > > Tables with mostly TTL columns tend to suffer from high droppable tombstone > ratio, which results in higher read latency, cpu utilization, and disk usage. > Expired TTL data become tombstones, and the nature of purging tombstones > during compaction (due to checking for overlapping SSTables) make them > susceptible to surviving much longer than expected. A table option to purge > TTL on expiration would address this issue, by preventing them from becoming > tombstones. A boolean purge_ttl_on_expiration table setting would allow users > to easily turn the feature on or off. > Being more aggressive with gc_grace could also address the problem of long > lasting tombstones, but that would affect tombstones from deletes as well. > Even if a purged [expired] cell is revived via repair from a node that hasn't > yet compacted away the cell, it would be revived as an expiring cell with the > same localDeletionTime, so reads should properly handle them. As well, it > would be purged in the next compaction. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
