[jira] [Commented] (CASSANDRA-6666) Avoid accumulating tombstones after partial hint replay
[ https://issues.apache.org/jira/browse/CASSANDRA-?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14151127#comment-14151127 ] Donald Smith commented on CASSANDRA-: - I know this is moot because of the redesign of hints, but I want to understand this. OK, if a hint was successfully delivered, then I can see how a tombstone would be useful for causing deletion of *older* instances in other sstables. But if a hint timed-out (tombstone), then any older instance will also have timed out (presumably). So, could tombstones be deleted in that case (timeout)? Perhaps a timed out cell IS a tombstone, but my point is: I don't see why they need to take up space. > Avoid accumulating tombstones after partial hint replay > --- > > Key: CASSANDRA- > URL: https://issues.apache.org/jira/browse/CASSANDRA- > Project: Cassandra > Issue Type: Bug > Components: Core >Reporter: Jonathan Ellis >Assignee: Jonathan Ellis >Priority: Minor > Labels: hintedhandoff > Attachments: .txt, cassandra_system.log.debug.gz > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-6666) Avoid accumulating tombstones after partial hint replay
[ https://issues.apache.org/jira/browse/CASSANDRA-?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14141257#comment-14141257 ] Jonathan Ellis commented on CASSANDRA-: --- How else would you mark it delivered? (See CASSANDRA-6230 for a fix that is out of scope here.) > Avoid accumulating tombstones after partial hint replay > --- > > Key: CASSANDRA- > URL: https://issues.apache.org/jira/browse/CASSANDRA- > Project: Cassandra > Issue Type: Bug > Components: Core >Reporter: Jonathan Ellis >Assignee: Jonathan Ellis >Priority: Minor > Labels: hintedhandoff > Attachments: .txt, cassandra_system.log.debug.gz > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-6666) Avoid accumulating tombstones after partial hint replay
[ https://issues.apache.org/jira/browse/CASSANDRA-?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14141256#comment-14141256 ] Aleksey Yeschenko commented on CASSANDRA-: -- You can't just omit them, because multiple sstables. Anyway, I'm closing the ticket as duplicate of CASSANDRA-6230, because there is no bullet-proof way to fix the issue until then, unfortunately. > Avoid accumulating tombstones after partial hint replay > --- > > Key: CASSANDRA- > URL: https://issues.apache.org/jira/browse/CASSANDRA- > Project: Cassandra > Issue Type: Bug > Components: Core >Reporter: Jonathan Ellis >Assignee: Jonathan Ellis >Priority: Minor > Labels: hintedhandoff > Attachments: .txt, cassandra_system.log.debug.gz > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-6666) Avoid accumulating tombstones after partial hint replay
[ https://issues.apache.org/jira/browse/CASSANDRA-?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14141239#comment-14141239 ] Donald Smith commented on CASSANDRA-: - Perhaps this question is inappropriate here. But can tombstones be completely omitted for system.hints, given that they're not replicated and given that only internal code modifies them in normal operation? If a hint is delivered successfully, why does it need a tombstone at all? If it times out, then cassandra is going to give up on delivering it. So, again, why does it need a tombstone? On the cassandra irc channel, several people speculated that the cassandra developers didn't want to make a *special case* for system.hints. Also, system.hints has *gc_grace_seconds=0*, so they won't survive a compaction, presumably. I realize that in C* 3.0, tombstones will be moved out of tables, but I still am perplexed why tombstones are needed at all for hints. My apologies if this is a dumb question. > Avoid accumulating tombstones after partial hint replay > --- > > Key: CASSANDRA- > URL: https://issues.apache.org/jira/browse/CASSANDRA- > Project: Cassandra > Issue Type: Bug > Components: Core >Reporter: Jonathan Ellis >Assignee: Jonathan Ellis >Priority: Minor > Labels: hintedhandoff > Fix For: 2.0.11 > > Attachments: .txt, cassandra_system.log.debug.gz > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-6666) Avoid accumulating tombstones after partial hint replay
[
https://issues.apache.org/jira/browse/CASSANDRA-?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14078171#comment-14078171
]
Vishal Mehta commented on CASSANDRA-:
-
Hello Every,
Please pardon my ignorance, since I am writing first time in opensource bug
report.
Recently I think I hit this bug because I saw similar symptoms in my 3 node
cassandra setup. Where I am running a test with around 12K qps (inserts in 3
different tables) with TTL set to 1 hour and keyspace has GC seconds set to
14400 (4 hours).
So tests eventually runs to a point where Cassandra sees Tombstones more than
100K and it crashes with following exception in
/var/log/cassandra/cassandra.log.
{noformat}
ERROR 13:23:56,747 Scanned over 10 tombstones in system.hints; query
aborted (see tombstone_fail_threshold)
ERROR 13:23:56,962 Exception in thread Thread[HintedHandoff:1,1,main]
org.apache.cassandra.db.filter.TombstoneOverwhelmingException
at
org.apache.cassandra.db.filter.SliceQueryFilter.collectReducedColumns(SliceQueryFilter.java:202)
at
org.apache.cassandra.db.filter.QueryFilter.collateColumns(QueryFilter.java:122)
at
org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:80)
at
org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:72)
at
org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:297)
at
org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:53)
at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1547)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1376)
at
org.apache.cassandra.db.HintedHandOffManager.doDeliverHintsToEndpoint(HintedHandOffManager.java:373)
at
org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:330)
at
org.apache.cassandra.db.HintedHandOffManager.access$300(HintedHandOffManager.java:91)
at
org.apache.cassandra.db.HintedHandOffManager$5.run(HintedHandOffManager.java:547)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:724)
INFO 13:24:00,987 No gossip backlog; proceeding
{noformat}
*Note:* Is it plausible to keep GC seconds closer to TTLs? Also I could see one
of the node deleted all the records from disk and freed up the space, where as
other two nodes never deleted their tombstones.
> Avoid accumulating tombstones after partial hint replay
> ---
>
> Key: CASSANDRA-
> URL: https://issues.apache.org/jira/browse/CASSANDRA-
> Project: Cassandra
> Issue Type: Bug
> Components: Core
>Reporter: Jonathan Ellis
>Assignee: Jonathan Ellis
>Priority: Minor
> Labels: hintedhandoff
> Fix For: 2.0.10
>
> Attachments: .txt, cassandra_system.log.debug.gz
>
>
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (CASSANDRA-6666) Avoid accumulating tombstones after partial hint replay
[ https://issues.apache.org/jira/browse/CASSANDRA-?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14026545#comment-14026545 ] Jonathan Ellis commented on CASSANDRA-: --- Because the hints manager does compactions manually, only when it knows it will be useful (after hint delivery). The generic strategies would waste a ton of effort, especially LCS. > Avoid accumulating tombstones after partial hint replay > --- > > Key: CASSANDRA- > URL: https://issues.apache.org/jira/browse/CASSANDRA- > Project: Cassandra > Issue Type: Bug > Components: Core >Reporter: Jonathan Ellis >Assignee: Jonathan Ellis >Priority: Minor > Labels: hintedhandoff > Fix For: 1.2.17, 2.0.9 > > Attachments: .txt, cassandra_system.log.debug.gz > > -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (CASSANDRA-6666) Avoid accumulating tombstones after partial hint replay
[
https://issues.apache.org/jira/browse/CASSANDRA-?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14026537#comment-14026537
]
Walid Darwish commented on CASSANDRA-:
--
Just wondering why compaction is disabled on the hints table by default (and
only on that table across the system keyspace)?
is that for performance reasons? Can this be turned on safely?
Also, can this be changed safely to levelled compaction?
compaction= {'enabled': 'false', 'class': 'SizeTieredCompactionStrategy'}
I am using 2.0.7
> Avoid accumulating tombstones after partial hint replay
> ---
>
> Key: CASSANDRA-
> URL: https://issues.apache.org/jira/browse/CASSANDRA-
> Project: Cassandra
> Issue Type: Bug
> Components: Core
>Reporter: Jonathan Ellis
>Assignee: Jonathan Ellis
>Priority: Minor
> Labels: hintedhandoff
> Fix For: 1.2.17, 2.0.9
>
> Attachments: .txt, cassandra_system.log.debug.gz
>
>
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (CASSANDRA-6666) Avoid accumulating tombstones after partial hint replay
[
https://issues.apache.org/jira/browse/CASSANDRA-?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13978389#comment-13978389
]
Jonathan Ellis commented on CASSANDRA-:
---
The sstable is a single partition with cells like this:
{noformat}
{"key": "3933deebcbec4fc5b4be8d7cb5b62e5c",
"columns":
[["2c0bd697-b391-11e3-96a7-ad88e7f21869:7",1395692184,1395692184185,"d"],
["2c0bd698-b391-11e3-96a7-ad88e7f21869:7",1395692184,1395692184185,"d"],
["2c0bd699-b391-11e3-96a7-ad88e7f21869:7",1395692184,1395692184185,"d"],
["2c0bd69a-b391-11e3-96a7-ad88e7f21869:7",1395692184,1395692184185,"d"],
["2c0bd69b-b391-11e3-96a7-ad88e7f21869:7",1395692184,1395692184185,"d"],
{noformat}
I.e., a lot of tombstones deleted Mar 24 15:16:24 2014. So I would have
expected the compaction that ran at 16:40 to have collected them.
> Avoid accumulating tombstones after partial hint replay
> ---
>
> Key: CASSANDRA-
> URL: https://issues.apache.org/jira/browse/CASSANDRA-
> Project: Cassandra
> Issue Type: Bug
> Components: Core
>Reporter: Jonathan Ellis
>Assignee: Jonathan Ellis
>Priority: Minor
> Labels: hintedhandoff
> Fix For: 1.2.16, 2.0.6
>
> Attachments: .txt, cassandra_system.log.debug.gz
>
>
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (CASSANDRA-6666) Avoid accumulating tombstones after partial hint replay
[ https://issues.apache.org/jira/browse/CASSANDRA-?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13978255#comment-13978255 ] Jonathan Ellis commented on CASSANDRA-: --- [~wdarwish] Nope. [~iamaleksey] do you have any explanation for this before we debug log the hell out of hasIrrelevantData + shouldPurge? > Avoid accumulating tombstones after partial hint replay > --- > > Key: CASSANDRA- > URL: https://issues.apache.org/jira/browse/CASSANDRA- > Project: Cassandra > Issue Type: Bug > Components: Core >Reporter: Jonathan Ellis >Assignee: Jonathan Ellis >Priority: Minor > Labels: hintedhandoff > Fix For: 1.2.16, 2.0.6 > > Attachments: .txt, cassandra_system.log.debug.gz > > -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (CASSANDRA-6666) Avoid accumulating tombstones after partial hint replay
[ https://issues.apache.org/jira/browse/CASSANDRA-?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13978190#comment-13978190 ] Walid Darwish commented on CASSANDRA-: -- Jonathan, I am following up on this on behalf of Anne. Any luck with the log files uploaded? > Avoid accumulating tombstones after partial hint replay > --- > > Key: CASSANDRA- > URL: https://issues.apache.org/jira/browse/CASSANDRA- > Project: Cassandra > Issue Type: Bug > Components: Core >Reporter: Jonathan Ellis >Assignee: Jonathan Ellis >Priority: Minor > Labels: hintedhandoff > Fix For: 1.2.16, 2.0.6 > > Attachments: .txt, cassandra_system.log.debug.gz > > -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (CASSANDRA-6666) Avoid accumulating tombstones after partial hint replay
[ https://issues.apache.org/jira/browse/CASSANDRA-?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13950812#comment-13950812 ] Anne Sullivan commented on CASSANDRA-: -- Files uploaded to scp server - thanks. > Avoid accumulating tombstones after partial hint replay > --- > > Key: CASSANDRA- > URL: https://issues.apache.org/jira/browse/CASSANDRA- > Project: Cassandra > Issue Type: Bug > Components: Core >Reporter: Jonathan Ellis >Assignee: Jonathan Ellis >Priority: Minor > Labels: hintedhandoff > Fix For: 1.2.16, 2.0.6 > > Attachments: .txt, cassandra_system.log.debug.gz > > -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (CASSANDRA-6666) Avoid accumulating tombstones after partial hint replay
[ https://issues.apache.org/jira/browse/CASSANDRA-?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13941048#comment-13941048 ] Jonathan Ellis commented on CASSANDRA-: --- So, most of it got compacted away, but the rest did not and even survived another compaction. Can you attach the surviving sstable? If you're worried about exposing sensitive data I can arrange a private location. > Avoid accumulating tombstones after partial hint replay > --- > > Key: CASSANDRA- > URL: https://issues.apache.org/jira/browse/CASSANDRA- > Project: Cassandra > Issue Type: Bug > Components: Core >Reporter: Jonathan Ellis >Assignee: Jonathan Ellis >Priority: Minor > Labels: hintedhandoff > Fix For: 1.2.16, 2.0.6 > > Attachments: .txt, cassandra_system.log.debug.gz > > -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (CASSANDRA-6666) Avoid accumulating tombstones after partial hint replay
[
https://issues.apache.org/jira/browse/CASSANDRA-?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13940757#comment-13940757
]
Anne Sullivan commented on CASSANDRA-:
--
Attached debug logs (cassandra_system.log.debug.gz)
- two node cluster, RF=2 (node A / node B)
- bring node B down for 30 minutes, back online @ 2014-03-19 10:15:00
- hinted handoff starts @ 2014-03-19 10:18:00
- hinted handoff finishes successfully @ 2014-03-19 10:59:04:
INFO [HintedHandoff:1] 2014-03-19 10:59:04,434 HintedHandOffManager.java (line
378) Finished hinted handoff of 1903040 rows to endpoint /###.###.###.###
- @ 2014-03-19 10:59:16,428 user defined compaction compacts down to one
sstable (expected result = no sstables / empty set):
INFO [CompactionExecutor:3421] 2014-03-19 10:59:16,428 CompactionTask.java
(line 275) Compacted 4 sstables to
[/path/to/db/data/system/hints/system-hints-jb-49,]. 172,043,512 bytes to
6,365,114 (~3% of original) in 11,991ms = 0.506233MB/s. 4 total partitions
merged to 1. Partition merge counts were {4:1, }
- @ 2014-03-19 11:08:08 another hinted handoff starts (presumably because the
hints table isn't empty, and it's checking every 10 minutes) and reports 0 live
cells:
WARN [HintedHandoff:2] 2014-03-19 11:08:08,639 SliceQueryFilter.java (line 210)
Read 0 live and 75750 tombstoned cells in system.hints (see
tombstone_warn_threshold)
INFO [HintedHandoff:2] 2014-03-19 11:08:08,639 HintedHandOffManager.java (line
378) Finished hinted handoff of 0 rows to endpoint /###.###.###.###
- user defined compaction again leaves behind a single sstable:
INFO [CompactionExecutor:3438] 2014-03-19 11:08:08,963 CompactionTask.java
(line 275) Compacted 1 sstables to
[/path/to/db/data/system/hints/system-hints-jb-50,]. 6,365,114 bytes to
6,365,114 (~100% of original) in 322ms = 18.851695MB/s. 1 total partitions
merged to 1. Partition merge counts were {1:1, }
Partial output from running "cqlsh> describe keyspace system", to confirm that
gc_grace_seconds=0, so expecting tombstones to be removed on compaction:
CREATE TABLE hints (
target_id uuid,
hint_id timeuuid,
message_version int,
mutation blob,
PRIMARY KEY (target_id, hint_id, message_version)
) WITH COMPACT STORAGE AND
bloom_filter_fp_chance=0.01 AND
caching='KEYS_ONLY' AND
comment='hints awaiting delivery' AND
dclocal_read_repair_chance=0.00 AND
gc_grace_seconds=0 AND
index_interval=128 AND
read_repair_chance=0.00 AND
replicate_on_write='true' AND
populate_io_cache_on_flush='false' AND
default_time_to_live=0 AND
speculative_retry='99.0PERCENTILE' AND
memtable_flush_period_in_ms=360 AND
compaction={'enabled': 'false', 'class': 'SizeTieredCompactionStrategy'} AND
compression={'sstable_compression': 'LZ4Compressor'};
> Avoid accumulating tombstones after partial hint replay
> ---
>
> Key: CASSANDRA-
> URL: https://issues.apache.org/jira/browse/CASSANDRA-
> Project: Cassandra
> Issue Type: Bug
> Components: Core
>Reporter: Jonathan Ellis
>Assignee: Jonathan Ellis
>Priority: Minor
> Labels: hintedhandoff
> Fix For: 1.2.16, 2.0.6
>
> Attachments: .txt, cassandra_system.log.debug.gz
>
>
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (CASSANDRA-6666) Avoid accumulating tombstones after partial hint replay
[ https://issues.apache.org/jira/browse/CASSANDRA-?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13933837#comment-13933837 ] Jonathan Ellis commented on CASSANDRA-: --- Can you attach a log where this happens with debug enabled on o.a.c.db.compaction? > Avoid accumulating tombstones after partial hint replay > --- > > Key: CASSANDRA- > URL: https://issues.apache.org/jira/browse/CASSANDRA- > Project: Cassandra > Issue Type: Bug > Components: Core >Reporter: Jonathan Ellis >Assignee: Jonathan Ellis >Priority: Minor > Labels: hintedhandoff > Fix For: 1.2.16, 2.0.6 > > Attachments: .txt > > -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (CASSANDRA-6666) Avoid accumulating tombstones after partial hint replay
[ https://issues.apache.org/jira/browse/CASSANDRA-?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13918081#comment-13918081 ] Anne Sullivan commented on CASSANDRA-: -- With this fix, we're seeing a case where the user-defined compaction that runs post-hint-replay is not removing the sstables as expected. The hinted handoff finishes successfully, but some sstables remain (100% tombstones) and then the 10 minute check for hints keeps trying to start HH and failing with the tombstone exception. Running a manual compaction clears them up. It occurs 100% of the time if we have some larger sstables in the hints CF ... so 4 small sstables ~ 10MB and 2 sstables > 100MB, the 2 that are large are never processed by the user-defined compaction. > Avoid accumulating tombstones after partial hint replay > --- > > Key: CASSANDRA- > URL: https://issues.apache.org/jira/browse/CASSANDRA- > Project: Cassandra > Issue Type: Bug > Components: Core >Reporter: Jonathan Ellis >Assignee: Jonathan Ellis >Priority: Minor > Labels: hintedhandoff > Fix For: 1.2.16, 2.0.6 > > Attachments: .txt > > -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (CASSANDRA-6666) Avoid accumulating tombstones after partial hint replay
[ https://issues.apache.org/jira/browse/CASSANDRA-?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13907926#comment-13907926 ] graham sanderson commented on CASSANDRA-: - We have seen an issue where we get into the >100K tombstone state with the hints table also; It has happened more than once (worked around in 2.0.5 with manual compaction on all nodes of system.hints). One quick question... this patch will hopefully prevent us getting into this state in many cases, unless you have a huge number of hints for a node that is down for a very long time: since there is no auto-compaction, a large number of hints may have expired from TTL, thus preventing any further hint delivery. This is OK if you bring the node back up as a fresh one, otherwise, I guess suggesting manual compaction is fine, or we could do a compact().get() on seeing a TombstoneOverwhelmingException? > Avoid accumulating tombstones after partial hint replay > --- > > Key: CASSANDRA- > URL: https://issues.apache.org/jira/browse/CASSANDRA- > Project: Cassandra > Issue Type: Bug > Components: Core >Reporter: Jonathan Ellis >Assignee: Jonathan Ellis >Priority: Minor > Labels: hintedhandoff > Fix For: 1.2.16, 2.0.6 > > Attachments: .txt > > -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (CASSANDRA-6666) Avoid accumulating tombstones after partial hint replay
[ https://issues.apache.org/jira/browse/CASSANDRA-?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13894481#comment-13894481 ] Aleksey Yeschenko commented on CASSANDRA-: -- LGTM Nit: 1.2 has DatabaseDescriptor.getTombstoneDebugThreshold(), not DatabaseDescriptor.getTombstoneWarnThreshold() > Avoid accumulating tombstones after partial hint replay > --- > > Key: CASSANDRA- > URL: https://issues.apache.org/jira/browse/CASSANDRA- > Project: Cassandra > Issue Type: Bug > Components: Core >Reporter: Jonathan Ellis >Assignee: Jonathan Ellis >Priority: Minor > Labels: hintedhandoff > Fix For: 1.2.16, 2.0.6 > > Attachments: .txt > > -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (CASSANDRA-6666) Avoid accumulating tombstones after partial hint replay
[ https://issues.apache.org/jira/browse/CASSANDRA-?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13893506#comment-13893506 ] Anne Sullivan commented on CASSANDRA-: -- Didn't realize that, thanks for the info. I'll have to try to reproduce the repair issue then as it must have been something different causing it. > Avoid accumulating tombstones after partial hint replay > --- > > Key: CASSANDRA- > URL: https://issues.apache.org/jira/browse/CASSANDRA- > Project: Cassandra > Issue Type: Bug > Components: Core >Reporter: Jonathan Ellis >Assignee: Jonathan Ellis >Priority: Minor > Labels: hintedhandoff > Fix For: 1.2.16, 2.0.6 > > Attachments: .txt > > -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (CASSANDRA-6666) Avoid accumulating tombstones after partial hint replay
[ https://issues.apache.org/jira/browse/CASSANDRA-?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13893502#comment-13893502 ] Jonathan Ellis commented on CASSANDRA-: --- Hints (and other local system tables) are neither replicated nor repaired. > Avoid accumulating tombstones after partial hint replay > --- > > Key: CASSANDRA- > URL: https://issues.apache.org/jira/browse/CASSANDRA- > Project: Cassandra > Issue Type: Bug > Components: Core >Reporter: Jonathan Ellis >Assignee: Jonathan Ellis >Priority: Minor > Labels: hintedhandoff > Fix For: 1.2.16, 2.0.6 > > Attachments: .txt > > -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (CASSANDRA-6666) Avoid accumulating tombstones after partial hint replay
[ https://issues.apache.org/jira/browse/CASSANDRA-?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13893503#comment-13893503 ] Anne Sullivan commented on CASSANDRA-: -- Sorry, just re-read the patch, it will fix the nodetool repair problem too. Thanks. > Avoid accumulating tombstones after partial hint replay > --- > > Key: CASSANDRA- > URL: https://issues.apache.org/jira/browse/CASSANDRA- > Project: Cassandra > Issue Type: Bug > Components: Core >Reporter: Jonathan Ellis >Assignee: Jonathan Ellis >Priority: Minor > Labels: hintedhandoff > Fix For: 1.2.16, 2.0.6 > > Attachments: .txt > > -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (CASSANDRA-6666) Avoid accumulating tombstones after partial hint replay
[ https://issues.apache.org/jira/browse/CASSANDRA-?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13893491#comment-13893491 ] Anne Sullivan commented on CASSANDRA-: -- Will this patch basically force a compaction? I think it also becomes a problem for nodetool repair - if you run a nodetool repair while the hints are being replayed, the repair will fail on reading the hint row if > 100K tombstones present > Avoid accumulating tombstones after partial hint replay > --- > > Key: CASSANDRA- > URL: https://issues.apache.org/jira/browse/CASSANDRA- > Project: Cassandra > Issue Type: Bug > Components: Core >Reporter: Jonathan Ellis >Assignee: Jonathan Ellis >Priority: Minor > Labels: hintedhandoff > Fix For: 1.2.16, 2.0.6 > > Attachments: .txt > > -- This message was sent by Atlassian JIRA (v6.1.5#6160)
