[
https://issues.apache.org/jira/browse/CASSANDRA-6998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13994200#comment-13994200
]
Muhammad Adel commented on CASSANDRA-6998:
------------------------------------------
Sorry if I didn't make myself clear enough. I will try to explain what this
patch does:
1-Normally, the page size passed to SliceQueryFilter represents number of live
columns returned. The SliceQueryFilter returns a mix of live columns and
tombstones which are not eligible for garbage collection yet. If the number of
tombstones seen during constructing the query result exceeds the threshold, an
exception is thrown.
2-When retrieving hinted handoff data, use the SliceQueryFilter in a different
way. The page size will represent the total number of columns returned, live
columns count + tombstones count. Even tombstones that are eligible for garbage
collection are returned. The check for the tombstones threshold is removed. No
memory overload will take place since the size of data processed and returned
is controlled and predictable (exactly the page size).
3-This will send all data in the hinted handoff table to the recovering node
allowing for higher data consistency even if the node was down for a time
larger than the gc_grace period. On the other hand it can cause a higher
network load.
This is a suggested way to deal with the issue of consistency of deleted hinted
handoff data. It is not just an addressing for the exception problem, because I
think the exception problem is a symptom for the more delicate issue of deleted
hinted handoff.
> HintedHandoff - expired hints may block future hints deliveries
> ---------------------------------------------------------------
>
> Key: CASSANDRA-6998
> URL: https://issues.apache.org/jira/browse/CASSANDRA-6998
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: - cluster of two DCs: DC1, DC2
> - keyspace using NetworkTopologyStrategy (replication factors for both DCs)
> - heavy load (write:read, 100:1) with LOCAL_QUORUM using Java driver setup
> with DC awareness, writing to DC1
> Reporter: Scooletz
> Labels: HintedHandoff, TTL
> Fix For: 2.0.3
>
> Attachments: 6998
>
>
> For tests purposes, DC2 was shut down for 1 day. The _hints_ table was filled
> with millions of rows. Now, when _HintedHandOffManager_ tries to
> _doDeliverHintsToEndpoint_ it queries the store with
> QueryFilter.getSliceFilter which counts deleted (TTLed) cells and throws
> org.apache.cassandra.db.filter.TombstoneOverwhelmingException.
> Throwing this exception stops the manager from running compaction as it is
> run only after successful handoff. This leaves the HH practically disabled
> till administrator runs truncateAllHints.
> Wouldn't it be nicer if on
> org.apache.cassandra.db.filter.TombstoneOverwhelmingException run compaction?
> That would remove TTLed hints leaving whole HH mechanism in a healthy state.
> The stacktrace is:
> {quote}
> org.apache.cassandra.db.filter.TombstoneOverwhelmingException
> at
> org.apache.cassandra.db.filter.SliceQueryFilter.collectReducedColumns(SliceQueryFilter.java:201)
> at
> org.apache.cassandra.db.filter.QueryFilter.collateColumns(QueryFilter.java:122)
> at
> org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:80)
> at
> org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:72)
> at
> org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:297)
> at
> org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:53)
> at
> org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1487)
> at
> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1306)
> at
> org.apache.cassandra.db.HintedHandOffManager.doDeliverHintsToEndpoint(HintedHandOffManager.java:351)
> at
> org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:309)
> at
> org.apache.cassandra.db.HintedHandOffManager.access$300(HintedHandOffManager.java:92)
> at
> org.apache.cassandra.db.HintedHandOffManager$4.run(HintedHandOffManager.java:530)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:722)
> {quote}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
