[
https://issues.apache.org/jira/browse/CASSANDRA-21681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18115093#comment-18115093
]
Stefan Miklosovic commented on CASSANDRA-21681:
-----------------------------------------------
Before this patch, I have seen that the line from time of new dictionary
promoted forked into two, where the first one went up aggresively. That was the
result of one node not getting the latest dictionary because of the issues
described in this ticket. After the patch, all three nodes got the newest dict
immediately so two lines collapsed into one (from cca 75 minutes).
> Compression dictionary distribution can silently leave a node on a stale
> dictionary
> -----------------------------------------------------------------------------------
>
> Key: CASSANDRA-21681
> URL: https://issues.apache.org/jira/browse/CASSANDRA-21681
> Project: Apache Cassandra
> Issue Type: Bug
> Components: Feature/Compression
> Reporter: Stefan Miklosovic
> Assignee: Stefan Miklosovic
> Priority: Normal
> Fix For: 6.x, 7.x
>
> Attachments: screenshot-1.png
>
>
> h2. Problem
>
> Dictionaries live in \{{system_distributed.compression_dictionaries}} and
> propagate via a
> fire-and-forget \{{CompressionDictionaryUpdateMessage}}. Every read/write used
> {\{ConsistencyLevel.ONE}} and every failure along the path is silent: the
> write may land on one replica ahead of the (unordered) notification; a
> recipient reading at ONE can hit a replica without the row yet and get
> nothing; \{{retrieveCompressionDictionary}} swallows exceptions as \{{null}}
> unlogged; and \{{CompressionDictionaryEventHandler}} feeds that \{{null}} to
> \{{cache.add()}} (a no-op).
>
> The node then keeps its old dictionary, silently, until the periodic refresh
> (\{{compression_dictionary_refresh_interval}}, default 3600s) - so after a
> workload shift it can write an hour of SSTables with a stale dictionary while
> its peers have moved on
>
> h2. Changes
>
> # *ONE -> QUORUM* for dictionary store/retrieve/delete/list in
> \{{SystemDistributedKeyspace}}: a QUORUM write cannot be missed by a later
> QUORUM read, removing the race rather than papering over
> it. (View-build queries and the \{{ANY}} write elsewhere are deliberately
> untouched.)
> # *Failed reads are logged*: the retrieves now WARN on a genuine read failure
> - otherwise an unavailable replica is indistinguishable from a table with no
> dictionary.
> # *No \{{null}} to the cache*: on notify-but-can't-read,
> \{{CompressionDictionaryEventHandler}} WARNs
> and returns instead of calling \{{cache.add(null)}}, stating that the node
> keeps its current
> dictionary until the next refresh.
>
> h2. Empty result != failure
>
> {\{UntypedResultSet.one()}} throws on an empty result set, which is the
> normal state for any table whose dictionary has not been stored yet. Routing
> that through (2)'s catch would fire the WARN on every refresh/eval tick and
> bury the real failures it exists to surface (measured: 7 spurious
> \{{IllegalStateException}} warnings in 36 min on one table). The retrieves
> therefore check \{{isEmpty()}} and return \{{null}} on the normal path,
> leaving the catch for genuine failures.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
