Stefan Miklosovic created CASSANDRA-21681:
---------------------------------------------
Summary: Compression dictionary distribution can silently leave a
node on a stale dictionary
Key: CASSANDRA-21681
URL: https://issues.apache.org/jira/browse/CASSANDRA-21681
Project: Apache Cassandra
Issue Type: Bug
Components: Feature/Compression
Reporter: Stefan Miklosovic
h2. Problem
Dictionaries live in \{{system_distributed.compression_dictionaries}} and
propagate via a
fire-and-forget \{{CompressionDictionaryUpdateMessage}}. Every read/write used
{\{ConsistencyLevel.ONE}} and every failure along the path is silent: the write
may land on one replica ahead of the (unordered) notification; a recipient
reading at ONE can hit a replica without the row yet and get nothing;
\{{retrieveCompressionDictionary}} swallows exceptions as \{{null}} unlogged;
and \{{CompressionDictionaryEventHandler}} feeds that \{{null}} to
\{{cache.add()}} (a no-op).
The node then keeps its old dictionary, silently, until the periodic refresh
(\{{compression_dictionary_refresh_interval}}, default 3600s) - so after a
workload shift it can write an hour of SSTables with a stale dictionary while
its peers have moved on
h2. Changes
# *ONE -> QUORUM* for dictionary store/retrieve/delete/list in
\{{SystemDistributedKeyspace}}: a QUORUM write cannot be missed by a later
QUORUM read, removing the race rather than papering over
it. (View-build queries and the \{{ANY}} write elsewhere are deliberately
untouched.)
# *Failed reads are logged*: the retrieves now WARN on a genuine read failure -
otherwise an unavailable replica is indistinguishable from a table with no
dictionary.
# *No \{{null}} to the cache*: on notify-but-can't-read,
\{{CompressionDictionaryEventHandler}} WARNs
and returns instead of calling \{{cache.add(null)}}, stating that the node
keeps its current
dictionary until the next refresh.
h2. Empty result != failure
{\{UntypedResultSet.one()}} throws on an empty result set, which is the normal
state for any table whose dictionary has not been stored yet. Routing that
through (2)'s catch would fire the WARN on every refresh/eval tick and bury the
real failures it exists to surface (measured: 7 spurious
\{{IllegalStateException}} warnings in 36 min on one table). The retrieves
therefore check \{{isEmpty()}} and return \{{null}} on the normal path, leaving
the catch for genuine failures.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
