[
https://issues.apache.org/jira/browse/CASSANDRA-21727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jeremiah Jordan updated CASSANDRA-21727:
----------------------------------------
Bug Category: Parent values: Correctness(12982)Level 1 values:
Unrecoverable Corruption / Loss(13161)
Complexity: Normal
Discovered By: Adhoc Test
Fix Version/s: 5.0.x
6.0
7.0
Severity: Normal
Status: Open (was: Triage Needed)
> SAI vector graph is broken after an interrupt during the wait for a permit
> --------------------------------------------------------------------------
>
> Key: CASSANDRA-21727
> URL: https://issues.apache.org/jira/browse/CASSANDRA-21727
> Project: Apache Cassandra
> Issue Type: Bug
> Components: Feature/SAI
> Reporter: Jeremiah Jordan
> Assignee: Jeremiah Jordan
> Priority: Normal
> Fix For: 5.0.x, 6.0, 7.0
>
>
> h2. Problem
> CASSANDRA-21644 added a semaphore to {{OnHeapGraph.add}}. The writer gets the
> semaphore too late. Before the wait, the writer already added the new vector
> to {{postingsMap}}, {{vectorValues}} and {{postingsByOrdinal}}. It also
> already wrote the vector to {{vectorsByKey}}.
> If the writer receives an interrupt during the wait, the wait throws
> {{UncheckedInterruptedException}}. The insert stops before {{addGraphNode}}.
> The result is as follows:
> * The vector is in {{postingsMap}}, {{vectorValues}} and
> {{postingsByOrdinal}}, but it is not in the graph.
> * A retry of the same row finds the vector in {{postingsMap}}. The retry adds
> only the key. It does not add the vector to the graph.
> * A vector search does not find the row.
> * At flush, the assertion in {{OnHeapGraph.writeData}} fails. This assertion
> checks that each ordinal is in the graph. If the JVM runs without assertions,
> no error occurs and the index is incorrect.
> * For an update, {{vectorsByKey}} contains the new vector, but the row keeps
> the old vector. The brute-force search for {{ORDER BY}} uses the new vector
> to score the row.
> h2. Steps to reproduce
> # Create an {{OnHeapGraph}} with a memtable for which
> {{limitsConcurrentWritesTo}} returns false.
> # Set the interrupt flag of the thread.
> # Call {{add}} with a new vector. The call throws
> {{UncheckedInterruptedException}}.
> # Clear the interrupt flag. Call {{add}} again for the same row and vector.
> # Search the graph for the vector.
> Expected result: the search finds the vector. Actual result: {{size()}}
> returns 1, but the search finds no vector.
> h2. Proposed fix
> Change {{OnHeapGraph.add}} as follows:
> # Read the entry for the vector from {{postingsMap}}.
> # If the vector is new, get the permit before any change to the data. This
> includes {{vectorsByKey}}.
> # Release the permit in one {{finally}} block.
> With this order, an interrupt during the wait does not change the graph or
> {{vectorsByKey}}. A retry then adds the vector to the graph.
> Do not try to roll back the changes after the interrupt. Before the rollback,
> other writers can add keys for the same vector and send acknowledgements to
> clients. Also, the graph cannot reuse an ordinal after it assigns that
> ordinal.
> Add tests for these conditions:
> * After an interrupt during the wait for a permit, the graph,
> {{vectorsByKey}} and the number of permits do not change. A retry adds the
> vector to the graph.
> * If two writers add the same new vector, the writer that does not add the
> vector returns its permit.
> * If more writers than permits add the same vectors at the same time, all
> inserts complete.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
