[
https://issues.apache.org/jira/browse/CASSANDRA-16047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18114508#comment-18114508
]
Mykyta Bozhenko commented on CASSANDRA-16047:
---------------------------------------------
Patch: https://github.com/apache/cassandra/pull/5149
Tracker.addSSTables publishes new readers into the live set before linking them
into backups/, and holds no reference while linking. A compaction that picks
the reader up in between can obsolete and delete its files while createLinks is
still iterating components; the exists() check before each link does not close
that window, which is why the failure looks environmental and is timing
dependent.
The patch takes a Ref on each reader (tryRef) for the duration of createLinks;
a reader whose lifecycle has already ended is skipped as a whole rather than
losing individual components.
One behaviour change to flag: in SSTableReader.createLinks, a missing component
is now skipped only if the format regenerates it on load (BIG: FILTER, SUMMARY;
BTI: FILTER). Any other missing component throws UncheckedIOException with a
NoSuchFileException cause instead of being silently skipped, so a snapshot or
backup can no longer succeed with an incomplete, unloadable set of files. It is
a plain exception, not an FSError, so the disk failure policy is not involved.
Reproduced against unpatched production: the new test fails with
NoSuchFileException on backups/pa-1-big-Statistics.db, matching the exception
in the description.
Looking for a reviewer.
> Potential race condition in creating hard link when incremental backup is
> turned on
> -----------------------------------------------------------------------------------
>
> Key: CASSANDRA-16047
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16047
> Project: Apache Cassandra
> Issue Type: Bug
> Components: Local/SSTable
> Reporter: Wei Deng
> Assignee: Mykyta Bozhenko
> Priority: Urgent
> Attachments: incremental_backup_hardlink_exception.jpg,
> incremental_backup_hardlink_exception1.jpg
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> It seems that there is a race condition in creating hard link if incremental
> backup is turned on.
> The following screenshot was captured in a production cluster running
> Cassandra 3.0.15 after turning on incremental backup. When this
> {{NoSuchFileException}} happens, due to the {{FSWriteError}} and the default
> disk failure policy, the JVM will be shutdown, so it's a pretty critical bug.
> !incremental_backup_hardlink_exception.jpg!
> Due to the risk of causing production database downtime (if similar issue
> happens on multiple nodes in a short time frame), and same exception causing
> JVM shutdown multiple times already, incremental backup had to be turned off
> for now, but this is not an ideal situation.
> !incremental_backup_hardlink_exception1.jpg!
> The deployment is on a public cloud environment with EBS-like disks that are
> backed by SSD with decent latency, throughput and IOPS, so it is hard to
> think the culprit being in the OS and IO layer. Based on the second
> screenshot above, this is a low flush traffic {{system.size_estimates}}
> table, so compaction of the source SSTable doesn't seem to be at play here.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
