[
https://issues.apache.org/jira/browse/CASSANDRA-7066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14647460#comment-14647460
]
Stefania commented on CASSANDRA-7066:
-------------------------------------
[~benedict] thank you for your comments. I've updated the description and
implemented the incremental checksum checks. You may want to review all
together later or take a quick look at what we do when a check fails, which is
not much, just leave the files there and log an error, not sure what a panic
error is exactly?
bq. Perhaps we should offer a tool, like sstablelister, to offline complete any
transaction log files, which can be specified as prep-work for backup restore
sstablelister will return all tx logs as well as temporary files:
{{sstablelister -o -t tmp ks table}}, so they can delete tx logs and tmp files
with one command, which is equivalent to offline complete.
I still need clarifications regarding the last point:
bq. I agree about individual files over-complicating things, however we _could
_ perhaps take the maximum last modified of all files with that descriptor, and
then delete the files in ascending order of last modified. This would just help
the log files be robust to backups being copied in with these files still
present and referencing them as old, and also to some potential effects of
problems like CASSANDRA-9908.
Here we are talking about old files only, right? Because new files haven't even
been created yet when trackNew() is called and it must be like so to ensure we
don't have any window where physical temporary files exist that are not
tracked. So we would take the last modified of all files with that descriptor
when obsoleted is called and log this time along with the descriptor? Then when
it's time to delete old files we refuse to delete any more recent files, do I
understand this correctly?
[~nickmbailey] your points are well noted:
bq. The snapshot command should create a full backup of a keyspace/table on the
node. The directories created from the snapshot should be all that is required
to restore that keyspace/table on that node to the point in time that the
snapshot was taken.
The snapshot command creates hard links to the sstable files, it won't look at
transaction log files. The new lifecycle transaction ensures the sstables live
for a cfs are valid and not temporary.
bq. A snapshot should be restorable either via the sstableloader tool or by
manually copying the files from the snapshot in to place (given the same
schema/topology). If copying the files into place manually, restarting the node
or making an additional call to load the sstables may be required.
If copying the files manually users will have to remove any partial transaction
log files and their temporary files. This can be done via sstablelister with
one command, for example {{for x in $(sstablelister -o -t tmp ks table); do rm
$x; done)}}. sstable loader is implemented via streaming so the uploaded
sstables will automatically get a new generation and won't clash with older
sstables involved in transactions.
bq. When using the sstableloader tool I should be able to restore data taken
from a snapshot regardless of what data exists on the node or is currently
being written.
As per above, this should be the case.
> Simplify (and unify) cleanup of compaction leftovers
> ----------------------------------------------------
>
> Key: CASSANDRA-7066
> URL: https://issues.apache.org/jira/browse/CASSANDRA-7066
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
> Reporter: Benedict
> Assignee: Stefania
> Priority: Minor
> Labels: benedict-to-commit, compaction
> Fix For: 3.0 alpha 1
>
> Attachments: 7066.txt
>
>
> Currently we manage a list of in-progress compactions in a system table,
> which we use to cleanup incomplete compactions when we're done. The problem
> with this is that 1) it's a bit clunky (and leaves us in positions where we
> can unnecessarily cleanup completed files, or conversely not cleanup files
> that have been superceded); and 2) it's only used for a regular compaction -
> no other compaction types are guarded in the same way, so can result in
> duplication if we fail before deleting the replacements.
> I'd like to see each sstable store in its metadata its direct ancestors, and
> on startup we simply delete any sstables that occur in the union of all
> ancestor sets. This way as soon as we finish writing we're capable of
> cleaning up any leftovers, so we never get duplication. It's also much easier
> to reason about.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
