[
https://issues.apache.org/jira/browse/CASSANDRA-21173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18063600#comment-18063600
]
Matt Byrd commented on CASSANDRA-21173:
---------------------------------------
Thanks so much for responding with so much detail, I think I follow, but I
realize there is something I should have clarified better.
We have a mix of table folders with and without table-id. (not even considering
snapshot directories)
The node in question has been upgraded over a long period of time, so it was
upgraded from effectively 1.2 -> 2.0 -> 2.1 -> 3.0 -> 4.1 (over many years)
and then just more recently to 5.0.
Upon upgrade to 5.0 running nodetool clearsnapshot does not clear any snapshot
already created or created thereafter, as is the case for lots of other
clusters/nodes and unless I'm missing something, anyone else who's upgraded
clusters up to 5.0 from that far back in time. We effectively have several
hundred clusters of this form.
It doesn't cause eventual 100% disk usage, as we have tooling which removes the
snapshots out of band of Cassandra in a separate sidecar process.
I did consider whether we should try and find some sort of way of renaming the
folders of said clusters to include the table id, but that felt inherently
risky and perhaps out of scope.
Moving data around on disk on startup and renaming things feels a bit fraught
with subtle data correctness concerns and also the sort of out of band tooling
(for example various backup solutions, which incidentally is how this JIRA
ticket came about).
Though it would obviously be good to have everything have the same
structure/format.
It does seem like these two folder formats are already handled in
Directories.java(see constructor), so there is effectively some kind of support
for clusters which have existed since 2.1.
Maybe in a future version you could re-name these directories automatically,
but it would probably be a much more complicated change than this one and also
wouldn't fix the state of things in 5.0.
I guess at least in 5.1 we can still nodetool clearsnapshot for those we create
whilst the jvm is still running.
Does that better help frame the motivation here?
> Snapshots from tables without table-id embedded in their folder name are not
> loaded by SnapshotLoader
> -----------------------------------------------------------------------------------------------------
>
> Key: CASSANDRA-21173
> URL: https://issues.apache.org/jira/browse/CASSANDRA-21173
> Project: Apache Cassandra
> Issue Type: Bug
> Components: Local/Snapshots, Local/Startup and Shutdown
> Reporter: Matt Byrd
> Assignee: Matt Byrd
> Priority: Normal
> Fix For: 5.0.x, 5.1, 6.x
>
> Attachments: ci_summary_trunk_mbyrd_CASSANDRA-21173.html
>
>
> Tables created prior to 2.1 do not have a table-id embedded in their table
> folder name.
> This is handled correctly in Directories.java (see constructor) unfortunately
> in SnapshotLoader, we use a regex which attempts to extract the table-id and
> hence skips over any tables created prior to 2.1.
> The end result is that these tables are not visible in list snapshot and more
> importantly cannot be cleared via nodetool clearsnapshot. This was noticed
> upon major upgrade to 5.0.
> I've observed this on 5.0, from reading the code it appears likely improved
> in 5.1, in that it now requires a restart in addition to trigger.
> Some related tickets:
> Introduction of table-id and backwards compatible handling of old folders
> originally here:
> https://issues.apache.org/jira/browse/CASSANDRA-5202
> Machinery to list snapshots which doesn’t handle old format was added here:
> https://issues.apache.org/jira/browse/CASSANDRA-16843
> https://github.com/apache/cassandra/commit/31aa17a2a3b18bdda723123cad811f075287807d
> There was some discussion at the time of not handling pre 2.1 tables here:
> https://issues.apache.org/jira/browse/CASSANDRA-16843?focusedCommentId=17440088&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17440088
> Then nodetool clearsnapshot stopped working here with:
> https://issues.apache.org/jira/browse/CASSANDRA-17757
> Things improve a bit in 5.1 with
> https://issues.apache.org/jira/browse/CASSANDRA-18111
> Now we no longer try and load the snapshots via SnapshotLoader in entirety
> before deciding if we can clear them, but instead make use of
> SnapshotManager. Whilst snapshots taken while the jvm is running are now
> visible and clearable, from reading upon restart we lose that information and
> cannot view/clear snapshots created before the restart.
> One solution to handle these pre 2.1 tables, is to include the table-id in
> the manifest.json, then we'll be able to read this information if not
> available from folder name upon restart.
> Another possibility which doesn't fix as many problems, is just to expose via
> jmx/nodetool
> something to allow operators to bypass the snapshot loading mechanism and
> directly clear the old pre-2.1 snapshots.
> A more involved and risky change would be to somehow think about how we
> migrate all this existing data in different folder structures to new
> consistent folder structure, but this seems quite involved and would likely
> deserve it's own JIRA at least.
> I have a patch locally against trunk for the first approach, just storing the
> tableId in the manifest.json, which does this and will run it against CI.
> I'll have a further think about if there are any other approaches, if anyone
> has any ideas let me know.
> Another thing to consider is where we should apply this change.
> Probably at a minimum 5.0, since that's where one can no longer nodetool
> clearsnapshot on certain tables and the effect is a bit worse there than in
> 5.1.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
