[jira] [Commented] (CASSANDRA-16843) List snapshots of dropped tables
[
https://issues.apache.org/jira/browse/CASSANDRA-16843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17530261#comment-17530261
]
Paulo Motta commented on CASSANDRA-16843:
-
Since CI looks good, I merged this as
[31aa17a2a3b18bdda723123cad811f075287807d|https://github.com/apache/cassandra/commit/31aa17a2a3b18bdda723123cad811f075287807d]
to trunk.
Thanks all for the prompt reviews!
> List snapshots of dropped tables
>
>
> Key: CASSANDRA-16843
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16843
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Snapshots
>Reporter: James Brown
>Assignee: Paulo Motta
>Priority: Normal
> Fix For: 4.1
>
> Time Spent: 5h
> Remaining Estimate: 0h
>
> Auto snapshots from dropped tables don't seem to show up in {{nodetool
> listsnapshots}} (even though they do get cleared by {{nodetool
> clearsnapshot}}). This makes them kind of annoying to clean up, since you
> need to muck about in the data directory to find them.
> Erick on the mailing list said that this seems to be an oversight and that
> clearsnapshot was fixed by
> [CASSANDRA-6418|https://issues.apache.org/jira/browse/CASSANDRA-6418].
> I reproduced this both on 3.11.11 and 4.0.0.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-16843) List snapshots of dropped tables
[
https://issues.apache.org/jira/browse/CASSANDRA-16843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17530256#comment-17530256
]
Paulo Motta commented on CASSANDRA-16843:
-
Created CASSANDRA-17588 to address *Total TrueDiskSpaceUsed* computation.
bq. Let's ship this then on a reasonably clean build.
CI looks
[good|https://ci-cassandra.apache.org/view/patches/job/Cassandra-devbranch/1646/].
Can we merge?
> List snapshots of dropped tables
>
>
> Key: CASSANDRA-16843
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16843
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Snapshots
>Reporter: James Brown
>Assignee: Paulo Motta
>Priority: Normal
> Fix For: 4.1
>
> Time Spent: 5h
> Remaining Estimate: 0h
>
> Auto snapshots from dropped tables don't seem to show up in {{nodetool
> listsnapshots}} (even though they do get cleared by {{nodetool
> clearsnapshot}}). This makes them kind of annoying to clean up, since you
> need to muck about in the data directory to find them.
> Erick on the mailing list said that this seems to be an oversight and that
> clearsnapshot was fixed by
> [CASSANDRA-6418|https://issues.apache.org/jira/browse/CASSANDRA-6418].
> I reproduced this both on 3.11.11 and 4.0.0.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-16843) List snapshots of dropped tables
[
https://issues.apache.org/jira/browse/CASSANDRA-16843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17530254#comment-17530254
]
Stefan Miklosovic commented on CASSANDRA-16843:
---
Yes, that's how I understood it, probably not communicated properly, "take care
of the bugs" not like "remaining bugs in this ticket" but "other bugs related
to this subsystem we just stumbled upon while discussing this."
Let's ship this then on a reasonably clean build.
> List snapshots of dropped tables
>
>
> Key: CASSANDRA-16843
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16843
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Snapshots
>Reporter: James Brown
>Assignee: Paulo Motta
>Priority: Normal
> Fix For: 4.1
>
> Time Spent: 5h
> Remaining Estimate: 0h
>
> Auto snapshots from dropped tables don't seem to show up in {{nodetool
> listsnapshots}} (even though they do get cleared by {{nodetool
> clearsnapshot}}). This makes them kind of annoying to clean up, since you
> need to muck about in the data directory to find them.
> Erick on the mailing list said that this seems to be an oversight and that
> clearsnapshot was fixed by
> [CASSANDRA-6418|https://issues.apache.org/jira/browse/CASSANDRA-6418].
> I reproduced this both on 3.11.11 and 4.0.0.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-16843) List snapshots of dropped tables
[
https://issues.apache.org/jira/browse/CASSANDRA-16843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17530252#comment-17530252
]
Paulo Motta commented on CASSANDRA-16843:
-
{quote}So what is your strategy, you want to merge this and in the freeze you
take care of the bugs? In that case I think that might fly.
{quote}
The minor bug on the *Total TrueDiskSpaceUsed* computation was not introduced
by this patch, it is like that on all versions below trunk. We can fix it
whenever we want, but it probably shouldn't block this.
> List snapshots of dropped tables
>
>
> Key: CASSANDRA-16843
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16843
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Snapshots
>Reporter: James Brown
>Assignee: Paulo Motta
>Priority: Normal
> Fix For: 4.1
>
> Time Spent: 5h
> Remaining Estimate: 0h
>
> Auto snapshots from dropped tables don't seem to show up in {{nodetool
> listsnapshots}} (even though they do get cleared by {{nodetool
> clearsnapshot}}). This makes them kind of annoying to clean up, since you
> need to muck about in the data directory to find them.
> Erick on the mailing list said that this seems to be an oversight and that
> clearsnapshot was fixed by
> [CASSANDRA-6418|https://issues.apache.org/jira/browse/CASSANDRA-6418].
> I reproduced this both on 3.11.11 and 4.0.0.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-16843) List snapshots of dropped tables
[
https://issues.apache.org/jira/browse/CASSANDRA-16843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17530251#comment-17530251
]
Paulo Motta commented on CASSANDRA-16843:
-
In order to make things easier to understand, since this is indeed a bit
confusing:
Each individual snapshot has two attributes:
* {*}Size on disk{*}: the amount of disk space used by a specific snapshot.
* *TrueDiskSpaceUsed:* the amount of *extra* disk space used by a snapshot
(snapshot_size - live_sstables)
Before this patch, the *TrueDiskSpace* metric was incorrectly calculated,
because:
a) It did not include the size of manifest and schema files.
b) It did not include the size of secondary indexes.
The behavior above was fixed by this patch.
This means that when you create a snapshot on ccm:
* *Size on disk* should be equal to the size of the snapshots
* *TrueDiskSpaceUsed* should only include the size of manifest+schema file,
since there are live sstables, so the snapshot is not taking more space to
store these sstables which are hard-linked.
When you drop the table, the snapshot no longer has live data so *Size on disk*
= *TrueDiskSpaceUsed* (this was not the case before this patch, because
*TrueDiskSpaceUsed* did not previously include the size of manifest and schema
files, so *Size on disk* > *TrueDiskSpaceUsed.*
In addition to that, the *Total TrueDiskSpaceUsed* is computed by a different
code path, and still has the following limitation:
a) It does not include the size of manifest and schema files.
Fixing the *Total TrueDiskSpaceUsed* metric is out of the scope of this patch
because required additional changes. It will be fixed on a follow-up ticket.
> List snapshots of dropped tables
>
>
> Key: CASSANDRA-16843
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16843
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Snapshots
>Reporter: James Brown
>Assignee: Paulo Motta
>Priority: Normal
> Fix For: 4.1
>
> Time Spent: 5h
> Remaining Estimate: 0h
>
> Auto snapshots from dropped tables don't seem to show up in {{nodetool
> listsnapshots}} (even though they do get cleared by {{nodetool
> clearsnapshot}}). This makes them kind of annoying to clean up, since you
> need to muck about in the data directory to find them.
> Erick on the mailing list said that this seems to be an oversight and that
> clearsnapshot was fixed by
> [CASSANDRA-6418|https://issues.apache.org/jira/browse/CASSANDRA-6418].
> I reproduced this both on 3.11.11 and 4.0.0.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-16843) List snapshots of dropped tables
[
https://issues.apache.org/jira/browse/CASSANDRA-16843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17530249#comment-17530249
]
Stefan Miklosovic commented on CASSANDRA-16843:
---
_When the table is dropped the true size is equal to the size on disk because
no sstables are live._
Right ... Ok I think this explains it.
_True disk space is zero because all sstables are live, so snapshots are not
taking any extra space._
Ok, I compacted the table and took a snapshot again and it started to be
non-zero.
Ok so it seems we are good after all, minus that bug you mentioned.
So what is your strategy, you want to merge this and in the freeze you take
care of the bugs? In that case I think that might fly.
> List snapshots of dropped tables
>
>
> Key: CASSANDRA-16843
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16843
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Snapshots
>Reporter: James Brown
>Assignee: Paulo Motta
>Priority: Normal
> Fix For: 4.1
>
> Time Spent: 5h
> Remaining Estimate: 0h
>
> Auto snapshots from dropped tables don't seem to show up in {{nodetool
> listsnapshots}} (even though they do get cleared by {{nodetool
> clearsnapshot}}). This makes them kind of annoying to clean up, since you
> need to muck about in the data directory to find them.
> Erick on the mailing list said that this seems to be an oversight and that
> clearsnapshot was fixed by
> [CASSANDRA-6418|https://issues.apache.org/jira/browse/CASSANDRA-6418].
> I reproduced this both on 3.11.11 and 4.0.0.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-16843) List snapshots of dropped tables
[
https://issues.apache.org/jira/browse/CASSANDRA-16843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17530243#comment-17530243
]
Paulo Motta commented on CASSANDRA-16843:
-
{quote}listsnapshots gives me "Total TrueDiskSpaceUsed: 0 bytes" irrelevant how
many snapshots I have
{quote}
True disk space is zero because all sstables are live, so snapshots are not
taking any extra space.
This is working as expected. It's the current behavior on trunk and it is not
changed by this patch.
However there is another bug there, because "Total TrueDiskSpaceUsed" should
include sizes of schema and manifest files.
I do not touch the code path that is responsible for computing "*Total*
TrueDiskSpaceUsed" on this patch. The true disk space should also include
manifest and schema files, but fixing that requires a reworking of how the
*total* true size is computed, and is out of the scope of this patch.
I fixed this on the extended version of this patch that will be posted later.
> List snapshots of dropped tables
>
>
> Key: CASSANDRA-16843
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16843
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Snapshots
>Reporter: James Brown
>Assignee: Paulo Motta
>Priority: Normal
> Fix For: 4.1
>
> Time Spent: 5h
> Remaining Estimate: 0h
>
> Auto snapshots from dropped tables don't seem to show up in {{nodetool
> listsnapshots}} (even though they do get cleared by {{nodetool
> clearsnapshot}}). This makes them kind of annoying to clean up, since you
> need to muck about in the data directory to find them.
> Erick on the mailing list said that this seems to be an oversight and that
> clearsnapshot was fixed by
> [CASSANDRA-6418|https://issues.apache.org/jira/browse/CASSANDRA-6418].
> I reproduced this both on 3.11.11 and 4.0.0.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-16843) List snapshots of dropped tables
[
https://issues.apache.org/jira/browse/CASSANDRA-16843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17530240#comment-17530240
]
Paulo Motta commented on CASSANDRA-16843:
-
{quote}Not sure if it is a bug but it is weird - when I list snapshots, it
gives me some values in True Size / Size on Disk but when I drop that table and
I list snapshots again, these sizes are different.
{quote}
This is not a bug. The true size should be very small when the table is live,
because live sstables are skipped and the true size only accounts for the
manifest and schema file sizes.
When the table is dropped the true size is equal to the size on disk because no
sstables are live.
> List snapshots of dropped tables
>
>
> Key: CASSANDRA-16843
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16843
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Snapshots
>Reporter: James Brown
>Assignee: Paulo Motta
>Priority: Normal
> Fix For: 4.1
>
> Time Spent: 4h 40m
> Remaining Estimate: 0h
>
> Auto snapshots from dropped tables don't seem to show up in {{nodetool
> listsnapshots}} (even though they do get cleared by {{nodetool
> clearsnapshot}}). This makes them kind of annoying to clean up, since you
> need to muck about in the data directory to find them.
> Erick on the mailing list said that this seems to be an oversight and that
> clearsnapshot was fixed by
> [CASSANDRA-6418|https://issues.apache.org/jira/browse/CASSANDRA-6418].
> I reproduced this both on 3.11.11 and 4.0.0.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-16843) List snapshots of dropped tables
[
https://issues.apache.org/jira/browse/CASSANDRA-16843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17530239#comment-17530239
]
Paulo Motta commented on CASSANDRA-16843:
-
Can you paste the full output for the "listsnapshots" value that you're
observing? this would help debug.
I remember testing before submitting and it was working as expected. Let me try
again.
> List snapshots of dropped tables
>
>
> Key: CASSANDRA-16843
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16843
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Snapshots
>Reporter: James Brown
>Assignee: Paulo Motta
>Priority: Normal
> Fix For: 4.1
>
> Time Spent: 4h 40m
> Remaining Estimate: 0h
>
> Auto snapshots from dropped tables don't seem to show up in {{nodetool
> listsnapshots}} (even though they do get cleared by {{nodetool
> clearsnapshot}}). This makes them kind of annoying to clean up, since you
> need to muck about in the data directory to find them.
> Erick on the mailing list said that this seems to be an oversight and that
> clearsnapshot was fixed by
> [CASSANDRA-6418|https://issues.apache.org/jira/browse/CASSANDRA-6418].
> I reproduced this both on 3.11.11 and 4.0.0.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-16843) List snapshots of dropped tables
[
https://issues.apache.org/jira/browse/CASSANDRA-16843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17530236#comment-17530236
]
Stefan Miklosovic commented on CASSANDRA-16843:
---
I think I hit few bugs:
1) create a table, insert some data
2) create a snapshot with TTL like 1 minute
3) drop that table
4) list snapshots, you will see dropped snaphot there as well as snaphot with
TTL
However, it will not remove the snapshot with TTL after 1 minute if that table
is dropped. In other words, it will not expire snapshots on tables which do not
exist anymore.
2) listsnapshots gives me "Total TrueDiskSpaceUsed: 0 bytes" irrelevant how
many snapshots I have
3) Not sure if it is a bug but it is weird - when I list snapshots, it gives me
some values in True Size / Size on Disk but when I drope that able and I list
snapshots again, these sizes are different.
I am -1 on this as it is in the current state, unfortunately. We can still
merge this if these problems are addressed before the freeze. I will be online
occasionaly during weekend too.
> List snapshots of dropped tables
>
>
> Key: CASSANDRA-16843
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16843
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Snapshots
>Reporter: James Brown
>Assignee: Paulo Motta
>Priority: Normal
> Fix For: 4.1
>
> Time Spent: 4h 40m
> Remaining Estimate: 0h
>
> Auto snapshots from dropped tables don't seem to show up in {{nodetool
> listsnapshots}} (even though they do get cleared by {{nodetool
> clearsnapshot}}). This makes them kind of annoying to clean up, since you
> need to muck about in the data directory to find them.
> Erick on the mailing list said that this seems to be an oversight and that
> clearsnapshot was fixed by
> [CASSANDRA-6418|https://issues.apache.org/jira/browse/CASSANDRA-6418].
> I reproduced this both on 3.11.11 and 4.0.0.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-16843) List snapshots of dropped tables
[
https://issues.apache.org/jira/browse/CASSANDRA-16843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17530045#comment-17530045
]
Brandon Williams commented on CASSANDRA-16843:
--
Was just looking and it looks good to me after the latest commit. I'll check CI
when it's complete but I'm +1 if that doesn't have any new failures.
> List snapshots of dropped tables
>
>
> Key: CASSANDRA-16843
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16843
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Snapshots
>Reporter: James Brown
>Assignee: Paulo Motta
>Priority: Normal
> Fix For: 4.1
>
> Time Spent: 3h
> Remaining Estimate: 0h
>
> Auto snapshots from dropped tables don't seem to show up in {{nodetool
> listsnapshots}} (even though they do get cleared by {{nodetool
> clearsnapshot}}). This makes them kind of annoying to clean up, since you
> need to muck about in the data directory to find them.
> Erick on the mailing list said that this seems to be an oversight and that
> clearsnapshot was fixed by
> [CASSANDRA-6418|https://issues.apache.org/jira/browse/CASSANDRA-6418].
> I reproduced this both on 3.11.11 and 4.0.0.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-16843) List snapshots of dropped tables
[
https://issues.apache.org/jira/browse/CASSANDRA-16843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17530042#comment-17530042
]
Paulo Motta commented on CASSANDRA-16843:
-
Squashed and rebased patch after review comments, prepared for commit:
|[squashed
commit|https://github.com/apache/cassandra/commit/01a7d1e1fcb63f1413e5e9dd96f3ca2bbb829f3d]|[tests|https://ci-cassandra.apache.org/view/patches/job/Cassandra-devbranch/1646/]|
Feel free to take a sanity check look if you have cycles [~brandon.williams].
> List snapshots of dropped tables
>
>
> Key: CASSANDRA-16843
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16843
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Snapshots
>Reporter: James Brown
>Assignee: Paulo Motta
>Priority: Normal
> Fix For: 4.1
>
> Time Spent: 3h
> Remaining Estimate: 0h
>
> Auto snapshots from dropped tables don't seem to show up in {{nodetool
> listsnapshots}} (even though they do get cleared by {{nodetool
> clearsnapshot}}). This makes them kind of annoying to clean up, since you
> need to muck about in the data directory to find them.
> Erick on the mailing list said that this seems to be an oversight and that
> clearsnapshot was fixed by
> [CASSANDRA-6418|https://issues.apache.org/jira/browse/CASSANDRA-6418].
> I reproduced this both on 3.11.11 and 4.0.0.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-16843) List snapshots of dropped tables
[
https://issues.apache.org/jira/browse/CASSANDRA-16843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17530032#comment-17530032
]
Paulo Motta commented on CASSANDRA-16843:
-
bq. I do not mind if we merge this now but the proximity of the freeze makes me
uncomfortable a bit.
This patch in its current state is fairly low risk, since it only updates the
implementation of "nodetool listsnapshots" without touching other components.
Furthermore this area is pretty well covered by unit tests and dtests.
I realized the original larger patch had a higher risk as it was also touching
other areas like snapshot cleanup and ephemeral snapshots, so I decided to
split the original patch to only contain the minimum viable set. I will create
tickets for follow-up work for 4.2.
bq. I reviewed it and put my comments there.
Thanks for the prompt review. Addressed comments [on this
commit|https://github.com/apache/cassandra/pull/1595/commits/d1834c88c362f3804f9817e6d2cd80c6fd0e8d4b].
[CI
failures|https://ci-cassandra.apache.org/view/patches/job/Cassandra-devbranch/1645/testReport/]
look unrelated.
I will squash + rebase and re-submit CI shortly.
> List snapshots of dropped tables
>
>
> Key: CASSANDRA-16843
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16843
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Snapshots
>Reporter: James Brown
>Assignee: Paulo Motta
>Priority: Normal
> Fix For: 4.1
>
> Time Spent: 3h
> Remaining Estimate: 0h
>
> Auto snapshots from dropped tables don't seem to show up in {{nodetool
> listsnapshots}} (even though they do get cleared by {{nodetool
> clearsnapshot}}). This makes them kind of annoying to clean up, since you
> need to muck about in the data directory to find them.
> Erick on the mailing list said that this seems to be an oversight and that
> clearsnapshot was fixed by
> [CASSANDRA-6418|https://issues.apache.org/jira/browse/CASSANDRA-6418].
> I reproduced this both on 3.11.11 and 4.0.0.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-16843) List snapshots of dropped tables
[
https://issues.apache.org/jira/browse/CASSANDRA-16843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17529955#comment-17529955
]
Stefan Miklosovic commented on CASSANDRA-16843:
---
I reviewed it and put my comments there. I ll return to this in the late
evening today CEST time to see if there was some progress. I do not mind if we
merge this now but the proximity of the freeze makes me uncomfortable a bit.
However, if we find there are some minor issues with this during freeze, we can
still fix it there as well.
> List snapshots of dropped tables
>
>
> Key: CASSANDRA-16843
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16843
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Snapshots
>Reporter: James Brown
>Assignee: Paulo Motta
>Priority: Normal
> Fix For: 4.1
>
> Time Spent: 2h 50m
> Remaining Estimate: 0h
>
> Auto snapshots from dropped tables don't seem to show up in {{nodetool
> listsnapshots}} (even though they do get cleared by {{nodetool
> clearsnapshot}}). This makes them kind of annoying to clean up, since you
> need to muck about in the data directory to find them.
> Erick on the mailing list said that this seems to be an oversight and that
> clearsnapshot was fixed by
> [CASSANDRA-6418|https://issues.apache.org/jira/browse/CASSANDRA-6418].
> I reproduced this both on 3.11.11 and 4.0.0.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-16843) List snapshots of dropped tables
[
https://issues.apache.org/jira/browse/CASSANDRA-16843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17529681#comment-17529681
]
Paulo Motta commented on CASSANDRA-16843:
-
[~brandon.williams] [~smiklosovic] This is finally ready for a final round of
review and I apologize for the delay. Please check the 2 previous comments for
context.
Even though I'd like to get this in, I will understand if you're not able to
get to this before the 4.1 freeze.
|[trunk|https://github.com/apache/cassandra/pull/1595]|[tests|https://ci-cassandra.apache.org/view/patches/job/Cassandra-devbranch/1645/]|
> List snapshots of dropped tables
>
>
> Key: CASSANDRA-16843
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16843
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Snapshots
>Reporter: James Brown
>Assignee: Paulo Motta
>Priority: Normal
> Fix For: 4.1
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Auto snapshots from dropped tables don't seem to show up in {{nodetool
> listsnapshots}} (even though they do get cleared by {{nodetool
> clearsnapshot}}). This makes them kind of annoying to clean up, since you
> need to muck about in the data directory to find them.
> Erick on the mailing list said that this seems to be an oversight and that
> clearsnapshot was fixed by
> [CASSANDRA-6418|https://issues.apache.org/jira/browse/CASSANDRA-6418].
> I reproduced this both on 3.11.11 and 4.0.0.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-16843) List snapshots of dropped tables
[
https://issues.apache.org/jira/browse/CASSANDRA-16843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17529680#comment-17529680
]
Paulo Motta commented on CASSANDRA-16843:
-
The reason why snapshots of "dropped tables" are omitted from the "nodetool
listsnapshots" output above is because the prior implementation relied on the
mechanics of
[ColumnFamilyStore|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/db/ColumnFamilyStore.java#L2240=]
and
[Directories|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/db/Directories.java#L964=]
to list snapshots.
Since dropped tables no longer have an associated {{ColumnFamilyStore}} object,
it's not possible to list snapshots of dropped tables in the current
implementation.
[This patch|https://github.com/apache/cassandra/pull/1595] re-architects the
snapshot listing logic to be fully decoupled from
{{{}ColumnFamilyStore{}}}/{{{}Directories{}}} and rely solely on the snapshot
directory structure, which currently has this format:
* {{$data_dir/$ks_name/$table_name-$table_uuid/snapshots/$tag}}
The new snapshot discovery logic is mostly contained in the
[SnapshotLoader|https://github.com/apache/cassandra/blob/993190ada5b65b79c5b7ca707d436a6ceff7abcf/src/java/org/apache/cassandra/service/snapshot/SnapshotLoader.java]
class, which traverses the data directory [looking for snapshot directories
matching the pattern
above|https://github.com/apache/cassandra/blob/993190ada5b65b79c5b7ca707d436a6ceff7abcf/src/java/org/apache/cassandra/service/snapshot/SnapshotLoader.java#L102=].
I updated
[StorageService.getSnapshotDetails|https://github.com/apache/cassandra/pull/1595/files#diff-9bf2c26bc294ef9085e16bf287490223665eaa2eb8ec24bcf5bd8653c713644bR4131]
which is used by {{nodetool listsnapshots}} to use new {{SnapshotLoader}}
class to load snapshots.
The snapshot true size computation was previously dependent on logic from
[Directories|https://github.com/apache/cassandra/blob/bb3749f2bb8282f67375c67712d8e3ca1f085879/src/java/org/apache/cassandra/db/Directories.java#L1153],
so in order to fully decouple snapshot listing from {{{}Directories{}}}, I
[simplified the computation of snapshot true
size|https://github.com/apache/cassandra/pull/1595/files#diff-7d6d1bafcad95c5715c91c9065a4a8c58c3d5c98d0699d9c913717f5c0086bb7L114]
to only include files which do not have a corresponding "live" file on
{{{}$data_dir/$ks_name/$table_name-$table_uuid{}}}.
This simplification to the snapshot true size computation fixed two additional
issues with the previous implementation (illustrated with examples in the
previous comment):
1) Snapshot true size did not include "schema.cql" and "manifest.json" sizes
2) Snapshot true size did not include secondary indexes (CASSANDRA-17357)
I performed other simplifications and refactorings along the way, but given the
proximity to the 4.1 freeze, I prepared a leaner version of the original patch
to facilitate review.
After this is merged I will prepare another set of follow-up patches (for next
release) with refactorings and simplifications in the snapshot management
module that will be enabled by this change.
Testing:
- [dtest to check if snapshot of dropped tables are included on
listsnapshots|https://github.com/apache/cassandra/pull/1595/files#diff-35dcc7dbb180da51d4f548e79f31ba45fb7beb7dbeec27663053817619efff1bR195]
-
[SnapshotLoaderTest|https://github.com/apache/cassandra/blob/993190ada5b65b79c5b7ca707d436a6ceff7abcf/test/unit/org/apache/cassandra/service/snapshot/SnapshotLoaderTest.java]
- [Test to check that manifest and schema file sizes are included in true size
computation|https://github.com/apache/cassandra/pull/1595/files#diff-ef5be0b69d0440b76021282c4b24bad69770ef9419be260df2169f49921db377R291]
- [Update DirectoriesTest.testSecondaryIndexDirectories to include 2i on true
size
computation|https://github.com/apache/cassandra/pull/1595/files#diff-1948a455b59a97d8d1ab3d2cb5388190c1cbb8e8081e3ac97bfc0c51a7ef64e3R421]
- [testGetLiveFileFromSnapshotFile (used by new true size
computation)|https://github.com/apache/cassandra/pull/1595/files#diff-d349fb289ec10bece5531f1630cd2bcc55665b5cf3cd59cfcfb4dc93f288a571R233]
> List snapshots of dropped tables
>
>
> Key: CASSANDRA-16843
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16843
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Snapshots
>Reporter: James Brown
>Assignee: Paulo Motta
>Priority: Normal
> Fix For: 4.1
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Auto snapshots from dropped tables don't seem to show up in {{nodetool
> listsnapshots}} (even though they do get cleared by {{nodetool
> clearsnapshot}}). This makes them kind of annoying to clean up, since
[jira] [Commented] (CASSANDRA-16843) List snapshots of dropped tables
[
https://issues.apache.org/jira/browse/CASSANDRA-16843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17529609#comment-17529609
]
Paulo Motta commented on CASSANDRA-16843:
-
To provide some contextualization and recap before going into the
implementation details, please find a little summary of what end-user changes
will be made visible by this patch.
This is the current output of {{nodetool listsnapshots}} before this patch:
{noformat}
Snapshot Details:
Snapshot name Keyspace name Column family name True size Size on disk Creation
timeExpiration time
test ksindexed_table 9.83 KiB 21.22 KiB
2022-04-26T19:13:20.102Z
test ksmy_table 9.83 KiB 10.76 KiB
2022-04-26T19:13:20.102Z
Total TrueDiskSpaceUsed: 19.65 KiB
{noformat}
*The main problem being solved by this patch is that snapshots from dropped
tables are omitted from this output.*
In addition to this, there are 2 additional issues with the previous output:
1) Snapshot "true size" column does not include {{manifest.json}} and
{{schema.cql}} file sizes. This can be observed by the mismatching numbers in
the "true size" and "size on disk" columns of {{my_table}}.
2) Snapshot "true size" of table with secondary index ({{indexed_table}}) does
not include secondary index files (CASSANDRA-17357). This can be observed by
the "true size" being 9.83 KiB while the "size on disk" is 21.22 KiB.
After this patch, the following output is displayed for the same data:
{noformat}
Snapshot Details:
Snapshot name Keyspace name Column family name True size Size
on disk Creation timeExpiration time
test ksindexed_table 21.22 KiB 21.22
KiB2022-04-26T19:13:20.102Z
test ksmy_table 10.76 KiB 10.76
KiB2022-04-26T19:13:20.102Z
dropped-1650997415751-my_table ksmy_table 989 bytes 989
bytes2022-04-26T18:23:35.751Z
Total TrueDiskSpaceUsed: 32.95 KiB
{noformat}
The new output after this patch shows the snapshot "true size" equal to the
"size on disk" when there are no live sstables.
(will follow-up with implementation details on next comment)
> List snapshots of dropped tables
>
>
> Key: CASSANDRA-16843
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16843
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Snapshots
>Reporter: James Brown
>Assignee: Paulo Motta
>Priority: Normal
> Fix For: 4.1
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Auto snapshots from dropped tables don't seem to show up in {{nodetool
> listsnapshots}} (even though they do get cleared by {{nodetool
> clearsnapshot}}). This makes them kind of annoying to clean up, since you
> need to muck about in the data directory to find them.
> Erick on the mailing list said that this seems to be an oversight and that
> clearsnapshot was fixed by
> [CASSANDRA-6418|https://issues.apache.org/jira/browse/CASSANDRA-6418].
> I reproduced this both on 3.11.11 and 4.0.0.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-16843) List snapshots of dropped tables
[
https://issues.apache.org/jira/browse/CASSANDRA-16843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17529043#comment-17529043
]
Paulo Motta commented on CASSANDRA-16843:
-
It's in progress, not ready for review yet.
> List snapshots of dropped tables
>
>
> Key: CASSANDRA-16843
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16843
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Snapshots
>Reporter: James Brown
>Assignee: Paulo Motta
>Priority: Normal
> Fix For: 4.1
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Auto snapshots from dropped tables don't seem to show up in {{nodetool
> listsnapshots}} (even though they do get cleared by {{nodetool
> clearsnapshot}}). This makes them kind of annoying to clean up, since you
> need to muck about in the data directory to find them.
> Erick on the mailing list said that this seems to be an oversight and that
> clearsnapshot was fixed by
> [CASSANDRA-6418|https://issues.apache.org/jira/browse/CASSANDRA-6418].
> I reproduced this both on 3.11.11 and 4.0.0.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-16843) List snapshots of dropped tables
[
https://issues.apache.org/jira/browse/CASSANDRA-16843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17529036#comment-17529036
]
Stefan Miklosovic commented on CASSANDRA-16843:
---
Is this ready to be reviewed? I am not sure if this is still indeed in "in
progress".
> List snapshots of dropped tables
>
>
> Key: CASSANDRA-16843
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16843
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Snapshots
>Reporter: James Brown
>Assignee: Paulo Motta
>Priority: Normal
> Fix For: 4.1
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Auto snapshots from dropped tables don't seem to show up in {{nodetool
> listsnapshots}} (even though they do get cleared by {{nodetool
> clearsnapshot}}). This makes them kind of annoying to clean up, since you
> need to muck about in the data directory to find them.
> Erick on the mailing list said that this seems to be an oversight and that
> clearsnapshot was fixed by
> [CASSANDRA-6418|https://issues.apache.org/jira/browse/CASSANDRA-6418].
> I reproduced this both on 3.11.11 and 4.0.0.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-16843) List snapshots of dropped tables
[
https://issues.apache.org/jira/browse/CASSANDRA-16843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17528209#comment-17528209
]
Paulo Motta commented on CASSANDRA-16843:
-
Submitted CI with intermediate patch to gather initial results:
https://ci-cassandra.apache.org/view/patches/job/Cassandra-devbranch/1631/
> List snapshots of dropped tables
>
>
> Key: CASSANDRA-16843
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16843
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Snapshots
>Reporter: James Brown
>Assignee: Paulo Motta
>Priority: Normal
> Fix For: 4.1
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Auto snapshots from dropped tables don't seem to show up in {{nodetool
> listsnapshots}} (even though they do get cleared by {{nodetool
> clearsnapshot}}). This makes them kind of annoying to clean up, since you
> need to muck about in the data directory to find them.
> Erick on the mailing list said that this seems to be an oversight and that
> clearsnapshot was fixed by
> [CASSANDRA-6418|https://issues.apache.org/jira/browse/CASSANDRA-6418].
> I reproduced this both on 3.11.11 and 4.0.0.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-16843) List snapshots of dropped tables
[
https://issues.apache.org/jira/browse/CASSANDRA-16843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17469440#comment-17469440
]
Brandon Williams commented on CASSANDRA-16843:
--
bq. I will create a fix to this by moving this logic which is dependent of
ColumnFamilyStore out of the TableSnapshot class.
I'm +1 on this especially, as I recall it being problematic when I looked at
this earlier.
> List snapshots of dropped tables
>
>
> Key: CASSANDRA-16843
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16843
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Snapshots
>Reporter: James Brown
>Assignee: Paulo Motta
>Priority: Normal
> Fix For: 3.11.x, 4.0.x
>
>
> Auto snapshots from dropped tables don't seem to show up in {{nodetool
> listsnapshots}} (even though they do get cleared by {{nodetool
> clearsnapshot}}). This makes them kind of annoying to clean up, since you
> need to muck about in the data directory to find them.
> Erick on the mailing list said that this seems to be an oversight and that
> clearsnapshot was fixed by
> [CASSANDRA-6418|https://issues.apache.org/jira/browse/CASSANDRA-6418].
> I reproduced this both on 3.11.11 and 4.0.0.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-16843) List snapshots of dropped tables
[
https://issues.apache.org/jira/browse/CASSANDRA-16843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17441848#comment-17441848
]
Brandon Williams commented on CASSANDRA-16843:
--
This all makes sense and looks good so far to me.
bq. One potential issue is handling legacy data directories (2.x series iirc)
which do not contain the UUID part. I'm not sure if these are still supported,
if so we may need to handle these.
I don't think this should be an issue, at least not in trunk.
> List snapshots of dropped tables
>
>
> Key: CASSANDRA-16843
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16843
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Snapshots
>Reporter: James Brown
>Assignee: Paulo Motta
>Priority: Normal
> Fix For: 3.11.x, 4.0.x
>
>
> Auto snapshots from dropped tables don't seem to show up in {{nodetool
> listsnapshots}} (even though they do get cleared by {{nodetool
> clearsnapshot}}). This makes them kind of annoying to clean up, since you
> need to muck about in the data directory to find them.
> Erick on the mailing list said that this seems to be an oversight and that
> clearsnapshot was fixed by
> [CASSANDRA-6418|https://issues.apache.org/jira/browse/CASSANDRA-6418].
> I reproduced this both on 3.11.11 and 4.0.0.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-16843) List snapshots of dropped tables
[
https://issues.apache.org/jira/browse/CASSANDRA-16843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17440088#comment-17440088
]
Paulo Motta commented on CASSANDRA-16843:
-
Added an initial trunk patch [on this
PR|https://github.com/apache/cassandra/pull/1305] decoupling snapshot loading
logic from {{ColumnFamilyStore}} which enables listing snapshots of dropped
tables.
The basic idea is to search all data file locations for snapshot directories
matching
{{{}${data_dir}/${ks_name}/${table_name}-${table_uuid}/snapshots/{tag{,
optionally with a json manifest ({{{}manifest.json{}}}) and feed these to
{{SnapshotManager}} which will keep track of live snapshots listed by
{{{}nodetool listsnapshots{}}}.
One potential issue is handling legacy data directories (2.x series iirc) which
do not contain the UUID part. I'm not sure if these are still supported, if so
we may need to handle these.
The snapshot searching logic is done by {{SnapshotFinder}} on [this
commit|https://github.com/pauloricardomg/cassandra/commit/6b104afed5cd190ab43e97bad622feda5db5d2df]
(with a few tests).
[This
commit|https://github.com/pauloricardomg/cassandra/commit/ec53e8a5cec9dd58f574663c4ab48780de7feb4a]
updates {{SnapshotManager}} initialization logic to use {{SnapshotFinder}}
instead of {{Keyspace/ColumnFamilyStore}} to load snapshots into memory during
startup.
The {{SnapshotManager}} class originally stored only expiring snapshots (added
by CASSANDRA-16789), but after this patch it keeps an in-memory view of all
live snapshots (expiring and non-expiring). As new snapshots are created they
are registered on this class by {{ColumnFamilyStore}} via
{{{}SnapshotManager.addSnapshot{}}}. In the future I want move snapshot
creation logic completely out of {{ColumnFamilyStore}} leaving there only
sstable hardlinking logic.
[This
commit|https://github.com/pauloricardomg/cassandra/commit/c614f88b817145e83237f61975e0e8a1b4b71cff]
makes snapshot listing use {{{}SnapshotManager.getSnapshots{}}}. This enables
listing of dropped snapshots which is tested by the test [added on this
commit|https://github.com/pauloricardomg/cassandra/commit/1fb024e6ab726da5c42f7937a8e97540d43fd7fd].
The previous change broke
{{org.apache.cassandra.distributed.test.SnapshotsTest.testManualSnapshotCleanup}}
because cleared snapshots were not removed from {{{}SnapshotManager{}}}. [This
commit|https://github.com/pauloricardomg/cassandra/commit/2b1ec31885908b1199a93127668b2a4fd422a2c6]
fixes this by moving snapshot cleanup logic to {{{}SnapshotManager{}}}.
Finally [this
commit|https://github.com/pauloricardomg/cassandra/commit/6edb90ac025690566a7a1ebc138a6198bb8a9c28]
fixes an NPE when computing the true size disk bytes of snapshots since these
are not currently computed for snapshots loaded via {{{}SnapshotFinder{}}}. I
will create a fix to this by moving this logic which is dependent of
{{ColumnFamilyStore}} out of the {{TableSnapshot}} class.
I wanted to hear your initial feedback on the approach before addressing todos,
working on edge cases (ie. secondary indexes, dropped tables with same names,
snapshot tag uniqueness) and cleanup (ie. remove dead snapshot handling code).
Can you take a look [~brandon.williams], [~stefan.miklosovic] ?
> List snapshots of dropped tables
>
>
> Key: CASSANDRA-16843
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16843
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Snapshots
>Reporter: James Brown
>Assignee: Paulo Motta
>Priority: Normal
> Fix For: 3.11.x, 4.0.x
>
>
> Auto snapshots from dropped tables don't seem to show up in {{nodetool
> listsnapshots}} (even though they do get cleared by {{nodetool
> clearsnapshot}}). This makes them kind of annoying to clean up, since you
> need to muck about in the data directory to find them.
> Erick on the mailing list said that this seems to be an oversight and that
> clearsnapshot was fixed by
> [CASSANDRA-6418|https://issues.apache.org/jira/browse/CASSANDRA-6418].
> I reproduced this both on 3.11.11 and 4.0.0.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
