[jira] [Comment Edited] (CASSANDRA-16619) Loss of commit log data possible after sstable ingest
[ https://issues.apache.org/jira/browse/CASSANDRA-16619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17528376#comment-17528376 ] Doug Whitfield edited comment on CASSANDRA-16619 at 4/26/22 7:47 PM: - oops, I was wanting to do a search for things since 3.11.9 but I changed this bug. I clearly need more coffee. going to see if I can figure out what it was. UPDATE: Unfortunately, this is not on Wayback...digging. UPDATE: History tab to the rescue was (Author: douglasawh): oops, I was wanting to do a search for things since 3.11.9 but I changed this bug. I clearly need more coffee. going to see if I can figure out what it was. UPDATE: Unfortunately, this is not on Wayback...digging. > Loss of commit log data possible after sstable ingest > - > > Key: CASSANDRA-16619 > URL: https://issues.apache.org/jira/browse/CASSANDRA-16619 > Project: Cassandra > Issue Type: Bug > Components: Local/Commit Log >Reporter: Jacek Lewandowski >Assignee: Jacek Lewandowski >Priority: Normal > Fix For: 3.0.25, 3.11.11, 4.0-rc2, 4.0 > > Time Spent: 2h > Remaining Estimate: 0h > > SSTable metadata contains commit log positions of the sstable. These > positions are used to filter out mutations from the commit log on restart and > only make sense for the node on which the data was flushed. > If an SSTable is moved between nodes they may cover regions that the > receiving node has not yet flushed, and result in valid data being lost > should these sections of the commit log need to be replayed. > Solution: > The chosen solution introduces a new sstable metadata (StatsMetadata) - > originatingHostId (UUID), which is the local host id of the node on which the > sstable was created, or null if not known. Commit log intervals from an > sstable are taken into account during Commit Log replay only when the > originatingHostId of the sstable matches the local node's hostId. > For new sstables the originatingHostId is set according to StorageService's > local hostId. > For compacted sstables the originatingHostId set according to > StorageService's local hostId, and only commit log intervals from local > sstables is preserved in the resulting sstable. > discovered by [~jakubzytka] -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-16619) Loss of commit log data possible after sstable ingest
[ https://issues.apache.org/jira/browse/CASSANDRA-16619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17528376#comment-17528376 ] Doug Whitfield edited comment on CASSANDRA-16619 at 4/26/22 7:45 PM: - oops, I was wanting to do a search for things since 3.11.9 but I changed this bug. I clearly need more coffee. going to see if I can figure out what it was. UPDATE: Unfortunately, this is not on Wayback...digging. was (Author: douglasawh): oops, I was wanting to do a search for things since 3.11.9 but I changed this bug. I clearly need more coffee. going to see if I can figure out what it was. > Loss of commit log data possible after sstable ingest > - > > Key: CASSANDRA-16619 > URL: https://issues.apache.org/jira/browse/CASSANDRA-16619 > Project: Cassandra > Issue Type: Bug > Components: Local/Commit Log >Reporter: Jacek Lewandowski >Assignee: Jacek Lewandowski >Priority: Normal > Fix For: 3.0.25, 3.11.11, 4.0-rc2, 4.0 > > Time Spent: 2h > Remaining Estimate: 0h > > SSTable metadata contains commit log positions of the sstable. These > positions are used to filter out mutations from the commit log on restart and > only make sense for the node on which the data was flushed. > If an SSTable is moved between nodes they may cover regions that the > receiving node has not yet flushed, and result in valid data being lost > should these sections of the commit log need to be replayed. > Solution: > The chosen solution introduces a new sstable metadata (StatsMetadata) - > originatingHostId (UUID), which is the local host id of the node on which the > sstable was created, or null if not known. Commit log intervals from an > sstable are taken into account during Commit Log replay only when the > originatingHostId of the sstable matches the local node's hostId. > For new sstables the originatingHostId is set according to StorageService's > local hostId. > For compacted sstables the originatingHostId set according to > StorageService's local hostId, and only commit log intervals from local > sstables is preserved in the resulting sstable. > discovered by [~jakubzytka] -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-16619) Loss of commit log data possible after sstable ingest
[ https://issues.apache.org/jira/browse/CASSANDRA-16619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17450645#comment-17450645 ] Stefan Miklosovic edited comment on CASSANDRA-16619 at 11/29/21, 6:11 PM: -- I ll keep an eye on this, I was trying to reproduce it but I cant :D But I am absolutely sure I was getting this, multiple times in a row. I could not wrap my head around this. I am not sure if it is happening non-deterministically or what. was (Author: smiklosovic): I ll keep an eye on this, I was trying to reproduce it but I cant :D But I am absolutely sure I was getting this, multiple times in a row. I could not wrapy my head around this. I am not sure if it is happening non-deterministically or what. > Loss of commit log data possible after sstable ingest > - > > Key: CASSANDRA-16619 > URL: https://issues.apache.org/jira/browse/CASSANDRA-16619 > Project: Cassandra > Issue Type: Bug > Components: Local/Commit Log >Reporter: Jacek Lewandowski >Assignee: Jacek Lewandowski >Priority: Normal > Fix For: 3.0.25, 3.11.11, 4.0-rc2, 4.0 > > Time Spent: 2h > Remaining Estimate: 0h > > SSTable metadata contains commit log positions of the sstable. These > positions are used to filter out mutations from the commit log on restart and > only make sense for the node on which the data was flushed. > If an SSTable is moved between nodes they may cover regions that the > receiving node has not yet flushed, and result in valid data being lost > should these sections of the commit log need to be replayed. > Solution: > The chosen solution introduces a new sstable metadata (StatsMetadata) - > originatingHostId (UUID), which is the local host id of the node on which the > sstable was created, or null if not known. Commit log intervals from an > sstable are taken into account during Commit Log replay only when the > originatingHostId of the sstable matches the local node's hostId. > For new sstables the originatingHostId is set according to StorageService's > local hostId. > For compacted sstables the originatingHostId set according to > StorageService's local hostId, and only commit log intervals from local > sstables is preserved in the resulting sstable. > discovered by [~jakubzytka] -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-16619) Loss of commit log data possible after sstable ingest
[
https://issues.apache.org/jira/browse/CASSANDRA-16619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17450494#comment-17450494
]
Branimir Lambov edited comment on CASSANDRA-16619 at 11/29/21, 2:36 PM:
Rechecking the code, the point in time in restores is applied in
{{MutationInitiator.initiateMutation}}
[here|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/db/commitlog/CommitLogReplayer.java#L247].
What this patch may change is the filtering by commit log position done [later
in the same
method|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/db/commitlog/CommitLogReplayer.java#L265].
I still do not see any evidence that this patch is affecting PIT restore in any
way, other than making replay slower because it could be needlessly replaying
mutations that are already present in sstables. Granted, the latter is an issue
and may warrant risking correctness by flagging this out, but reusing the same
ID through porting CASSANDRA-14582 to earlier versions is most probably a
better solution.
was (Author: blambov):
Rechecking the code, the point in time in restores is applied in
{{MutationInitiator.initiateMutation}}
[here|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/db/commitlog/CommitLogReplayer.java#L247].
What this patch may change is the filtering by commit log position done [later
in the same
method|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/db/commitlog/CommitLogReplayer.java#L265].
I still do not see any evidence that this patch is affecting PIT restore in any
way, other than making replay slower because it could be needlessly replaying
mutations that are already present in sstables. Granted, the latter is an issue
and may warrant risking correctness by flagging this out, but reusing the same
ID as in CASSANDRA-14582 is most probably a better solution.
> Loss of commit log data possible after sstable ingest
> -
>
> Key: CASSANDRA-16619
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16619
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Commit Log
>Reporter: Jacek Lewandowski
>Assignee: Jacek Lewandowski
>Priority: Normal
> Fix For: 3.0.25, 3.11.11, 4.0-rc2, 4.0
>
> Time Spent: 2h
> Remaining Estimate: 0h
>
> SSTable metadata contains commit log positions of the sstable. These
> positions are used to filter out mutations from the commit log on restart and
> only make sense for the node on which the data was flushed.
> If an SSTable is moved between nodes they may cover regions that the
> receiving node has not yet flushed, and result in valid data being lost
> should these sections of the commit log need to be replayed.
> Solution:
> The chosen solution introduces a new sstable metadata (StatsMetadata) -
> originatingHostId (UUID), which is the local host id of the node on which the
> sstable was created, or null if not known. Commit log intervals from an
> sstable are taken into account during Commit Log replay only when the
> originatingHostId of the sstable matches the local node's hostId.
> For new sstables the originatingHostId is set according to StorageService's
> local hostId.
> For compacted sstables the originatingHostId set according to
> StorageService's local hostId, and only commit log intervals from local
> sstables is preserved in the resulting sstable.
> discovered by [~jakubzytka]
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-16619) Loss of commit log data possible after sstable ingest
[ https://issues.apache.org/jira/browse/CASSANDRA-16619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17450439#comment-17450439 ] Branimir Lambov edited comment on CASSANDRA-16619 at 11/29/21, 1:14 PM: Could you elaborate why this change would cause the restore point to be ignored? was (Author: blambov): Could you elaborate why this change would cause the restore point would be ignored? > Loss of commit log data possible after sstable ingest > - > > Key: CASSANDRA-16619 > URL: https://issues.apache.org/jira/browse/CASSANDRA-16619 > Project: Cassandra > Issue Type: Bug > Components: Local/Commit Log >Reporter: Jacek Lewandowski >Assignee: Jacek Lewandowski >Priority: Normal > Fix For: 3.0.25, 3.11.11, 4.0-rc2, 4.0 > > Time Spent: 2h > Remaining Estimate: 0h > > SSTable metadata contains commit log positions of the sstable. These > positions are used to filter out mutations from the commit log on restart and > only make sense for the node on which the data was flushed. > If an SSTable is moved between nodes they may cover regions that the > receiving node has not yet flushed, and result in valid data being lost > should these sections of the commit log need to be replayed. > Solution: > The chosen solution introduces a new sstable metadata (StatsMetadata) - > originatingHostId (UUID), which is the local host id of the node on which the > sstable was created, or null if not known. Commit log intervals from an > sstable are taken into account during Commit Log replay only when the > originatingHostId of the sstable matches the local node's hostId. > For new sstables the originatingHostId is set according to StorageService's > local hostId. > For compacted sstables the originatingHostId set according to > StorageService's local hostId, and only commit log intervals from local > sstables is preserved in the resulting sstable. > discovered by [~jakubzytka] -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-16619) Loss of commit log data possible after sstable ingest
[ https://issues.apache.org/jira/browse/CASSANDRA-16619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17450366#comment-17450366 ] Stefan Miklosovic edited comment on CASSANDRA-16619 at 11/29/21, 11:20 AM: --- No, I think I am still right in what I wrote. It breaks point-in-time restoration. If you set property "restore_point_in_time" in commitlog_archiving.properties, with this patch it is just ignored and all is replayed. So you do not have any possibility to replay to some point in time for a completely new node. I still consider this to be a regression. Lets say you have a cluster of 5 nodes you are taking snapshots of regularly as well as all commit logs in between and all this stuff is uploaded to some backup storage. Then on restore, you want to regenerate the cluster as it was, point-in-time precision. With backup, I have also backed up the information about tokens so I set initial_tokens field in yaml etc. So when I put it all in place, SSTables will match the nodes (because tokens have not changed) and I want to replay the commit logs, but since host ids are generated and I cant set them up on my own as shown in the other ticket I linked, for previous versions, pit restore will stop to work properly even my tokens and all that stuff is just completely fine and I know what I do. This was all possible to do previously. Now I end up with all data being replayed even I do not want that. was (Author: smiklosovic): No, I think I am still right in what I wrote. It breaks point-in-time restoration. If you set property "restore_point_in_time" in commitlog_archiving.properties, with this patch it is just ignored and all is replayed. So you do not have any possibility to replay to some point in time for a completely new node. I still consider this to be a regression. Lets say you have a cluster of 5 nodes you are taking snapshots of regularly as well as all commit logs in between and all this stuff is uploaded to some backup storage. Then on restore, you want to regenerate the cluster as it was, point-in-time precision. With backup, I have also backed up the information about tokens so I set initial_tokens field in yaml etc. So when I put it all in place, SSTables will match the nodes (because tokens have not changed) and I want to replay the commit logs, but since host ids are generated and I cant set them up on my own as shown in the other ticket I linked, for previous versions, pit restore will stop work propertly even my tokens and all that stuff is just completely fine and I know what I do. This was all possible to do previously. Now I end up with all data being replay even I do not want that. > Loss of commit log data possible after sstable ingest > - > > Key: CASSANDRA-16619 > URL: https://issues.apache.org/jira/browse/CASSANDRA-16619 > Project: Cassandra > Issue Type: Bug > Components: Local/Commit Log >Reporter: Jacek Lewandowski >Assignee: Jacek Lewandowski >Priority: Normal > Fix For: 3.0.25, 3.11.11, 4.0-rc2, 4.0 > > Time Spent: 2h > Remaining Estimate: 0h > > SSTable metadata contains commit log positions of the sstable. These > positions are used to filter out mutations from the commit log on restart and > only make sense for the node on which the data was flushed. > If an SSTable is moved between nodes they may cover regions that the > receiving node has not yet flushed, and result in valid data being lost > should these sections of the commit log need to be replayed. > Solution: > The chosen solution introduces a new sstable metadata (StatsMetadata) - > originatingHostId (UUID), which is the local host id of the node on which the > sstable was created, or null if not known. Commit log intervals from an > sstable are taken into account during Commit Log replay only when the > originatingHostId of the sstable matches the local node's hostId. > For new sstables the originatingHostId is set according to StorageService's > local hostId. > For compacted sstables the originatingHostId set according to > StorageService's local hostId, and only commit log intervals from local > sstables is preserved in the resulting sstable. > discovered by [~jakubzytka] -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-16619) Loss of commit log data possible after sstable ingest
[ https://issues.apache.org/jira/browse/CASSANDRA-16619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17450366#comment-17450366 ] Stefan Miklosovic edited comment on CASSANDRA-16619 at 11/29/21, 11:13 AM: --- No, I think I am still right in what I wrote. It breaks point-in-time restoration. If you set property "restore_point_in_time" in commitlog_archiving.properties, with this patch it is just ignored and all is replayed. So you do not have any possibility to replay to some point in time for a completely new node. I still consider this to be a regression. Lets say you have a cluster of 5 nodes you are taking snapshots of regularly as well as all commit logs in between and all this stuff is uploaded to some backup storage. Then on restore, you want to regenerate the cluster as it was, point-in-time precision. With backup, I have also backed up the information about tokens so I set initial_tokens field in yaml etc. So when I put it all in place, SSTables will match the nodes (because tokens have not changed) and I want to replay the commit logs, but since host ids are generated and I cant set them up on my own as shown in the other ticket I linked, for previous versions, pit restore will stop work propertly even my tokens and all that stuff is just completely fine and I know what I do. This was all possible to do previously. Now I end up with all data being replay even I do not want that. was (Author: smiklosovic): No, I think I am still right in what I wrote. It breaks point-in-time restoration. If you set property "restore_point_in_time" in commitlog_archiving.properties, with this patch it is just ignored and all is replayed. So you do not have any possibility to replay to some point in time for a completely new node. I still consider this to be a regression. > Loss of commit log data possible after sstable ingest > - > > Key: CASSANDRA-16619 > URL: https://issues.apache.org/jira/browse/CASSANDRA-16619 > Project: Cassandra > Issue Type: Bug > Components: Local/Commit Log >Reporter: Jacek Lewandowski >Assignee: Jacek Lewandowski >Priority: Normal > Fix For: 3.0.25, 3.11.11, 4.0-rc2, 4.0 > > Time Spent: 2h > Remaining Estimate: 0h > > SSTable metadata contains commit log positions of the sstable. These > positions are used to filter out mutations from the commit log on restart and > only make sense for the node on which the data was flushed. > If an SSTable is moved between nodes they may cover regions that the > receiving node has not yet flushed, and result in valid data being lost > should these sections of the commit log need to be replayed. > Solution: > The chosen solution introduces a new sstable metadata (StatsMetadata) - > originatingHostId (UUID), which is the local host id of the node on which the > sstable was created, or null if not known. Commit log intervals from an > sstable are taken into account during Commit Log replay only when the > originatingHostId of the sstable matches the local node's hostId. > For new sstables the originatingHostId is set according to StorageService's > local hostId. > For compacted sstables the originatingHostId set according to > StorageService's local hostId, and only commit log intervals from local > sstables is preserved in the resulting sstable. > discovered by [~jakubzytka] -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-16619) Loss of commit log data possible after sstable ingest
[ https://issues.apache.org/jira/browse/CASSANDRA-16619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17450284#comment-17450284 ] Stefan Miklosovic edited comment on CASSANDRA-16619 at 11/29/21, 9:11 AM: -- I think this feature is broken on the scenario when I have a commit log I want to replay for a completely diffent / new node which has UUID not matching the one in the commit log. The way how to workaround this will be in 4.1 as done in https://issues.apache.org/jira/browse/CASSANDRA-14582 however I am afraid that point in time restoration of all other Cassandra versions, since this patch was introduced, is broken. was (Author: smiklosovic): I think this feature is broken on the scenario when I have a commit log I want to replay for a completely diffent / new node which has UUID not matching the one in the commit log. There is a way how to workaround this will be in 4.1 as done in https://issues.apache.org/jira/browse/CASSANDRA-14582 however I am afraid that point in time restoration of all other Cassandra versions, since this patch was introduced, is broken. > Loss of commit log data possible after sstable ingest > - > > Key: CASSANDRA-16619 > URL: https://issues.apache.org/jira/browse/CASSANDRA-16619 > Project: Cassandra > Issue Type: Bug > Components: Local/Commit Log >Reporter: Jacek Lewandowski >Assignee: Jacek Lewandowski >Priority: Normal > Fix For: 3.0.25, 3.11.11, 4.0-rc2, 4.0 > > Time Spent: 2h > Remaining Estimate: 0h > > SSTable metadata contains commit log positions of the sstable. These > positions are used to filter out mutations from the commit log on restart and > only make sense for the node on which the data was flushed. > If an SSTable is moved between nodes they may cover regions that the > receiving node has not yet flushed, and result in valid data being lost > should these sections of the commit log need to be replayed. > Solution: > The chosen solution introduces a new sstable metadata (StatsMetadata) - > originatingHostId (UUID), which is the local host id of the node on which the > sstable was created, or null if not known. Commit log intervals from an > sstable are taken into account during Commit Log replay only when the > originatingHostId of the sstable matches the local node's hostId. > For new sstables the originatingHostId is set according to StorageService's > local hostId. > For compacted sstables the originatingHostId set according to > StorageService's local hostId, and only commit log intervals from local > sstables is preserved in the resulting sstable. > discovered by [~jakubzytka] -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-16619) Loss of commit log data possible after sstable ingest
[ https://issues.apache.org/jira/browse/CASSANDRA-16619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17449594#comment-17449594 ] Stefan Miklosovic edited comment on CASSANDRA-16619 at 11/26/21, 3:03 PM: -- How am I supposed to do a point-in-time restoration using a commit log from the other node when it gets skipped as shown in the previous comment? That is by means of pointing Cassandra to dir with logs and modifying commitlog_archiving.properties. was (Author: smiklosovic): How am I supposed to do a point-in-time restoration using a commit log from the other node when it gets skipped as shown in the previous comment? > Loss of commit log data possible after sstable ingest > - > > Key: CASSANDRA-16619 > URL: https://issues.apache.org/jira/browse/CASSANDRA-16619 > Project: Cassandra > Issue Type: Bug > Components: Local/Commit Log >Reporter: Jacek Lewandowski >Assignee: Jacek Lewandowski >Priority: Normal > Fix For: 3.0.25, 3.11.11, 4.0-rc2, 4.0 > > Time Spent: 2h > Remaining Estimate: 0h > > SSTable metadata contains commit log positions of the sstable. These > positions are used to filter out mutations from the commit log on restart and > only make sense for the node on which the data was flushed. > If an SSTable is moved between nodes they may cover regions that the > receiving node has not yet flushed, and result in valid data being lost > should these sections of the commit log need to be replayed. > Solution: > The chosen solution introduces a new sstable metadata (StatsMetadata) - > originatingHostId (UUID), which is the local host id of the node on which the > sstable was created, or null if not known. Commit log intervals from an > sstable are taken into account during Commit Log replay only when the > originatingHostId of the sstable matches the local node's hostId. > For new sstables the originatingHostId is set according to StorageService's > local hostId. > For compacted sstables the originatingHostId set according to > StorageService's local hostId, and only commit log intervals from local > sstables is preserved in the resulting sstable. > discovered by [~jakubzytka] -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-16619) Loss of commit log data possible after sstable ingest
[ https://issues.apache.org/jira/browse/CASSANDRA-16619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17449594#comment-17449594 ] Stefan Miklosovic edited comment on CASSANDRA-16619 at 11/26/21, 2:45 PM: -- How am I supposed to do a point-in-time restoration using a commit log from the other node when it gets skipped as shown in the previous comment? was (Author: smiklosovic): How am I supposed to do a point-in-time restoration using a commit log from the other node when it gets skipped as show in the previous comment? > Loss of commit log data possible after sstable ingest > - > > Key: CASSANDRA-16619 > URL: https://issues.apache.org/jira/browse/CASSANDRA-16619 > Project: Cassandra > Issue Type: Bug > Components: Local/Commit Log >Reporter: Jacek Lewandowski >Assignee: Jacek Lewandowski >Priority: Normal > Fix For: 3.0.25, 3.11.11, 4.0-rc2, 4.0 > > Time Spent: 2h > Remaining Estimate: 0h > > SSTable metadata contains commit log positions of the sstable. These > positions are used to filter out mutations from the commit log on restart and > only make sense for the node on which the data was flushed. > If an SSTable is moved between nodes they may cover regions that the > receiving node has not yet flushed, and result in valid data being lost > should these sections of the commit log need to be replayed. > Solution: > The chosen solution introduces a new sstable metadata (StatsMetadata) - > originatingHostId (UUID), which is the local host id of the node on which the > sstable was created, or null if not known. Commit log intervals from an > sstable are taken into account during Commit Log replay only when the > originatingHostId of the sstable matches the local node's hostId. > For new sstables the originatingHostId is set according to StorageService's > local hostId. > For compacted sstables the originatingHostId set according to > StorageService's local hostId, and only commit log intervals from local > sstables is preserved in the resulting sstable. > discovered by [~jakubzytka] -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-16619) Loss of commit log data possible after sstable ingest
[ https://issues.apache.org/jira/browse/CASSANDRA-16619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17345956#comment-17345956 ] Benjamin Lerer edited comment on CASSANDRA-16619 at 5/17/21, 8:20 AM: -- The patch looks good to me. Sorry, for missing that part in the review. I will commit was (Author: blerer): The patch looks good to me. > Loss of commit log data possible after sstable ingest > - > > Key: CASSANDRA-16619 > URL: https://issues.apache.org/jira/browse/CASSANDRA-16619 > Project: Cassandra > Issue Type: Bug > Components: Local/Commit Log >Reporter: Jacek Lewandowski >Assignee: Jacek Lewandowski >Priority: Normal > Fix For: 3.0.25, 3.11.11, 4.0-rc2, 4.0 > > Time Spent: 40m > Remaining Estimate: 0h > > SSTable metadata contains commit log positions of the sstable. These > positions are used to filter out mutations from the commit log on restart and > only make sense for the node on which the data was flushed. > If an SSTable is moved between nodes they may cover regions that the > receiving node has not yet flushed, and result in valid data being lost > should these sections of the commit log need to be replayed. > Solution: > The chosen solution introduces a new sstable metadata (StatsMetadata) - > originatingHostId (UUID), which is the local host id of the node on which the > sstable was created, or null if not known. Commit log intervals from an > sstable are taken into account during Commit Log replay only when the > originatingHostId of the sstable matches the local node's hostId. > For new sstables the originatingHostId is set according to StorageService's > local hostId. > For compacted sstables the originatingHostId set according to > StorageService's local hostId, and only commit log intervals from local > sstables is preserved in the resulting sstable. > discovered by [~jakubzytka] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-16619) Loss of commit log data possible after sstable ingest
[ https://issues.apache.org/jira/browse/CASSANDRA-16619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17343859#comment-17343859 ] Jacek Lewandowski edited comment on CASSANDRA-16619 at 5/13/21, 1:30 PM: - Unfortunately I missed moving the originating host id to the end in 4.0 PR here is the fix: https://github.com/apache/cassandra/pull/1007 was (Author: jlewandowski): Unfortunately I missed moving the originating host id to the end in 4.0 PR > Loss of commit log data possible after sstable ingest > - > > Key: CASSANDRA-16619 > URL: https://issues.apache.org/jira/browse/CASSANDRA-16619 > Project: Cassandra > Issue Type: Bug > Components: Local/Commit Log >Reporter: Jacek Lewandowski >Assignee: Jacek Lewandowski >Priority: Normal > Fix For: 3.0.25, 3.11.11, 4.0-rc2, 4.0 > > Time Spent: 40m > Remaining Estimate: 0h > > SSTable metadata contains commit log positions of the sstable. These > positions are used to filter out mutations from the commit log on restart and > only make sense for the node on which the data was flushed. > If an SSTable is moved between nodes they may cover regions that the > receiving node has not yet flushed, and result in valid data being lost > should these sections of the commit log need to be replayed. > Solution: > The chosen solution introduces a new sstable metadata (StatsMetadata) - > originatingHostId (UUID), which is the local host id of the node on which the > sstable was created, or null if not known. Commit log intervals from an > sstable are taken into account during Commit Log replay only when the > originatingHostId of the sstable matches the local node's hostId. > For new sstables the originatingHostId is set according to StorageService's > local hostId. > For compacted sstables the originatingHostId set according to > StorageService's local hostId, and only commit log intervals from local > sstables is preserved in the resulting sstable. > discovered by [~jakubzytka] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-16619) Loss of commit log data possible after sstable ingest
[ https://issues.apache.org/jira/browse/CASSANDRA-16619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17339547#comment-17339547 ] Benjamin Lerer edited comment on CASSANDRA-16619 at 5/5/21, 9:18 AM: - +1 Thanks for all the hard work [~jlewandowski] was (Author: blerer): +1. > Loss of commit log data possible after sstable ingest > - > > Key: CASSANDRA-16619 > URL: https://issues.apache.org/jira/browse/CASSANDRA-16619 > Project: Cassandra > Issue Type: Bug > Components: Local/Commit Log >Reporter: Jacek Lewandowski >Assignee: Jacek Lewandowski >Priority: Normal > Fix For: 4.0, 3.0.x, 3.11.x > > Time Spent: 0.5h > Remaining Estimate: 0h > > SSTable metadata contains commit log positions of the sstable. These > positions are used to filter out mutations from the commit log on restart and > only make sense for the node on which the data was flushed. > If an SSTable is moved between nodes they may cover regions that the > receiving node has not yet flushed, and result in valid data being lost > should these sections of the commit log need to be replayed. > Solution: > The chosen solution introduces a new sstable metadata (StatsMetadata) - > originatingHostId (UUID), which is the local host id of the node on which the > sstable was created, or null if not known. Commit log intervals from an > sstable are taken into account during Commit Log replay only when the > originatingHostId of the sstable matches the local node's hostId. > For new sstables the originatingHostId is set according to StorageService's > local hostId. > For compacted sstables the originatingHostId set according to > StorageService's local hostId, and only commit log intervals from local > sstables is preserved in the resulting sstable. > discovered by [~jakubzytka] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-16619) Loss of commit log data possible after sstable ingest
[ https://issues.apache.org/jira/browse/CASSANDRA-16619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17337318#comment-17337318 ] Jacek Lewandowski edited comment on CASSANDRA-16619 at 5/4/21, 10:10 PM: - Started CI tests: https://ci-cassandra.apache.org/view/patches/job/Cassandra-devbranch/728/ (4.0) - eventually looks good I guess https://ci-cassandra.apache.org/view/patches/job/Cassandra-devbranch/729/ (3.11) https://ci-cassandra.apache.org/view/patches/job/Cassandra-devbranch/724/ (3.0) was (Author: jlewandowski): Started CI tests: https://ci-cassandra.apache.org/view/patches/job/Cassandra-devbranch/728/ (4.0) - eventually looks good I guess https://ci-cassandra.apache.org/view/patches/job/Cassandra-devbranch/725/ (3.11) https://ci-cassandra.apache.org/view/patches/job/Cassandra-devbranch/724/ (3.0) > Loss of commit log data possible after sstable ingest > - > > Key: CASSANDRA-16619 > URL: https://issues.apache.org/jira/browse/CASSANDRA-16619 > Project: Cassandra > Issue Type: Bug > Components: Local/Commit Log >Reporter: Jacek Lewandowski >Assignee: Jacek Lewandowski >Priority: Normal > Fix For: 4.0, 3.0.x, 3.11.x > > Time Spent: 0.5h > Remaining Estimate: 0h > > SSTable metadata contains commit log positions of the sstable. These > positions are used to filter out mutations from the commit log on restart and > only make sense for the node on which the data was flushed. > If an SSTable is moved between nodes they may cover regions that the > receiving node has not yet flushed, and result in valid data being lost > should these sections of the commit log need to be replayed. > Solution: > The chosen solution introduces a new sstable metadata (StatsMetadata) - > originatingHostId (UUID), which is the local host id of the node on which the > sstable was created, or null if not known. Commit log intervals from an > sstable are taken into account during Commit Log replay only when the > originatingHostId of the sstable matches the local node's hostId. > For new sstables the originatingHostId is set according to StorageService's > local hostId. > For compacted sstables the originatingHostId set according to > StorageService's local hostId, and only commit log intervals from local > sstables is preserved in the resulting sstable. > discovered by [~jakubzytka] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-16619) Loss of commit log data possible after sstable ingest
[ https://issues.apache.org/jira/browse/CASSANDRA-16619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17337318#comment-17337318 ] Jacek Lewandowski edited comment on CASSANDRA-16619 at 5/4/21, 2:38 PM: Started CI tests: https://ci-cassandra.apache.org/view/patches/job/Cassandra-devbranch/728/ (4.0) - eventually looks good I guess https://ci-cassandra.apache.org/view/patches/job/Cassandra-devbranch/725/ (3.11) https://ci-cassandra.apache.org/view/patches/job/Cassandra-devbranch/724/ (3.0) was (Author: jlewandowski): Started CI tests: https://ci-cassandra.apache.org/view/patches/job/Cassandra-devbranch/727/ (4.0) https://ci-cassandra.apache.org/view/patches/job/Cassandra-devbranch/725/ (3.11) https://ci-cassandra.apache.org/view/patches/job/Cassandra-devbranch/724/ (3.0) > Loss of commit log data possible after sstable ingest > - > > Key: CASSANDRA-16619 > URL: https://issues.apache.org/jira/browse/CASSANDRA-16619 > Project: Cassandra > Issue Type: Bug > Components: Local/Commit Log >Reporter: Jacek Lewandowski >Assignee: Jacek Lewandowski >Priority: Normal > Fix For: 4.0, 3.0.x, 3.11.x > > Time Spent: 0.5h > Remaining Estimate: 0h > > SSTable metadata contains commit log positions of the sstable. These > positions are used to filter out mutations from the commit log on restart and > only make sense for the node on which the data was flushed. > If an SSTable is moved between nodes they may cover regions that the > receiving node has not yet flushed, and result in valid data being lost > should these sections of the commit log need to be replayed. > Solution: > The chosen solution introduces a new sstable metadata (StatsMetadata) - > originatingHostId (UUID), which is the local host id of the node on which the > sstable was created, or null if not known. Commit log intervals from an > sstable are taken into account during Commit Log replay only when the > originatingHostId of the sstable matches the local node's hostId. > For new sstables the originatingHostId is set according to StorageService's > local hostId. > For compacted sstables the originatingHostId set according to > StorageService's local hostId, and only commit log intervals from local > sstables is preserved in the resulting sstable. > discovered by [~jakubzytka] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-16619) Loss of commit log data possible after sstable ingest
[ https://issues.apache.org/jira/browse/CASSANDRA-16619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17337318#comment-17337318 ] Jacek Lewandowski edited comment on CASSANDRA-16619 at 5/3/21, 7:56 PM: Started CI tests: https://ci-cassandra.apache.org/view/patches/job/Cassandra-devbranch/727/ (4.0) https://ci-cassandra.apache.org/view/patches/job/Cassandra-devbranch/725/ (3.11) https://ci-cassandra.apache.org/view/patches/job/Cassandra-devbranch/724/ (3.0) was (Author: jlewandowski): Started CI tests: https://ci-cassandra.apache.org/view/patches/job/Cassandra-devbranch/723/ (4.0) https://ci-cassandra.apache.org/view/patches/job/Cassandra-devbranch/725/ (3.11) https://ci-cassandra.apache.org/view/patches/job/Cassandra-devbranch/724/ (3.0) > Loss of commit log data possible after sstable ingest > - > > Key: CASSANDRA-16619 > URL: https://issues.apache.org/jira/browse/CASSANDRA-16619 > Project: Cassandra > Issue Type: Bug > Components: Local/Commit Log >Reporter: Jacek Lewandowski >Assignee: Jacek Lewandowski >Priority: Normal > Fix For: 4.0, 3.0.x, 3.11.x > > Time Spent: 0.5h > Remaining Estimate: 0h > > SSTable metadata contains commit log positions of the sstable. These > positions are used to filter out mutations from the commit log on restart and > only make sense for the node on which the data was flushed. > If an SSTable is moved between nodes they may cover regions that the > receiving node has not yet flushed, and result in valid data being lost > should these sections of the commit log need to be replayed. > Solution: > The chosen solution introduces a new sstable metadata (StatsMetadata) - > originatingHostId (UUID), which is the local host id of the node on which the > sstable was created, or null if not known. Commit log intervals from an > sstable are taken into account during Commit Log replay only when the > originatingHostId of the sstable matches the local node's hostId. > For new sstables the originatingHostId is set according to StorageService's > local hostId. > For compacted sstables the originatingHostId set according to > StorageService's local hostId, and only commit log intervals from local > sstables is preserved in the resulting sstable. > discovered by [~jakubzytka] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-16619) Loss of commit log data possible after sstable ingest
[ https://issues.apache.org/jira/browse/CASSANDRA-16619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17337318#comment-17337318 ] Jacek Lewandowski edited comment on CASSANDRA-16619 at 5/2/21, 8:07 AM: Started CI tests: https://ci-cassandra.apache.org/view/patches/job/Cassandra-devbranch/723/ (4.0) https://ci-cassandra.apache.org/view/patches/job/Cassandra-devbranch/725/ (3.11) https://ci-cassandra.apache.org/view/patches/job/Cassandra-devbranch/724/ (3.0) was (Author: jlewandowski): Started CI tests: https://ci-cassandra.apache.org/view/patches/job/Cassandra-devbranch/720/ https://ci-cassandra.apache.org/view/patches/job/Cassandra-devbranch/721/ https://ci-cassandra.apache.org/view/patches/job/Cassandra-devbranch/722/ > Loss of commit log data possible after sstable ingest > - > > Key: CASSANDRA-16619 > URL: https://issues.apache.org/jira/browse/CASSANDRA-16619 > Project: Cassandra > Issue Type: Bug > Components: Local/Commit Log >Reporter: Jacek Lewandowski >Assignee: Jacek Lewandowski >Priority: Normal > Fix For: 4.0, 3.0.x, 3.11.x > > Time Spent: 0.5h > Remaining Estimate: 0h > > SSTable metadata contains commit log positions of the sstable. These > positions are used to filter out mutations from the commit log on restart and > only make sense for the node on which the data was flushed. > If an SSTable is moved between nodes they may cover regions that the > receiving node has not yet flushed, and result in valid data being lost > should these sections of the commit log need to be replayed. > Solution: > The chosen solution introduces a new sstable metadata (StatsMetadata) - > originatingHostId (UUID), which is the local host id of the node on which the > sstable was created, or null if not known. Commit log intervals from an > sstable are taken into account during Commit Log replay only when the > originatingHostId of the sstable matches the local node's hostId. > For new sstables the originatingHostId is set according to StorageService's > local hostId. > For compacted sstables the originatingHostId set according to > StorageService's local hostId, and only commit log intervals from local > sstables is preserved in the resulting sstable. > discovered by [~jakubzytka] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-16619) Loss of commit log data possible after sstable ingest
[
https://issues.apache.org/jira/browse/CASSANDRA-16619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17332479#comment-17332479
]
Benjamin Lerer edited comment on CASSANDRA-16619 at 4/26/21, 3:07 PM:
--
As {{4.0-rc1}} has been released we need to provide forward compatibility for
data. I do not think it is the case with the current patch. Users using
{{4.0-rc1}} will have an {{na}} format *without* hostID whereas when they will
deploy {{4.0-rc2}} they will have an {{na}} format *with* hostID which will
fail to read the {{na}} SSTables from the {{4.0-rc1}}. To fix that I think that
we need to introduce a new {{nb}} version.
Regarding the patches, we should refactor the
{{MetadataSerializerTest.testXReadY}} tests to ensure that they test all the
possible combinations not only some of them.
was (Author: blerer):
As {{4.0-rc1}} has been released we need to provide forward compatibility for
data. I do not think it is the case with the current patch. Users using
{{4.0-rc1}} will have an {{na}} format *without* hostID whereas when they will
deploy {{4.0-rc2}} they will have an {{na}} format *with* hostID which will
fail to read the {{na}} SSTables from the {{4.0-rc1}}. To fix that I think that
we need to introduce a new {{nb}} version.
Regarding the patches should refactor the {{MetadataSerializerTest.testXReadY}}
tests to ensure that they test all the possible combinations not only some of
them.
> Loss of commit log data possible after sstable ingest
> -
>
> Key: CASSANDRA-16619
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16619
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Commit Log
>Reporter: Jacek Lewandowski
>Assignee: Jacek Lewandowski
>Priority: Normal
> Fix For: 4.0, 3.0.x, 3.11.x
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> SSTable metadata contains commit log positions of the sstable. These
> positions are used to filter out mutations from the commit log on restart and
> only make sense for the node on which the data was flushed.
> If an SSTable is moved between nodes they may cover regions that the
> receiving node has not yet flushed, and result in valid data being lost
> should these sections of the commit log need to be replayed.
> Solution:
> The chosen solution introduces a new sstable metadata (StatsMetadata) -
> originatingHostId (UUID), which is the local host id of the node on which the
> sstable was created, or null if not known. Commit log intervals from an
> sstable are taken into account during Commit Log replay only when the
> originatingHostId of the sstable matches the local node's hostId.
> For new sstables the originatingHostId is set according to StorageService's
> local hostId.
> For compacted sstables the originatingHostId set according to
> StorageService's local hostId, and only commit log intervals from local
> sstables is preserved in the resulting sstable.
> discovered by [~jakubzytka]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-16619) Loss of commit log data possible after sstable ingest
[
https://issues.apache.org/jira/browse/CASSANDRA-16619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17332479#comment-17332479
]
Benjamin Lerer edited comment on CASSANDRA-16619 at 4/26/21, 3:06 PM:
--
As {{4.0-rc1}} has been released by consequence we need to provide forward
compatibility for data. I do not think it is the case with the current patch.
Users using {{4.0-rc1}} will have an {{na}} format *without* hostID whereas
when they will deploy {{4.0-rc2}} they will have an {{na}} format *with* hostID
which will fail to read the {{na}} SSTables from the {{4.0-rc1}}. To fix that I
think that we need to introduce a new {{nb}} version.
Regarding the patches should refactor the {{MetadataSerializerTest.testXReadY}}
tests to ensure that they test all the possible combinations not only some of
them.
was (Author: blerer):
As {{4.0-rc1}} has been released by consequence we need to provide forward
compatibility for data. I do not think it is the case with the current patch.
Users using {{4.0-rc1}} will have an {{na}} format *without* hostID whereas
when they will deploy {{4.0-rc2}} they will have an {{na}} format *with* hostID
which will fail to read the {{na}} SSTables from the {{4.0-rc1}}. To fix that I
think that we need to introduce a new {{nb}} version.
Regarding the patches should refactor the testMcReadMc
> Loss of commit log data possible after sstable ingest
> -
>
> Key: CASSANDRA-16619
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16619
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Commit Log
>Reporter: Jacek Lewandowski
>Assignee: Jacek Lewandowski
>Priority: Normal
> Fix For: 4.0, 3.0.x, 3.11.x
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> SSTable metadata contains commit log positions of the sstable. These
> positions are used to filter out mutations from the commit log on restart and
> only make sense for the node on which the data was flushed.
> If an SSTable is moved between nodes they may cover regions that the
> receiving node has not yet flushed, and result in valid data being lost
> should these sections of the commit log need to be replayed.
> Solution:
> The chosen solution introduces a new sstable metadata (StatsMetadata) -
> originatingHostId (UUID), which is the local host id of the node on which the
> sstable was created, or null if not known. Commit log intervals from an
> sstable are taken into account during Commit Log replay only when the
> originatingHostId of the sstable matches the local node's hostId.
> For new sstables the originatingHostId is set according to StorageService's
> local hostId.
> For compacted sstables the originatingHostId set according to
> StorageService's local hostId, and only commit log intervals from local
> sstables is preserved in the resulting sstable.
> discovered by [~jakubzytka]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-16619) Loss of commit log data possible after sstable ingest
[
https://issues.apache.org/jira/browse/CASSANDRA-16619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17332479#comment-17332479
]
Benjamin Lerer edited comment on CASSANDRA-16619 at 4/26/21, 3:06 PM:
--
As {{4.0-rc1}} has been released we need to provide forward compatibility for
data. I do not think it is the case with the current patch. Users using
{{4.0-rc1}} will have an {{na}} format *without* hostID whereas when they will
deploy {{4.0-rc2}} they will have an {{na}} format *with* hostID which will
fail to read the {{na}} SSTables from the {{4.0-rc1}}. To fix that I think that
we need to introduce a new {{nb}} version.
Regarding the patches should refactor the {{MetadataSerializerTest.testXReadY}}
tests to ensure that they test all the possible combinations not only some of
them.
was (Author: blerer):
As {{4.0-rc1}} has been released by consequence we need to provide forward
compatibility for data. I do not think it is the case with the current patch.
Users using {{4.0-rc1}} will have an {{na}} format *without* hostID whereas
when they will deploy {{4.0-rc2}} they will have an {{na}} format *with* hostID
which will fail to read the {{na}} SSTables from the {{4.0-rc1}}. To fix that I
think that we need to introduce a new {{nb}} version.
Regarding the patches should refactor the {{MetadataSerializerTest.testXReadY}}
tests to ensure that they test all the possible combinations not only some of
them.
> Loss of commit log data possible after sstable ingest
> -
>
> Key: CASSANDRA-16619
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16619
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Commit Log
>Reporter: Jacek Lewandowski
>Assignee: Jacek Lewandowski
>Priority: Normal
> Fix For: 4.0, 3.0.x, 3.11.x
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> SSTable metadata contains commit log positions of the sstable. These
> positions are used to filter out mutations from the commit log on restart and
> only make sense for the node on which the data was flushed.
> If an SSTable is moved between nodes they may cover regions that the
> receiving node has not yet flushed, and result in valid data being lost
> should these sections of the commit log need to be replayed.
> Solution:
> The chosen solution introduces a new sstable metadata (StatsMetadata) -
> originatingHostId (UUID), which is the local host id of the node on which the
> sstable was created, or null if not known. Commit log intervals from an
> sstable are taken into account during Commit Log replay only when the
> originatingHostId of the sstable matches the local node's hostId.
> For new sstables the originatingHostId is set according to StorageService's
> local hostId.
> For compacted sstables the originatingHostId set according to
> StorageService's local hostId, and only commit log intervals from local
> sstables is preserved in the resulting sstable.
> discovered by [~jakubzytka]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-16619) Loss of commit log data possible after sstable ingest
[ https://issues.apache.org/jira/browse/CASSANDRA-16619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17326907#comment-17326907 ] Yifan Cai edited comment on CASSANDRA-16619 at 4/21/21, 9:41 PM: - ZCS streams a file as-is and w/o loading it into memory, hence fast. To remove a field metadata, a node needs to load the file into memory when receiving from remote. I think it is an expected behavior with ZCS. To distinguish, adding the original hostID in the metadata sounds valid. -- edit -- Talked with [~dcapwell] on slack. In the case of ZCS, the sstable metadata is updated after flushing the bytes. See [here.|https://github.com/apache/cassandra/blob/0fd8f0a52fbd69c47d073373abfe7d2437bbd9ca/src/java/org/apache/cassandra/db/streaming/CassandraEntireSSTableStreamReader.java#L142] Currently, it does not reset the commitLogInterval. But it is possible to just add a step to reset, and avoid updating the sstable format, i.e. add a new field. was (Author: yifanc): ZCS streams a file as-is and w/o loading it into memory, hence fast. To remove a field metadata, a node needs to load the file into memory when receiving from remote. I think it is an expected behavior with ZCS. To distinguish, adding the original hostID in the metadata sounds valid. > Loss of commit log data possible after sstable ingest > - > > Key: CASSANDRA-16619 > URL: https://issues.apache.org/jira/browse/CASSANDRA-16619 > Project: Cassandra > Issue Type: Bug > Components: Local/Commit Log >Reporter: Jacek Lewandowski >Assignee: Jacek Lewandowski >Priority: Normal > Fix For: 3.0.x, 3.11.x, 4.0.x > > Time Spent: 0.5h > Remaining Estimate: 0h > > SSTable metadata contains commit log positions of the sstable. These > positions are used to filter out mutations from the commit log on restart and > only make sense for the node on which the data was flushed. > If an SSTable is moved between nodes they may cover regions that the > receiving node has not yet flushed, and result in valid data being lost > should these sections of the commit log need to be replayed. > Solution: > The chosen solution introduces a new sstable metadata (StatsMetadata) - > originatingHostId (UUID), which is the local host id of the node on which the > sstable was created, or null if not known. Commit log intervals from an > sstable are taken into account during Commit Log replay only when the > originatingHostId of the sstable matches the local node's hostId. > For new sstables the originatingHostId is set according to StorageService's > local hostId. > For compacted sstables the originatingHostId set according to > StorageService's local hostId, and only commit log intervals from local > sstables is preserved in the resulting sstable. > discovered by [~jakubzytka] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-16619) Loss of commit log data possible after sstable ingest
[ https://issues.apache.org/jira/browse/CASSANDRA-16619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17326916#comment-17326916 ] Jeremiah Jordan edited comment on CASSANDRA-16619 at 4/21/21, 9:10 PM: --- [~dcapwell] this isn't just about ZCS. Any backup/restore process that copied an SSTable around is also affected. If a given node did not create the CommitLogPosition information then it needs to be ignored when loading the sstable. Hence the proposal to store the hostid in the metadata. If the hostid in the sstable doesn't match the hostid of the current node, then you can just ignore that erroneous information. This affects 3.x clusters as well, not just 4.x with ZCS. was (Author: jjordan): [~dcapwell] this isn't just about ZCS. Any backup/restore process that copied an SSTable around is also affected. If a given node did not create the CommitLogPosition information then it needs to be ignored when loading the sstable. > Loss of commit log data possible after sstable ingest > - > > Key: CASSANDRA-16619 > URL: https://issues.apache.org/jira/browse/CASSANDRA-16619 > Project: Cassandra > Issue Type: Bug > Components: Local/Commit Log >Reporter: Jacek Lewandowski >Assignee: Jacek Lewandowski >Priority: Normal > Fix For: 3.0.x, 3.11.x, 4.0.x > > Time Spent: 0.5h > Remaining Estimate: 0h > > SSTable metadata contains commit log positions of the sstable. These > positions are used to filter out mutations from the commit log on restart and > only make sense for the node on which the data was flushed. > If an SSTable is moved between nodes they may cover regions that the > receiving node has not yet flushed, and result in valid data being lost > should these sections of the commit log need to be replayed. > Solution: > The chosen solution introduces a new sstable metadata (StatsMetadata) - > originatingHostId (UUID), which is the local host id of the node on which the > sstable was created, or null if not known. Commit log intervals from an > sstable are taken into account during Commit Log replay only when the > originatingHostId of the sstable matches the local node's hostId. > For new sstables the originatingHostId is set according to StorageService's > local hostId. > For compacted sstables the originatingHostId set according to > StorageService's local hostId, and only commit log intervals from local > sstables is preserved in the resulting sstable. > discovered by [~jakubzytka] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-16619) Loss of commit log data possible after sstable ingest
[ https://issues.apache.org/jira/browse/CASSANDRA-16619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17326899#comment-17326899 ] David Capwell edited comment on CASSANDRA-16619 at 4/21/21, 8:34 PM: - that sounds like a bug in zero-copy streaming then, ideally we should strip that info out before adding the SSTables was (Author: dcapwell): that sounds like a bug in zero-copy streaming then, ideally we should strip that info out before adding > Loss of commit log data possible after sstable ingest > - > > Key: CASSANDRA-16619 > URL: https://issues.apache.org/jira/browse/CASSANDRA-16619 > Project: Cassandra > Issue Type: Bug > Components: Local/Commit Log >Reporter: Jacek Lewandowski >Assignee: Jacek Lewandowski >Priority: Normal > Fix For: 3.0.x, 3.11.x, 4.0.x > > Time Spent: 20m > Remaining Estimate: 0h > > SSTable metadata contains commit log positions of the sstable. These > positions are used to filter out mutations from the commit log on restart and > only make sense for the node on which the data was flushed. > If an SSTable is moved between nodes they may cover regions that the > receiving node has not yet flushed, and result in valid data being lost > should these sections of the commit log need to be replayed. > Solution: > The chosen solution introduces a new sstable metadata (StatsMetadata) - > originatingHostId (UUID), which is the local host id of the node on which the > sstable was created, or null if not known. Commit log intervals from an > sstable are taken into account during Commit Log replay only when the > originatingHostId of the sstable matches the local node's hostId. > For new sstables the originatingHostId is set according to StorageService's > local hostId. > For compacted sstables the originatingHostId set according to > StorageService's local hostId, and only commit log intervals from local > sstables is preserved in the resulting sstable. > discovered by [~jakubzytka] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
