[jira] [Commented] (CASSANDRA-17180) Implement startup check to prevent Cassandra start to spread zombie data
[
https://issues.apache.org/jira/browse/CASSANDRA-17180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17524978#comment-17524978
]
Stefan Miklosovic commented on CASSANDRA-17180:
---
After spending more time on this, I identified an issue I am not sure how to
solve. I have not detected this by my unit tests because I was, more or less,
mocking it but once I actually tried it on the running node, to my surprise it
was not detecting the tables which should be causing violations.
The reason this is happening is that when I am about to read the gc grace
period from the table, I firstly iterate over all keyspaces like this:
{code}
Schema.instance.getUserKeyspaces()
{code}
and then on each such returned keyspace, I iterate over its tables, reading gc
grace period parameter.
The problem with this approach is that Schema.instance.getUserKeyspaces()
returns no keyspaces. The reason why it is happening is that by the time
startup checks are executed, it is way too early for this stuff to be fully
populated yet.
Interestingly enough, there is another check later, called
"checkSystemKeyspaceState" which is using this, which is not empty.
"Schema.instance.getTablesAndViews(SchemaConstants.SYSTEM_KEYSPACE_NAME)"
This internally translates to:
{code}
public Iterable getTablesAndViews(String keyspaceName)
{
Preconditions.checkNotNull(keyspaceName);
KeyspaceMetadata ksm = ObjectUtils.getFirstNonNull(() ->
distributedKeyspaces.getNullable(keyspaceName),
() ->
localKeyspaces.getNullable(keyspaceName));
Preconditions.checkNotNull(ksm, "Keyspace %s not found", keyspaceName);
return ksm.tablesAndViews();
}
{code}
So here, localKeyspaces are populated, but distributedKeyspaces are not. The
difference is that local keyspaces are populated directly in the constructor of
Schema like
{code}
private Schema()
{
this.online = isDaemonInitialized();
this.localKeyspaces = (FORCE_LOAD_LOCAL_KEYSPACES ||
isDaemonInitialized() || isToolInitialized())
? Keyspaces.of(SchemaKeyspace.metadata(),
SystemKeyspace.metadata())
: Keyspaces.none();
{code}
on the other hand, distributed keyspaces I am interested in seem to be
populated as the result of schema migrations which happen later after all
checks are run.
Other approaches like calling "Keyspace.nonSystem" or similar translate
internally to same "empty distributedKeyspaces" problem. I could manually parse
what is on the disk but then I am not sure how to get table metadata from it
and this approach does not seem robust enough anyway.
[~brandon.williams] [~paulo] do you have any idea how to get metadata
information so early in the booting sequence. If rendered impossible, I think
we have to completely abandon statup check approach and do it after schema is
initialised fully, but on the other hand, we can not do it too late to still be
able to somehow prevent the node from running.
> Implement startup check to prevent Cassandra start to spread zombie data
>
>
> Key: CASSANDRA-17180
> URL: https://issues.apache.org/jira/browse/CASSANDRA-17180
> Project: Cassandra
> Issue Type: New Feature
> Components: Legacy/Observability
>Reporter: Stefan Miklosovic
>Assignee: Stefan Miklosovic
>Priority: Normal
> Time Spent: 9.5h
> Remaining Estimate: 0h
>
> As already discussed on ML, it would be nice to have a service which would
> periodically write timestamp to a file signalling it is up / running.
> Then, on the startup, we would read this file and we would determine if there
> is some table which gc grace is behind this time and we would fail the start
> so we would prevent zombie data to be likely spread around a cluster.
> https://lists.apache.org/thread/w4w5t2hlcrvqhgdwww61hgg58qz13glw
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-17180) Implement startup check to prevent Cassandra start to spread zombie data
[ https://issues.apache.org/jira/browse/CASSANDRA-17180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17524999#comment-17524999 ] Stefan Miklosovic commented on CASSANDRA-17180: --- I think it is viable to do via "SchemaKeyspace.fetchNonSystemKeyspaces()". It will make a query and it will return Keyspaces with all metadata etc. However, I identified that this method has a bug in it. It will not return only user keyspaces but it will return also system_traces.events and system_traces.sessions so SchemaConstants#LOCAL_SYSTEM_KEYSPACE_NAMES is likely missing these two. I ll take a look. One more detail to mention is that SchemaKeyspace.fetchNonSystemKeyspaces is package protected, I had to make it public. There are other methods of the similar fashion which are private too. I am not sure I can make this method publicly visible without any conseqencies yet. > Implement startup check to prevent Cassandra start to spread zombie data > > > Key: CASSANDRA-17180 > URL: https://issues.apache.org/jira/browse/CASSANDRA-17180 > Project: Cassandra > Issue Type: New Feature > Components: Legacy/Observability >Reporter: Stefan Miklosovic >Assignee: Stefan Miklosovic >Priority: Normal > Time Spent: 9.5h > Remaining Estimate: 0h > > As already discussed on ML, it would be nice to have a service which would > periodically write timestamp to a file signalling it is up / running. > Then, on the startup, we would read this file and we would determine if there > is some table which gc grace is behind this time and we would fail the start > so we would prevent zombie data to be likely spread around a cluster. > https://lists.apache.org/thread/w4w5t2hlcrvqhgdwww61hgg58qz13glw -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-17180) Implement startup check to prevent Cassandra start to spread zombie data
[ https://issues.apache.org/jira/browse/CASSANDRA-17180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17525057#comment-17525057 ] Stefan Miklosovic commented on CASSANDRA-17180: --- Changes from the last review: 1) added prefix "check_" to checks in the configuration 2) updated documentation in cassandra.yaml 3) fixed retrieval of table metadata 4) ignoring tables which have gc = 0 5) moved heartbeating scheduling after checks are done 6) renamed default name of heartbeat file from ".cassandra-heartbeat" to "cassandra-heartbeat" (it is not hidden anymore). > Implement startup check to prevent Cassandra start to spread zombie data > > > Key: CASSANDRA-17180 > URL: https://issues.apache.org/jira/browse/CASSANDRA-17180 > Project: Cassandra > Issue Type: New Feature > Components: Legacy/Observability >Reporter: Stefan Miklosovic >Assignee: Stefan Miklosovic >Priority: Normal > Time Spent: 9.5h > Remaining Estimate: 0h > > As already discussed on ML, it would be nice to have a service which would > periodically write timestamp to a file signalling it is up / running. > Then, on the startup, we would read this file and we would determine if there > is some table which gc grace is behind this time and we would fail the start > so we would prevent zombie data to be likely spread around a cluster. > https://lists.apache.org/thread/w4w5t2hlcrvqhgdwww61hgg58qz13glw -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-17180) Implement startup check to prevent Cassandra start to spread zombie data
[ https://issues.apache.org/jira/browse/CASSANDRA-17180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17525316#comment-17525316 ] Paulo Motta commented on CASSANDRA-17180: - I cannot review this today but I *hope* to follow-up this by tomorrow or Friday if nobody else gets to it before. > Implement startup check to prevent Cassandra start to spread zombie data > > > Key: CASSANDRA-17180 > URL: https://issues.apache.org/jira/browse/CASSANDRA-17180 > Project: Cassandra > Issue Type: New Feature > Components: Legacy/Observability >Reporter: Stefan Miklosovic >Assignee: Stefan Miklosovic >Priority: Normal > Time Spent: 9.5h > Remaining Estimate: 0h > > As already discussed on ML, it would be nice to have a service which would > periodically write timestamp to a file signalling it is up / running. > Then, on the startup, we would read this file and we would determine if there > is some table which gc grace is behind this time and we would fail the start > so we would prevent zombie data to be likely spread around a cluster. > https://lists.apache.org/thread/w4w5t2hlcrvqhgdwww61hgg58qz13glw -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-17180) Implement startup check to prevent Cassandra start to spread zombie data
[
https://issues.apache.org/jira/browse/CASSANDRA-17180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17526105#comment-17526105
]
Paulo Motta commented on CASSANDRA-17180:
-
Thanks for addressing initial comments. Finally found some time to look into
this more deeply. Please find some follow-up comments below:
* I think safety checks should be enabled by default, as long as people can
disable it easily. Should we make this startup check enabled by default? We
could improve the error message when the check fails to mention the properties
to disable the check ({{startup_checks.check_data_resurrection.enabled=false}})
or ignore specific keyspace/tables ({{excluded_tables}}/{{excluded_keyspaces}})?
* I didn't like check-specific logic on CassandraDaemon to schedule the
heartbeat. I implemented this
[suggestion|https://github.com/apache/cassandra/commit/0b3557dd43255538942a86f63dec4c36272f25e9]
to move check post-action to StartupCheck class - what do you think?
* Can we rename {{GcGraceSecondsOnStartupCheck}} class to
{{CheckDataResurrection}} to be consistent with the check name ?
* Can we make the default heartbeat file be stored on the storage directory
(ie. {{DD.getLocalSystemKeyspacesDataFileLocations()}}) ? In some deployments
the cassandra directory is non-writable.
* I don't like adding [custom
logic|https://github.com/apache/cassandra/pull/1351/files#diff-f375982492d2426d26da68e105a44d397568be76361e8156fe299e875b8041ffR214]
to read/write the hearbeat file - since this is error-prone and we're just
interested in the timestamp value and not the file format. Can we just use
[File.setLastModified|https://docs.oracle.com/javase/7/docs/api/java/io/File.html#setLastModified(long)]
and
[File.lastModified|https://docs.oracle.com/javase/7/docs/api/java/io/File.html#lastModified()]
to read/write the heartbeat instead?
> Implement startup check to prevent Cassandra start to spread zombie data
>
>
> Key: CASSANDRA-17180
> URL: https://issues.apache.org/jira/browse/CASSANDRA-17180
> Project: Cassandra
> Issue Type: New Feature
> Components: Legacy/Observability
>Reporter: Stefan Miklosovic
>Assignee: Stefan Miklosovic
>Priority: Normal
> Time Spent: 9.5h
> Remaining Estimate: 0h
>
> As already discussed on ML, it would be nice to have a service which would
> periodically write timestamp to a file signalling it is up / running.
> Then, on the startup, we would read this file and we would determine if there
> is some table which gc grace is behind this time and we would fail the start
> so we would prevent zombie data to be likely spread around a cluster.
> https://lists.apache.org/thread/w4w5t2hlcrvqhgdwww61hgg58qz13glw
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-17180) Implement startup check to prevent Cassandra start to spread zombie data
[ https://issues.apache.org/jira/browse/CASSANDRA-17180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17526115#comment-17526115 ] Paulo Motta commented on CASSANDRA-17180: - bq. After spending more time on this, I identified an issue Nice catch! bq. I have not detected this by my unit tests because I was, more or less, mocking it but once I actually tried it on the running node, to my surprise it was not detecting the tables which should be causing violations. Can we create a (in-jvm or python) dtest to ensure this is being properly tested and any future regressions caught? bq. I think it is viable to do via "SchemaKeyspace.fetchNonSystemKeyspaces()". Sounds good to me. bq. I am not sure I can make this method publicly visible without any conseqencies yet. I think this should be fine. bq. On the other hand, it will check tables in "system_distributed" as well as "system_auth". These tables do not have gc = 0 and they are not excluded from fetchNonSystemKeyspaces call. that's ok, it's probably a good idea to check these tables anyway. > Implement startup check to prevent Cassandra start to spread zombie data > > > Key: CASSANDRA-17180 > URL: https://issues.apache.org/jira/browse/CASSANDRA-17180 > Project: Cassandra > Issue Type: New Feature > Components: Legacy/Observability >Reporter: Stefan Miklosovic >Assignee: Stefan Miklosovic >Priority: Normal > Time Spent: 9.5h > Remaining Estimate: 0h > > As already discussed on ML, it would be nice to have a service which would > periodically write timestamp to a file signalling it is up / running. > Then, on the startup, we would read this file and we would determine if there > is some table which gc grace is behind this time and we would fail the start > so we would prevent zombie data to be likely spread around a cluster. > https://lists.apache.org/thread/w4w5t2hlcrvqhgdwww61hgg58qz13glw -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-17180) Implement startup check to prevent Cassandra start to spread zombie data
[
https://issues.apache.org/jira/browse/CASSANDRA-17180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17526431#comment-17526431
]
Paulo Motta commented on CASSANDRA-17180:
-
{quote}Can we just use File.setLastModified and File.lastModified to read/write
the heartbeat instead?
{quote}
alternatively we can just write a JSON similar to the snapshot manifest, since
we can use existing JSON utilities to read/write the hearbeat file without
needing to implement a custom parser. something like this:
{noformat}
{"last_heartbeat": "2022-04-22T13:33:41Z"}
{noformat}
we could later augment this json with more info if the need arises.
WDYT?
> Implement startup check to prevent Cassandra start to spread zombie data
>
>
> Key: CASSANDRA-17180
> URL: https://issues.apache.org/jira/browse/CASSANDRA-17180
> Project: Cassandra
> Issue Type: New Feature
> Components: Legacy/Observability
>Reporter: Stefan Miklosovic
>Assignee: Stefan Miklosovic
>Priority: Normal
> Time Spent: 9.5h
> Remaining Estimate: 0h
>
> As already discussed on ML, it would be nice to have a service which would
> periodically write timestamp to a file signalling it is up / running.
> Then, on the startup, we would read this file and we would determine if there
> is some table which gc grace is behind this time and we would fail the start
> so we would prevent zombie data to be likely spread around a cluster.
> https://lists.apache.org/thread/w4w5t2hlcrvqhgdwww61hgg58qz13glw
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-17180) Implement startup check to prevent Cassandra start to spread zombie data
[ https://issues.apache.org/jira/browse/CASSANDRA-17180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17526440#comment-17526440 ] Brandon Williams commented on CASSANDRA-17180: -- That sounds like a great idea to me. > Implement startup check to prevent Cassandra start to spread zombie data > > > Key: CASSANDRA-17180 > URL: https://issues.apache.org/jira/browse/CASSANDRA-17180 > Project: Cassandra > Issue Type: New Feature > Components: Legacy/Observability >Reporter: Stefan Miklosovic >Assignee: Stefan Miklosovic >Priority: Normal > Time Spent: 9.5h > Remaining Estimate: 0h > > As already discussed on ML, it would be nice to have a service which would > periodically write timestamp to a file signalling it is up / running. > Then, on the startup, we would read this file and we would determine if there > is some table which gc grace is behind this time and we would fail the start > so we would prevent zombie data to be likely spread around a cluster. > https://lists.apache.org/thread/w4w5t2hlcrvqhgdwww61hgg58qz13glw -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-17180) Implement startup check to prevent Cassandra start to spread zombie data
[
https://issues.apache.org/jira/browse/CASSANDRA-17180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17526452#comment-17526452
]
Stefan Miklosovic commented on CASSANDRA-17180:
---
[~paulo] thanks for finally looking into it, I ll deal with it over the weekend
to finally move this over the line.
I had implemented something similar to your postActions idea but Brandon's
opinion was that we are inventing just something else here. But I see you moved
that "execute post actions loop" after all checks are verified in
CassandraDaemon instead of having it in StartupChecks.verify directly. I am
fine with your take on that, is Brandon too?
Good to know this is going to check system_distributed and system_auth too.
As for the default place of the heartbeat file, thats good point. Maybe we
should go a little bit wild here and we might save it to /tmp/ ? I think that
has the most guarantee of being writable. I do not like the fact that there is
suddenly some file in area for sstables / tables. Other existing software might
have a problem with this. For example when you are backuping, you would need to
what ... exclude or include that file? It depends how people look at these
backups etc. For that reason I would place it somewhere else. But if we
place it to /tmp, and you have more than one node running on the same machine,
there will be the clash as two nodes happen to write to the same file {_}by
default{_}. In that case we would have to make that file name unique, e.g. by
including node's id. What is your take on this?
Yes we can rename that class.
I do not mind to start to write JSON into that file, but ... how do you want to
parse that file? I still need to read it / check it and so on. By what you
would like to replace all that logic?
> Implement startup check to prevent Cassandra start to spread zombie data
>
>
> Key: CASSANDRA-17180
> URL: https://issues.apache.org/jira/browse/CASSANDRA-17180
> Project: Cassandra
> Issue Type: New Feature
> Components: Legacy/Observability
>Reporter: Stefan Miklosovic
>Assignee: Stefan Miklosovic
>Priority: Normal
> Time Spent: 9.5h
> Remaining Estimate: 0h
>
> As already discussed on ML, it would be nice to have a service which would
> periodically write timestamp to a file signalling it is up / running.
> Then, on the startup, we would read this file and we would determine if there
> is some table which gc grace is behind this time and we would fail the start
> so we would prevent zombie data to be likely spread around a cluster.
> https://lists.apache.org/thread/w4w5t2hlcrvqhgdwww61hgg58qz13glw
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-17180) Implement startup check to prevent Cassandra start to spread zombie data
[ https://issues.apache.org/jira/browse/CASSANDRA-17180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17526459#comment-17526459 ] Brandon Williams commented on CASSANDRA-17180: -- bq. I am fine with your take on that, is Brandon too? I'm fine with the majority here. bq. Maybe we should go a little bit wild here and we might save it to /tmp/ ? That sounds unworkable to me. There's no guarantee of durability until the next startup, people override tmpdir, etc. bq. I do not mind to start to write JSON into that file, but ... how do you want to parse that file? Can't we parse it like any other JSON file? > Implement startup check to prevent Cassandra start to spread zombie data > > > Key: CASSANDRA-17180 > URL: https://issues.apache.org/jira/browse/CASSANDRA-17180 > Project: Cassandra > Issue Type: New Feature > Components: Legacy/Observability >Reporter: Stefan Miklosovic >Assignee: Stefan Miklosovic >Priority: Normal > Time Spent: 9.5h > Remaining Estimate: 0h > > As already discussed on ML, it would be nice to have a service which would > periodically write timestamp to a file signalling it is up / running. > Then, on the startup, we would read this file and we would determine if there > is some table which gc grace is behind this time and we would fail the start > so we would prevent zombie data to be likely spread around a cluster. > https://lists.apache.org/thread/w4w5t2hlcrvqhgdwww61hgg58qz13glw -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-17180) Implement startup check to prevent Cassandra start to spread zombie data
[ https://issues.apache.org/jira/browse/CASSANDRA-17180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17526463#comment-17526463 ] Stefan Miklosovic commented on CASSANDRA-17180: --- Yes, I realised that /tmp/ problem just now ... ahh. > Can't we parse it like any other JSON file? Ah right, I know what you mean. So the two outstanding questions is the default place of this file and if it should be enabled by default. > Implement startup check to prevent Cassandra start to spread zombie data > > > Key: CASSANDRA-17180 > URL: https://issues.apache.org/jira/browse/CASSANDRA-17180 > Project: Cassandra > Issue Type: New Feature > Components: Legacy/Observability >Reporter: Stefan Miklosovic >Assignee: Stefan Miklosovic >Priority: Normal > Time Spent: 9.5h > Remaining Estimate: 0h > > As already discussed on ML, it would be nice to have a service which would > periodically write timestamp to a file signalling it is up / running. > Then, on the startup, we would read this file and we would determine if there > is some table which gc grace is behind this time and we would fail the start > so we would prevent zombie data to be likely spread around a cluster. > https://lists.apache.org/thread/w4w5t2hlcrvqhgdwww61hgg58qz13glw -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-17180) Implement startup check to prevent Cassandra start to spread zombie data
[ https://issues.apache.org/jira/browse/CASSANDRA-17180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17526467#comment-17526467 ] Stefan Miklosovic commented on CASSANDRA-17180: --- I think we need to really place it to a data dir because /tmp is not durable enough and other Cassandra dir might not be writable. The only durable & writable is data dir. > Implement startup check to prevent Cassandra start to spread zombie data > > > Key: CASSANDRA-17180 > URL: https://issues.apache.org/jira/browse/CASSANDRA-17180 > Project: Cassandra > Issue Type: New Feature > Components: Legacy/Observability >Reporter: Stefan Miklosovic >Assignee: Stefan Miklosovic >Priority: Normal > Time Spent: 9.5h > Remaining Estimate: 0h > > As already discussed on ML, it would be nice to have a service which would > periodically write timestamp to a file signalling it is up / running. > Then, on the startup, we would read this file and we would determine if there > is some table which gc grace is behind this time and we would fail the start > so we would prevent zombie data to be likely spread around a cluster. > https://lists.apache.org/thread/w4w5t2hlcrvqhgdwww61hgg58qz13glw -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-17180) Implement startup check to prevent Cassandra start to spread zombie data
[ https://issues.apache.org/jira/browse/CASSANDRA-17180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17526470#comment-17526470 ] Brandon Williams commented on CASSANDRA-17180: -- Data dir in the system keyspace makes the most sense to me. The system ks is generally not backed up/restored since that's a bad idea, so if the file is there is won't accidentally be restored and cause a problem. > Implement startup check to prevent Cassandra start to spread zombie data > > > Key: CASSANDRA-17180 > URL: https://issues.apache.org/jira/browse/CASSANDRA-17180 > Project: Cassandra > Issue Type: New Feature > Components: Legacy/Observability >Reporter: Stefan Miklosovic >Assignee: Stefan Miklosovic >Priority: Normal > Time Spent: 9.5h > Remaining Estimate: 0h > > As already discussed on ML, it would be nice to have a service which would > periodically write timestamp to a file signalling it is up / running. > Then, on the startup, we would read this file and we would determine if there > is some table which gc grace is behind this time and we would fail the start > so we would prevent zombie data to be likely spread around a cluster. > https://lists.apache.org/thread/w4w5t2hlcrvqhgdwww61hgg58qz13glw -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-17180) Implement startup check to prevent Cassandra start to spread zombie data
[ https://issues.apache.org/jira/browse/CASSANDRA-17180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17527698#comment-17527698 ] Stefan Miklosovic commented on CASSANDRA-17180: --- j8 [https://app.circleci.com/pipelines/github/instaclustr/cassandra/941/workflows/e9d5a89c-78a6-42ef-96e3-cd32dcacf47d] pr [https://github.com/apache/cassandra/pull/1351/commits] j8_jvm_dtests_vnode build has failed because it timeouted as I somehow set parallelism to 1, I ll run it on commit, does matter, good enough to review, I ll add java 11 to as well. Heartbeat json stuff is based on what we did with SnapshotManifest, same staff, copy-paste basically. this is "enabled by default" branch: [https://github.com/instaclustr/cassandra/tree/CASSANDRA-17180-enabled-by-default] It is on par with the other one, just one commit on top to make it enabled by default. > Implement startup check to prevent Cassandra start to spread zombie data > > > Key: CASSANDRA-17180 > URL: https://issues.apache.org/jira/browse/CASSANDRA-17180 > Project: Cassandra > Issue Type: New Feature > Components: Legacy/Observability >Reporter: Stefan Miklosovic >Assignee: Stefan Miklosovic >Priority: Normal > Fix For: 4.1 > > Time Spent: 9.5h > Remaining Estimate: 0h > > As already discussed on ML, it would be nice to have a service which would > periodically write timestamp to a file signalling it is up / running. > Then, on the startup, we would read this file and we would determine if there > is some table which gc grace is behind this time and we would fail the start > so we would prevent zombie data to be likely spread around a cluster. > https://lists.apache.org/thread/w4w5t2hlcrvqhgdwww61hgg58qz13glw -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-17180) Implement startup check to prevent Cassandra start to spread zombie data
[ https://issues.apache.org/jira/browse/CASSANDRA-17180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17528103#comment-17528103 ] Stefan Miklosovic commented on CASSANDRA-17180: --- I was thinking about enablement by default and I do not want to enable it by default yet. Let's just live with this for a while and gather more feedback from the field first. What I am saying is that let's mature it first a bit before crossing that bridge. > Implement startup check to prevent Cassandra start to spread zombie data > > > Key: CASSANDRA-17180 > URL: https://issues.apache.org/jira/browse/CASSANDRA-17180 > Project: Cassandra > Issue Type: New Feature > Components: Legacy/Observability >Reporter: Stefan Miklosovic >Assignee: Stefan Miklosovic >Priority: Normal > Fix For: 4.1 > > Time Spent: 11h 20m > Remaining Estimate: 0h > > As already discussed on ML, it would be nice to have a service which would > periodically write timestamp to a file signalling it is up / running. > Then, on the startup, we would read this file and we would determine if there > is some table which gc grace is behind this time and we would fail the start > so we would prevent zombie data to be likely spread around a cluster. > https://lists.apache.org/thread/w4w5t2hlcrvqhgdwww61hgg58qz13glw -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-17180) Implement startup check to prevent Cassandra start to spread zombie data
[
https://issues.apache.org/jira/browse/CASSANDRA-17180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17528907#comment-17528907
]
Josh McKenzie commented on CASSANDRA-17180:
---
{quote}I was thinking about enablement by default and I do not want to enable
it by default yet.
{quote}
My .02: I agree w/you re: keeping this disabled by default.
Ultimately a marker file that we use to parse if a node did anything within
gc_grace is all just a symptom of a bigger problem: us relying on gc_grace to
determine a window of time after which it's safe to delete tombstones. The
fundamental design is deeply, _deeply_ flawed. While this ticket here will
provide an optional band-aid for operators for certain use-cases to prevent
zombie data, I'd like to see us tackle this design in a more holistic fashion
that makes tickets like this unnecessary and makes purging more deterministic.
Been on my mind for a bit and I planned to bring it up on the ML after 4.1
branch and test burn down.
> Implement startup check to prevent Cassandra start to spread zombie data
>
>
> Key: CASSANDRA-17180
> URL: https://issues.apache.org/jira/browse/CASSANDRA-17180
> Project: Cassandra
> Issue Type: New Feature
> Components: Legacy/Observability
>Reporter: Stefan Miklosovic
>Assignee: Stefan Miklosovic
>Priority: Normal
> Fix For: 4.1
>
> Time Spent: 11h 20m
> Remaining Estimate: 0h
>
> As already discussed on ML, it would be nice to have a service which would
> periodically write timestamp to a file signalling it is up / running.
> Then, on the startup, we would read this file and we would determine if there
> is some table which gc grace is behind this time and we would fail the start
> so we would prevent zombie data to be likely spread around a cluster.
> https://lists.apache.org/thread/w4w5t2hlcrvqhgdwww61hgg58qz13glw
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-17180) Implement startup check to prevent Cassandra start to spread zombie data
[ https://issues.apache.org/jira/browse/CASSANDRA-17180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17529121#comment-17529121 ] Josh McKenzie commented on CASSANDRA-17180: --- No criticism of this work, just asking that we leave it disabled by default. > Implement startup check to prevent Cassandra start to spread zombie data > > > Key: CASSANDRA-17180 > URL: https://issues.apache.org/jira/browse/CASSANDRA-17180 > Project: Cassandra > Issue Type: New Feature > Components: Legacy/Observability >Reporter: Stefan Miklosovic >Assignee: Stefan Miklosovic >Priority: Normal > Fix For: 4.1 > > Time Spent: 11h 20m > Remaining Estimate: 0h > > As already discussed on ML, it would be nice to have a service which would > periodically write timestamp to a file signalling it is up / running. > Then, on the startup, we would read this file and we would determine if there > is some table which gc grace is behind this time and we would fail the start > so we would prevent zombie data to be likely spread around a cluster. > https://lists.apache.org/thread/w4w5t2hlcrvqhgdwww61hgg58qz13glw -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-17180) Implement startup check to prevent Cassandra start to spread zombie data
[ https://issues.apache.org/jira/browse/CASSANDRA-17180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17529253#comment-17529253 ] Stefan Miklosovic commented on CASSANDRA-17180: --- That is great to hear. So, all I am waiting for is the formal +1 from either of reviewers. Since I addressed all recent comments / ideas in the last review, dotted my i's and crossed my t's, I do not know how else I could improve this patch. The build is there as well. As we are close to 1st May, I would really appreciate if we resolved this till Friday 12pm CEST (cca 6pm EST) as weekends are for ... doing nothing. > Implement startup check to prevent Cassandra start to spread zombie data > > > Key: CASSANDRA-17180 > URL: https://issues.apache.org/jira/browse/CASSANDRA-17180 > Project: Cassandra > Issue Type: New Feature > Components: Legacy/Observability >Reporter: Stefan Miklosovic >Assignee: Stefan Miklosovic >Priority: Normal > Fix For: 4.1 > > Time Spent: 11h 20m > Remaining Estimate: 0h > > As already discussed on ML, it would be nice to have a service which would > periodically write timestamp to a file signalling it is up / running. > Then, on the startup, we would read this file and we would determine if there > is some table which gc grace is behind this time and we would fail the start > so we would prevent zombie data to be likely spread around a cluster. > https://lists.apache.org/thread/w4w5t2hlcrvqhgdwww61hgg58qz13glw -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-17180) Implement startup check to prevent Cassandra start to spread zombie data
[ https://issues.apache.org/jira/browse/CASSANDRA-17180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17529587#comment-17529587 ] Brandon Williams commented on CASSANDRA-17180: -- Left minor comments but this is mostly ready to go. > Implement startup check to prevent Cassandra start to spread zombie data > > > Key: CASSANDRA-17180 > URL: https://issues.apache.org/jira/browse/CASSANDRA-17180 > Project: Cassandra > Issue Type: New Feature > Components: Legacy/Observability >Reporter: Stefan Miklosovic >Assignee: Stefan Miklosovic >Priority: Normal > Fix For: 4.1 > > Time Spent: 11.5h > Remaining Estimate: 0h > > As already discussed on ML, it would be nice to have a service which would > periodically write timestamp to a file signalling it is up / running. > Then, on the startup, we would read this file and we would determine if there > is some table which gc grace is behind this time and we would fail the start > so we would prevent zombie data to be likely spread around a cluster. > https://lists.apache.org/thread/w4w5t2hlcrvqhgdwww61hgg58qz13glw -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-17180) Implement startup check to prevent Cassandra start to spread zombie data
[ https://issues.apache.org/jira/browse/CASSANDRA-17180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17529588#comment-17529588 ] Paulo Motta commented on CASSANDRA-17180: - I plan to take a final look at this today. > Implement startup check to prevent Cassandra start to spread zombie data > > > Key: CASSANDRA-17180 > URL: https://issues.apache.org/jira/browse/CASSANDRA-17180 > Project: Cassandra > Issue Type: New Feature > Components: Legacy/Observability >Reporter: Stefan Miklosovic >Assignee: Stefan Miklosovic >Priority: Normal > Fix For: 4.1 > > Time Spent: 11.5h > Remaining Estimate: 0h > > As already discussed on ML, it would be nice to have a service which would > periodically write timestamp to a file signalling it is up / running. > Then, on the startup, we would read this file and we would determine if there > is some table which gc grace is behind this time and we would fail the start > so we would prevent zombie data to be likely spread around a cluster. > https://lists.apache.org/thread/w4w5t2hlcrvqhgdwww61hgg58qz13glw -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-17180) Implement startup check to prevent Cassandra start to spread zombie data
[
https://issues.apache.org/jira/browse/CASSANDRA-17180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17529711#comment-17529711
]
Paulo Motta commented on CASSANDRA-17180:
-
Added a few minor nit comments on the PR.
While testing this locally, I noticed that the heartbeat timestamp was being
written as:
{noformat}
{"last_heartbeat":1651195035.63800}%
{noformat}
This problem also happens with the snapshot manifest. I created [this
commit|https://github.com/apache/cassandra/commit/f5a4e7345501c47593d723f0224c28e26ebe7b64]
to fix this and unify json parsing on {{{}FBUtilities{}}}. Can you check this
and incorporate if it looks good?
After fixing this the heartbeat is written as which is iso8601:
{noformat}
{"last_heartbeat":"2022-04-29T01:25:22.906Z"}%
{noformat}
Got this output when testing manually, so working as expected :
{noformat}
ERROR [main] 2022-04-28 22:18:47,017 CassandraDaemon.java:896 - There are
tables for which gcGraceSeconds is older then the lastly known time Cassandra
node was up based on its heartbeat
/user/.ccm/test/node1/data0/cassandra-heartbeat with timestamp
2022-04-29T01:17:15.638Z. Cassandra node will not start as it would likely
introduce data consistency issues (zombies etc). Please resolve these issues
manually, then remove the heartbeat and start the node again. Invalid tables:
ks.indexed_table
{noformat}
Feel free to commit this after addressing outstanding comments + green CI.
> Implement startup check to prevent Cassandra start to spread zombie data
>
>
> Key: CASSANDRA-17180
> URL: https://issues.apache.org/jira/browse/CASSANDRA-17180
> Project: Cassandra
> Issue Type: New Feature
> Components: Legacy/Observability
>Reporter: Stefan Miklosovic
>Assignee: Stefan Miklosovic
>Priority: Normal
> Fix For: 4.1
>
> Time Spent: 12h
> Remaining Estimate: 0h
>
> As already discussed on ML, it would be nice to have a service which would
> periodically write timestamp to a file signalling it is up / running.
> Then, on the startup, we would read this file and we would determine if there
> is some table which gc grace is behind this time and we would fail the start
> so we would prevent zombie data to be likely spread around a cluster.
> https://lists.apache.org/thread/w4w5t2hlcrvqhgdwww61hgg58qz13glw
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-17180) Implement startup check to prevent Cassandra start to spread zombie data
[ https://issues.apache.org/jira/browse/CASSANDRA-17180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17529843#comment-17529843 ] Stefan Miklosovic commented on CASSANDRA-17180: --- Thanks both for the review. I noticed we are duplicating the object mapper as it was in snapshots as well but I wanted to address this later to take a look what other object mappers we use elsewhere and I would unify it in next ticket. I checked it and besides these object mappers in snapshot manifest and heartbeat file, there is one more in org.apache.cassandra.cql3.Json. That object mapper is reading only, it does not write anything, so it is nicely replaceable by this one unified object mapper where we only changed serialisation of Instant which does not clash with how object mapper in JSON is deserialising stuff. However, I would really leave this out as it is out of the scope of this ticket and might be addressed later. Anyway, I incorporated the changes of Paulo to have unification done at least on these two mappers. > Implement startup check to prevent Cassandra start to spread zombie data > > > Key: CASSANDRA-17180 > URL: https://issues.apache.org/jira/browse/CASSANDRA-17180 > Project: Cassandra > Issue Type: New Feature > Components: Legacy/Observability >Reporter: Stefan Miklosovic >Assignee: Stefan Miklosovic >Priority: Normal > Fix For: 4.1 > > Time Spent: 12h 10m > Remaining Estimate: 0h > > As already discussed on ML, it would be nice to have a service which would > periodically write timestamp to a file signalling it is up / running. > Then, on the startup, we would read this file and we would determine if there > is some table which gc grace is behind this time and we would fail the start > so we would prevent zombie data to be likely spread around a cluster. > https://lists.apache.org/thread/w4w5t2hlcrvqhgdwww61hgg58qz13glw -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-17180) Implement startup check to prevent Cassandra start to spread zombie data
[ https://issues.apache.org/jira/browse/CASSANDRA-17180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17529919#comment-17529919 ] Stefan Miklosovic commented on CASSANDRA-17180: --- Thanks [~paulo] for the effort even we had some rough period here. Really appreciate it. Maybe planning the work more in advance would be better for everybody to avoid deadlines like this. I would rather finish it a week earlier just to have some peace in mind. > Implement startup check to prevent Cassandra start to spread zombie data > > > Key: CASSANDRA-17180 > URL: https://issues.apache.org/jira/browse/CASSANDRA-17180 > Project: Cassandra > Issue Type: New Feature > Components: Legacy/Observability >Reporter: Stefan Miklosovic >Assignee: Stefan Miklosovic >Priority: Normal > Fix For: 4.1 > > Time Spent: 12h 10m > Remaining Estimate: 0h > > As already discussed on ML, it would be nice to have a service which would > periodically write timestamp to a file signalling it is up / running. > Then, on the startup, we would read this file and we would determine if there > is some table which gc grace is behind this time and we would fail the start > so we would prevent zombie data to be likely spread around a cluster. > https://lists.apache.org/thread/w4w5t2hlcrvqhgdwww61hgg58qz13glw -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
