[jira] [Commented] (CASSANDRA-6862) Schema versions mismatch on large ring
[ https://issues.apache.org/jira/browse/CASSANDRA-6862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15279521#comment-15279521 ] Michael Fong commented on CASSANDRA-6862: - Hi, We have recently seen schema version mismatch issue while doing rolling upgrade from 1.2.19 to 2.0.17. What is worse is that the mismatch could lead to a rapid and massive message exchange of schema information across nodes, and sometimes may lead to node OOM. I opened a ticket regarding this at CASSANDRA-11748 Have you observed the heap over usage on exchanging schema information before? > Schema versions mismatch on large ring > -- > > Key: CASSANDRA-6862 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6862 > Project: Cassandra > Issue Type: Bug > Environment: 2.0 >Reporter: Oleg Anastasyev >Assignee: Oleg Anastasyev > Fix For: 2.0.7 > > Attachments: 6862-v2.txt, SchemaVersionLiveColumns.txt > > > We have a large cluster with several hundreds nodes in 1 ring. Sometimes, > especially after massive restarts, schema versions reported on different > nodes mismatch. > Investigated and found, that difference is not in schema itsself, but in > thombstones (different nodes have different set of thombstones applied to > system tables). > Fixed by digesting schema with thombstones removed first. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-6862) Schema versions mismatch on large ring
[
https://issues.apache.org/jira/browse/CASSANDRA-6862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14195950#comment-14195950
]
Aleksey Yeschenko commented on CASSANDRA-6862:
--
bq. Also I think you are saying that 2.0 code will have a schema change and
there will be a schema disagreement after first node is upgraded. But that will
settle after all machines have the new schema.
They won't. They will only settle on a single schema once every single node has
been upgraded - because we don't exchange schema (no push, no pull) between
nodes with different major versions - because of internal migrations like that
(if you try to push those new columns created from aliases to 1.2 nodes, your
1.2 node will not be able to handle it).
So this issue doesn't introduce the problem you are talking about - because
it's already there, since CASSANDRA-5125, which we aren't going to revert.
The only solution is to wait until CASSANDRA-6038 happens - after that, we'll
have schema push/pull/digests done in a separately versioned manner (and
instead of having a single current schema version, have a map {3.0 :
f8d559cd7fc723fc176e914d6d71c6f3, 3.1 : f6b405436ca4cb04fd2b4b7be3e8571e}).
Schema versions mismatch on large ring
--
Key: CASSANDRA-6862
URL: https://issues.apache.org/jira/browse/CASSANDRA-6862
Project: Cassandra
Issue Type: Bug
Components: Core
Environment: 2.0
Reporter: Oleg Anastasyev
Assignee: Oleg Anastasyev
Fix For: 2.0.7
Attachments: 6862-v2.txt, SchemaVersionLiveColumns.txt
We have a large cluster with several hundreds nodes in 1 ring. Sometimes,
especially after massive restarts, schema versions reported on different
nodes mismatch.
Investigated and found, that difference is not in schema itsself, but in
thombstones (different nodes have different set of thombstones applied to
system tables).
Fixed by digesting schema with thombstones removed first.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-6862) Schema versions mismatch on large ring
[ https://issues.apache.org/jira/browse/CASSANDRA-6862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14196449#comment-14196449 ] sankalp kohli commented on CASSANDRA-6862: -- Make sense. Thanks Schema versions mismatch on large ring -- Key: CASSANDRA-6862 URL: https://issues.apache.org/jira/browse/CASSANDRA-6862 Project: Cassandra Issue Type: Bug Components: Core Environment: 2.0 Reporter: Oleg Anastasyev Assignee: Oleg Anastasyev Fix For: 2.0.7 Attachments: 6862-v2.txt, SchemaVersionLiveColumns.txt We have a large cluster with several hundreds nodes in 1 ring. Sometimes, especially after massive restarts, schema versions reported on different nodes mismatch. Investigated and found, that difference is not in schema itsself, but in thombstones (different nodes have different set of thombstones applied to system tables). Fixed by digesting schema with thombstones removed first. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-6862) Schema versions mismatch on large ring
[ https://issues.apache.org/jira/browse/CASSANDRA-6862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14195242#comment-14195242 ] sankalp kohli commented on CASSANDRA-6862: -- Also, the proper fix here was probably to figure out why different nodes had differing tombstones, and fix that, instead of excluding them from digest altogether. Different nodes don't have differing tombstones. The problem is that 1.2 machines are adding the tombstones in its schema version and 2.0 are not. This will cause schema to never agree even with same tombstones in the ring. This is different from schema changes due to 1.2 to 2.0 upgrade which settles once it is propagated in the ring. Am I missing something? Schema versions mismatch on large ring -- Key: CASSANDRA-6862 URL: https://issues.apache.org/jira/browse/CASSANDRA-6862 Project: Cassandra Issue Type: Bug Components: Core Environment: 2.0 Reporter: Oleg Anastasyev Assignee: Oleg Anastasyev Fix For: 2.0.7 Attachments: 6862-v2.txt, SchemaVersionLiveColumns.txt We have a large cluster with several hundreds nodes in 1 ring. Sometimes, especially after massive restarts, schema versions reported on different nodes mismatch. Investigated and found, that difference is not in schema itsself, but in thombstones (different nodes have different set of thombstones applied to system tables). Fixed by digesting schema with thombstones removed first. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-6862) Schema versions mismatch on large ring
[ https://issues.apache.org/jira/browse/CASSANDRA-6862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14195265#comment-14195265 ] Aleksey Yeschenko commented on CASSANDRA-6862: -- One of us is, because I've lost the thread. What do you think the current issue is, under what circumstances does it happen, and why? Schema versions mismatch on large ring -- Key: CASSANDRA-6862 URL: https://issues.apache.org/jira/browse/CASSANDRA-6862 Project: Cassandra Issue Type: Bug Components: Core Environment: 2.0 Reporter: Oleg Anastasyev Assignee: Oleg Anastasyev Fix For: 2.0.7 Attachments: 6862-v2.txt, SchemaVersionLiveColumns.txt We have a large cluster with several hundreds nodes in 1 ring. Sometimes, especially after massive restarts, schema versions reported on different nodes mismatch. Investigated and found, that difference is not in schema itsself, but in thombstones (different nodes have different set of thombstones applied to system tables). Fixed by digesting schema with thombstones removed first. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-6862) Schema versions mismatch on large ring
[ https://issues.apache.org/jira/browse/CASSANDRA-6862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14195533#comment-14195533 ] sankalp kohli commented on CASSANDRA-6862: -- Say there are tombstones in the tables storing schema. They are all within GC grace(written in last 7 days) and is successfully replicated on all machines. You are upgrading a 1.2.19 cluster to 2.0.11. You upgrade one node to 2.0, the 2.0 node will compute the schema version without using tombstones. All 1.2 nodes are computing the schema version using tombstones. This will cause schema disagreement leading to schema pulls. Even after schema pull, things won't settle since schema versions will differ. This will happen till the entire cluster is upgraded to 2.0. For large clusters, this will be bad as it takes a long time to upgrade. So this is the problem. Also I think you are saying that 2.0 code will have a schema change and there will be a schema disagreement after first node is upgraded. But that will settle after all machines have the new schema. In this, schema will bounce around till all machines have been upgraded. Schema versions mismatch on large ring -- Key: CASSANDRA-6862 URL: https://issues.apache.org/jira/browse/CASSANDRA-6862 Project: Cassandra Issue Type: Bug Components: Core Environment: 2.0 Reporter: Oleg Anastasyev Assignee: Oleg Anastasyev Fix For: 2.0.7 Attachments: 6862-v2.txt, SchemaVersionLiveColumns.txt We have a large cluster with several hundreds nodes in 1 ring. Sometimes, especially after massive restarts, schema versions reported on different nodes mismatch. Investigated and found, that difference is not in schema itsself, but in thombstones (different nodes have different set of thombstones applied to system tables). Fixed by digesting schema with thombstones removed first. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-6862) Schema versions mismatch on large ring
[ https://issues.apache.org/jira/browse/CASSANDRA-6862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14193125#comment-14193125 ] Aleksey Yeschenko commented on CASSANDRA-6862: -- I'm not saying it's not a problem - just that this particular issue is irrelevant here. Because as soon as 1.2 node upgrades to 2.0 and starts up, we'll make change to its schema (conversion of aliases to proper columns) and schema versions will change anyway, tombstones or not. Schema versions mismatch on large ring -- Key: CASSANDRA-6862 URL: https://issues.apache.org/jira/browse/CASSANDRA-6862 Project: Cassandra Issue Type: Bug Components: Core Environment: 2.0 Reporter: Oleg Anastasyev Assignee: Oleg Anastasyev Fix For: 2.0.7 Attachments: 6862-v2.txt, SchemaVersionLiveColumns.txt We have a large cluster with several hundreds nodes in 1 ring. Sometimes, especially after massive restarts, schema versions reported on different nodes mismatch. Investigated and found, that difference is not in schema itsself, but in thombstones (different nodes have different set of thombstones applied to system tables). Fixed by digesting schema with thombstones removed first. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-6862) Schema versions mismatch on large ring
[ https://issues.apache.org/jira/browse/CASSANDRA-6862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14193127#comment-14193127 ] Aleksey Yeschenko commented on CASSANDRA-6862: -- Also, the proper fix here was probably to figure out why different nodes had differing tombstones, and fix that, instead of excluding them from digest altogether. Schema versions mismatch on large ring -- Key: CASSANDRA-6862 URL: https://issues.apache.org/jira/browse/CASSANDRA-6862 Project: Cassandra Issue Type: Bug Components: Core Environment: 2.0 Reporter: Oleg Anastasyev Assignee: Oleg Anastasyev Fix For: 2.0.7 Attachments: 6862-v2.txt, SchemaVersionLiveColumns.txt We have a large cluster with several hundreds nodes in 1 ring. Sometimes, especially after massive restarts, schema versions reported on different nodes mismatch. Investigated and found, that difference is not in schema itsself, but in thombstones (different nodes have different set of thombstones applied to system tables). Fixed by digesting schema with thombstones removed first. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-6862) Schema versions mismatch on large ring
[ https://issues.apache.org/jira/browse/CASSANDRA-6862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14192141#comment-14192141 ] sankalp kohli commented on CASSANDRA-6862: -- This will cause problems for 1.2 to 2.0 upgrade as 1.2 will add tombstones and 2.0 won't. Schema versions mismatch on large ring -- Key: CASSANDRA-6862 URL: https://issues.apache.org/jira/browse/CASSANDRA-6862 Project: Cassandra Issue Type: Bug Components: Core Environment: 2.0 Reporter: Oleg Anastasyev Assignee: Oleg Anastasyev Fix For: 2.0.7 Attachments: 6862-v2.txt, SchemaVersionLiveColumns.txt We have a large cluster with several hundreds nodes in 1 ring. Sometimes, especially after massive restarts, schema versions reported on different nodes mismatch. Investigated and found, that difference is not in schema itsself, but in thombstones (different nodes have different set of thombstones applied to system tables). Fixed by digesting schema with thombstones removed first. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-6862) Schema versions mismatch on large ring
[ https://issues.apache.org/jira/browse/CASSANDRA-6862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14192312#comment-14192312 ] Aleksey Yeschenko commented on CASSANDRA-6862: -- bq. This will cause problems for 1.2 to 2.0 upgrade as 1.2 will add tombstones and 2.0 won't. Which is completely irrelevant, because of the startup migration of CASSANDRA-5125 (see SystemKeyspace#copyAllAliasesToColumnsProper()) 3.0 to 3.1 upgrade will be the first to get rid of the issue entirely, making schema pulls, pushes, and digests fully flawless. Schema versions mismatch on large ring -- Key: CASSANDRA-6862 URL: https://issues.apache.org/jira/browse/CASSANDRA-6862 Project: Cassandra Issue Type: Bug Components: Core Environment: 2.0 Reporter: Oleg Anastasyev Assignee: Oleg Anastasyev Fix For: 2.0.7 Attachments: 6862-v2.txt, SchemaVersionLiveColumns.txt We have a large cluster with several hundreds nodes in 1 ring. Sometimes, especially after massive restarts, schema versions reported on different nodes mismatch. Investigated and found, that difference is not in schema itsself, but in thombstones (different nodes have different set of thombstones applied to system tables). Fixed by digesting schema with thombstones removed first. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-6862) Schema versions mismatch on large ring
[ https://issues.apache.org/jira/browse/CASSANDRA-6862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14192423#comment-14192423 ] sankalp kohli commented on CASSANDRA-6862: -- Why do you think it is not a problem from 1.2 to 2.0? Can you please elaborate? Schema versions mismatch on large ring -- Key: CASSANDRA-6862 URL: https://issues.apache.org/jira/browse/CASSANDRA-6862 Project: Cassandra Issue Type: Bug Components: Core Environment: 2.0 Reporter: Oleg Anastasyev Assignee: Oleg Anastasyev Fix For: 2.0.7 Attachments: 6862-v2.txt, SchemaVersionLiveColumns.txt We have a large cluster with several hundreds nodes in 1 ring. Sometimes, especially after massive restarts, schema versions reported on different nodes mismatch. Investigated and found, that difference is not in schema itsself, but in thombstones (different nodes have different set of thombstones applied to system tables). Fixed by digesting schema with thombstones removed first. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-6862) Schema versions mismatch on large ring
[ https://issues.apache.org/jira/browse/CASSANDRA-6862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14012922#comment-14012922 ] Robert Coli commented on CASSANDRA-6862: Does this affect versions before 2.0 as well? Or is since here 2.0.0, like CASSANDRA-6863? Schema versions mismatch on large ring -- Key: CASSANDRA-6862 URL: https://issues.apache.org/jira/browse/CASSANDRA-6862 Project: Cassandra Issue Type: Bug Components: Core Environment: 2.0 Reporter: Oleg Anastasyev Assignee: Oleg Anastasyev Fix For: 2.0.7 Attachments: 6862-v2.txt, SchemaVersionLiveColumns.txt We have a large cluster with several hundreds nodes in 1 ring. Sometimes, especially after massive restarts, schema versions reported on different nodes mismatch. Investigated and found, that difference is not in schema itsself, but in thombstones (different nodes have different set of thombstones applied to system tables). Fixed by digesting schema with thombstones removed first. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (CASSANDRA-6862) Schema versions mismatch on large ring
[ https://issues.apache.org/jira/browse/CASSANDRA-6862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13940217#comment-13940217 ] Jonathan Ellis commented on CASSANDRA-6862: --- committed, will follow up in 6863 Schema versions mismatch on large ring -- Key: CASSANDRA-6862 URL: https://issues.apache.org/jira/browse/CASSANDRA-6862 Project: Cassandra Issue Type: Bug Components: Core Environment: 2.0 Reporter: Oleg Anastasyev Assignee: Oleg Anastasyev Fix For: 2.0.7 Attachments: 6862-v2.txt, SchemaVersionLiveColumns.txt We have a large cluster with several hundreds nodes in 1 ring. Sometimes, especially after massive restarts, schema versions reported on different nodes mismatch. Investigated and found, that difference is not in schema itsself, but in thombstones (different nodes have different set of thombstones applied to system tables). Fixed by digesting schema with thombstones removed first. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (CASSANDRA-6862) Schema versions mismatch on large ring
[ https://issues.apache.org/jira/browse/CASSANDRA-6862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13939327#comment-13939327 ] Oleg Anastasyev commented on CASSANDRA-6862: agree. figured out the same an hur earlier. can fix this by the way in CASSANDRA-6863 or open a separate ticket about this ? Schema versions mismatch on large ring -- Key: CASSANDRA-6862 URL: https://issues.apache.org/jira/browse/CASSANDRA-6862 Project: Cassandra Issue Type: Bug Components: Core Environment: 2.0 Reporter: Oleg Anastasyev Assignee: Oleg Anastasyev Fix For: 2.0.7 Attachments: 6862-v2.txt, SchemaVersionLiveColumns.txt We have a large cluster with several hundreds nodes in 1 ring. Sometimes, especially after massive restarts, schema versions reported on different nodes mismatch. Investigated and found, that difference is not in schema itsself, but in thombstones (different nodes have different set of thombstones applied to system tables). Fixed by digesting schema with thombstones removed first. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (CASSANDRA-6862) Schema versions mismatch on large ring
[ https://issues.apache.org/jira/browse/CASSANDRA-6862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13939329#comment-13939329 ] Oleg Anastasyev commented on CASSANDRA-6862: patch LGTM Schema versions mismatch on large ring -- Key: CASSANDRA-6862 URL: https://issues.apache.org/jira/browse/CASSANDRA-6862 Project: Cassandra Issue Type: Bug Components: Core Environment: 2.0 Reporter: Oleg Anastasyev Assignee: Oleg Anastasyev Fix For: 2.0.7 Attachments: 6862-v2.txt, SchemaVersionLiveColumns.txt We have a large cluster with several hundreds nodes in 1 ring. Sometimes, especially after massive restarts, schema versions reported on different nodes mismatch. Investigated and found, that difference is not in schema itsself, but in thombstones (different nodes have different set of thombstones applied to system tables). Fixed by digesting schema with thombstones removed first. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (CASSANDRA-6862) Schema versions mismatch on large ring
[ https://issues.apache.org/jira/browse/CASSANDRA-6862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13937912#comment-13937912 ] Jonathan Ellis commented on CASSANDRA-6862: --- I think the right conclusion from that is we need to fix the digest to include row + range tombstones :) Schema versions mismatch on large ring -- Key: CASSANDRA-6862 URL: https://issues.apache.org/jira/browse/CASSANDRA-6862 Project: Cassandra Issue Type: Bug Components: Core Environment: 2.0 Reporter: Oleg Anastasyev Assignee: Oleg Anastasyev Fix For: 2.0.7 Attachments: 6862-v2.txt, SchemaVersionLiveColumns.txt We have a large cluster with several hundreds nodes in 1 ring. Sometimes, especially after massive restarts, schema versions reported on different nodes mismatch. Investigated and found, that difference is not in schema itsself, but in thombstones (different nodes have different set of thombstones applied to system tables). Fixed by digesting schema with thombstones removed first. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (CASSANDRA-6862) Schema versions mismatch on large ring
[ https://issues.apache.org/jira/browse/CASSANDRA-6862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13935542#comment-13935542 ] Jonathan Ellis commented on CASSANDRA-6862: --- Should probably also remove row-level and range tombstones? v2 attached. Schema versions mismatch on large ring -- Key: CASSANDRA-6862 URL: https://issues.apache.org/jira/browse/CASSANDRA-6862 Project: Cassandra Issue Type: Bug Components: Core Environment: 2.0 Reporter: Oleg Anastasyev Fix For: 2.0.7 Attachments: 6862-v2.txt, SchemaVersionLiveColumns.txt We have a large cluster with several hundreds nodes in 1 ring. Sometimes, especially after massive restarts, schema versions reported on different nodes mismatch. Investigated and found, that difference is not in schema itsself, but in thombstones (different nodes have different set of thombstones applied to system tables). Fixed by digesting schema with thombstones removed first. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (CASSANDRA-6862) Schema versions mismatch on large ring
[ https://issues.apache.org/jira/browse/CASSANDRA-6862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13935804#comment-13935804 ] Oleg Anastasyev commented on CASSANDRA-6862: LGTM. Oh those range tombstones, everybody forget about them...;-) Schema versions mismatch on large ring -- Key: CASSANDRA-6862 URL: https://issues.apache.org/jira/browse/CASSANDRA-6862 Project: Cassandra Issue Type: Bug Components: Core Environment: 2.0 Reporter: Oleg Anastasyev Assignee: Oleg Anastasyev Fix For: 2.0.7 Attachments: 6862-v2.txt, SchemaVersionLiveColumns.txt We have a large cluster with several hundreds nodes in 1 ring. Sometimes, especially after massive restarts, schema versions reported on different nodes mismatch. Investigated and found, that difference is not in schema itsself, but in thombstones (different nodes have different set of thombstones applied to system tables). Fixed by digesting schema with thombstones removed first. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (CASSANDRA-6862) Schema versions mismatch on large ring
[ https://issues.apache.org/jira/browse/CASSANDRA-6862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13935832#comment-13935832 ] Oleg Anastasyev commented on CASSANDRA-6862: row level and range tombstones does not participate in digest calculation AFAIK, so this is not neccessary Schema versions mismatch on large ring -- Key: CASSANDRA-6862 URL: https://issues.apache.org/jira/browse/CASSANDRA-6862 Project: Cassandra Issue Type: Bug Components: Core Environment: 2.0 Reporter: Oleg Anastasyev Assignee: Oleg Anastasyev Fix For: 2.0.7 Attachments: 6862-v2.txt, SchemaVersionLiveColumns.txt We have a large cluster with several hundreds nodes in 1 ring. Sometimes, especially after massive restarts, schema versions reported on different nodes mismatch. Investigated and found, that difference is not in schema itsself, but in thombstones (different nodes have different set of thombstones applied to system tables). Fixed by digesting schema with thombstones removed first. -- This message was sent by Atlassian JIRA (v6.2#6252)
