[
https://issues.apache.org/jira/browse/CASSANDRA-13441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16006613#comment-16006613
]
Julius Žaromskis edited comment on CASSANDRA-13441 at 5/11/17 3:22 PM:
-----------------------------------------------------------------------
Hi, any workaround for this issue? I've hit this after upgrading from 3.0.9 to
3.0.13 and doing sstableupgrade. Noticed weird disk write patterns and started
seeing migration tasks bouncing around. I've only managed to update first of
the 3 nodes. Migrations tasks have stopped after I've rebooted first node.
{noformat}
Cluster Information:
Name: cloud.zaromskis.lt cluster
Snitch: org.apache.cassandra.locator.DynamicEndpointSnitch
Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
Schema versions:
600b7268-d42a-3b72-8706-093b6c8cfaff: [10.240.0.6]
77a40699-8e9e-35aa-834e-68c32e40a45a: [10.240.0.7, 10.240.0.8]
{noformat}
{noformat}
Datacenter: dc1
===============
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host ID
Rack
UN 10.240.0.6 284.95 GB 256 63.4%
d0d83d9d-0dec-45cd-9ca9-93515fa131f3 rack1
UN 10.240.0.7 288.53 GB 256 64.1%
6d9709a0-0e10-46a1-9afa-d106b74ca9e0 rack1
UN 10.240.0.8 326.31 GB 256 72.5%
5c969700-8bd9-49a4-9772-1284439f8364 rack1
{noformat}
The migrations are in fact executed, as I can see that on the disk, new files
are created every second in system keyspace. Why would cluster settle on the
same schema version then?
The schema version of first node would not propagate to other nodes. I'm afraid
further upgrades might create new schema versions? I can't afford to lose any
data. Any advise?
was (Author: juliuszaromskis):
Hi, any workaround for this issue? I've hit this after upgrading from 3.0.9 to
3.0.13 and doing sstableupgrade. Noticed weird disk write patterns and started
seeing migration tasks bouncing around. I've only managed to update first of
the 3 nodes. Migrations tasks have stopped after I've rebooted first node.
{noformat}
Cluster Information:
Name: cloud.zaromskis.lt cluster
Snitch: org.apache.cassandra.locator.DynamicEndpointSnitch
Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
Schema versions:
600b7268-d42a-3b72-8706-093b6c8cfaff: [10.240.0.6]
77a40699-8e9e-35aa-834e-68c32e40a45a: [10.240.0.7, 10.240.0.8]
{noformat}
{noformat}
Datacenter: dc1
===============
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host ID
Rack
UN 10.240.0.6 284.95 GB 256 63.4%
d0d83d9d-0dec-45cd-9ca9-93515fa131f3 rack1
UN 10.240.0.7 288.53 GB 256 64.1%
6d9709a0-0e10-46a1-9afa-d106b74ca9e0 rack1
UN 10.240.0.8 326.31 GB 256 72.5%
5c969700-8bd9-49a4-9772-1284439f8364 rack1
{noformat}
The schema version of first node would not propagate to other nodes. I'm afraid
further upgrades might create new schema versions? I can't afford to lose any
data. Any advise?
> Schema version changes for each upgraded node in a rolling upgrade, causing
> migration storms
> --------------------------------------------------------------------------------------------
>
> Key: CASSANDRA-13441
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13441
> Project: Cassandra
> Issue Type: Bug
> Components: Schema
> Reporter: Jeff Jirsa
> Assignee: Jeff Jirsa
> Fix For: 3.0.14, 3.11.0, 4.0
>
>
> In versions < 3.0, during a rolling upgrade (say 2.0 -> 2.1), the first node
> to upgrade to 2.1 would add the new tables, setting the new 2.1 version ID,
> and subsequently upgraded hosts would settle on that version.
> When a 3.0 node upgrades and writes its own new-in-3.0 system tables, it'll
> write the same tables that exist in the schema with brand new timestamps. As
> written, this will cause all nodes in the cluster to change schema (to the
> version with the newest timestamp). On a sufficiently large cluster with a
> non-trivial schema, this could cause (literally) millions of migration tasks
> to needlessly bounce across the cluster.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
