1) You should not perform any streaming operations (repair, bootstrap,
decommission) in the middle of an upgrade. Note that an upgrade is not
complete until you have completed upgradesstables on all nodes in the
cluster.
2) No streaming involved with writes so it's not an issue.
3) It doesn't matter whichever way you do it. My personal experience is
that it's best to do it as you go but YMMV.
4) It depends on a lot of factors including type of disks (e.g. SSDs vs
HDDs), data model, access patterns, cluster load, etc. The only way you'll
be able to estimate it is by running your own tests.
5) There is no "max" time but it is preferable that you complete the
upgrades in the shortest amount of time. Until you have completed
upgradesstables on all nodes, there is a performance hit with reading older
generations of sstables. I'm sure you're about to ask "how much perf hit?"
and the answer is "test it".
6) It is not advisable to perform schema changes in mixed-mode -- the
schema version on upgraded nodes are different and there will be a mismatch
until all nodes are upgraded.
Good luck!
On Mon, Aug 14, 2017 at 12:14 PM, Park Wu <park...@yahoo.com.invalid> wrote:
> Hi, folks: I am planning to upgrade our production from dsc 2.0.16 to
> 2.1.18 for 2 DC (20 nodes each, 600GB per node). Few questions:
> 1), what happen when doing rolling upgrade. Let's say we only upgrade one
> node to new version, before upgrade sstable, the data coming in will stay
> in the node and not be able to stream to other nodes?
> 2), What if I have very active writes? how much data it can hold until it
> sees other nodes with new version so it can stream?
> 3), Should I upgrade sstable when all nodes in one DC upgraded? or wait
> until all 2 DC upgraded?
> 4), any idea or experience how long it will take to upgrade sstable for
> 600 GB data on each node?
> 5), what is the max time I can take for rolling upgrade on each DC?
> 6), I was doing a test with 3-nodes cluster, one node with 2.1.18, other
> two are 2.0.16. I got a warning on the node with newer version when I tried
> to create keyspace and insert some sample data:
> "Warning: schema version mismatch detected, which might be caused by DOWN
> nodes; if this is not the case, check the schema versions of your nodes in
> system.local and system.peers. OperationTimedOut: errors={},
> last_host=xxx"
> But data upserted successfully, even not be seen on other nodes. Any
> suggestion?
> Great thanks for any help or comments!
> - Park
>
