Thanks Sean. I have automation in place that can put the new binary and restart the node to a newer version as quickly as possible. upgradesstables is I/O intensive and it takes time and is proportional to the data on the node. Given these constraints, is there a risk due to prolonged upgradesstables?
On Tue, Dec 4, 2018 at 12:20 PM Durity, Sean R <sean_r_dur...@homedepot.com> wrote: > We have had great success with Cassandra upgrades with applications > staying on-line. It is one of the strongest benefits of Cassandra. A couple > things I incorporate into upgrades: > > - The main task is getting the new binaries loaded, then > restarting the node – in a rolling fashion. Get this done as quickly as > possible > > - Streaming between versions is usually problematic. So, I never > do any node additions or decommissions during an upgrade > > - With applications running, there is not an acceptable back-out > plan (either lose data or take a long outage or both), so we are always > going forward. So, lower life cycle testing is important before hitting > production > > - Upgrading is a more frequent activity, so get the > process/automation in place. The upgrade process should not be a reason to > delay, especially for minor version upgrades that might be quickly > necessary (security issue or bug fix). > > > > > > Sean Durity > > > > *From:* Shravan R <skr...@gmail.com> > *Sent:* Tuesday, December 04, 2018 12:22 PM > *To:* user@cassandra.apache.org > *Subject:* [EXTERNAL] Re: upgrade Apache Cassandra 2.1.9 to 3.0.9 > > > > Thanks Jeff. I tried to bootstrap a 3.x node to a partially upgraded > cluster (2.1.9 + 3.x) and I was *not* able to do so. The schema never > settled. > > > > How does the below approach sound like? > > 1. Update the software binary on all nodes to use cassandra-3.x upon a > restart. > 2. Restart all nodes in a rolling fashion > 3. Run nodetool upgradesstables in a rolling fashion > > > > Is there a risk on pending nodetool upgradesstables? > > > > On Sun, Dec 2, 2018 at 2:12 AM Jeff Jirsa <jji...@gmail.com> wrote: > > > > > On Dec 2, 2018, at 12:40 PM, Shravan R <skr...@gmail.com> wrote: > > Marc/Dimitry/Jon - greatly appreciate your feedback. I will look into the > version part that you suggested. The reason to go direct to 3.x is to take > a bi leap and reduce overall effort to upgrade a large cluster (development > included). > > > > I have these questions from my original post. Appreciate if you could shed > some light and point me in the right direction. > > > > 1) How do deal with decommissioning a 2.1.9 node in a partially upgraded > cluster? > > > > If any of the replicas have already upgraded, which is almost guaranteed > if you’re using vnodes, It’s hard / you don’t. You’d basically upgrade > everything else and then deal with it. If a host fails mid upgrade you’ll > likely have some period of unavailables while you bounce the replicas to > finish, then you can decom > > > > > > > > 2) How to bootstrap a 3.x node to a partially upgraded cluster? > > > > This may work fine, but test it because I’m not certain. It should be able > to read the 2.1 and 3.0 sstables that’ll stream so it’ll just work > > > > 3) Is there an alternative approach to the upgrade large clusters. i.e > instead of going through nodetool upgradesstables on each node in rolling > fashion > > > > Bounce them all as quickly as is practical, do the upgradesstables after > the bounces complete > > > > > > > > > > > > On Sat, Dec 1, 2018 at 1:03 PM Jonathan Haddad <j...@jonhaddad.com> wrote: > > Dmitry is right. Generally speaking always go with the latest bug fix > release. > > > > On Sat, Dec 1, 2018 at 10:14 AM Dmitry Saprykin <saprykin.dmi...@gmail.com> > wrote: > > See more here > > https://issues.apache.org/jira/plugins/servlet/mobile#issue/CASSANDRA-13004 > <https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_plugins_servlet_mobile-23issue_CASSANDRA-2D13004&d=DwMFaQ&c=MtgQEAMQGqekjTjiAhkudQ&r=aC_gxC6z_4f9GLlbWiKzHm1vucZTtVYWDDvyLkh8IaQ&m=--WtdKaRCohgTv7Y6px-TdcK2xJFB9oaDOSfdoBQ8D0&s=8csmPWgUEWao6E4wthrG_-BX5a2OQJKXpkKtFLjSPlI&e=> > > > > On Sat, Dec 1, 2018 at 1:02 PM Dmitry Saprykin <saprykin.dmi...@gmail.com> > wrote: > > Even more, 3.0.9 is a terrible target choice by itself. It has a nasty bug > corrupting sstables on alter. > > > > On Sat, Dec 1, 2018 at 11:55 AM Marc Selwan <marc.sel...@datastax.com> > wrote: > > Hi Shravan, > > > > Did you upgrade Apache Cassandra 2.1.9 to the latest patch release before > doing the major upgrade? It's generally favorable to go to the latest patch > release as often times they include fixes that smooth over the upgrade > process. There are hundreds of bug fixes between 2.1.9 and 2.1.20 (current > version) > > > > Best, > > Marc > > > > On Fri, Nov 30, 2018 at 3:13 PM Shravan R <skr...@gmail.com> wrote: > > Hello, > > > > I am planning to upgrade Apache Cassandra 2.1.9 to Apache Cassandra-3.0.9. > I came up with the version based on [1]. I followed upgrade steps as in > [2]. I was testing the same in the lab and encountered issues (streaming > just fails and hangs for ever) with bootstrapping a 3.0.9 node on a > partially upgraded cluster. [50% of nodes on 2.1.9 and 50% on 3.0.9]. The > production cluster that I am supporting is pretty large and I am > anticipating to end up in a situation like this (Hope not) and would like > to be prepared. > > > > 1) How do deal with decommissioning a 2.1.9 node in a partially upgraded > cluster? > > 2) How to bootstrap a 3.x node to a partially upgraded cluster? > > 3) Is there an alternative approach to the upgrade large clusters. i.e > instead of going through nodetool upgradesstables on each node in rolling > fashion > > > > > > As per [1] the general restriction is to avoid decommissioning or adding > nodes but in reality there can be failures or maintenance that warrants us > to do so. > > > > > > Please point me in the right direction. > > > > > > Thanks, > > Shravan > > > > > > [1] > https://docs.datastax.com/en/upgrade/doc/upgrade/datastax_enterprise/upgdDSE50.html#upgdDSE50__cstar-version-change > <https://urldefense.proofpoint.com/v2/url?u=https-3A__docs.datastax.com_en_upgrade_doc_upgrade_datastax-5Fenterprise_upgdDSE50.html-23upgdDSE50-5F-5Fcstar-2Dversion-2Dchange&d=DwMFaQ&c=MtgQEAMQGqekjTjiAhkudQ&r=aC_gxC6z_4f9GLlbWiKzHm1vucZTtVYWDDvyLkh8IaQ&m=--WtdKaRCohgTv7Y6px-TdcK2xJFB9oaDOSfdoBQ8D0&s=IV10uiJPS2fyCHGYw7yR90A8cxcpjy9Is40YUWgPvF0&e=> > > > > [2] > https://myopsblog.wordpress.com/2017/12/04/upgrade-cassandra-cluster-from-2-x-to-3-x/ > <https://urldefense.proofpoint.com/v2/url?u=https-3A__myopsblog.wordpress.com_2017_12_04_upgrade-2Dcassandra-2Dcluster-2Dfrom-2D2-2Dx-2Dto-2D3-2Dx_&d=DwMFaQ&c=adz96Xi0w1RHqtPMowiL2g&r=E6NVfMr2TIhW42QMfARTvsfCLtdF-oEA3KfAQRfVZdk&m=zbxL9Z9UjZMSVoHeue5w2ch4V1n65VR39w0_ysPWhBc&s=Ef6f6CfzIk0DBt3xD3fBmBhsfU8Yc2lv7YnIgiTWLMg&e=> > > > > -- > > Marc Selwan | DataStax | Product Management | (925) 413-7079 > > > > > > -- > > Jon Haddad > http://www.rustyrazorblade.com > <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.rustyrazorblade.com&d=DwMFaQ&c=MtgQEAMQGqekjTjiAhkudQ&r=aC_gxC6z_4f9GLlbWiKzHm1vucZTtVYWDDvyLkh8IaQ&m=--WtdKaRCohgTv7Y6px-TdcK2xJFB9oaDOSfdoBQ8D0&s=HPfRIl5knAn9tpUyT06P6YaktRTDwyFSMO7r4GHXBzY&e=> > twitter: rustyrazorblade > > > ------------------------------ > > The information in this Internet Email is confidential and may be legally > privileged. It is intended solely for the addressee. Access to this Email > by anyone else is unauthorized. If you are not the intended recipient, any > disclosure, copying, distribution or any action taken or omitted to be > taken in reliance on it, is prohibited and may be unlawful. When addressed > to our clients any opinions or advice contained in this Email are subject > to the terms and conditions expressed in any applicable governing The Home > Depot terms of business or client engagement letter. The Home Depot > disclaims all responsibility and liability for the accuracy and content of > this attachment and for any damages or losses arising from any > inaccuracies, errors, viruses, e.g., worms, trojan horses, etc., or other > items of a destructive nature, which may be contained in this attachment > and shall not be liable for direct, indirect, consequential or special > damages in connection with this e-mail message or its attachment. >
