Fwd: Re: How to gracefully decommission a highly loaded node?

[onmstester onmstester]

After a long time stuck in LEAVING, and "not doing any streams", i killed 
Cassandra process and restart it, then again ran nodetool decommission 
(Datastax recipe for stuck decommission), now it says, LEAVING, "unbootstrap 
$(the node id)" What's going on? Should i forget about decommission and just 
remove the node? There is an issue to make decommission resumable: 
https://issues.apache.org/jira/browse/CASSANDRA-12008 but i couldn't figure out 
how this suppose to work? I was expecting that after restarting 
stucked-decommission-cassandra, it resume the decommissioning process, but the 
node became UN after restart. Sent using Zoho Mail  Forwarded 
message  From : Simon Fontana Oscarsson 
 To : 
"user@cassandra.apache.org" Date : Tue, 04 Dec 2018 
15:20:15 +0330 Subject : Re: How to gracefully decommission a highly loaded 
node?  Forwarded message  Hi, If it already uses 100 % 
CPU I have a hard time seeing it being able to do a decomission while serving 
requests. If you have a lot of free space I would first try nodetool 
disableautocompaction. If you don't see any progress in nodetool netstats you 
can also disablebinary, disablethrift and disablehandoff to stop serving client 
requests.  -- SIMON FONTANA OSCARSSON
Software Developer

Ericsson
Ölandsgatan 1
37133 Karlskrona, Sweden
simon.fontana.oscars...@ericsson.com
www.ericsson.com On tis, 2018-12-04 at 14:21 +0330, onmstester onmstester 
wrote: One node suddenly uses 100% CPU, i suspect hardware problems and do not 
have time to trace that, so decided to just remove the node from the cluster, 
but although the node state changed to UL, but no sign of Leaving: the node is 
still compacting and flushing memtables, writing mutations and CPU is 100% for 
hours since. Is there any means to force a Cassandra Node to just decommission 
and stop doing normal things? Due to W.CL=ONE, i can not use removenode and 
shutdown the node Best Regards Sent using Zoho Mail

smime.p7s
Description: Binary data

-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org

Re: Migrating from DSE5.1.2 to Opensource cassandra

[dinesh.jo...@yahoo.com.INVALID]

Thanks, nice summary of the overall process.
Dinesh 

On Tuesday, December 4, 2018, 9:38:47 PM EST, Jonathan Koppenhofer 
 wrote:  
 
 Unfortunately, we found this to be a little tricky. We did migrations from DSE 
4.8 and 5.0 to OSS 3.0.x, so you may run into additional issues. I will also 
say your best option may be to install a fresh cluster and stream the data. 
This wasn't feasible for us at the size and scale in the time frames and 
infrastructure restrictions we had. I will have to review my notes for more 
detail, but off the top of my head, for an in place migration...
Pre-upgrade* Be sure you are not using any Enterprise features like Search or 
Graph. Not only are there not equivalent features in open source, but theses 
features require proprietary classes to be in the classpath, or Cassandra will 
not even start up.* By default, I think DSE uses their own custom 
authenticators, authorizors, and such. Make sure what you are doing has an open 
source equivalent.* The DSE system keyapaces use custom replication strategies. 
Convert these to NTS before upgrade.* Otherwise, follow the same processes you 
would do before an upgrade (repair, snapshot, etc)
Upgrade* The easy part is just replacing the binaries as you would in normal 
upgrade. Drain and stop the existing node first. You can also do this same 
process in a rolling fashion to maintain availability. In our case, we were 
doing an in-place upgrade and reusing the same IPs* DSE unfortunately creates a 
custom column in a system table that requires you to remove one (or more) 
system tables (peers?) to be able to start the node. You delete these system 
tables by  removing the sstbles on disk while the node is down. This is a bit 
of a headache if using vnodes. As we are using vnodes, it required us to 
manually specify num tokens, and the specific tokens the node was responsible 
for in Cassandra.yaml. You have to do this before you start the node. If not 
using vnodes, this is simpler, but we used vnodes. Again, I'll double check my 
notes. Once the node is up, you can revert to your normal vnodes/num tokens 
settings.
Post upgrade:* Drop DSE system tables.
I'll revert with more detail if needed.

On Tue, Dec 4, 2018, 5:46 PM Nandakishore Tokala 

Re: Migrating from DSE5.1.2 to Opensource cassandra

[Jonathan Koppenhofer]

Unfortunately, we found this to be a little tricky. We did migrations from
DSE 4.8 and 5.0 to OSS 3.0.x, so you may run into additional issues. I will
also say your best option may be to install a fresh cluster and stream the
data. This wasn't feasible for us at the size and scale in the time frames
and infrastructure restrictions we had. I will have to review my notes for
more detail, but off the top of my head, for an in place migration...

Pre-upgrade
* Be sure you are not using any Enterprise features like Search or Graph.
Not only are there not equivalent features in open source, but theses
features require proprietary classes to be in the classpath, or Cassandra
will not even start up.
* By default, I think DSE uses their own custom authenticators,
authorizors, and such. Make sure what you are doing has an open source
equivalent.
* The DSE system keyapaces use custom replication strategies. Convert these
to NTS before upgrade.
* Otherwise, follow the same processes you would do before an upgrade
(repair, snapshot, etc)

Upgrade
* The easy part is just replacing the binaries as you would in normal
upgrade. Drain and stop the existing node first. You can also do this same
process in a rolling fashion to maintain availability. In our case, we were
doing an in-place upgrade and reusing the same IPs
* DSE unfortunately creates a custom column in a system table that requires
you to remove one (or more) system tables (peers?) to be able to start the
node. You delete these system tables by  removing the sstbles on disk while
the node is down. This is a bit of a headache if using vnodes. As we are
using vnodes, it required us to manually specify num tokens, and the
specific tokens the node was responsible for in Cassandra.yaml. You have to
do this before you start the node. If not using vnodes, this is simpler,
but we used vnodes. Again, I'll double check my notes. Once the node is up,
you can revert to your normal vnodes/num tokens settings.

Post upgrade:
* Drop DSE system tables.

I'll revert with more detail if needed.

On Tue, Dec 4, 2018, 5:46 PM Nandakishore Tokala <
nandakishore.tok...@gmail.com wrote:

> HI All,
>
> we are migrating from DSE to open source Cassandra. if anyone has recently
> migrated, Can you please share their experience, steps you followed and
> challenges you guys faced.
>
> we want to migrate to the same computable version in open source, can you
> give us version number(even with the minor version) for DSE 5.1.2
>
> 5.1 DSE production-certified 3.10 + enhancements 3.4 + enhancements big m
>
> --
> Thanks & Regards,
> Nanda Kishore
>

Migrating from DSE5.1.2 to Opensource cassandra

[Nandakishore Tokala]

HI All,

we are migrating from DSE to open source Cassandra. if anyone has recently
migrated, Can you please share their experience, steps you followed and
challenges you guys faced.

we want to migrate to the same computable version in open source, can you
give us version number(even with the minor version) for DSE 5.1.2

5.1 DSE production-certified 3.10 + enhancements 3.4 + enhancements big m

-- 
Thanks & Regards,
Nanda Kishore

Re: [EXTERNAL] Re: upgrade Apache Cassandra 2.1.9 to 3.0.9

[Shravan R]

Thanks Sean. I have automation in place that can put the new binary and
restart the node to a newer version as quickly as possible. upgradesstables
is I/O intensive and it takes time and is proportional to the data on the
node. Given these constraints, is there a risk due to prolonged
upgradesstables?

On Tue, Dec 4, 2018 at 12:20 PM Durity, Sean R 
wrote:

> We have had great success with Cassandra upgrades with applications
> staying on-line. It is one of the strongest benefits of Cassandra. A couple
> things I incorporate into upgrades:
>
> -  The main task is getting the new binaries loaded, then
> restarting the node – in a rolling fashion. Get this done as quickly as
> possible
>
> -  Streaming between versions is usually problematic. So, I never
> do any node additions or decommissions during an upgrade
>
> -  With applications running, there is not an acceptable back-out
> plan (either lose data or take a long outage or both), so we are always
> going forward. So, lower life cycle testing is important before hitting
> production
>
> -  Upgrading is a more frequent activity, so get the
> process/automation in place. The upgrade process should not be a reason to
> delay, especially for minor version upgrades that might be quickly
> necessary (security issue or bug fix).
>
>
>
>
>
> Sean Durity
>
>
>
> *From:* Shravan R 
> *Sent:* Tuesday, December 04, 2018 12:22 PM
> *To:* user@cassandra.apache.org
> *Subject:* [EXTERNAL] Re: upgrade Apache Cassandra 2.1.9 to 3.0.9
>
>
>
> Thanks Jeff. I tried to bootstrap a 3.x node to a partially upgraded
> cluster (2.1.9 + 3.x) and I was *not* able to do so. The schema never
> settled.
>
>
>
> How does the below approach sound like?
>
>1. Update the software binary on all nodes to use cassandra-3.x upon a
>restart.
>2. Restart all nodes in a rolling fashion
>3. Run nodetool upgradesstables in a rolling fashion
>
>
>
> Is there a risk on pending nodetool upgradesstables?
>
>
>
> On Sun, Dec 2, 2018 at 2:12 AM Jeff Jirsa  wrote:
>
>
>
>
> On Dec 2, 2018, at 12:40 PM, Shravan R  wrote:
>
> Marc/Dimitry/Jon - greatly appreciate your feedback. I will look into the
> version part that you suggested. The reason to go direct to 3.x is to take
> a bi leap and reduce overall effort to upgrade a large cluster (development
> included).
>
>
>
> I have these questions from my original post. Appreciate if you could shed
> some light and point me in the right direction.
>
>
>
> 1) How do deal with decommissioning a 2.1.9 node in a partially upgraded
> cluster?
>
>
>
> If any of the replicas have already upgraded, which is almost guaranteed
> if you’re using vnodes, It’s hard / you don’t. You’d basically upgrade
> everything else and then deal with it. If a host fails mid upgrade you’ll
> likely have some period of unavailables while you bounce the replicas to
> finish, then you can decom
>
>
>
>
>
>
>
> 2) How to bootstrap a 3.x node to a partially upgraded cluster?
>
>
>
> This may work fine, but test it because I’m not certain. It should be able
> to read the 2.1 and 3.0 sstables that’ll stream so it’ll just work
>
>
>
> 3) Is there an alternative approach to the upgrade large clusters. i.e
> instead of going through nodetool upgradesstables on each node in rolling
> fashion
>
>
>
> Bounce them all as quickly as is practical, do the upgradesstables after
> the bounces complete
>
>
>
>
>
>
>
>
>
>
>
> On Sat, Dec 1, 2018 at 1:03 PM Jonathan Haddad  wrote:
>
> Dmitry is right. Generally speaking always go with the latest bug fix
> release.
>
>
>
> On Sat, Dec 1, 2018 at 10:14 AM Dmitry Saprykin 
> wrote:
>
> See more here
>
> https://issues.apache.org/jira/plugins/servlet/mobile#issue/CASSANDRA-13004
> 
>
>
>
> On Sat, Dec 1, 2018 at 1:02 PM Dmitry Saprykin 
> wrote:
>
> Even more, 3.0.9 is a terrible target choice by itself. It has a nasty bug
> corrupting sstables on alter.
>
>
>
> On Sat, Dec 1, 2018 at 11:55 AM Marc Selwan 
> wrote:
>
> Hi Shravan,
>
>
>
> Did you upgrade Apache Cassandra 2.1.9 to the latest patch release before
> doing the major upgrade? It's generally favorable to go to the latest patch
> release as often times they include fixes that smooth over the upgrade
> process. There are hundreds of bug fixes between 2.1.9 and 2.1.20 (current
> version)
>
>
>
> Best,
>
> Marc
>
>
>
> On Fri, Nov 30, 2018 at 3:13 PM Shravan R  wrote:
>
> Hello,
>
>
>
> I am planning to upgrade Apache Cassandra 2.1.9 to Apache Cassandra-3.0.9.
> I came up with the version based on [1]. I followed upgrade steps as in
> [2]. I was testing the same in the lab and encountered issues (streaming
> just fails and hangs for ever) with bootstrapping a

Pulling in Cassandra metrics in Graphite

[Saha, Sushanta K]

Anyone is aware of a blog/reference/doc that can give me step by step to
accomplish this? Appreciate the help. Thanks in advance!

 Sushanta

RE: [EXTERNAL] Cassandra Upgrade Plan 2.2.4 to 3.11.3

[Durity, Sean R]

See my recent post for some additional points. But I wanted to encourage you to 
look at the in-place upgrade on your existing hardware. No need to add a DC to 
try and upgrade. The cluster will handle reads and writes with nodes of 
different versions – no problems. I have done this many times on many clusters.

Also, I tell my teams there is no real back-out after we get the first node 
upgraded. This is because any new data is being written in the new sstable 
format (assuming the version has a new sstable format) – whether inserts or 
compaction. Any snapshot of the cluster pre-upgrade is now obsolete. Test 
thoroughly, then go forward as quickly as possible.


Sean Durity

From: Devaki, Srinivas 
Sent: Sunday, December 02, 2018 9:24 AM
To: user@cassandra.apache.org
Subject: [EXTERNAL] Cassandra Upgrade Plan 2.2.4 to 3.11.3

Hi everyone,

I have planned out our org's cassandra upgrade plan and want to make sure if it 
seems fine.

Details Existing Cluster:
* Cassandra 2.2.4
* 8 nodes with 32G ram and 12G max heap allocated to cassandra
* 4 nodes in each rack

1. Ensured all clients to use LOCAL_* consistency levels and all traffic to 
"old" dc
2. Add new cluster as "new" dc with cassandra 2.2.4
  2.1 update conf on all nodes in "old" dc
  2.2 rolling restart the "old" dc
3. Alter tables with similar replication factor on the "new" dc
4. cassandra repair on all nodes in "new" dc
5. upgrade each node in "new" dc to cassandra 3.11.3 (and upgradesstables)
6. switch all clients to connect to new cluster
7. repair all new nodes once more
8. alter tables to replication only on new dc
9. remove "old" dc

and I have some doubts on the same plan
D1. can i just join 3.11.3 cluster as "new" dc in the 2.2.4 cluster?
D2. how does rolling upgrade work, as in within the same cluster how can 2 
versions coexist?

Will be grateful if you could review this plan.

PS: following this plan to ensure that I can revert back to old behaviour at 
any step

Thanks
Srinivas Devaki
SRE/SDE at Zomato






The information in this Internet Email is confidential and may be legally 
privileged. It is intended solely for the addressee. Access to this Email by 
anyone else is unauthorized. If you are not the intended recipient, any 
disclosure, copying, distribution or any action taken or omitted to be taken in 
reliance on it, is prohibited and may be unlawful. When addressed to our 
clients any opinions or advice contained in this Email are subject to the terms 
and conditions expressed in any applicable governing The Home Depot terms of 
business or client engagement letter. The Home Depot disclaims all 
responsibility and liability for the accuracy and content of this attachment 
and for any damages or losses arising from any inaccuracies, errors, viruses, 
e.g., worms, trojan horses, etc., or other items of a destructive nature, which 
may be contained in this attachment and shall not be liable for direct, 
indirect, consequential or special damages in connection with this e-mail 
message or its attachment.

Re: upgrade Apache Cassandra 2.1.9 to 3.0.9

[Jeff Jirsa]

Schema won’t be transferred cross-majors


-- 
Jeff Jirsa


> On Dec 4, 2018, at 10:51 PM, Shravan R  wrote:
> 
> Thanks Jeff. I tried to bootstrap a 3.x node to a partially upgraded cluster 
> (2.1.9 + 3.x) and I was not able to do so. The schema never settled. 
> 
> How does the below approach sound like?
> Update the software binary on all nodes to use cassandra-3.x upon a restart.
> Restart all nodes in a rolling fashion
> Run nodetool upgradesstables in a rolling fashion
> 
> Is there a risk on pending nodetool upgradesstables? 
> 
>> On Sun, Dec 2, 2018 at 2:12 AM Jeff Jirsa  wrote:
>> 
>> 
>>> On Dec 2, 2018, at 12:40 PM, Shravan R  wrote:
>>> 
>>> Marc/Dimitry/Jon - greatly appreciate your feedback. I will look into the 
>>> version part that you suggested. The reason to go direct to 3.x is to take 
>>> a bi leap and reduce overall effort to upgrade a large cluster (development 
>>> included). 
>>> 
>>> I have these questions from my original post. Appreciate if you could shed 
>>> some light and point me in the right direction.
>>> 
>>> 1) How do deal with decommissioning a 2.1.9 node in a partially upgraded 
>>> cluster?
>> 
>> If any of the replicas have already upgraded, which is almost guaranteed if 
>> you’re using vnodes, It’s hard / you don’t. You’d basically upgrade 
>> everything else and then deal with it. If a host fails mid upgrade you’ll 
>> likely have some period of unavailables while you bounce the replicas to 
>> finish, then you can decom
>> 
>> 
>> 
>>> 2) How to bootstrap a 3.x node to a partially upgraded cluster? 
>> 
>> This may work fine, but test it because I’m not certain. It should be able 
>> to read the 2.1 and 3.0 sstables that’ll stream so it’ll just work
>> 
>>> 3) Is there an alternative approach to the upgrade large clusters. i.e 
>>> instead of going through nodetool upgradesstables on each node in rolling 
>>> fashion
>> 
>> Bounce them all as quickly as is practical, do the upgradesstables after the 
>> bounces complete
>> 
>> 
>> 
>>> 
>>> 
 On Sat, Dec 1, 2018 at 1:03 PM Jonathan Haddad  wrote:
 Dmitry is right. Generally speaking always go with the latest bug fix 
 release. 
 
> On Sat, Dec 1, 2018 at 10:14 AM Dmitry Saprykin 
>  wrote:
> See more here 
> https://issues.apache.org/jira/plugins/servlet/mobile#issue/CASSANDRA-13004
> 
>> On Sat, Dec 1, 2018 at 1:02 PM Dmitry Saprykin 
>>  wrote:
>> Even more, 3.0.9 is a terrible target choice by itself. It has a nasty 
>> bug corrupting sstables on alter.
>> 
>>> On Sat, Dec 1, 2018 at 11:55 AM Marc Selwan  
>>> wrote:
>>> Hi Shravan, 
>>> 
>>> Did you upgrade Apache Cassandra 2.1.9 to the latest patch release 
>>> before doing the major upgrade? It's generally favorable to go to the 
>>> latest patch release as often times they include fixes that smooth over 
>>> the upgrade process. There are hundreds of bug fixes between 2.1.9 and 
>>> 2.1.20 (current version) 
>>> 
>>> Best,
>>> Marc
>>> 
 On Fri, Nov 30, 2018 at 3:13 PM Shravan R  wrote:
 Hello,
 
 I am planning to upgrade Apache Cassandra 2.1.9 to Apache 
 Cassandra-3.0.9. I came up with the version based on [1]. I followed 
 upgrade steps as in [2]. I was testing the same in the lab and 
 encountered issues (streaming just fails and hangs for ever) with 
 bootstrapping a 3.0.9 node on a partially upgraded cluster. [50% of 
 nodes on 2.1.9 and 50% on 3.0.9]. The production cluster that I am 
 supporting is pretty large and I am anticipating to end up in a 
 situation like this (Hope not) and would like to be prepared.
 
 1) How do deal with decommissioning a 2.1.9 node in a partially 
 upgraded cluster?
 2) How to bootstrap a 3.x node to a partially upgraded cluster? 
 3) Is there an alternative approach to the upgrade large clusters. i.e 
 instead of going through nodetool upgradesstables on each node in 
 rolling fashion
 
 
 As per [1] the general restriction is to avoid decommissioning or 
 adding nodes but in reality there can be failures or maintenance that 
 warrants us to do so. 
 
 
 Please point me in the right direction.
 
 
 Thanks,
 Shravan
 
 
 [1] 
 https://docs.datastax.com/en/upgrade/doc/upgrade/datastax_enterprise/upgdDSE50.html#upgdDSE50__cstar-version-change
 
 [2] 
 https://myopsblog.wordpress.com/2017/12/04/upgrade-cassandra-cluster-from-2-x-to-3-x/
 
>>> -- 
>>> Marc Selwan | DataStax | Product Management | (925) 413-7079 
>>> 
>>> 
 -- 
 Jon Haddad
 http://www.rustyrazorblade.com
 twitter: rustyrazorblade

RE: [EXTERNAL] Re: upgrade Apache Cassandra 2.1.9 to 3.0.9

[Durity, Sean R]

We have had great success with Cassandra upgrades with applications staying 
on-line. It is one of the strongest benefits of Cassandra. A couple things I 
incorporate into upgrades:

-  The main task is getting the new binaries loaded, then restarting 
the node – in a rolling fashion. Get this done as quickly as possible

-  Streaming between versions is usually problematic. So, I never do 
any node additions or decommissions during an upgrade

-  With applications running, there is not an acceptable back-out plan 
(either lose data or take a long outage or both), so we are always going 
forward. So, lower life cycle testing is important before hitting production

-  Upgrading is a more frequent activity, so get the process/automation 
in place. The upgrade process should not be a reason to delay, especially for 
minor version upgrades that might be quickly necessary (security issue or bug 
fix).


Sean Durity

From: Shravan R 
Sent: Tuesday, December 04, 2018 12:22 PM
To: user@cassandra.apache.org
Subject: [EXTERNAL] Re: upgrade Apache Cassandra 2.1.9 to 3.0.9

Thanks Jeff. I tried to bootstrap a 3.x node to a partially upgraded cluster 
(2.1.9 + 3.x) and I was not able to do so. The schema never settled.

How does the below approach sound like?

  1.  Update the software binary on all nodes to use cassandra-3.x upon a 
restart.
  2.  Restart all nodes in a rolling fashion
  3.  Run nodetool upgradesstables in a rolling fashion

Is there a risk on pending nodetool upgradesstables?

On Sun, Dec 2, 2018 at 2:12 AM Jeff Jirsa 
mailto:jji...@gmail.com>> wrote:


On Dec 2, 2018, at 12:40 PM, Shravan R 
mailto:skr...@gmail.com>> wrote:
Marc/Dimitry/Jon - greatly appreciate your feedback. I will look into the 
version part that you suggested. The reason to go direct to 3.x is to take a bi 
leap and reduce overall effort to upgrade a large cluster (development 
included).

I have these questions from my original post. Appreciate if you could shed some 
light and point me in the right direction.

1) How do deal with decommissioning a 2.1.9 node in a partially upgraded 
cluster?

If any of the replicas have already upgraded, which is almost guaranteed if 
you’re using vnodes, It’s hard / you don’t. You’d basically upgrade everything 
else and then deal with it. If a host fails mid upgrade you’ll likely have some 
period of unavailables while you bounce the replicas to finish, then you can 
decom




2) How to bootstrap a 3.x node to a partially upgraded cluster?

This may work fine, but test it because I’m not certain. It should be able to 
read the 2.1 and 3.0 sstables that’ll stream so it’ll just work


3) Is there an alternative approach to the upgrade large clusters. i.e instead 
of going through nodetool upgradesstables on each node in rolling fashion

Bounce them all as quickly as is practical, do the upgradesstables after the 
bounces complete






On Sat, Dec 1, 2018 at 1:03 PM Jonathan Haddad 
mailto:j...@jonhaddad.com>> wrote:
Dmitry is right. Generally speaking always go with the latest bug fix release.

On Sat, Dec 1, 2018 at 10:14 AM Dmitry Saprykin 
mailto:saprykin.dmi...@gmail.com>> wrote:
See more here
https://issues.apache.org/jira/plugins/servlet/mobile#issue/CASSANDRA-13004

On Sat, Dec 1, 2018 at 1:02 PM Dmitry Saprykin 
mailto:saprykin.dmi...@gmail.com>> wrote:
Even more, 3.0.9 is a terrible target choice by itself. It has a nasty bug 
corrupting sstables on alter.

On Sat, Dec 1, 2018 at 11:55 AM Marc Selwan 
mailto:marc.sel...@datastax.com>> wrote:
Hi Shravan,

Did you upgrade Apache Cassandra 2.1.9 to the latest patch release before doing 
the major upgrade? It's generally favorable to go to the latest patch release 
as often times they include fixes that smooth over the upgrade process. There 
are hundreds of bug fixes between 2.1.9 and 2.1.20 (current version)

Best,
Marc

On Fri, Nov 30, 2018 at 3:13 PM Shravan R 
mailto:skr...@gmail.com>> wrote:
Hello,

I am planning to upgrade Apache Cassandra 2.1.9 to Apache Cassandra-3.0.9. I 
came up with the version based on [1]. I followed upgrade steps as in [2]. I 
was testing the same in the lab and encountered issues (streaming just fails 
and hangs for ever) with bootstrapping a 3.0.9 node on a partially upgraded 
cluster. [50% of nodes on 2.1.9 and 50% on 3.0.9]. The production cluster that 
I am supporting is pretty large and I am anticipating to end up in a situation 
like this (Hope not) and would like to be prepared.

1) How do deal with decommissioning a 2.1.9 node in a partially upgraded 
cluster?
2) How to bootstrap a 3.x node to a partially upgraded cluster?
3) Is there an alternative approach to the upgrad

Re: upgrade Apache Cassandra 2.1.9 to 3.0.9

[Shravan R]

Thanks Jeff. I tried to bootstrap a 3.x node to a partially upgraded
cluster (2.1.9 + 3.x) and I was *not* able to do so. The schema never
settled.

How does the below approach sound like?

   1. Update the software binary on all nodes to use cassandra-3.x upon a
   restart.
   2. Restart all nodes in a rolling fashion
   3. Run nodetool upgradesstables in a rolling fashion


Is there a risk on pending nodetool upgradesstables?

On Sun, Dec 2, 2018 at 2:12 AM Jeff Jirsa  wrote:

>
>
> On Dec 2, 2018, at 12:40 PM, Shravan R  wrote:
>
> Marc/Dimitry/Jon - greatly appreciate your feedback. I will look into the
> version part that you suggested. The reason to go direct to 3.x is to take
> a bi leap and reduce overall effort to upgrade a large cluster (development
> included).
>
> I have these questions from my original post. Appreciate if you could shed
> some light and point me in the right direction.
>
> 1) How do deal with decommissioning a 2.1.9 node in a partially upgraded
> cluster?
>
>
> If any of the replicas have already upgraded, which is almost guaranteed
> if you’re using vnodes, It’s hard / you don’t. You’d basically upgrade
> everything else and then deal with it. If a host fails mid upgrade you’ll
> likely have some period of unavailables while you bounce the replicas to
> finish, then you can decom
>
>
>
> 2) How to bootstrap a 3.x node to a partially upgraded cluster?
>
>
> This may work fine, but test it because I’m not certain. It should be able
> to read the 2.1 and 3.0 sstables that’ll stream so it’ll just work
>
> 3) Is there an alternative approach to the upgrade large clusters. i.e
> instead of going through nodetool upgradesstables on each node in rolling
> fashion
>
>
> Bounce them all as quickly as is practical, do the upgradesstables after
> the bounces complete
>
>
>
>
>
> On Sat, Dec 1, 2018 at 1:03 PM Jonathan Haddad  wrote:
>
>> Dmitry is right. Generally speaking always go with the latest bug fix
>> release.
>>
>> On Sat, Dec 1, 2018 at 10:14 AM Dmitry Saprykin <
>> saprykin.dmi...@gmail.com> wrote:
>>
>>> See more here
>>>
>>> https://issues.apache.org/jira/plugins/servlet/mobile#issue/CASSANDRA-13004
>>>
>>> On Sat, Dec 1, 2018 at 1:02 PM Dmitry Saprykin <
>>> saprykin.dmi...@gmail.com> wrote:
>>>
 Even more, 3.0.9 is a terrible target choice by itself. It has a nasty
 bug corrupting sstables on alter.

 On Sat, Dec 1, 2018 at 11:55 AM Marc Selwan 
 wrote:

> Hi Shravan,
>
> Did you upgrade Apache Cassandra 2.1.9 to the latest patch release
> before doing the major upgrade? It's generally favorable to go to the
> latest patch release as often times they include fixes that smooth over 
> the
> upgrade process. There are hundreds of bug fixes between 2.1.9 and 2.1.20
> (current version)
>
> Best,
> Marc
>
> On Fri, Nov 30, 2018 at 3:13 PM Shravan R  wrote:
>
>> Hello,
>>
>> I am planning to upgrade Apache Cassandra 2.1.9 to Apache
>> Cassandra-3.0.9. I came up with the version based on [1]. I followed
>> upgrade steps as in [2]. I was testing the same in the lab and 
>> encountered
>> issues (streaming just fails and hangs for ever) with bootstrapping a 
>> 3.0.9
>> node on a partially upgraded cluster. [50% of nodes on 2.1.9 and 50% on
>> 3.0.9]. The production cluster that I am supporting is pretty large and I
>> am anticipating to end up in a situation like this (Hope not) and would
>> like to be prepared.
>>
>> 1) How do deal with decommissioning a 2.1.9 node in a partially
>> upgraded cluster?
>> 2) How to bootstrap a 3.x node to a partially upgraded cluster?
>> 3) Is there an alternative approach to the upgrade large clusters.
>> i.e instead of going through nodetool upgradesstables on each node in
>> rolling fashion
>>
>>
>> As per [1] the general restriction is to avoid decommissioning or
>> adding nodes but in reality there can be failures or maintenance that
>> warrants us to do so.
>>
>>
>> Please point me in the right direction.
>>
>>
>> Thanks,
>> Shravan
>>
>>
>> [1]
>> https://docs.datastax.com/en/upgrade/doc/upgrade/datastax_enterprise/upgdDSE50.html#upgdDSE50__cstar-version-change
>>
>> [2]
>> https://myopsblog.wordpress.com/2017/12/04/upgrade-cassandra-cluster-from-2-x-to-3-x/
>> 
>>
>> --
> Marc Selwan | DataStax | Product Management | (925) 413-7079
>
>
> --
>> Jon Haddad
>> http://www.rustyrazorblade.com
>> twitter: rustyrazorblade
>>
>

CompactionExecutor hangup and MemtableFlushWriter blocked

[qmzhang]

Hi guys, my cassandra cluster run in strange state. 
all nodes CompactionExecutor hangup and blocked. MemtableFlushWriter blocked 
and pending task keep increasing. 
All nodes. nodetool tpstats output like :
Pool Name Active   Pending  Completed   Blocked  
All time blocked
ReadStage  0 05269585 0 
0
MiscStage  0 0  0 0 
0
CompactionExecutor1212  13083 0 
0
MutationStage  0 03406883 0 
0
GossipStage0 0 322048 0 
0
RequestResponseStage   0 0 644613 0 
0
ReadRepairStage0 0  0 0 
0
CounterMutationStage   0 0  0 0 
0
MemtablePostFlush  147468 0 
0
ValidationExecutor 0 0 10 0 
0
MemtableFlushWriter147435 0 
0
ViewMutationStage  0 0  0 0 
0
CacheCleanupExecutor   0 0  0 0 
0
MemtableReclaimMemory  0 0435 0 
0
PendingRangeCalculator 0 0 10 0 
0
SecondaryIndexManagement   0 0  0 0 
0
HintsDispatcher0 0  0 0 
0
PerDiskMemtableFlushWriter_1   0 0428 0 
0
Native-Transport-Requests  0 0   10920163 0 
4
PerDiskMemtableFlushWriter_2   0 0428 0 
0
MigrationStage 0 0  0 0 
0
PerDiskMemtableFlushWriter_0   0 0435 0 
0
Sampler0 0  0 0 
0
PerDiskMemtableFlushWriter_5   0 0428 0 
0
InternalResponseStage  0 0 10 0 
0
PerDiskMemtableFlushWriter_6   0 0428 0 
0
PerDiskMemtableFlushWriter_3   0 0428 0 
0
PerDiskMemtableFlushWriter_4   0 0428 0 
0
AntiEntropyStage   0 0 40 0 
0
PerDiskMemtableFlushWriter_7   0 0428 0 
0


Message type   Dropped
READ 0
RANGE_SLICE  0
_TRACE   0
HINT 0
MUTATION 0
COUNTER_MUTATION 0
BATCH_STORE  0
BATCH_REMOVE 0
REQUEST_RESPONSE 0
PAGED_RANGE  0
READ_REPAIR  0



Restart cluster dose work.
Look forward to your reply.thanks

Re: Cassandra C++ driver for C++98

[Michael Penick]

We don't have access to that compiler. If you could try it for us that
would be greatly appreciated. Pull requests on GitHub are more than welcome.

Mike

On Tue, Nov 20, 2018 at 2:48 PM Amit Plaha  wrote:

> Thanks for the response, Michael!
>
> I could build the driver on UNIX using gcc/g++ but is it possible to build
> the driver using Sunpro CC compiler? I'm running in to issues. What it
> looks like is that some parts of the code are only buildable on GCC. Please
> let me know whether Sunpro CC compiler can be used instead.
>
> On Tue, Oct 30, 2018 at 1:36 PM Michael Penick <
> michael.pen...@datastax.com> wrote:
>
>> Pretty much any version including the most current.
>>
>> On Mon, Oct 29, 2018 at 4:29 PM, Amit Plaha  wrote:
>>
>>> Hi Mike,
>>>
>>> Thanks for the response. Can you let me know which version of the driver
>>> can build with C++98?
>>>
>>> Regards,
>>> Amit
>>>
>>> On Fri, Oct 26, 2018 at 8:53 AM Michael Penick <
>>> michael.pen...@datastax.com> wrote:
>>>
 Those changes where for testing code only.


 https://github.com/datastax/cpp-driver/commit/ffc9bbd8747b43ad5dcef749fe4c63ff245fcf74
 

 The driver has compiled with fine w/ C++98 for some time now. The
 encoding/decoding doesn't make any assumptions about endianess so it should
 work fine on big-endian system, but it's not officially supported (YMMV).

 Mike




 On Thu, Oct 25, 2018 at 3:17 PM, Amit Plaha 
 wrote:

> Hi All,
>
> Is there any Cassandra C++ driver that works with C++98 and is also
> compatible with UNIX big-endian?
>
> I found this issue: https://datastax-oss.atlassian.net/browse/CPP-692
> 
> which seems to have been resolved, but not sure if this is exactly what 
> I'm
> looking for. Does this make the DSE driver compatible with C++98?
>
> Any pointers will be appreciated. Thanks!
>
> Regards,
> Amit
>


>>

Re: How to gracefully decommission a highly loaded node?

[Simon Fontana Oscarsson]

Hi,
If it already uses 100 % CPU I have a hard time seeing it being able to
do a decomission while serving requests. If you have a lot of free
space I would first try nodetool disableautocompaction. If you don't
see any progress in nodetool netstats you can also disablebinary,
disablethrift and disablehandoff to stop serving client requests. 

-- 
SIMON FONTANA OSCARSSON
Software Developer

Ericsson
Ölandsgatan 1
37133 Karlskrona, Sweden
simon.fontana.oscars...@ericsson.com
www.ericsson.com
On tis, 2018-12-04 at 14:21 +0330, onmstester onmstester wrote:
> One node suddenly uses 100% CPU, i suspect hardware problems and do
> not have time to trace that, so decided to just remove the node from
> the cluster, but although the node state changed to UL, but no sign
> of Leaving: the node is still compacting and flushing memtables,
> writing mutations and CPU is 100% for hours since.
> Is there any means to force a Cassandra Node to just decommission and
> stop doing normal things?
> Due to W.CL=ONE, i can not use removenode and shutdown the node
> 
> Best Regards
> Sent using Zoho Mail
> 
> 

smime.p7s
Description: S/MIME cryptographic signature

How to gracefully decommission a highly loaded node?

[onmstester onmstester]

One node suddenly uses 100% CPU, i suspect hardware problems and do not have 
time to trace that, so decided to just remove the node from the cluster, but 
although the node state changed to UL, but no sign of Leaving: the node is 
still compacting and flushing memtables, writing mutations and CPU is 100% for 
hours since. Is there any means to force a Cassandra Node to just decommission 
and stop doing normal things? Due to W.CL=ONE, i can not use removenode and 
shutdown the node Best Regards Sent using Zoho Mail

Re: Data storage space unbalance issue

[Eunsu Kim]

Thanks to you, I found a good tool called Reaper.(http://cassandra-reaper.io/ 
)

I will try it.


> On 4 Dec 2018, at 3:30 PM, Elliott Sims  wrote:
> 
> It depends on the type of repair, but you'll want to make sure all the data 
> is where it should be before running cleanup.  Somewhat related, if you're 
> not running regular repairs already, you should be.  You can do it via cron, 
> but I strongly suggest checking out Reaper.
> 
> On Wed, Nov 28, 2018, 8:05 PM Eunsu Kim   wrote:
> Thank you for your response.
> 
> I will run repair from datacenter2 with your advice. Do I have to run repair 
> on every node in datacenter2?
> 
> There is no snapshot when checked with nodetool listsnaphosts.
> 
> Thank you.
> 
>> On 29 Nov 2018, at 4:31 AM, Elliott Sims > > wrote:
>> 
>> I think you answered your own question, sort of.
>> 
>> When you expand a cluster, it copies the appropriate rows to the new node(s) 
>> but doesn't automatically remove them from the old nodes.  When you ran 
>> cleanup on datacenter1, it cleared out those old extra copies.  I would 
>> suggest running a repair first for safety on datacenter2, then a "nodetool 
>> cleanup" on those hosts.  
>> 
>> Also run "nodetool snapshot" to make sure you don't have any old snapshots 
>> sitting around taking up space.
>> 
>> On Wed, Nov 28, 2018 at 5:29 AM Eunsu Kim > > wrote:
>> (I am sending the previous mail again because it seems that it has not been 
>> sent properly.)
>> 
>> HI experts,
>> 
>> I am running 2 datacenters each containing five nodes. (total 10 nodes, all 
>> 3.11.3)
>> 
>> My data is stored one at each data center. (REPLICATION = { 'class' : 
>> 'org.apache.cassandra.locator.NetworkTopologyStrategy', 'datacenter1': '1', 
>> 'datacenter2': '1’ })
>> 
>> Most of my data have a short TTL(14days). The gc_grace_seconds value for all 
>> tables is also 600sec.
>> 
>> I expect the two data centers to use the same size but datacenter2 is using 
>> more size. It seems that the datas of datacenter2 is rarely deleted. While 
>> the disk usage for datacenter1 remains constant, the disk usage for 
>> datacenter2 continues to grow.
>> 
>> ——
>> Datacenter: datacenter1
>> ===
>> Status=Up/Down
>> |/ State=Normal/Leaving/Joining/Moving
>> --  Address   Load   Tokens   Owns (effective)  Host ID  
>>  Rack
>> UN  10.61.58.228  925.48 GiB  256  21.5% 
>> 60d1bac8-b4d6-4e02-a05f-badee0bb36f5  rack1
>> UN  10.61.58.167  840 GiB256  20.0% 
>> a04fc77a-907f-490c-971c-4e1f964c7b14  rack1
>> UN  10.61.75.86   1.13 TiB   256  19.3% 
>> 618c101b-036d-42e7-bf9f-2bcbd429cbd1  rack1
>> UN  10.61.59.22   844.19 GiB  256  20.0% 
>> d8a4a165-13f0-4f4a-9278-4024730b8116  rack1
>> UN  10.61.59.82   737.88 GiB  256  19.2% 
>> 054a4eb5-6d1c-46fa-b550-34da610da4e0  rack1
>> Datacenter: datacenter2
>> ===
>> Status=Up/Down
>> |/ State=Normal/Leaving/Joining/Moving
>> --  Address   Load   Tokens   Owns (effective)  Host ID  
>>  Rack
>> UN  10.42.6.120   1.11 TiB   256  18.6% 
>> 69f15be0-e5a1-474e-87cf-b063e6854402  rack1
>> UN  10.42.5.207   1.17 TiB   256  20.0% 
>> f78bdce5-cb01-47e0-90b9-fcc31568e49e  rack1
>> UN  10.42.6.471.01 TiB   256  20.1% 
>> 3ff93b47-2c15-4e1a-a4ea-2596f26b4281  rack1
>> UN  10.42.6.481007.67 GiB  256  20.4% 
>> 8cbbe76d-6496-403a-8b09-fe6812c9dea2  rack1
>> UN  10.42.5.208   1.29 TiB   256  20.9% 
>> 4aa96c6a-6083-417f-a58a-ec847bcbfc7e  rack1
>> --
>> 
>> A few days ago, one node of datacenter1 broke down and replaced it, and I 
>> worked on rebuild, repair, and cleanup.
>> 
>> 
>> What else can I do?
>> 
>> Thank you in advance.
>