Perhaps have a read here?
https://docs.datastax.com/en/cassandra-oss/3.x/cassandra/operations/opsAddNodeToCluster.html
On 04/04/2023 06:41, David Tinker wrote:
Ok. Have to psych myself up to the add node task a bit. Didn't go well
the first time round!
Tasks
- Make sure the new node is not in seeds list!
- Check cluster name, listen address, rpc address
- Give it its own rack in cassandra-rackdc.properties
- Delete cassandra-topology.properties if it exists
- Make sure no compactions are on the go
- rm -rf /var/lib/cassandra/*
- rm /data/cassandra/commitlog/* (this is on different disk)
- systemctl start cassandra
And it should start streaming data from the other nodes and join the
cluster. Anything else I have to watch out for? Tx.
On Tue, Apr 4, 2023 at 5:25 AM Jeff Jirsa <jji...@gmail.com> wrote:
Because executing “removenode” streamed extra data from live nodes
to the “gaining” replica
Oversimplified (if you had one token per node)
If you start with A B C
Then add D
D should bootstrap a range from each of A B and C, but at the end,
some of the data that was A B C becomes B C D
When you removenode, you tell B and C to send data back to A.
A B and C will eventually contact that data away. Eventually.
If you get around to adding D again, running “cleanup” when you’re
done (successfully) will remove a lot of it.
On Apr 3, 2023, at 8:14 PM, David Tinker <david.tin...@gmail.com>
wrote:
Looks like the remove has sorted things out. Thanks.
One thing I am wondering about is why the nodes are carrying a
lot more data? The loads were about 2.7T before, now 3.4T.
# nodetool status
Datacenter: dc1
===============
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host ID
Rack
UN xxx.xxx.xxx.105 3.4 TiB 256 100.0%
afd02287-3f88-4c6f-8b27-06f7a8192402 rack3
UN xxx.xxx.xxx.253 3.34 TiB 256 100.0%
e1af72be-e5df-4c6b-a124-c7bc48c6602a rack2
UN xxx.xxx.xxx.107 3.44 TiB 256 100.0%
ab72f017-be96-41d2-9bef-a551dec2c7b5 rack1
On Mon, Apr 3, 2023 at 5:42 PM Bowen Song via user
<user@cassandra.apache.org> wrote:
That's correct. nodetool removenode is strongly preferred
when your node is already down. If the node is still
functional, use nodetool decommission on the node instead.
On 03/04/2023 16:32, Jeff Jirsa wrote:
FWIW, `nodetool decommission` is strongly preferred.
`nodetool removenode` is designed to be run when a host is
offline. Only decommission is guaranteed to maintain
consistency / correctness, and removemode probably streams a
lot more data around than decommission.
On Mon, Apr 3, 2023 at 6:47 AM Bowen Song via user
<user@cassandra.apache.org> wrote:
Use nodetool removenode is strongly preferred in most
circumstances, and only resort to assassinate if you do
not care about data consistency or you know there won't
be any consistency issue (e.g. no new writes and did not
run nodetool cleanup).
Since the size of data on the new node is small,
nodetool removenode should finish fairly quickly and
bring your cluster back.
Next time when you are doing something like this again,
please test it out on a non-production environment, make
sure everything works as expected before moving onto the
production.
On 03/04/2023 06:28, David Tinker wrote:
Should I use assassinate or removenode? Given that
there is some data on the node. Or will that be found
on the other nodes? Sorry for all the questions but I
really don't want to mess up.
On Mon, Apr 3, 2023 at 7:21 AM Carlos Diaz
<crdiaz...@gmail.com> wrote:
That's what nodetool assassinte will do.
On Sun, Apr 2, 2023 at 10:19 PM David Tinker
<david.tin...@gmail.com> wrote:
Is it possible for me to remove the node from
the cluster i.e. to undo this mess and get the
cluster operating again?
On Mon, Apr 3, 2023 at 7:13 AM Carlos Diaz
<crdiaz...@gmail.com> wrote:
You can leave it in the seed list of the
other nodes, just make sure it's not
included in this node's seed list.
However, if you do decide to fix the issue
with the racks first assassinate this node
(nodetool assassinate <ip>), and update the
rack name before you restart.
On Sun, Apr 2, 2023 at 10:06 PM David
Tinker <david.tin...@gmail.com> wrote:
It is also in the seeds list for the
other nodes. Should I remove it from
those, restart them one at a time, then
restart it?
/etc/cassandra # grep -i bootstrap *
doesn't show anything so I don't think
I have auto_bootstrap false.
Thanks very much for the help.
On Mon, Apr 3, 2023 at 7:01 AM Carlos
Diaz <crdiaz...@gmail.com> wrote:
Just remove it from the seed list
in the cassandra.yaml file and
restart the node. Make sure that
auto_bootstrap is set to true first
though.
On Sun, Apr 2, 2023 at 9:59 PM
David Tinker
<david.tin...@gmail.com> wrote:
So likely because I made it a
seed node when I added it to
the cluster it didn't do the
bootstrap process. How can I
recover this?
On Mon, Apr 3, 2023 at 6:41 AM
David Tinker
<david.tin...@gmail.com> wrote:
Yes replication factor is 3.
I ran nodetool repair -pr
on all the nodes (one at a
time) and am still having
issues getting data back
from queries.
I did make the new node a
seed node.
Re "rack4": I assumed that
was just an indication as
to the physical location of
the server for redundancy.
This one is separate from
the others so I used rack4.
On Mon, Apr 3, 2023 at
6:30 AM Carlos Diaz
<crdiaz...@gmail.com> wrote:
I'm assuming that your
replication factor is
3. If that's the case,
did you intentionally
put this node in rack
4? Typically, you want
to add nodes in
multiples of your
replication factor in
order to keep the
"racks" balanced. In
other words, this node
should have been added
to rack 1, 2 or 3.
Having said that, you
should be able to
easily fix your problem
by running a nodetool
repair -pr on the new
node.
On Sun, Apr 2, 2023 at
8:16 PM David Tinker
<david.tin...@gmail.com>
wrote:
Hi All
I recently added a
node to my 3 node
Cassandra 4.0.5
cluster and now
many reads are not
returning rows!
What do I need to
do to fix this?
There weren't any
errors in the logs
or other problems
that I could see. I
expected the
cluster to balance
itself but this
hasn't happened
(yet?). The nodes
are similar so I
have num_tokens=256
for each. I am
using the
Murmur3Partitioner.
# nodetool status
Datacenter: dc1
===============
Status=Up/Down
|/
State=Normal/Leaving/Joining/Moving
-- Address
Load Tokens
Owns (effective)
Host ID Rack
UN xxx.xxx.xxx.105
2.65 TiB 256
72.9%
afd02287-3f88-4c6f-8b27-06f7a8192402
rack3
UN xxx.xxx.xxx.253
2.6 TiB 256
73.9%
e1af72be-e5df-4c6b-a124-c7bc48c6602a
rack2
UN xxx.xxx.xxx.24
93.82 KiB 256
80.0%
c4e8b4a0-f014-45e6-afb4-648aad4f8500
rack4
UN xxx.xxx.xxx.107
2.65 TiB 256
73.2%
ab72f017-be96-41d2-9bef-a551dec2c7b5
rack1
# nodetool netstats
Mode: NORMAL
Not sending any
streams.
Read Repair Statistics:
Attempted: 0
Mismatch (Blocking): 0
Mismatch
(Background): 0
Pool Name Active
Pending Completed
Dropped
Large messages
n/a 0 71754 0
Small messages
n/a 0 8398184 14
Gossip messages
n/a 0
1303634 0
# nodetool ring
Datacenter: dc1
==========
Address
Rack Status
State Load
Owns Token
9189523899826545641
xxx.xxx.xxx.24
rack4 Up
Normal 93.82 KiB
79.95%
-9194674091837769168
xxx.xxx.xxx.107
rack1 Up
Normal 2.65 TiB
73.25%
-9168781258594813088
xxx.xxx.xxx.253
rack2 Up
Normal 2.6 TiB
73.92%
-9163037340977721917
xxx.xxx.xxx.105
rack3 Up
Normal 2.65 TiB
72.88%
-9148860739730046229
xxx.xxx.xxx.107
rack1 Up
Normal 2.65 TiB
73.25%
-9125240034139323535
xxx.xxx.xxx.253
rack2 Up
Normal 2.6 TiB
73.92%
-9112518853051755414
xxx.xxx.xxx.105
rack3 Up
Normal 2.65 TiB
72.88%
-9100516173422432134
...
This is causing a
serious production
issue. Please help
if you can.
Thanks
David
