Thanks. Yes my big screwup here was to make the new node a seed node so it didn't get any data. I am going to add 3 more nodes, one at a time when the cluster has finished with the remove and everything seems stable. Should I do a full repair first or is the remove node operation basically doing that?
Re the racks. I just asked that question on this list and the answer was that adding the new nodes as rack4, rack5 and rack6 is fine. They are all on separate physical racks. Is that ok? On Mon, Apr 3, 2023 at 5:16 PM Aaron Ploetz <aaronplo...@gmail.com> wrote: > The time it takes to stream data off of a node varies by network, cloud > region, and other factors. So it's not unheard of for it to take a bit to > finish. > > Just thought I'd mention that auto_bootstrap is true by default. So if > you're not setting it, the node should bootstrap as long as it's not a seed > node. > > As for the rack issue, yes, it's a good idea to keep your racks in > multiples of your RF. When performing token ownership calculations, > Cassandra takes rack designation into consideration. It tries to ensure > that multiple replicas for a row are not placed in the same rack. TBH - > I'd build out two more nodes to have 6 nodes across 3 racks (2 in each), > just to ensure even distribution. Otherwise, you might notice that the > nodes sharing a rack will consume disk at a different rate than the nodes > which have their own rack. > > On Mon, Apr 3, 2023 at 8:57 AM David Tinker <david.tin...@gmail.com> > wrote: > >> Thanks. Hmm, the remove has been busy for hours but seems to be >> progressing. >> >> I have been running this on the nodes to monitor progress: >> # nodetool netstats | grep Already >> Receiving 92 files, 843934103369 bytes total. Already received 82 >> files (89.13%), 590204687299 bytes total (69.93%) >> Sending 84 files, 860198753783 bytes total. Already sent 56 files >> (66.67%), 307038785732 bytes total (35.69%) >> Sending 78 files, 815573435637 bytes total. Already sent 56 files >> (71.79%), 313079823738 bytes total (38.39%) >> >> The percentages are ticking up. >> >> # nodetool ring | head -20 >> Datacenter: dc1 >> ========== >> Address Rack Status State Load Owns >> Token >> >> 9189523899826545641 >> xxx.xxx.xxx..24 rack4 Down Leaving 26.62 GiB 79.95% >> -9194674091837769168 >> xxx.xxx.xxx.107 rack1 Up Normal 2.68 TiB 73.25% >> -9168781258594813088 >> xxx.xxx.xxx.253 rack2 Up Normal 2.63 TiB 73.92% >> -9163037340977721917 >> xxx.xxx.xxx.105 rack3 Up Normal 2.68 TiB 72.88% >> -9148860739730046229 >> >> >> On Mon, Apr 3, 2023 at 3:46 PM Bowen Song via user < >> user@cassandra.apache.org> wrote: >> >>> Use nodetool removenode is strongly preferred in most circumstances, >>> and only resort to assassinate if you do not care about data >>> consistency or you know there won't be any consistency issue (e.g. no new >>> writes and did not run nodetool cleanup). >>> >>> Since the size of data on the new node is small, nodetool removenode >>> should finish fairly quickly and bring your cluster back. >>> >>> Next time when you are doing something like this again, please test it >>> out on a non-production environment, make sure everything works as expected >>> before moving onto the production. >>> >>> >>> On 03/04/2023 06:28, David Tinker wrote: >>> >>> Should I use assassinate or removenode? Given that there is some data on >>> the node. Or will that be found on the other nodes? Sorry for all the >>> questions but I really don't want to mess up. >>> >>> On Mon, Apr 3, 2023 at 7:21 AM Carlos Diaz <crdiaz...@gmail.com> wrote: >>> >>>> That's what nodetool assassinte will do. >>>> >>>> On Sun, Apr 2, 2023 at 10:19 PM David Tinker <david.tin...@gmail.com> >>>> wrote: >>>> >>>>> Is it possible for me to remove the node from the cluster i.e. to undo >>>>> this mess and get the cluster operating again? >>>>> >>>>> On Mon, Apr 3, 2023 at 7:13 AM Carlos Diaz <crdiaz...@gmail.com> >>>>> wrote: >>>>> >>>>>> You can leave it in the seed list of the other nodes, just make sure >>>>>> it's not included in this node's seed list. However, if you do decide to >>>>>> fix the issue with the racks first assassinate this node (nodetool >>>>>> assassinate <ip>), and update the rack name before you restart. >>>>>> >>>>>> On Sun, Apr 2, 2023 at 10:06 PM David Tinker <david.tin...@gmail.com> >>>>>> wrote: >>>>>> >>>>>>> It is also in the seeds list for the other nodes. Should I remove it >>>>>>> from those, restart them one at a time, then restart it? >>>>>>> >>>>>>> /etc/cassandra # grep -i bootstrap * >>>>>>> doesn't show anything so I don't think I have auto_bootstrap false. >>>>>>> >>>>>>> Thanks very much for the help. >>>>>>> >>>>>>> >>>>>>> On Mon, Apr 3, 2023 at 7:01 AM Carlos Diaz <crdiaz...@gmail.com> >>>>>>> wrote: >>>>>>> >>>>>>>> Just remove it from the seed list in the cassandra.yaml file and >>>>>>>> restart the node. Make sure that auto_bootstrap is set to true first >>>>>>>> though. >>>>>>>> >>>>>>>> On Sun, Apr 2, 2023 at 9:59 PM David Tinker <david.tin...@gmail.com> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> So likely because I made it a seed node when I added it to the >>>>>>>>> cluster it didn't do the bootstrap process. How can I recover this? >>>>>>>>> >>>>>>>>> On Mon, Apr 3, 2023 at 6:41 AM David Tinker < >>>>>>>>> david.tin...@gmail.com> wrote: >>>>>>>>> >>>>>>>>>> Yes replication factor is 3. >>>>>>>>>> >>>>>>>>>> I ran nodetool repair -pr on all the nodes (one at a time) and >>>>>>>>>> am still having issues getting data back from queries. >>>>>>>>>> >>>>>>>>>> I did make the new node a seed node. >>>>>>>>>> >>>>>>>>>> Re "rack4": I assumed that was just an indication as to the >>>>>>>>>> physical location of the server for redundancy. This one is separate >>>>>>>>>> from >>>>>>>>>> the others so I used rack4. >>>>>>>>>> >>>>>>>>>> On Mon, Apr 3, 2023 at 6:30 AM Carlos Diaz <crdiaz...@gmail.com> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> I'm assuming that your replication factor is 3. If that's the >>>>>>>>>>> case, did you intentionally put this node in rack 4? Typically, >>>>>>>>>>> you want >>>>>>>>>>> to add nodes in multiples of your replication factor in order to >>>>>>>>>>> keep the >>>>>>>>>>> "racks" balanced. In other words, this node should have been added >>>>>>>>>>> to rack >>>>>>>>>>> 1, 2 or 3. >>>>>>>>>>> >>>>>>>>>>> Having said that, you should be able to easily fix your problem >>>>>>>>>>> by running a nodetool repair -pr on the new node. >>>>>>>>>>> >>>>>>>>>>> On Sun, Apr 2, 2023 at 8:16 PM David Tinker < >>>>>>>>>>> david.tin...@gmail.com> wrote: >>>>>>>>>>> >>>>>>>>>>>> Hi All >>>>>>>>>>>> >>>>>>>>>>>> I recently added a node to my 3 node Cassandra 4.0.5 cluster >>>>>>>>>>>> and now many reads are not returning rows! What do I need to do to >>>>>>>>>>>> fix >>>>>>>>>>>> this? There weren't any errors in the logs or other problems that >>>>>>>>>>>> I could >>>>>>>>>>>> see. I expected the cluster to balance itself but this hasn't >>>>>>>>>>>> happened >>>>>>>>>>>> (yet?). The nodes are similar so I have num_tokens=256 for each. I >>>>>>>>>>>> am using >>>>>>>>>>>> the Murmur3Partitioner. >>>>>>>>>>>> >>>>>>>>>>>> # nodetool status >>>>>>>>>>>> Datacenter: dc1 >>>>>>>>>>>> =============== >>>>>>>>>>>> Status=Up/Down >>>>>>>>>>>> |/ State=Normal/Leaving/Joining/Moving >>>>>>>>>>>> -- Address Load Tokens Owns (effective) Host >>>>>>>>>>>> ID Rack >>>>>>>>>>>> UN xxx.xxx.xxx.105 2.65 TiB 256 72.9% >>>>>>>>>>>> afd02287-3f88-4c6f-8b27-06f7a8192402 rack3 >>>>>>>>>>>> UN xxx.xxx.xxx.253 2.6 TiB 256 73.9% >>>>>>>>>>>> e1af72be-e5df-4c6b-a124-c7bc48c6602a rack2 >>>>>>>>>>>> UN xxx.xxx.xxx.24 93.82 KiB 256 80.0% >>>>>>>>>>>> c4e8b4a0-f014-45e6-afb4-648aad4f8500 rack4 >>>>>>>>>>>> UN xxx.xxx.xxx.107 2.65 TiB 256 73.2% >>>>>>>>>>>> ab72f017-be96-41d2-9bef-a551dec2c7b5 rack1 >>>>>>>>>>>> >>>>>>>>>>>> # nodetool netstats >>>>>>>>>>>> Mode: NORMAL >>>>>>>>>>>> Not sending any streams. >>>>>>>>>>>> Read Repair Statistics: >>>>>>>>>>>> Attempted: 0 >>>>>>>>>>>> Mismatch (Blocking): 0 >>>>>>>>>>>> Mismatch (Background): 0 >>>>>>>>>>>> Pool Name Active Pending Completed >>>>>>>>>>>> Dropped >>>>>>>>>>>> Large messages n/a 0 71754 >>>>>>>>>>>> 0 >>>>>>>>>>>> Small messages n/a 0 8398184 >>>>>>>>>>>> 14 >>>>>>>>>>>> Gossip messages n/a 0 1303634 >>>>>>>>>>>> 0 >>>>>>>>>>>> >>>>>>>>>>>> # nodetool ring >>>>>>>>>>>> Datacenter: dc1 >>>>>>>>>>>> ========== >>>>>>>>>>>> Address Rack Status State Load >>>>>>>>>>>> Owns Token >>>>>>>>>>>> >>>>>>>>>>>> 9189523899826545641 >>>>>>>>>>>> xxx.xxx.xxx.24 rack4 Up Normal 93.82 KiB >>>>>>>>>>>> 79.95% -9194674091837769168 >>>>>>>>>>>> xxx.xxx.xxx.107 rack1 Up Normal 2.65 TiB >>>>>>>>>>>> 73.25% -9168781258594813088 >>>>>>>>>>>> xxx.xxx.xxx.253 rack2 Up Normal 2.6 TiB >>>>>>>>>>>> 73.92% -9163037340977721917 >>>>>>>>>>>> xxx.xxx.xxx.105 rack3 Up Normal 2.65 TiB >>>>>>>>>>>> 72.88% -9148860739730046229 >>>>>>>>>>>> xxx.xxx.xxx.107 rack1 Up Normal 2.65 TiB >>>>>>>>>>>> 73.25% -9125240034139323535 >>>>>>>>>>>> xxx.xxx.xxx.253 rack2 Up Normal 2.6 TiB >>>>>>>>>>>> 73.92% -9112518853051755414 >>>>>>>>>>>> xxx.xxx.xxx.105 rack3 Up Normal 2.65 TiB >>>>>>>>>>>> 72.88% -9100516173422432134 >>>>>>>>>>>> ... >>>>>>>>>>>> >>>>>>>>>>>> This is causing a serious production issue. Please help if you >>>>>>>>>>>> can. >>>>>>>>>>>> >>>>>>>>>>>> Thanks >>>>>>>>>>>> David >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>
