Re: New seed node in the cluster immediately UN without passing for UJ state
> > Just follow up to your statement: > Limiting the seeds to 2 per DC means : > A) Each node in a DC has at least 2 seeds and those seeds belong to the > same DC > or > B) Each node in a DC has at least 2 seeds even across different DC > I apologise for the ambiguity of my previous response, I see it now. :) The recommendation is to pick 2 nodes in each DC to designate as seeds. For example if you had 3 DCs in your cluster, your seeds list might look like: - seeds: "DC1_IP1, DC1_IP2, DC2_IP1, DC2_IP2, DC3_IP1, DC3_IP2" I hope that makes sense. Cheers!
RE: New seed node in the cluster immediately UN without passing for UJ state
Seed node doesn’t bootstrap so if new node were to act as seed node, official recommendations are to boot strap ‘new’ node first , only after that list that node as seed. Seed nodes are usually same across cluster nodes. You can designate 2 nodes as seed per dc in order to mitigate network latency for bootstrapping node ie. Say you have 2dcs you designate 4 nodes (in total) as seed – 2 from each dc. ~Asad From: Sergio [mailto:lapostadiser...@gmail.com] Sent: Tuesday, February 25, 2020 6:19 PM To: user@cassandra.apache.org Cc: erik.rami...@datastax.com Subject: Re: New seed node in the cluster immediately UN without passing for UJ state Hi Erick! Just follow up to your statement: Limiting the seeds to 2 per DC means : A) Each node in a DC has at least 2 seeds and those seeds belong to the same DC or B) Each node in a DC has at least 2 seeds even across different DC Thanks, Sergio Il giorno gio 13 feb 2020 alle ore 19:46 Erick Ramirez mailto:erick.rami...@datastax.com>> ha scritto: Not a problem. And I've just responded on the new thread. Cheers!
Re: New seed node in the cluster immediately UN without passing for UJ state
Hi Erick! Just follow up to your statement: Limiting the seeds to 2 per DC means : A) Each node in a DC has at least 2 seeds and those seeds belong to the same DC or B) Each node in a DC has at least 2 seeds even across different DC Thanks, Sergio Il giorno gio 13 feb 2020 alle ore 19:46 Erick Ramirez < erick.rami...@datastax.com> ha scritto: > Not a problem. And I've just responded on the new thread. Cheers! > >>
Re: New seed node in the cluster immediately UN without passing for UJ state
Not a problem. And I've just responded on the new thread. Cheers! >
Re: New seed node in the cluster immediately UN without passing for UJ state
Thank you very much for this helpful information! I opened a new thread for the other question :) Sergio Il giorno gio 13 feb 2020 alle ore 19:22 Erick Ramirez < erick.rami...@datastax.com> ha scritto: > I want to have more than one seed node in each DC, so unless I don't >> restart the node after changing the seed_list in that node it will not >> become the seed. > > > That's not really going to hurt you if you have other seeds in other DCs. > But if you're willing to take the hit from the restart then feel free to do > so. Just saying that it's not necessary to do it immediately so the option > is there for you. :) > > > Do I need to update the seed_list across all the nodes even in separate >> DCs and perform a rolling restart even across DCs or the restart should be >> happening only on the new node that I want as a seed? > > > You generally want to make the seeds list the same across all nodes in the > cluster. You want to avoid the situation where lots of nodes are used as > seeds by various nodes. Limiting the seeds to 2 per DC means that gossip > convergence will happen much faster. Cheers! > >>
Re: New seed node in the cluster immediately UN without passing for UJ state
> > I want to have more than one seed node in each DC, so unless I don't > restart the node after changing the seed_list in that node it will not > become the seed. That's not really going to hurt you if you have other seeds in other DCs. But if you're willing to take the hit from the restart then feel free to do so. Just saying that it's not necessary to do it immediately so the option is there for you. :) Do I need to update the seed_list across all the nodes even in separate DCs > and perform a rolling restart even across DCs or the restart should be > happening only on the new node that I want as a seed? You generally want to make the seeds list the same across all nodes in the cluster. You want to avoid the situation where lots of nodes are used as seeds by various nodes. Limiting the seeds to 2 per DC means that gossip convergence will happen much faster. Cheers! >
Re: New seed node in the cluster immediately UN without passing for UJ state
Right now yes I have one seed per DC. I want to have more than one seed node in each DC, so unless I don't restart the node after changing the seed_list in that node it will not become the seed. Do I need to update the seed_list across all the nodes even in separate DCs and perform a rolling restart even across DCs or the restart should be happening only on the new node that I want as a seed? The reason each Datacenter has: a seed from the current DC belongs to and a seed from the other DC. Thanks, Sergio Il giorno gio 13 feb 2020 alle ore 18:41 Erick Ramirez < erick.rami...@datastax.com> ha scritto: > 1) If I don't restart the node after changing the seed list this will >> never become the seed and I would like to be sure that I don't find my self >> in a spot where I don't have seed nodes and this means that I can not add a >> node in the cluster > > > Are you saying you only have 1 seed node in the seeds list of each node? > We recommend 2 nodes per DC as seeds -- if one node is down, there's still > another node in the local DC to contact. In the worst case scenario where 2 > nodes in the local DC are down, then nodes can contact seeds in other DCs. > > For the second item, could I make a small request? Since it's unrelated to > this thread, would you mind starting up a new email thread? It just makes > it easier for other users to follow the threads in the future if they're > searching for answers to similar questions. Cheers! > >>
Re: New seed node in the cluster immediately UN without passing for UJ state
> > 1) If I don't restart the node after changing the seed list this will > never become the seed and I would like to be sure that I don't find my self > in a spot where I don't have seed nodes and this means that I can not add a > node in the cluster Are you saying you only have 1 seed node in the seeds list of each node? We recommend 2 nodes per DC as seeds -- if one node is down, there's still another node in the local DC to contact. In the worst case scenario where 2 nodes in the local DC are down, then nodes can contact seeds in other DCs. For the second item, could I make a small request? Since it's unrelated to this thread, would you mind starting up a new email thread? It just makes it easier for other users to follow the threads in the future if they're searching for answers to similar questions. Cheers! >
Re: New seed node in the cluster immediately UN without passing for UJ state
Thank you very much for your response! 2 things: 1) If I don't restart the node after changing the seed list this will never become the seed and I would like to be sure that I don't find my self in a spot where I don't have seed nodes and this means that I can not add a node in the cluster 2) We have i3xlarge instances with data directory in the XFS filesystem that is ephemeral and hints, commit_log and saved_caches in the EBS volume. Whenever AWS is going to retire the instance due to degraded hardware performance is it better: Option 1) - Nodetool drain - Stop cassandra - Restart the machine from aws to be restored in a different VM from the hypervisor - Start Cassandra with -Dcassandra.replace_address OR Option 2) - Add a new node and wait for the NORMAL status - Decommission the one that is going to be retired - Run cleanup with cstar across the datacenters ? Thanks, Sergio Il giorno gio 13 feb 2020 alle ore 18:15 Erick Ramirez < erick.rami...@datastax.com> ha scritto: > I did decommission of this node and I did all the steps mentioned except >> the -Dcassandra.replace_address and now it is streaming correctly! > > > That works too but I was trying to avoid the rebalance operations (like > streaming to restore replica counts) since they can be expensive. > > So basically, if I want this new node as seed should I add its IP address >> after it joined the cluster and after >> - nodetool drain >> - restart cassandra? > > > There's no need to restart C* after updating the seeds list. It will just > take effect the next time you restart. > > I deactivated the future repair happening in the cluster while this node >> is joining. >> When you add a node is it better to stop the repair process? > > > It's not necessary to do so if you have sufficient capacity in your > cluster. Topology changes are just a normal part of a C* cluster's > operation just like repairs. But when you temporarily disable repairs, > existing nodes have more capacity to bootstrap a new node so there is a > benefit there. Cheers! > >>
Re: New seed node in the cluster immediately UN without passing for UJ state
> > I did decommission of this node and I did all the steps mentioned except > the -Dcassandra.replace_address and now it is streaming correctly! That works too but I was trying to avoid the rebalance operations (like streaming to restore replica counts) since they can be expensive. So basically, if I want this new node as seed should I add its IP address > after it joined the cluster and after > - nodetool drain > - restart cassandra? There's no need to restart C* after updating the seeds list. It will just take effect the next time you restart. I deactivated the future repair happening in the cluster while this node is > joining. > When you add a node is it better to stop the repair process? It's not necessary to do so if you have sufficient capacity in your cluster. Topology changes are just a normal part of a C* cluster's operation just like repairs. But when you temporarily disable repairs, existing nodes have more capacity to bootstrap a new node so there is a benefit there. Cheers! >
Re: New seed node in the cluster immediately UN without passing for UJ state
I did decommission of this node and I did all the steps mentioned except the -Dcassandra.replace_address and now it is streaming correctly! So basically, if I want this new node as seed should I add its IP address after it joined the cluster and after - nodetool drain - restart cassandra? I deactivated the future repair happening in the cluster while this node is joining. When you add a node is it better to stop the repair process? Thank you very much Erick! Best, Sergio Il giorno gio 13 feb 2020 alle ore 17:52 Erick Ramirez < erick.rami...@datastax.com> ha scritto: > Should I do something to fix it or leave as it? > > > It depends on what your intentions are. I would use the "replace" method > to build it correctly. At a high level: > - remove the IP from it's own seeds list > - delete the contents of data, commitlog and saved_caches > - add the replace flag in cassandra-env.sh ( > -Dcassandra.replace_address=its_own_ip) > - start C* > > That should allow the node to "replace itself" in the ring and prevent > expensive reshuffling/rebalancing of tokens. Cheers! > >>
Re: New seed node in the cluster immediately UN without passing for UJ state
> > Should I do something to fix it or leave as it? It depends on what your intentions are. I would use the "replace" method to build it correctly. At a high level: - remove the IP from it's own seeds list - delete the contents of data, commitlog and saved_caches - add the replace flag in cassandra-env.sh ( -Dcassandra.replace_address=its_own_ip) - start C* That should allow the node to "replace itself" in the ring and prevent expensive reshuffling/rebalancing of tokens. Cheers! >
Re: New seed node in the cluster immediately UN without passing for UJ state
Thanks for your fast reply! No repairs are running! https://cassandra.apache.org/doc/latest/faq/index.html#does-single-seed-mean-single-point-of-failure I added the node IP itself and the IP of existing seeds and I started Cassandra. So the right procedure is not to add in the seed list the new node and an already existing seed node and then start Cassandra? What should I do? I am running nodetool nestats and the streaming are happening from other nodes Thanks Il giorno gio 13 feb 2020 alle ore 17:39 Erick Ramirez < erick.rami...@datastax.com> ha scritto: > I wanted to add a new node in the cluster and it looks to be working fine >> but instead to wait for 2-3 hours data streaming like 100GB it immediately >> went to the UN (UP and NORMAL) state. >> > > Are you running a repair? I can't see how it's possibly receiving 100GB > since it won't bootstrap. >
Re: New seed node in the cluster immediately UN without passing for UJ state
> > I wanted to add a new node in the cluster and it looks to be working fine > but instead to wait for 2-3 hours data streaming like 100GB it immediately > went to the UN (UP and NORMAL) state. > Are you running a repair? I can't see how it's possibly receiving 100GB since it won't bootstrap.
Re: New seed node in the cluster immediately UN without passing for UJ state
Should I do something to fix it or leave as it? On Thu, Feb 13, 2020, 5:29 PM Jon Haddad wrote: > Seeds don't bootstrap, don't list new nodes as seeds. > > On Thu, Feb 13, 2020 at 5:23 PM Sergio wrote: > >> Hi guys! >> >> I don't know how but this is the first time that I see such behavior. I >> wanted to add a new node in the cluster and it looks to be working fine but >> instead to wait for 2-3 hours data streaming like 100GB it immediately went >> to the UN (UP and NORMAL) state. >> >> I saw a bunch of exception in the logs and WARN >> [MessagingService-Incoming-/10.1.17.126] 2020-02-14 01:08:07,812 >> IncomingTcpConnection.java:103 - UnknownColumnFamilyException reading from >> socket; closing >> org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find table >> for cfId a5af88d0-24f6-11e9-b009-95ed77b72f6e. If a table was just created, >> this is likely due to the schema not being fully propagated. Please wait >> for schema agreement on table creation. >> at >> org.apache.cassandra.config.CFMetaData$Serializer.deserialize(CFMetaData.java:1525) >> ~[apache-cassandra-3.11.5.jar:3.11.5] >> at >> org.apache.cassandra.db.partitions.PartitionUpdate$PartitionUpdateSerializer.deserialize30(PartitionUpdate.java:850) >> ~[apache-cassandra-3.11.5.jar:3.11.5] >> at >> org.apache.cassandra.db.partitions.PartitionUpdate$PartitionUpdateSerializer.deserialize(PartitionUpdate.java:825) >> ~[apache-cassandra-3.11.5.jar:3.11.5] >> at >> org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:415) >> ~[apache-cassandra-3.11.5.jar:3.11.5] >> at >> org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:434) >> ~[apache-cassandra-3.11.5.jar:3.11.5] >> at >> org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:371) >> ~[apache-cassandra-3.11.5.jar:3.11.5] >> at org.apache.cassandra.net.MessageIn.read(MessageIn.java:123) >> ~[apache-cassandra-3.11.5.jar:3.11.5] >> at >> org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:195) >> ~[apache-cassandra-3.11.5.jar:3.11.5] >> at >> org.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:183) >> ~[apache-cassandra-3.11.5.jar:3.11.5] >> at >> org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:94) >> ~[apache-cassandra-3.11.5.jar:3.11.5] >> >> but in the end, it is working... >> >> Suggestion? >> >> Thanks, >> >> Sergio >> >
Re: New seed node in the cluster immediately UN without passing for UJ state
Seeds don't bootstrap, don't list new nodes as seeds. On Thu, Feb 13, 2020 at 5:23 PM Sergio wrote: > Hi guys! > > I don't know how but this is the first time that I see such behavior. I > wanted to add a new node in the cluster and it looks to be working fine but > instead to wait for 2-3 hours data streaming like 100GB it immediately went > to the UN (UP and NORMAL) state. > > I saw a bunch of exception in the logs and WARN > [MessagingService-Incoming-/10.1.17.126] 2020-02-14 01:08:07,812 > IncomingTcpConnection.java:103 - UnknownColumnFamilyException reading from > socket; closing > org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find table > for cfId a5af88d0-24f6-11e9-b009-95ed77b72f6e. If a table was just created, > this is likely due to the schema not being fully propagated. Please wait > for schema agreement on table creation. > at > org.apache.cassandra.config.CFMetaData$Serializer.deserialize(CFMetaData.java:1525) > ~[apache-cassandra-3.11.5.jar:3.11.5] > at > org.apache.cassandra.db.partitions.PartitionUpdate$PartitionUpdateSerializer.deserialize30(PartitionUpdate.java:850) > ~[apache-cassandra-3.11.5.jar:3.11.5] > at > org.apache.cassandra.db.partitions.PartitionUpdate$PartitionUpdateSerializer.deserialize(PartitionUpdate.java:825) > ~[apache-cassandra-3.11.5.jar:3.11.5] > at > org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:415) > ~[apache-cassandra-3.11.5.jar:3.11.5] > at > org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:434) > ~[apache-cassandra-3.11.5.jar:3.11.5] > at > org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:371) > ~[apache-cassandra-3.11.5.jar:3.11.5] > at org.apache.cassandra.net.MessageIn.read(MessageIn.java:123) > ~[apache-cassandra-3.11.5.jar:3.11.5] > at > org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:195) > ~[apache-cassandra-3.11.5.jar:3.11.5] > at > org.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:183) > ~[apache-cassandra-3.11.5.jar:3.11.5] > at > org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:94) > ~[apache-cassandra-3.11.5.jar:3.11.5] > > but in the end, it is working... > > Suggestion? > > Thanks, > > Sergio >
