I am asking this because the only case where I see IP swap can occur is when two Cassandra nodes are running on the same K8S host node. I am evaluating how safe it is two run two Cassandra nodes on a single K8S host node.
*Totally Agree that swap is not the right word but one node still taking other nodes IP* Regards Manish On Tue, Aug 4, 2020 at 10:07 AM manish khandelwal < manishkhandelwa...@gmail.com> wrote: > But again if > Some Cassandra node (pod) with particular IP X is down, > Second Cassandra node (pod) tries to take the IP X of first Cassandra node, > Second Cassandra node should fail to join the cluster as the Cassandra > cluster will complain that IP X is already occupied. > In that sense actual swap of IPs can occur between two nodes and data > issues should not occur. > > I am asking this because the only case where I see IP swap can occur is > when two Cassandra nodes are running on the same K8S host node. I am > evaluating how safe it is two run two Cassandra nodes on a single K8S host > node. > > Regards > Manish > > > > On Tue, Aug 4, 2020 at 9:01 AM Christopher Bradford <bradfor...@gmail.com> > wrote: > >> In *most* k8s environments each Kubernetes worker receives its own >> dedicated CIDR range from the cluster’s CIDR space for allocating pod IP >> addresses. The issue described can occur when a k8s worker goes down then >> comes back up and the pods are rescheduled where either pod starts up with >> another pods previously used IP. >> >> They don’t necessarily have to swap 1:1 (ie one pod could use the other’s >> previous address while that pod receives a new address). Additionally it’s >> not a race condition of which container starts first. The k8s scheduler and >> kubelet daemon assign IPs to pods. >> >> On Mon, Aug 3, 2020 at 11:14 PM manish khandelwal < >> manishkhandelwa...@gmail.com> wrote: >> >>> I have started reading about how to deploy Cassandra with K8S. But as I >>>> read more I feel there are a lot of challenges in running Cassandra on K8s. >>>> Some of the challenges which I feel are >>>> >>>> 1. POD IPs identification - If the pods go down and when they come up >>>> their IPs change, how is it handled as we are dependent on IPs of Cassandra >>>> nodes for internode as well client server communication. >>>> >>> >>> Strictly safe to change an IP to an IP that is unused. >>> Strictly unsafe to use an IP that's already in the cluster (so if two >>> pods go down, having the first pod that comes up grab the IP of the second >>> pod is strictly dangerous and will violate consistency and maybe lose >>> data). >>> >>> *For point 2 (Strictly unsafe to use an IP) , if the first pod grabs the >>> IP of the second node then it should not be able to join the cluster.* >>> *So can IPs still be swapped? * >>> *When and how this IP swap can occur?* >>> >>> Regards >>> Manish >>> >>> On Mon, Jul 6, 2020 at 10:40 PM Jeff Jirsa <jji...@gmail.com> wrote: >>> >>>> >>>> >>>> On Mon, Jul 6, 2020 at 10:01 AM manish khandelwal < >>>> manishkhandelwa...@gmail.com> wrote: >>>> >>>>> I have started reading about how to deploy Cassandra with K8S. But as >>>>> I read more I feel there are a lot of challenges in running Cassandra on >>>>> K8s. Some of the challenges which I feel are >>>>> >>>>> 1. POD IPs identification - If the pods go down and when they come up >>>>> their IPs change, how is it handled as we are dependent on IPs of >>>>> Cassandra >>>>> nodes for internode as well client server communication. >>>>> >>>> >>>> Strictly safe to change an IP to an IP that is unused. >>>> Strictly unsafe to use an IP that's already in the cluster (so if two >>>> pods go down, having the first pod that comes up grab the IP of the second >>>> pod is strictly dangerous and will violate consistency and maybe lose >>>> data). >>>> >>>> >>>>> >>>>> 2. A K8S node can host a single pod. This is being done so that even >>>>> if the host goes down we have only one pod down case. With multiple pods >>>>> on >>>>> a single host there is a risk of traffic failures as consistency might not >>>>> be achieved. But if we keep two pods of the same rack on a single host >>>>> then >>>>> are we safe or is there any unknown risk? >>>>> >>>> >>>> This sounds like you're asking if rack aware snitches protect you from >>>> concurrent pods going down. Make sure you're using a rack aware snitch. >>>> >>>> >>>>> >>>>> 3. Seed discovery? Again as an extension of point 1, since IPs can >>>>> change, how we can manage seeds. >>>>> >>>> >>>> Either use DNS instead of static IPs, or use a seed provider that >>>> handles IPs changing. >>>> >>>> >>>>> >>>>> 4. Also I read a lot of use of Cassandra operators for maintaining a >>>>> Cassandra cluster on Kubernetes. I think that Cassandra Operator is like a >>>>> robot (automated admin) which works and acts like a norma admin will work. >>>>> I want to understand that how important is Cassandra operator and what if >>>>> we go on to production without one? >>>>> >>>>> Regards >>>>> Manish >>>>> >>>> -- >> >> Christopher Bradford >> >>
