RE: Rejection of peer join.
Great! Thanks Stuart. Appreciate the advise! Regards, Michael -Original Message- From: stu...@stuartbishop.net [mailto:stu...@stuartbishop.net] On Behalf Of Stuart Bishop Sent: Thursday, September 28, 2017 3:59 PM To: Michael Van Der Beek Cc: juju@lists.ubuntu.com Subject: Re: Rejection of peer join. On 28 September 2017 at 10:09, Michael Van Der Beek wrote: > Hi Stuart, > > Thanks for your insight. Greatly appreciate it. > > Last thing, when you said " I'm generally deploying charms to bare > metal and using local disk", Does that mean you manually install a > machine with Ubuntu, then in juju add-machine ssh:user@ Then juju > deploy --to ? Manually installing a machine with Ubuntu would work, but we provision the machine with MAAS. We then add it to an OpenStack Juju model using 'juju add-machine' as you suggest (the 'manual provider'). This lets us mix OpenStack VMs and bare metal in the same model. The downside is we have to handle network issues and firewall rules to the bare metal units ourselves. I expect we will change to using cross model relations when they are available between clouds, so we use a MAAS model for the bare metal units and connect them to OpenStack units in an OpenStack model. Ideally we could mix and match units from different cloud providers in the same model, but I believe that is a long way off. -- Stuart Bishop -- Juju mailing list Juju@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju
Re: Rejection of peer join.
On 28 September 2017 at 10:09, Michael Van Der Beek wrote: > Hi Stuart, > > Thanks for your insight. Greatly appreciate it. > > Last thing, when you said " I'm generally deploying charms to bare metal and > using local disk", > Does that mean you manually install a machine with Ubuntu, then in juju > add-machine ssh:user@ > Then juju deploy --to ? Manually installing a machine with Ubuntu would work, but we provision the machine with MAAS. We then add it to an OpenStack Juju model using 'juju add-machine' as you suggest (the 'manual provider'). This lets us mix OpenStack VMs and bare metal in the same model. The downside is we have to handle network issues and firewall rules to the bare metal units ourselves. I expect we will change to using cross model relations when they are available between clouds, so we use a MAAS model for the bare metal units and connect them to OpenStack units in an OpenStack model. Ideally we could mix and match units from different cloud providers in the same model, but I believe that is a long way off. -- Stuart Bishop -- Juju mailing list Juju@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju
RE: Rejection of peer join.
Hi Stuart, Thanks for your insight. Greatly appreciate it. Last thing, when you said " I'm generally deploying charms to bare metal and using local disk", Does that mean you manually install a machine with Ubuntu, then in juju add-machine ssh:user@ Then juju deploy --to ? Or you using juju deploy --to host.maas and in which landscape is already aware of a machine? I guess that is probably a dumb question, they both are generally the same, except one does it by bootp and the other is a manual install of ubuntu then juju provisions it Regards, Michael -Original Message- From: stu...@stuartbishop.net [mailto:stu...@stuartbishop.net] On Behalf Of Stuart Bishop Sent: Wednesday, September 27, 2017 8:47 PM To: Michael Van Der Beek Cc: juju@lists.ubuntu.com Subject: Re: Rejection of peer join. On 27 September 2017 at 14:50, Michael Van Der Beek wrote: > Hi Stuart, > > I think you misinterpreted what I was asking > > Assuming a pair of instance already are in relationship. > Lets assume we have drbd running between these two instances lets call it A-B. > If juju starts a 3rd instance C. I want to make sure the 3rd cannot join the > pair as drbd not supposed to have 3rd node. Although in theory you can create > a stack node for 3rd or more backups. > > So when the ha-relation-joined is triggered on either A or B. How do I tell > C, it is rejected from the join so that it doesn't screw up juju. I suppose > if A/B can send do a "relation-set" some thing to tell C, it is rejected. I would have C use relation-list to get a list of all the peers. If there are two or more units with a lower unit number, C knows it should not join, sets its status to 'blocked' and exit 0. There is no need to involve A or B in the conversation; Unit C can make the decision itself. Unit C could happily sit there in blocked state until either A or B departs, at which point C would see only one unit with a lower unit number and know it should join. You could even argue that the blocked state is incorrect with this design, and that unit C should actually set its status to active or maintenance, with a message stating it is a hot spare. The trick is that if someone does 'juju deploy -n3 thecharm', there is no guarantee on what order the units join the peer relation. So Unit C may think it should be active, and then a short while later sees other units join and will need to become inactive. If you need to avoid this situation, you are going to need to use Juju leadership to avoid these sorts of race conditions. There is only one leader at any point in time, so it can make decisions like which of the several units should be active or not without worrying about race conditions. But it sounds overly complex for your use case. > The issue for me, is how to scale if you have specific data in a set of > nodes. So you can ceph, or drbd or some cluster. So ceph will require 3 ceph > nodes, drbd two nodes and maybe galera cluster 3 nodes. > > So my idea is that there is already a loadbalance to scale. So my idea is > each time you want to scale you would add one or more pairs (assuming drbd) > to an already existing set of pairs. The load balancer will just redirect > data to specific pairs based on some logic (like modulus of the last octet of > customer IP which can give you 256 pairs). This is how we are doing on > physical machines. Haven't had a customer yet that requires more than 10,000 > tps for radius or 5 million concurrent sessions). Note I use pairs loosely in > this line as the pair if running galera cluster is 3 nodes instead of pair). > > I'm currently trying to figure how to do it on openstack. If you have some > recommendation for me to read/view on how people deal with scaling for very > high write IO to disk. Current for radius we are looking at near 95% writes > 5%reads. Nobody reads the data unless someone wants to know if user X is > currently logged in. If it was the other way around in (R/W) IO requirements > its much easier to scale. I'm generally deploying charms to bare metal and using local disk if there are any sort of non-trivial IO requirements. I believe the newer Juju storage features should allow you mount whatever sort of volumes you want from OpenStack (or any cloud provider), but I'm not familiar with the OpenStack specifics. -- Stuart Bishop -- Juju mailing list Juju@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju
Re: Rejection of peer join.
On 27 September 2017 at 14:50, Michael Van Der Beek wrote: > Hi Stuart, > > I think you misinterpreted what I was asking > > Assuming a pair of instance already are in relationship. > Lets assume we have drbd running between these two instances lets call it A-B. > If juju starts a 3rd instance C. I want to make sure the 3rd cannot join the > pair as drbd not supposed to have 3rd node. Although in theory you can create > a stack node for 3rd or more backups. > > So when the ha-relation-joined is triggered on either A or B. How do I tell > C, it is rejected from the join so that it doesn't screw up juju. I suppose > if A/B can send do a "relation-set" some thing to tell C, it is rejected. I would have C use relation-list to get a list of all the peers. If there are two or more units with a lower unit number, C knows it should not join, sets its status to 'blocked' and exit 0. There is no need to involve A or B in the conversation; Unit C can make the decision itself. Unit C could happily sit there in blocked state until either A or B departs, at which point C would see only one unit with a lower unit number and know it should join. You could even argue that the blocked state is incorrect with this design, and that unit C should actually set its status to active or maintenance, with a message stating it is a hot spare. The trick is that if someone does 'juju deploy -n3 thecharm', there is no guarantee on what order the units join the peer relation. So Unit C may think it should be active, and then a short while later sees other units join and will need to become inactive. If you need to avoid this situation, you are going to need to use Juju leadership to avoid these sorts of race conditions. There is only one leader at any point in time, so it can make decisions like which of the several units should be active or not without worrying about race conditions. But it sounds overly complex for your use case. > The issue for me, is how to scale if you have specific data in a set of > nodes. So you can ceph, or drbd or some cluster. So ceph will require 3 ceph > nodes, drbd two nodes and maybe galera cluster 3 nodes. > > So my idea is that there is already a loadbalance to scale. So my idea is > each time you want to scale you would add one or more pairs (assuming drbd) > to an already existing set of pairs. The load balancer will just redirect > data to specific pairs based on some logic (like modulus of the last octet of > customer IP which can give you 256 pairs). This is how we are doing on > physical machines. Haven't had a customer yet that requires more than 10,000 > tps for radius or 5 million concurrent sessions). Note I use pairs loosely in > this line as the pair if running galera cluster is 3 nodes instead of pair). > > I'm currently trying to figure how to do it on openstack. If you have some > recommendation for me to read/view on how people deal with scaling for very > high write IO to disk. Current for radius we are looking at near 95% writes > 5%reads. Nobody reads the data unless someone wants to know if user X is > currently logged in. If it was the other way around in (R/W) IO requirements > its much easier to scale. I'm generally deploying charms to bare metal and using local disk if there are any sort of non-trivial IO requirements. I believe the newer Juju storage features should allow you mount whatever sort of volumes you want from OpenStack (or any cloud provider), but I'm not familiar with the OpenStack specifics. -- Stuart Bishop -- Juju mailing list Juju@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju
RE: Rejection of peer join.
Hi Stuart, I think you misinterpreted what I was asking Assuming a pair of instance already are in relationship. Lets assume we have drbd running between these two instances lets call it A-B. If juju starts a 3rd instance C. I want to make sure the 3rd cannot join the pair as drbd not supposed to have 3rd node. Although in theory you can create a stack node for 3rd or more backups. So when the ha-relation-joined is triggered on either A or B. How do I tell C, it is rejected from the join so that it doesn't screw up juju. I suppose if A/B can send do a "relation-set" some thing to tell C, it is rejected. So A/B can now exit 0. And C can set blocked and exit 0. So would that create a satisfactory state in juju so that there is no relationship set but all exit okay. Now as to your other suggestion of starting C and allowing data to be synced to C, before dropping one node say A. That is possible as well, just means that you'll restrict at a higher number. The issue for me, is how to scale if you have specific data in a set of nodes. So you can ceph, or drbd or some cluster. So ceph will require 3 ceph nodes, drbd two nodes and maybe galera cluster 3 nodes. So my idea is that there is already a loadbalance to scale. So my idea is each time you want to scale you would add one or more pairs (assuming drbd) to an already existing set of pairs. The load balancer will just redirect data to specific pairs based on some logic (like modulus of the last octet of customer IP which can give you 256 pairs). This is how we are doing on physical machines. Haven't had a customer yet that requires more than 10,000 tps for radius or 5 million concurrent sessions). Note I use pairs loosely in this line as the pair if running galera cluster is 3 nodes instead of pair). I'm currently trying to figure how to do it on openstack. If you have some recommendation for me to read/view on how people deal with scaling for very high write IO to disk. Current for radius we are looking at near 95% writes 5%reads. Nobody reads the data unless someone wants to know if user X is currently logged in. If it was the other way around in (R/W) IO requirements its much easier to scale. Regards, Michael -Original Message- From: stu...@stuartbishop.net [mailto:stu...@stuartbishop.net] On Behalf Of Stuart Bishop Sent: Wednesday, September 27, 2017 2:38 PM To: Michael Van Der Beek Cc: juju@lists.ubuntu.com Subject: Re: Rejection of peer join. On 27 September 2017 at 10:02, Michael Van Der Beek wrote: > Hi All, > > I was thinking about a charm, that for peer relationship, supports > only two instances to form a peer. > > What would be the correct response to reject a 3rd instance trying to > join gracefully so that juju doesn’t complain of a failed setup? > > Exit 1 this will sure cause problems. Exit 0 but then juju doesn’t > know that the relationship is not set. Set the workload status of the unit that should not join to 'blocked' with a message explaining the problem, and Exit 0 (using the 'status-set' hook environment tool, or hookenv.status_set() in charm-helpers). The live peers should just ignore the third unit. If you do it right, it would be possible to add a third unit, then remove one of the live pair, and have everything migrate across to the new unit with only a small period of time without redundancy. > The idea is that I want to do failover within a pair of instances. > > I have a radius load balancer that will split packets based on a > modulus of same framedip (ip of client) or mac address or something. > > So each pair of radius would only hold sessions related to that > modulus result. So if I do modulus 4, I’ll have 4 pairs of instances. > > I’m not sure if running each pair (active-standby) support 1000tps is > viable up to a million sessions. You may not be sure, but maybe the person who is using your charm is? Or wants to test it? If it is no more effort, I'd suggest supporting an arbitrary number of units rather than just one or two. It could also provide a mechanism to migrate the Juju application to new units without ever losing redundancy by allowing the operator to bring a third unit online, wait for it to get in sync, and then drop one of the original units. -- Stuart Bishop -- Juju mailing list Juju@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju
Re: Rejection of peer join.
On 27 September 2017 at 10:02, Michael Van Der Beek wrote: > Hi All, > > I was thinking about a charm, that for peer relationship, supports only two > instances to form a peer. > > What would be the correct response to reject a 3rd instance trying to join > gracefully so that juju doesn’t complain of a failed setup? > > Exit 1 this will sure cause problems. Exit 0 but then juju doesn’t know that > the relationship is not set. Set the workload status of the unit that should not join to 'blocked' with a message explaining the problem, and Exit 0 (using the 'status-set' hook environment tool, or hookenv.status_set() in charm-helpers). The live peers should just ignore the third unit. If you do it right, it would be possible to add a third unit, then remove one of the live pair, and have everything migrate across to the new unit with only a small period of time without redundancy. > The idea is that I want to do failover within a pair of instances. > > I have a radius load balancer that will split packets based on a modulus of > same framedip (ip of client) or mac address or something. > > So each pair of radius would only hold sessions related to that modulus > result. So if I do modulus 4, I’ll have 4 pairs of instances. > > I’m not sure if running each pair (active-standby) support 1000tps is viable > up to a million sessions. You may not be sure, but maybe the person who is using your charm is? Or wants to test it? If it is no more effort, I'd suggest supporting an arbitrary number of units rather than just one or two. It could also provide a mechanism to migrate the Juju application to new units without ever losing redundancy by allowing the operator to bring a third unit online, wait for it to get in sync, and then drop one of the original units. -- Stuart Bishop -- Juju mailing list Juju@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju
Rejection of peer join.
Hi All, I was thinking about a charm, that for peer relationship, supports only two instances to form a peer. What would be the correct response to reject a 3rd instance trying to join gracefully so that juju doesn't complain of a failed setup? Exit 1 this will sure cause problems. Exit 0 but then juju doesn't know that the relationship is not set. Or is there some setting for peer in metadata.yaml that specifies only two instances at a time. Or this is not possibile, and must be chosen manually? The idea is that I want to do failover within a pair of instances. I have a radius load balancer that will split packets based on a modulus of same framedip (ip of client) or mac address or something. So each pair of radius would only hold sessions related to that modulus result. So if I do modulus 4, I'll have 4 pairs of instances. I'm not sure if running each pair (active-standby) support 1000tps is viable up to a million sessions. I can easily do that on an hardware platform, I'm not sure in an openstack environment with all its virtualization overheads is that still possible in a live environment. Thanks for your help. Regards, Michael -- Juju mailing list Juju@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju
