Very interesting discussion gents. Couple of questions/musings:
why is it necessary to have the management-private-<cloud id> group created by the manager of the cluster? Can't each Brooklyn node check what cloud it is on as it starts and then either create the management-private-<cloud id> or join it? It wasn't clear to me whether step 2: 2. Go to each cloud and update the "management-private-ingress-<cloud id>" group, adding the new public IP. was something that could be done automatically; again I would hope this is something that could avoid manual actions, where each new Brooklyn joining the cluster would announce itself and its address to the others (not sure how that can be done at present, maybe writing to the shared persistent store, and all HA Brooklyns monitor the store for new cluster members?), and when a new member joined then a Brooklyn in each cloud "<cloud id>" would update management-private-ingress-<cloud id> with the new public address. It seems to me that the most important unresolved question in the discussions is "Need to figure out what's the private IP. How do you tell apart public form private interfaces, what if the machine is part of multiple networks." Is there any way for Brooklyn to be able to tell its own public IP, other than to add it to configuration? Especially given that it may be sitting behind layers of NAT or proxies and may not actually be on a host with a network interface whose IP is the public one? Is there some external service that it could do a handshake with to make a best-guess of its own public IP, which might be good enough in many cases? Geoff On Wed, 15 Feb 2017 at 14:52 Svetoslav Neykov < svetoslav.ney...@cloudsoftcorp.com> wrote: > > What is the benefit of the "management-private-<cloud id>" group? It > seems superfluous to me if "management-private-ingress-<cloud id>" will > contain the public IPs of the members of the cluster. > > > Needed for in-cloud access. Even if the public IP is white-listed, local > access will use the private IP and will be rejected. So we either need that > or need to include the private IP along the public IP. But then: > * Need to figure out what's the private IP. How do you tell apart public > form private interfaces, what if the machine is part of multiple networks. > Do we just include all local interfaces in the SG (in addition to the > public)? > * Need to advertise it in the HA cluster metadata > * Do we include all private IPs in all clouds? Could allow someone to > impersonate the unused IPs. If not how do we tell which IP is for which > cloud? > > On one hand having the SG removes some cumbersome questions. On the other > removing it makes the cluster self-manageable. No need for external > intervention. > > > > > > On 15.02.2017 г., at 16:34, Sam Corbett <sam.corb...@cloudsoftcorp.com> > wrote: > > > > Thanks for clarifying. It was this cross-cloud case I was considering. > > > > What is the benefit of the "management-private-<cloud id>" group? It > seems superfluous to me if "management-private-ingress-<cloud id>" will > contain the public IPs of the members of the cluster. > > > > > > On 15/02/2017 14:16, Svetoslav Neykov wrote: > >> My response below was based on my thinking about in-cloud access > between Brooklyn and the managed machines. > >> When the access is cross-cloud we indeed need _c_ security groups. Each > one containing _n_ records with the public IPs of the HA cluster members. > These can be managed by Brooklyn though. > >> > >> To summarise (and get my thinking straight). In each cloud/availability > zone we could have: > >> 1. "management-private-<cloud id>" SG assigned on the HA cluster > member machines. No records in the SG. > >> 2. "management-private-ingress-<cloud id>" SG with first record > allowing all traffic from "management-private-<cloud id>" (above group) and > _n_ more records allowing all traffic coming from the public IPs of the HA > member nodes. This one is assigned to all managed entities. > >> > >> On adding a new member to the cluster: > >> 1. Assign "management-private-<cloud id>" to the new machine - this > is the only action that needs to be done by the manager of the cluster > >> 2. Go to each cloud and update the "management-private-ingress-<cloud > id>" group, adding the new public IP. > >> > >> Svet. > >> > >> > >>> On 15.02.2017 г., at 15:40, Svetoslav Neykov < > svetoslav.ney...@cloudsoftcorp.com> wrote: > >>> > >>>> You propose that the manager of the Brooklyn HA cluster maintain at > least _c_ security groups (one per security group scope - e.g. AWS EC2 > region - per cloud). > >>> Not quite. It's the number of "clouds HA cluster is deployed to". > That's the number of availability zones - usually a low number 1-2-3. > >>> > >>>> Each of these groups has _n_ records. When the HA cluster is resized > each group is modified to add or remove a record as appropriate. > >>> No, they are empty. Adding a new cluster member is just assigning the > SG to the new machine. Then for each managed entity there's a record in its > security group to allow traffic from the HA security group. > >>> For clouds where there are no running HA cluster members Brooklyn will > auto-create the security group and add it to entities' security groups but > it will be unused. > >>> > >>> The manager of the HA cluster can wait for Brooklyn to create the > "management" SG (which will contain the cluster ID in its name which is not > known in advance) and then assign it to the HA cluster members or create > one with a predefined name and configure the name in each HA instance. > >>> > >>> Svet. > >>> > >>> > >>> > >>>> On 15.02.2017 г., at 14:59, Sam Corbett < > sam.corb...@cloudsoftcorp.com> wrote: > >>>> > >>>> Interesting problem Svet. Your proposal is a neat way of sidestepping > the problem of updating many security groups as the set of HA nodes changes. > >>>> > >>>> Let me rephrase it to see if I understand correctly. > >>>> > >>>> Suppose we have: > >>>> * _n_ Brooklyn servers in an HA cluster (that could be across many > clouds / regions within a cloud) > >>>> * _c_ clouds that Brooklyn can deploy to > >>>> * _m_ instances across those clouds. > >>>> > >>>> We want to avoid the n+1th Brooklyn node requiring _m_ security group > updates. > >>>> > >>>> You propose that the manager of the Brooklyn HA cluster maintain at > least _c_ security groups (one per security group scope - e.g. AWS EC2 > region - per cloud). Each of these groups has _n_ records. When the HA > cluster is resized each group is modified to add or remove a record as > appropriate. > >>>> > >>>> Do I have this correct? > >>>> > >>>> Sam > >>>> > >>>> > >>>> On 14/02/2017 15:42, Svetoslav Neykov wrote: > >>>>> I'm trying to restrict access to the machines managed by Brooklyn > using security groups - tightening jclouds' default behaviour of opening > the "inboundPorts" to any source. > >>>>> Brooklyn obviously needs to have access to all managed machines. > This means it needs to figure out the address it uses to access each > machine and white list it in the machine's security group. > >>>>> This is kind of related to the email thread "[PROPOSAL] Separate > management addresses from the concept of an entity's public address" [1], > but in reverse. Instead of figuring out which machine IP to use I need to > do the reverse - which Brooklyn node IP will access the machine. > >>>>> It becomes more complicated when HA is introduced into the mix. Any > node that becomes a master needs to be able to access the machines. This > means the security groups need to be updated in such cases. > >>>>> > >>>>> Two questions follow: > >>>>> 1. How to determine which IP faces managed machines? There's no one > fixed answer here. Depending on the target cloud and location configuration > it varies. > >>>>> 2. How to keep the list of IPs from above point in sync, for each > of the members of the HA cluster? > >>>>> > >>>>> Don't think we can actually answer q1. That's why the solution I'm > thinking of is: > >>>>> * Always open the external IP to the machines. The external IP is > as reported by "LocalhostExternalIpLoader". > >>>>> * Assign a predefined SG to all machines in the HA cluster - > manually/out of band, since the machines are not managed by Brooklyn. Let > Brooklyn know the SG name, defaulting to "management-<cluster-id>". White > list the SG as a source for all managed machines. This will allow Brooklyn > to access managed machines on both the public and private IPs. It moves the > responsibility of assigning the SG to new HA member machines to whoever is > managing the Brooklyn cluster. We could then update the management SG with > **all** private IPs in the HA cluster (need to advertise them in the meta > data) or leave it again to the manager of the cluster. > >>>>> > >>>>> Would be really cool to have HA clusters manage/heal themselves. > >>>>> > >>>>> Tangentially related - [2] which IP do we use for the "url" field > int he HA member nodes metadata in REST API (currently empty for the Karaf > dist). If it's always the public IP then it doesn't work for private/VPN > instances . It is important for this to be the right one because: > >>>>> * Users are redirected to the master node > >>>>> * Automated systems need to know which is the current master. On > failover the old master (if still around) will redirect to the new master. > Workaround is to keep a local copy of the HA members and iterate over them > until it hits MASTER - but it's still important that the URLs are > accessible. > >>>>> > >>>>> Svet > >>>>> > >>>>> [1] > https://lists.apache.org/list.html?dev@brooklyn.apache.org:lte=1y:%5BPROPOSAL%5D%20Separate%20management%20addresses%20from%20the%20concept%20of%20an%20entity%27s%20public%20address > >>>>> [2] https://issues.apache.org/jira/browse/BROOKLYN-436 > > > >
