(+l...@google.com)
During the course of this discussion, it occured to me that the proposed
solution for case 3 may affect how the clients are distributed across the
grpclb load balancers.
Bill, in this scenario, the client will not know the IP addresses of the
balancers directly; instead, it will just know the proxy address and will
depend on the proxy to resolve the internal names of the balancers. This
means that even if there are many balancers, the client will not have
connections to all of them, but every time it connects to the proxy, it's
likely to get a different balancer. So if the individual balancer that it
happens to be talking to goes down, it will create a new connection to the
proxy, which will pick a new balancer task to talk to.
I think this is probably fine w.r.t. our conversations about properly
balancing load across the balancers, but I want to make sure that there's
no problem here that I might be missing. Does this sound reasonable to you?
Thanks!
On Fri, Jan 27, 2017 at 3:41 PM, Mark D. Roth <r...@google.com> wrote:
> On Fri, Jan 27, 2017 at 1:31 PM, Eric Anderson <ej...@google.com> wrote:
>
>> On Fri, Jan 27, 2017 at 12:16 PM, 'Mark D. Roth' via grpc.io <
>> grpc-io@googlegroups.com> wrote:
>>
>>> Yes. And that seems to agree with how the different proxy choosing logic
>>>> will work; the first primarily consumes hostnames and returns proxy
>>>> hostnames (which is http_proxy in C) and the second one primarily consumes
>>>> IPs and returns proxy IPs.
>>>>
>>>
>>> I don't think that's actually entirely correct. The first case doesn't
>>> consume anything; it unconditionally sets the hostname to be resolved.
>>>
>>
>> The first case will consume a hostname in Java. Observing the hostname is
>> necessary to fix the mixed internal/external in an expanded view of case 1.
>> Since the Java APIs support that mixed case, Java ends up needing to
>> support them. And if C ever needed to support the mixed case (which seems
>> likely to me), then it would also need to use the hostname.
>>
>>
>>> And the second case can consume either the hostname or the IP.
>>>
>>
>> And I wouldn't be surprised if only IP were used. We're not aware of a
>> user of it.
>>
>> This is more philosophical than practical,
>>>
>>
>> My further explanation there was meant to be more philosophical, as an
>> explanation that this "special case" is pretty normal and sort of agrees
>> with the rest of the design.
>>
>> But that philosophical debate aside, I think that we should focus on case
>>>>> 3, because that's a concrete case that we do want to support. So far, at
>>>>> least, I have not heard a workable proposal that does not require the
>>>>> proxy
>>>>> mapper to control the CONNECT argument (although I'm certainly still open
>>>>> to new proposals).
>>>>>
>>>>
>>>> I've provided two proposals. Neither of which seem debunked as of yet.
>>>> I could totally agree they may be worse than what you are proposing, but
>>>> the discussion hasn't gotten to that point. The mentioned security issue of
>>>> the first proposal seemed to ignore the fact that a reverse proxy could be
>>>> used to "protect" the LB, in an identical fashion to any forward proxy.
>>>>
>>>
>>> I don't quite understand the proposed reverse proxy approach. Can you
>>> explain how that would work in more detail?
>>>
>>
>> Case 3 as stated today (for contrasting)
>>
>> 1. client wants to connect to service.example.com
>> 2. do DNS SRV resolution for _grpclb._tcp.service.example.com; you
>> find it is a LB with name lb.example.com
>> 3. do a DNS resolution for lb.example.com, get IP 1.2.3.4
>> 4. ask the proxy mapper about IP 1.2.3.4, it recognizes the IP as the
>> proxy and says to use "CONNECT service.example.com" via proxy IP
>> 1.2.3.4
>> 5. connect to proxy 1.2.3.4, it performs internal resolution of
>> service.example.com and connects to one of the hosts
>>
>> That's not actually an accurate representation of how case 3 is proposed
> to work in the current document. The document is actually proposing the
> following:
>
> 1. client wants to connect to service.example.com
> 2. do DNS SRV resolution for _grpclb._tcp.service.example.com; you
> find it is a LB with name lb.example.com
> 3. do a DNS resolution for lb.example.com, get IP 1.2.3.4
> 4. ask the proxy mapper about IP 1.2.3.4; it recognizes the IP as the
> proxy and says to use "CONNECT lb.example.com" via proxy IP 1.2.3.4
> 5. connect to proxy 1.2.3.4 with "CONNECT lb.example.com"; proxy does
> internal name resolution and connects to one of the load balancers
> 6. send grpclb request; get response indicating that the backend
> server is 5.6.7.8
> 7. ask the proxy mapper about IP 5.6.7.8; it recognizes it as an
> internal IP address and says to use "CONNECT 5.6.7.8" via proxy IP 1.2.3.4
> 8. connect to proxy 1.2.3.4 with "CONNECT 5.6.7.8"; proxy connects to
> the specified backend server
>
> Remember that the goal of case 3 is to allow client-side per-call load
> balancing, despite not being able to resolve the internal names of the
> backend servers. Instead of getting those from DNS, we get them from the
> grpclb balancer.
>
>
>
>> Case 3 using reverse proxy for LB
>>
>> 1. client wants to connect to service.example.com
>> 2. do DNS SRV resolution for _grpclb._tcp.service.example.com; you
>> find it is a LB with name lb.example.com
>> 3. do a DNS resolution for lb.example.com, get IP 1.2.3.4
>> 4. (different starting here) connect to 1.2.3.4, which is a
>> transparent reverse proxy
>> 5. Perform an RPC to 1.2.3.4. Host header is lb.example.com. The
>> proxy performs internal mapping of lb.example.com to internal
>> addresses and connects to one of the hosts, forwarding the RPC.
>>
>> The reverse proxy approach is essentially what I originally suggested for
> case 3, but Julien argued that it would be a security problem.
>
> Keep in mind that in case 3, the grpclb load balancers and the server
> backends are in the same internal domain, with the same access
> restrictions. If we can't use a reverse proxy to access the server
> backends, I don't think we'll be able to do that for the grpclb balancers
> either.
>
> That having been said, as a security issue, Julien can address this
> directly.
>
>
>
>>
>> I agree that case 3 requires different parts of the system to be
>>> coordinated. For example, assuming that your proxy mapper implementation
>>> is getting the list of proxy addresses from a local file, you would need to
>>> first push an updated list that contains the new proxy address to all
>>> clients. Then, once all clients have been updated, you can add the new
>>> proxy to DNS.
>>>
>>
>> And the file needs to contain old proxy addresses that should be used for
>> detection but not be used.
>>
>> Okay. So we're on the same page there.
>>
>> I agree that this is cumbersome, but I think it's an inherent problem
>>> with case 3, because you need *some* way to configure the clients.
>>>
>>
>> I agree you need to be able to configure the clients. I understand that
>> something needs to tell the client what to do. My concern was the pain of
>> updating the proxy mapping list in concert with name resolution. And
>> because of that I would recommend implementors to use the magic IP, because
>> it has less operational overhead and less likelihood of failing.
>>
>> If you assume "one proxy" which has "one static IP" and everything is
>>>> hard-coded, then the design is fine. But that seems unlikely to describe a
>>>> productionized system. And that's why I would feel forced to use the "magic
>>>> IP" that it seems you have previously rejected.
>>>>
>>>
>>> There are a couple of reasons that I don't like the "magic IP"
>>> approach. First, it requires writing a custom resolver in addition to a
>>> custom proxy mapper,
>>>
>>
>> No, I'd just have DNS return the trash IP.
>>
>
> This actually makes me even less happy with the sentinel-value approach,
> because now we wouldn't just be using the value internally in a particular
> piece of software; we'd actually be publishing it in a way that would be
> very confusing when people were trying to debug the system from an
> operational perspective. ("Wait, why is the client even attempting to
> connect to the proxy, since DNS points it at this bogus IP address?")
>
>
>
>>
>> I'm not a big fan of "sentinel" values, since it's often hard to find a
>>> value that will never be used in real life.
>>>
>>
>> I would gladly accept a magic value instead of needing to make sure two
>> systems stay in sync and rollouts happen properly. And I would quickly
>> recommend that to others. And if I started explaining the gotchas of the
>> alternative, I'd expect them to quickly be thankful for the recommendation
>> since it is less code to write and less operational complexity.
>>
>
> I do see your point, but I think that the sentinel-value approach has
> operational downsides of its own. There are pros and cons here, so it
> basically boils down to a judgement call, and personally, I prefer the
> alternative that's currently outlined in the doc.
>
>
> Just thinking out loud here about whether there's another alternative --
> this is a purely brainstorming-level idea, so please feel free to shoot
> holes in it. What if we had another type of SRV record specifically for
> HTTP CONNECT proxy use? The presence of that record would tell the client
> to connect to that address and issue a CONNECT request using the originally
> looked up name. With that, case 3 would look something like this:
>
> 1. client wants to connect to service.example.com
> 2. do DNS SRV resolution for _grpclb._tcp.service.example.com; you
> find it is a LB with name lb.example.com
> 3. do DNS SRV resolution for _grpc_proxy._tcp.lb.example.com; you find
> it is a proxy with name proxy.example.com
> 4. do DNS lookup for proxy.example.com; get IP 1.2.3.4
> 5. connect to proxy 1.2.3.4 with "CONNECT lb.example.com"; proxy does
> internal name resolution and connects to one of the load balancers
> 6. send grpclb request; get response indicating that the backend
> server is 5.6.7.8
> 7. ask the proxy mapper about IP 5.6.7.8; it recognizes it as an
> internal IP address and says to use "CONNECT 5.6.7.8" via proxy IP 1.2.3.4
> 8. connect to proxy 1.2.3.4 with "CONNECT 5.6.7.8"; proxy connects to
> the specified backend server
>
> In this case, there's no proxy mapper involved in the grpclb connection,
> only for the backend connections, so the proxy mapper doesn't need to sync
> up with the resolver result (which would seem to ameliorate your concern).
> The down-sides are that there are more DNS lookups involved, and that we
> might need to extend the resolver API so that it can pass down richer
> information (not sure about that -- would need to think about this more
> fully).
>
> I'm not sure that this approach is really worth the additional complexity,
> but I figured I'd shoot it out there and see what you think. Thoughts...?
>
> --
> Mark D. Roth <r...@google.com>
> Software Engineer
> Google, Inc.
>
--
Mark D. Roth <r...@google.com>
Software Engineer
Google, Inc.
--
You received this message because you are subscribed to the Google Groups
"grpc.io" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to grpc-io+unsubscr...@googlegroups.com.
To post to this group, send email to grpc-io@googlegroups.com.
Visit this group at https://groups.google.com/group/grpc-io.
To view this discussion on the web visit
https://groups.google.com/d/msgid/grpc-io/CAJgPXp4tastoGRW9mDGw4A2Cxgy8ET6wfAehcb3t2tuNTk3HYQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
