> > * Even if there is a major breaking version, Flink releases major versions > too where it could be added > Netty framework locking is true but AFAIK there was a discussion to > rewrite the Netty stuff to a more sexy thing but there was no agreement to > do that. >
Flink major releases seem to happen even less frequently than Netty releases :( It would be unfortunate if a breaking Netty API change ended up in the FLINK-3957[1] catch-all 2.0 changes. All in all I would agree on making it experimental. > Thus I am happy with this compromise, thank you :) This would simply restrict use-cases where order is not important. Limiting > devs such an add way is no-go. > I think the only case to be made for imposing limitations would be to encourage devs to only use this API in very specific situations, otherwise to solve this in another way, and revisit the API if these limitations are met and alternatives do not work. That said, I am still trying to understand this specific Cloudera case – anything you can say about the limitations of its Flink setup (i.e, difficult to spawn sidecar processes (because of Yarn?)) would be greatly helpful to me and others without this bit of context. But I think the proposed priority function that you've added is a nice compromise as well, so +1 from my side with the proposal. I would only further suggest that we include the other options to this problem in the docs as the preferred approach, where possible. Thanks, Austin [1]: https://issues.apache.org/jira/browse/FLINK-3957 On Wed, Jun 30, 2021 at 10:25 AM Gabor Somogyi <gabor.g.somo...@gmail.com> wrote: > Answered here because the text started to be crowded. > > > It also locks Flink into the current major version of Netty (and the > Netty framework itself) for the foreseeable future. > It's not doing any Netty version locking because: > * Netty not necessarily will add breaking changes in major versions, the > API is quite stable > * Even if there is a major breaking version, Flink releases major versions > too where it could be added > Netty framework locking is true but AFAIK there was a discussion to > rewrite the Netty stuff to a more sexy thing but there was no agreement to > do that. > All in all I would agree on making it experimental. > > > why not restrict the service loader to only allow one? > This would simply restrict use-cases where order is not important. > Limiting devs such an add way is no-go. > I think the ordering came up multiple places which I think is a good > reason fill this gap with a priority function. > I've updated the doc and added it... > > BR, > G > > > On Wed, Jun 30, 2021 at 3:53 PM Austin Cawley-Edwards < > austin.caw...@gmail.com> wrote: > >> Hi Gabor, >> >> Thanks for your answers. I appreciate the explanations. Please see my >> responses + further questions below. >> >> >> * What stability semantics do you envision for this API? >>>> >>> As I foresee the API will be as stable as Netty API. Since there is >>> guarantee on no breaking changes between minor versions we can give the >>> same guarantee. >>> If for whatever reason we need to break it we can do it in major version >>> like every other open source project does. >>> >> >> * Does Flink expose dependencies’ APIs in other places? Since this >>>> exposes the Netty API, will this make it difficult to upgrade Netty? >>>> >>> I don't expect breaking changes between minor versions so such cases >>> there will be no issues. If there is a breaking change in major version >>> we need to wait Flink major version too. >>> >> >> To clarify, you are proposing this new API to have the same stability >> guarantees as @Public currently does? Where we will not introduce breaking >> changes unless absolutely necessary (and requiring a FLIP, etc.)? >> >> If this is the case, I think this puts the community in a tough position >> where we are forced to maintain compatibility with something that we do not >> have control over. It also locks Flink into the current major version of >> Netty (and the Netty framework itself) for the foreseeable future. >> >> I am saying we should not do this, perhaps this is the best solution to >> finding a good compromise here, but I am trying to discover + acknowledge >> the full implications of this proposal so they can be discussed. >> >> What do you think about marking this API as @Experimental and not >> guaranteeing stability between versions? Then, if we do decide we need to >> upgrade Netty (or move away from it), we can do so. >> >> * I share Till's concern about multiple factories – other HTTP middleware >>>> frameworks commonly support chaining middlewares. Since the proposed API >>>> does not include these features/guarantee ordering, do you see any reason >>>> to allow more than one factory? >>>> >>> I personally can't come up with a use-case where ordering is a must. I'm >>> not telling that this is not a valid use-case but adding a feature w/o >>> business rationale would include the maintenance cost (though I'm open to >>> add). >>> As I've seen Till also can't give example for that (please see the doc >>> comments). If you have anything in mind please share it and we can add >>> priority to the API. >>> There is another option too, namely we can be defensive and we can add >>> the priority right now. I would do this only if everybody states in mail >>> that it would be the best option, >>> otherwise I would stick to the original plan. >>> >> >> Let me try to come up with a use case: >> * Someone creates an authentication module for integrating with Google's >> OAuth and publishes it to flink-packages >> * Another person in another org wants to use Google OAuth and then add >> internal authorization based on the user >> * In this scenario, *Google OAuth must come before the internal >> authorization* >> * They place their module and the Google OAuth module to be picked up by >> the service loader >> * What happens? >> >> I do not think that the current proposal has a way to handle this, >> besides having the implementor of the internal authorization module bundle >> everything into one, as you have suggested. Since this is the only way to >> achieve order, why not restrict the service loader to only allow one? This >> way the API is explicit in what it supports. >> >> >> Let me know what you think, >> Austin >> >> >> On Wed, Jun 30, 2021 at 5:24 AM Gabor Somogyi <gabor.g.somo...@gmail.com> >> wrote: >> >>> Hi Austin, >>> >>> Please see my answers embedded down below. >>> >>> BR, >>> G >>> >>> >>> >>> On Tue, Jun 29, 2021 at 9:59 PM Austin Cawley-Edwards < >>> austin.caw...@gmail.com> wrote: >>> >>>> Hi all, >>>> >>>> Thanks for the updated proposal. I have a few questions about the API, >>>> please see below. >>>> >>>> * What stability semantics do you envision for this API? >>>> >>> As I foresee the API will be as stable as Netty API. Since there is >>> guarantee on no breaking changes between minor versions we can give the >>> same guarantee. >>> If for whatever reason we need to break it we can do it in major version >>> like every other open source project does. >>> >>> >>>> * Does Flink expose dependencies’ APIs in other places? Since this >>>> exposes the Netty API, will this make it difficult to upgrade Netty? >>>> >>> I don't expect breaking changes between minor versions so such cases >>> there will be no issues. If there is a breaking change in major version >>> we need to wait Flink major version too. >>> >>> >>>> * I share Till's concern about multiple factories – other HTTP >>>> middleware frameworks commonly support chaining middlewares. Since the >>>> proposed API does not include these features/guarantee ordering, do you see >>>> any reason to allow more than one factory? >>>> >>> I personally can't come up with a use-case where ordering is a must. I'm >>> not telling that this is not a valid use-case but adding a feature w/o >>> business rationale would include the maintenance cost (though I'm open to >>> add). >>> As I've seen Till also can't give example for that (please see the doc >>> comments). If you have anything in mind please share it and we can add >>> priority to the API. >>> There is another option too, namely we can be defensive and we can add >>> the priority right now. I would do this only if everybody states in mail >>> that it would be the best option, >>> otherwise I would stick to the original plan. >>> >>> >>>> >>>> Best, >>>> Austin >>>> >>>> On Tue, Jun 29, 2021 at 8:55 AM Márton Balassi < >>>> balassi.mar...@gmail.com> wrote: >>>> >>>>> Hi all, >>>>> >>>>> I commend Konstantin and Till when it comes to standing up for the >>>>> community values. >>>>> >>>>> Based on your feedback we are withdrawing the original proposal and >>>>> attaching a more general custom netty handler API proposal [1] written >>>>> by >>>>> G. The change necessary to the Flink repository is approximately 500 >>>>> lines >>>>> of code. [2] >>>>> >>>>> Please let us focus on discussing the details of this API and whether >>>>> it >>>>> covers the necessary use cases. >>>>> >>>>> [1] >>>>> >>>>> https://docs.google.com/document/d/1Idnw8YauMK1x_14iv0rVF0Hqm58J6Dg-hi-hEuL6hwM/edit#heading=h.ijcbce3c5gip >>>>> [2] >>>>> >>>>> https://github.com/gaborgsomogyi/flink/commit/942f23679ac21428bb87fc85557b9b443fcaf310 >>>>> >>>>> Thanks, >>>>> Marton >>>>> >>>>> On Wed, Jun 23, 2021 at 9:36 PM Austin Cawley-Edwards < >>>>> austin.caw...@gmail.com> wrote: >>>>> >>>>> > Hi all, >>>>> > >>>>> > Thanks, Konstantin and Till, for guiding the discussion. >>>>> > >>>>> > I was not aware of the results of the call with Konstantin and was >>>>> > attempting to resolve the unanswered questions before more, >>>>> potentially >>>>> > fruitless, work was done. >>>>> > >>>>> > I am also looking forward to the coming proposal, as well as >>>>> increasing my >>>>> > understanding of this specific use case + its limitations! >>>>> > >>>>> > Best, >>>>> > Austin >>>>> > >>>>> > On Tue, Jun 22, 2021 at 6:32 AM Till Rohrmann <trohrm...@apache.org> >>>>> > wrote: >>>>> > >>>>> > > Hi everyone, >>>>> > > >>>>> > > I do like the idea of keeping the actual change outside of Flink >>>>> but to >>>>> > > enable Flink to support such a use case (different authentication >>>>> > > mechanisms). I think this is a good compromise for the community >>>>> that >>>>> > > combines long-term maintainability with support for new use-cases. >>>>> I am >>>>> > > looking forward to your proposal. >>>>> > > >>>>> > > I also want to second Konstantin here that the tone of your last >>>>> email, >>>>> > > Marton, does not reflect the values and manners of the Flink >>>>> community >>>>> > and >>>>> > > is not representative of how we conduct discussions. Especially, >>>>> the more >>>>> > > senior community members should know this and act accordingly in >>>>> order to >>>>> > > be good role models for others in the community. Technical >>>>> discussions >>>>> > > should not be decided by who wields presumably the greatest >>>>> authority but >>>>> > > by the soundness of arguments and by what is the best solution for >>>>> a >>>>> > > problem. >>>>> > > >>>>> > > Let us now try to find the best solution for the problem at hand! >>>>> > > >>>>> > > Cheers, >>>>> > > Till >>>>> > > >>>>> > > On Tue, Jun 22, 2021 at 11:24 AM Konstantin Knauf < >>>>> kna...@apache.org> >>>>> > > wrote: >>>>> > > >>>>> > > > Hi everyone, >>>>> > > > >>>>> > > > First, Marton and I had a brief conversation yesterday offline >>>>> and >>>>> > > > discussed exploring the approach of exposing the authentication >>>>> > > > functionality via an API. So, I am looking forward to your >>>>> proposal in >>>>> > > that >>>>> > > > direction. The benefit of such a solution would be that it is >>>>> > extensible >>>>> > > > for others and it does add a smaller maintenance (in particular >>>>> > testing) >>>>> > > > footprint to Apache Flink itself. If we end up going down this >>>>> route, >>>>> > > > flink-packages.org would be a great way to promote these third >>>>> party >>>>> > > > "authentication modules". >>>>> > > > >>>>> > > > Second, Marton, I understand your frustration about the long >>>>> discussion >>>>> > > on >>>>> > > > this "simple matter", but the condescending tone of your last >>>>> mail >>>>> > feels >>>>> > > > uncalled for to me. Austin expressed a valid opinion on the >>>>> topic, >>>>> > which >>>>> > > is >>>>> > > > based on his experience from other Open Source frameworks (CNCF >>>>> > mostly). >>>>> > > I >>>>> > > > am sure you agree that it is important for Apache Flink to stay >>>>> open >>>>> > and >>>>> > > to >>>>> > > > consider different approaches and ideas and I don't think it >>>>> helps the >>>>> > > > culture of discussion to shoot it down like this ("This is where >>>>> this >>>>> > > > discussion stops."). >>>>> > > > >>>>> > > > Let's continue to move this discussion forward and I am sure >>>>> we'll >>>>> > find a >>>>> > > > consensus based on product and technological considerations. >>>>> > > > >>>>> > > > Thanks, >>>>> > > > >>>>> > > > Konstantin >>>>> > > > >>>>> > > > On Tue, Jun 22, 2021 at 9:31 AM Márton Balassi < >>>>> > balassi.mar...@gmail.com >>>>> > > > >>>>> > > > wrote: >>>>> > > > >>>>> > > > > Hi Austin, >>>>> > > > > >>>>> > > > > Thank you for your thoughts. This is where this discussion >>>>> stops. >>>>> > This >>>>> > > > > email thread already contains more characters than the >>>>> implementation >>>>> > > and >>>>> > > > > what is needed for the next 20 years of maintenance. >>>>> > > > > >>>>> > > > > It is great that you have a view on modern solutions and thank >>>>> you >>>>> > for >>>>> > > > > offering your help with brainstorming solutions. I am >>>>> responsible for >>>>> > > > Flink >>>>> > > > > at Cloudera and we do need an implementation like this and it >>>>> is in >>>>> > > fact >>>>> > > > > already in production at dozens of customers. We are open to >>>>> adapting >>>>> > > > that >>>>> > > > > to expose a more generic API (and keeping Kerberos to our >>>>> fork), to >>>>> > > > > contribute this to the community as others have asked for it >>>>> and to >>>>> > > > protect >>>>> > > > > ourselves from occasionally having to update this critical >>>>> > > implementation >>>>> > > > > path based on changes in the Apache codebase. I have worked >>>>> with >>>>> > close >>>>> > > > to a >>>>> > > > > hundred Big Data customers as a consultant and an engineering >>>>> manager >>>>> > > and >>>>> > > > > committed hundreds of changes to Apache Flink over the past >>>>> decade, >>>>> > > > please >>>>> > > > > trust my judgement on a simple matter like this. >>>>> > > > > >>>>> > > > > Please forgive me for referencing authority, this discussion >>>>> was >>>>> > > getting >>>>> > > > > out of hand. Please keep vigilant. >>>>> > > > > >>>>> > > > > Best, >>>>> > > > > Marton >>>>> > > > > >>>>> > > > > On Mon, Jun 21, 2021 at 10:50 PM Austin Cawley-Edwards < >>>>> > > > > austin.caw...@gmail.com> wrote: >>>>> > > > > >>>>> > > > > > Hi Gabor + Marton, >>>>> > > > > > >>>>> > > > > > I don't believe that the issue with this proposal is the >>>>> specific >>>>> > > > > mechanism >>>>> > > > > > proposed (Kerberos), but rather that it is not the level to >>>>> > implement >>>>> > > > it >>>>> > > > > at >>>>> > > > > > (Flink). I'm just one voice, so please take this with a >>>>> grain of >>>>> > > salt. >>>>> > > > > > >>>>> > > > > > In the other solutions previously noted there is no need to >>>>> > > instrument >>>>> > > > > > Flink which, in addition to reducing the maintenance burden, >>>>> > > provides a >>>>> > > > > > better, decoupled end result. >>>>> > > > > > >>>>> > > > > > IMO we should not add any new API in Flink for this use >>>>> case. I >>>>> > think >>>>> > > > it >>>>> > > > > is >>>>> > > > > > unfortunate and sympathize with the work that has already >>>>> been done >>>>> > > on >>>>> > > > > this >>>>> > > > > > feature – perhaps we could brainstorm ways to run this >>>>> alongside >>>>> > > Flink >>>>> > > > in >>>>> > > > > > your setup. Again, I don't think the proposed solution of an >>>>> > agnostic >>>>> > > > API >>>>> > > > > > would not work, nor is it a bad idea, but is not one that >>>>> will make >>>>> > > > Flink >>>>> > > > > > more compatible with the modern solutions to this problem. >>>>> > > > > > >>>>> > > > > > Best, >>>>> > > > > > Austin >>>>> > > > > > >>>>> > > > > > On Mon, Jun 21, 2021 at 2:18 PM Márton Balassi < >>>>> > > > balassi.mar...@gmail.com >>>>> > > > > > >>>>> > > > > > wrote: >>>>> > > > > > >>>>> > > > > > > Hi team, >>>>> > > > > > > >>>>> > > > > > > Thank you for your input. Based on this discussion I agree >>>>> with G >>>>> > > > that >>>>> > > > > > > selecting and standardizing on a specific strong >>>>> authentication >>>>> > > > > mechanism >>>>> > > > > > > is more challenging than the whole rest of the scope of >>>>> this >>>>> > > > > > authentication >>>>> > > > > > > story. :-) I suggest that G and I go back to the drawing >>>>> board >>>>> > and >>>>> > > > come >>>>> > > > > > up >>>>> > > > > > > with an API that can support multiple authentication >>>>> mechanisms, >>>>> > > and >>>>> > > > we >>>>> > > > > > > would only merge said API to Flink. Specific >>>>> implementations of >>>>> > it >>>>> > > > can >>>>> > > > > be >>>>> > > > > > > maintained outside of the project. This way we tackle the >>>>> main >>>>> > > > > challenge >>>>> > > > > > in >>>>> > > > > > > a truly minimal way. >>>>> > > > > > > >>>>> > > > > > > Best, >>>>> > > > > > > Marton >>>>> > > > > > > >>>>> > > > > > > On Mon, Jun 21, 2021 at 4:18 PM Gabor Somogyi < >>>>> > > > > gabor.g.somo...@gmail.com >>>>> > > > > > > >>>>> > > > > > > wrote: >>>>> > > > > > > >>>>> > > > > > > > Hi All, >>>>> > > > > > > > >>>>> > > > > > > > We see that adding any kind of specific authentication >>>>> raises >>>>> > > more >>>>> > > > > > > > questions than answers. >>>>> > > > > > > > What would be if a generic API would be added without >>>>> any real >>>>> > > > > > > > authentication logic? >>>>> > > > > > > > That way every provider can add its own protocol >>>>> implementation >>>>> > > as >>>>> > > > > > > > additional jar. >>>>> > > > > > > > >>>>> > > > > > > > BR, >>>>> > > > > > > > G >>>>> > > > > > > > >>>>> > > > > > > > >>>>> > > > > > > > On Thu, Jun 17, 2021 at 7:53 PM Austin Cawley-Edwards < >>>>> > > > > > > > austin.caw...@gmail.com> wrote: >>>>> > > > > > > > >>>>> > > > > > > >> Hi all, >>>>> > > > > > > >> >>>>> > > > > > > >> Sorry to be joining the conversation late. I'm also on >>>>> the >>>>> > side >>>>> > > of >>>>> > > > > > > >> Konstantin, generally, in that this seems to not be a >>>>> core >>>>> > goal >>>>> > > of >>>>> > > > > > Flink >>>>> > > > > > > >> as >>>>> > > > > > > >> a project and adds a maintenance burden. >>>>> > > > > > > >> >>>>> > > > > > > >> Would another con of Kerberos be that is likely a fading >>>>> > project >>>>> > > > in >>>>> > > > > > > terms >>>>> > > > > > > >> of network security? (serious question, please correct >>>>> me if >>>>> > > there >>>>> > > > > is >>>>> > > > > > > >> reason to believe it is gaining adoption) >>>>> > > > > > > >> >>>>> > > > > > > >> The point about Kerberos being independent of >>>>> infrastructure >>>>> > is >>>>> > > a >>>>> > > > > good >>>>> > > > > > > one >>>>> > > > > > > >> but is something that is also solved by modern sidecar >>>>> > proxies + >>>>> > > > > > service >>>>> > > > > > > >> meshes that can run across Kubernetes and bare-metal. >>>>> These >>>>> > > > > solutions >>>>> > > > > > > also >>>>> > > > > > > >> handle certificate provisioning, rotation, etc. in >>>>> addition to >>>>> > > > > > > >> higher-level >>>>> > > > > > > >> authorization policies. Some examples of projects with >>>>> this >>>>> > > > > "universal >>>>> > > > > > > >> infrastructure support" are Kuma[1] (CNCF Sandbox, I'm a >>>>> > > > maintainer) >>>>> > > > > > and >>>>> > > > > > > >> Istio[2] (Google). >>>>> > > > > > > >> >>>>> > > > > > > >> Wondering out loud: has anyone tried to run Flink on >>>>> top of >>>>> > > > > cilium[3], >>>>> > > > > > > >> which also provides zero-trust networking at the kernel >>>>> level >>>>> > > > > without >>>>> > > > > > > >> needing to instrument applications? This currently only >>>>> runs >>>>> > on >>>>> > > > > > > Kubernetes >>>>> > > > > > > >> on Linux, so that's a major limitation, but solves many >>>>> of the >>>>> > > > > request >>>>> > > > > > > >> forging concerns at all levels. >>>>> > > > > > > >> >>>>> > > > > > > >> Thanks, >>>>> > > > > > > >> Austin >>>>> > > > > > > >> >>>>> > > > > > > >> [1]: https://kuma.io/docs/1.1.6/quickstart/universal/ >>>>> > > > > > > >> [2]: >>>>> > > https://istio.io/latest/docs/setup/install/virtual-machine/ >>>>> > > > > > > >> [3]: https://cilium.io/ >>>>> > > > > > > >> >>>>> > > > > > > >> On Thu, Jun 17, 2021 at 1:50 PM Till Rohrmann < >>>>> > > > trohrm...@apache.org >>>>> > > > > > >>>>> > > > > > > >> wrote: >>>>> > > > > > > >> >>>>> > > > > > > >> > I left some comments in the Google document. It would >>>>> be >>>>> > great >>>>> > > > if >>>>> > > > > > > >> > someone from the community with security experience >>>>> could >>>>> > also >>>>> > > > > take >>>>> > > > > > a >>>>> > > > > > > >> look >>>>> > > > > > > >> > at it. Maybe Eron you have an opinion on the topic. >>>>> > > > > > > >> > >>>>> > > > > > > >> > Cheers, >>>>> > > > > > > >> > Till >>>>> > > > > > > >> > >>>>> > > > > > > >> > On Thu, Jun 17, 2021 at 6:57 PM Till Rohrmann < >>>>> > > > > trohrm...@apache.org >>>>> > > > > > > >>>>> > > > > > > >> > wrote: >>>>> > > > > > > >> > >>>>> > > > > > > >> > > Hi Gabor, >>>>> > > > > > > >> > > >>>>> > > > > > > >> > > I haven't found time to look into the updated FLIP >>>>> yet. >>>>> > I'll >>>>> > > > try >>>>> > > > > > to >>>>> > > > > > > >> do it >>>>> > > > > > > >> > > asap. >>>>> > > > > > > >> > > >>>>> > > > > > > >> > > Cheers, >>>>> > > > > > > >> > > Till >>>>> > > > > > > >> > > >>>>> > > > > > > >> > > On Wed, Jun 16, 2021 at 9:35 PM Konstantin Knauf < >>>>> > > > > > kna...@apache.org >>>>> > > > > > > > >>>>> > > > > > > >> > > wrote: >>>>> > > > > > > >> > > >>>>> > > > > > > >> > >> Hi Gabor, >>>>> > > > > > > >> > >> >>>>> > > > > > > >> > >> > However representing Kerberos as completely new >>>>> feature >>>>> > > is >>>>> > > > > not >>>>> > > > > > > true >>>>> > > > > > > >> > >> because >>>>> > > > > > > >> > >> it's already in since Flink makes authentication >>>>> at least >>>>> > > > with >>>>> > > > > > HDFS >>>>> > > > > > > >> and >>>>> > > > > > > >> > >> Hbase through Kerberos. >>>>> > > > > > > >> > >> >>>>> > > > > > > >> > >> True, that is one way to look at it, but there are >>>>> > > > differences, >>>>> > > > > > > too: >>>>> > > > > > > >> > >> Control Plane vs Data Plane, Core vs Connectors. >>>>> > > > > > > >> > >> >>>>> > > > > > > >> > >> > Adding OIDC or OAuth2 has the exact same >>>>> concerns what >>>>> > > > you've >>>>> > > > > > > guys >>>>> > > > > > > >> > just >>>>> > > > > > > >> > >> raised. Why exactly these? If you think this would >>>>> be >>>>> > > > > beneficial >>>>> > > > > > we >>>>> > > > > > > >> can >>>>> > > > > > > >> > >> discuss it in detail >>>>> > > > > > > >> > >> >>>>> > > > > > > >> > >> That's exactly my point. Once we start adding authx >>>>> > > support, >>>>> > > > we >>>>> > > > > > > will >>>>> > > > > > > >> > >> sooner or later discuss other options besides >>>>> Kerberos, >>>>> > > too. >>>>> > > > A >>>>> > > > > > user >>>>> > > > > > > >> who >>>>> > > > > > > >> > >> would like to use OAuth can not easily use >>>>> Kerberos, >>>>> > right? >>>>> > > > > > > >> > >> That is one of the reasons I am skeptical about >>>>> adding >>>>> > > > initial >>>>> > > > > > > authx >>>>> > > > > > > >> > >> support. >>>>> > > > > > > >> > >> >>>>> > > > > > > >> > >> > Related authorization you've mentioned it can be >>>>> > > > complicated >>>>> > > > > > over >>>>> > > > > > > >> > time. >>>>> > > > > > > >> > >> Can >>>>> > > > > > > >> > >> you show us an example? We've knowledge with >>>>> couple of >>>>> > open >>>>> > > > > > source >>>>> > > > > > > >> > >> components >>>>> > > > > > > >> > >> but authorization was never a horror complex >>>>> story. I >>>>> > > > > personally >>>>> > > > > > > have >>>>> > > > > > > >> > the >>>>> > > > > > > >> > >> most experience with Spark which I think is quite >>>>> simple >>>>> > > and >>>>> > > > > > > stable. >>>>> > > > > > > >> > Users >>>>> > > > > > > >> > >> can be viewers/admins >>>>> > > > > > > >> > >> and jobs started by others can't be modified. If >>>>> you can >>>>> > > > share >>>>> > > > > an >>>>> > > > > > > >> > example >>>>> > > > > > > >> > >> over-complication we can discuss on facts. >>>>> > > > > > > >> > >> >>>>> > > > > > > >> > >> Authorization is a new aspect that needs to be >>>>> considered >>>>> > > for >>>>> > > > > > every >>>>> > > > > > > >> > >> addition to the REST API. In the future users >>>>> might ask >>>>> > for >>>>> > > > > > > >> additional >>>>> > > > > > > >> > >> roles (e.g. an editor), user-defined roles and >>>>> you've >>>>> > > already >>>>> > > > > > > >> mentioned >>>>> > > > > > > >> > >> job-level permissions yourself. And keep in mind >>>>> that >>>>> > there >>>>> > > > > might >>>>> > > > > > > >> also >>>>> > > > > > > >> > be >>>>> > > > > > > >> > >> larger additions in the future like the >>>>> > flink-sql-gateway. >>>>> > > > > > > >> Contributions >>>>> > > > > > > >> > >> like this become more expensive the more aspects >>>>> we need >>>>> > to >>>>> > > > > > > consider. >>>>> > > > > > > >> > >> >>>>> > > > > > > >> > >> In general, I believe, it is important that the >>>>> community >>>>> > > > > focuses >>>>> > > > > > > its >>>>> > > > > > > >> > >> efforts where we can generate the most value to >>>>> the user >>>>> > > and >>>>> > > > - >>>>> > > > > > > >> > personally - >>>>> > > > > > > >> > >> I don't think there is much to gain by extending >>>>> Flink's >>>>> > > > scope >>>>> > > > > in >>>>> > > > > > > >> that >>>>> > > > > > > >> > >> direction. Of course, this is not black and white >>>>> and >>>>> > there >>>>> > > > are >>>>> > > > > > > other >>>>> > > > > > > >> > valid >>>>> > > > > > > >> > >> opinions. >>>>> > > > > > > >> > >> >>>>> > > > > > > >> > >> Thanks, >>>>> > > > > > > >> > >> >>>>> > > > > > > >> > >> Konstantin >>>>> > > > > > > >> > >> >>>>> > > > > > > >> > >> On Wed, Jun 16, 2021 at 7:38 PM Gabor Somogyi < >>>>> > > > > > > >> > gabor.g.somo...@gmail.com> >>>>> > > > > > > >> > >> wrote: >>>>> > > > > > > >> > >> >>>>> > > > > > > >> > >>> Hi Konstantin, >>>>> > > > > > > >> > >>> >>>>> > > > > > > >> > >>> Thanks for the response. Related new feature >>>>> > introduction >>>>> > > in >>>>> > > > > > case >>>>> > > > > > > of >>>>> > > > > > > >> > >>> Basic >>>>> > > > > > > >> > >>> auth I tend to agree, anything else can be chosen. >>>>> > > > > > > >> > >>> >>>>> > > > > > > >> > >>> However representing Kerberos as completely new >>>>> feature >>>>> > is >>>>> > > > not >>>>> > > > > > > true >>>>> > > > > > > >> > >>> because >>>>> > > > > > > >> > >>> it's already in since Flink makes authentication >>>>> at >>>>> > least >>>>> > > > with >>>>> > > > > > > HDFS >>>>> > > > > > > >> and >>>>> > > > > > > >> > >>> Hbase through Kerberos. >>>>> > > > > > > >> > >>> The main problem with the actual Kerberos >>>>> implementation >>>>> > > is >>>>> > > > > that >>>>> > > > > > > it >>>>> > > > > > > >> > >>> contains several bugs and only partially >>>>> implemented. >>>>> > > > > Following >>>>> > > > > > > your >>>>> > > > > > > >> > >>> suggestion can we agree that we >>>>> > > > > > > >> > >>> skip the Basic auth implementation and finish an >>>>> already >>>>> > > > > started >>>>> > > > > > > >> > Kerberos >>>>> > > > > > > >> > >>> story by adding History Server and Job Dashboard >>>>> > > > > authentication? >>>>> > > > > > > >> > >>> >>>>> > > > > > > >> > >>> Adding OIDC or OAuth2 has the exact same concerns >>>>> what >>>>> > > > you've >>>>> > > > > > guys >>>>> > > > > > > >> just >>>>> > > > > > > >> > >>> raised. Why exactly these? If you think this >>>>> would be >>>>> > > > > beneficial >>>>> > > > > > > we >>>>> > > > > > > >> can >>>>> > > > > > > >> > >>> discuss it in detail >>>>> > > > > > > >> > >>> but as a side story it would be good to finish a >>>>> halfway >>>>> > > > done >>>>> > > > > > > >> Kerberos >>>>> > > > > > > >> > >>> story. >>>>> > > > > > > >> > >>> >>>>> > > > > > > >> > >>> Related authorization you've mentioned it can be >>>>> > > complicated >>>>> > > > > > over >>>>> > > > > > > >> time. >>>>> > > > > > > >> > >>> Can >>>>> > > > > > > >> > >>> you show us an example? We've knowledge with >>>>> couple of >>>>> > > open >>>>> > > > > > source >>>>> > > > > > > >> > >>> components >>>>> > > > > > > >> > >>> but authorization was never a horror complex >>>>> story. I >>>>> > > > > personally >>>>> > > > > > > >> have >>>>> > > > > > > >> > the >>>>> > > > > > > >> > >>> most experience with Spark which I think is quite >>>>> simple >>>>> > > and >>>>> > > > > > > stable. >>>>> > > > > > > >> > >>> Users >>>>> > > > > > > >> > >>> can be viewers/admins >>>>> > > > > > > >> > >>> and jobs started by others can't be modified. If >>>>> you can >>>>> > > > share >>>>> > > > > > an >>>>> > > > > > > >> > example >>>>> > > > > > > >> > >>> over-complication we can discuss on facts. >>>>> > > > > > > >> > >>> >>>>> > > > > > > >> > >>> Thank you in advance! >>>>> > > > > > > >> > >>> >>>>> > > > > > > >> > >>> BR, >>>>> > > > > > > >> > >>> G >>>>> > > > > > > >> > >>> >>>>> > > > > > > >> > >>> >>>>> > > > > > > >> > >>> On Wed, Jun 16, 2021 at 5:42 PM Konstantin Knauf < >>>>> > > > > > > kna...@apache.org >>>>> > > > > > > >> > >>>>> > > > > > > >> > >>> wrote: >>>>> > > > > > > >> > >>> >>>>> > > > > > > >> > >>> > Hi everyone, >>>>> > > > > > > >> > >>> > >>>>> > > > > > > >> > >>> > sorry for joining late and thanks for the >>>>> insightful >>>>> > > > > > discussion. >>>>> > > > > > > >> > >>> > >>>>> > > > > > > >> > >>> > In general, I'd personally prefer not to >>>>> increase the >>>>> > > > > surface >>>>> > > > > > > >> area of >>>>> > > > > > > >> > >>> > Apache Flink unless there is a good reason. It >>>>> seems >>>>> > we >>>>> > > > all >>>>> > > > > > > agree >>>>> > > > > > > >> > that >>>>> > > > > > > >> > >>> > authx is not part of the core value proposition >>>>> of >>>>> > > Apache >>>>> > > > > > Flink, >>>>> > > > > > > >> so >>>>> > > > > > > >> > if >>>>> > > > > > > >> > >>> we >>>>> > > > > > > >> > >>> > can delegate this problem to a more specialized >>>>> tool, >>>>> > I >>>>> > > am >>>>> > > > > in >>>>> > > > > > > >> favor >>>>> > > > > > > >> > of >>>>> > > > > > > >> > >>> > that. Apache Flink is already huge and a lot of >>>>> work >>>>> > > goes >>>>> > > > > into >>>>> > > > > > > >> > >>> maintenance, >>>>> > > > > > > >> > >>> > so I personally have become more sensitive to >>>>> this >>>>> > > aspect >>>>> > > > > over >>>>> > > > > > > >> time. >>>>> > > > > > > >> > >>> > >>>>> > > > > > > >> > >>> > If we add support for Basic Auth and Kerberos >>>>> now, >>>>> > users >>>>> > > > > will >>>>> > > > > > > >> sooner >>>>> > > > > > > >> > or >>>>> > > > > > > >> > >>> > later ask for OIDC, LDAP, SAML,... I >>>>> acknowledge that >>>>> > > > > Kerberos >>>>> > > > > > > is >>>>> > > > > > > >> > >>> widely >>>>> > > > > > > >> > >>> > used in the corporate, on-premises context, but >>>>> isn't >>>>> > > the >>>>> > > > > > focus >>>>> > > > > > > >> > moving >>>>> > > > > > > >> > >>> more >>>>> > > > > > > >> > >>> > towards more web-friendly standards like >>>>> OIDC/OAuth >>>>> > 2.0? >>>>> > > > If >>>>> > > > > we >>>>> > > > > > > >> only >>>>> > > > > > > >> > >>> want to >>>>> > > > > > > >> > >>> > support a single protocol, there is an argument >>>>> to be >>>>> > > made >>>>> > > > > > that >>>>> > > > > > > it >>>>> > > > > > > >> > >>> should >>>>> > > > > > > >> > >>> > be OIDC and Dex [1,2] as a bridge to everything >>>>> else. >>>>> > > Have >>>>> > > > > > OIDC >>>>> > > > > > > or >>>>> > > > > > > >> > >>> OAuth2 >>>>> > > > > > > >> > >>> > been considered instead of Kerberos? How do you >>>>> see >>>>> > the >>>>> > > > > market >>>>> > > > > > > >> > moving? >>>>> > > > > > > >> > >>> But >>>>> > > > > > > >> > >>> > as I said before, in my opinion we can generate >>>>> more >>>>> > > value >>>>> > > > > by >>>>> > > > > > > >> > investing >>>>> > > > > > > >> > >>> > into other areas of Apache Flink. >>>>> > > > > > > >> > >>> > >>>>> > > > > > > >> > >>> > Authorization also has the potential to become >>>>> more >>>>> > > > > > fine-grained >>>>> > > > > > > >> and >>>>> > > > > > > >> > >>> > complex over time: you already mentioned >>>>> restricting >>>>> > the >>>>> > > > > > actions >>>>> > > > > > > >> > that a >>>>> > > > > > > >> > >>> > specific user can do in a cluster. >>>>> > > > > > > >> > >>> > >>>>> > > > > > > >> > >>> > Cheers, >>>>> > > > > > > >> > >>> > >>>>> > > > > > > >> > >>> > Konstantin >>>>> > > > > > > >> > >>> > >>>>> > > > > > > >> > >>> > [1] https://github.com/dexidp/dex >>>>> > > > > > > >> > >>> > [2] https://github.com/dexidp/dex/issues/1903 >>>>> > > > > > > >> > >>> > >>>>> > > > > > > >> > >>> > >>>>> > > > > > > >> > >>> > On Wed, Jun 16, 2021 at 11:44 AM Gabor Somogyi < >>>>> > > > > > > >> > >>> gabor.g.somo...@gmail.com> >>>>> > > > > > > >> > >>> > wrote: >>>>> > > > > > > >> > >>> > >>>>> > > > > > > >> > >>> >> Hi Till, >>>>> > > > > > > >> > >>> >> >>>>> > > > > > > >> > >>> >> Did you have the chance to take a look at the >>>>> doc? >>>>> > Not >>>>> > > > yet >>>>> > > > > > seen >>>>> > > > > > > >> any >>>>> > > > > > > >> > >>> >> update. >>>>> > > > > > > >> > >>> >> >>>>> > > > > > > >> > >>> >> BR, >>>>> > > > > > > >> > >>> >> G >>>>> > > > > > > >> > >>> >> >>>>> > > > > > > >> > >>> >> >>>>> > > > > > > >> > >>> >> On Wed, Jun 9, 2021 at 1:43 PM Till Rohrmann < >>>>> > > > > > > >> trohrm...@apache.org> >>>>> > > > > > > >> > >>> >> wrote: >>>>> > > > > > > >> > >>> >> >>>>> > > > > > > >> > >>> >> > Thanks for the update Gabor. I'll take a >>>>> look and >>>>> > > > respond >>>>> > > > > > in >>>>> > > > > > > >> the >>>>> > > > > > > >> > >>> >> document. >>>>> > > > > > > >> > >>> >> > >>>>> > > > > > > >> > >>> >> > Cheers, >>>>> > > > > > > >> > >>> >> > Till >>>>> > > > > > > >> > >>> >> > >>>>> > > > > > > >> > >>> >> > On Wed, Jun 9, 2021 at 12:59 PM Gabor >>>>> Somogyi < >>>>> > > > > > > >> > >>> >> gabor.g.somo...@gmail.com> >>>>> > > > > > > >> > >>> >> > wrote: >>>>> > > > > > > >> > >>> >> > >>>>> > > > > > > >> > >>> >> >> Hi Till, >>>>> > > > > > > >> > >>> >> >> >>>>> > > > > > > >> > >>> >> >> Your proxy suggestion has been considered >>>>> in-depth >>>>> > > and >>>>> > > > > > > updated >>>>> > > > > > > >> > the >>>>> > > > > > > >> > >>> FLIP >>>>> > > > > > > >> > >>> >> >> accordingly. >>>>> > > > > > > >> > >>> >> >> We've considered 2 proxy implementation >>>>> (Nginx and >>>>> > > > > Squid) >>>>> > > > > > > but >>>>> > > > > > > >> > >>> according >>>>> > > > > > > >> > >>> >> >> to our analysis and testing it's not >>>>> suitable for >>>>> > > the >>>>> > > > > > > >> mentioned >>>>> > > > > > > >> > >>> >> use-cases. >>>>> > > > > > > >> > >>> >> >> Please take a look at the rejected >>>>> alternatives >>>>> > for >>>>> > > > > > detailed >>>>> > > > > > > >> > >>> >> explanation. >>>>> > > > > > > >> > >>> >> >> >>>>> > > > > > > >> > >>> >> >> Thanks for your time in advance! >>>>> > > > > > > >> > >>> >> >> >>>>> > > > > > > >> > >>> >> >> BR, >>>>> > > > > > > >> > >>> >> >> G >>>>> > > > > > > >> > >>> >> >> >>>>> > > > > > > >> > >>> >> >> >>>>> > > > > > > >> > >>> >> >> On Fri, Jun 4, 2021 at 3:31 PM Till >>>>> Rohrmann < >>>>> > > > > > > >> > trohrm...@apache.org >>>>> > > > > > > >> > >>> > >>>>> > > > > > > >> > >>> >> >> wrote: >>>>> > > > > > > >> > >>> >> >> >>>>> > > > > > > >> > >>> >> >>> As I've said I am not a security expert and >>>>> > that's >>>>> > > > why >>>>> > > > > I >>>>> > > > > > > >> have to >>>>> > > > > > > >> > >>> ask >>>>> > > > > > > >> > >>> >> for >>>>> > > > > > > >> > >>> >> >>> clarification, Gabor. You are saying that >>>>> if we >>>>> > > > > > configure a >>>>> > > > > > > >> > >>> >> truststore for >>>>> > > > > > > >> > >>> >> >>> the REST endpoint with a single trusted >>>>> > certificate >>>>> > > > > which >>>>> > > > > > > has >>>>> > > > > > > >> > been >>>>> > > > > > > >> > >>> >> >>> generated by the operator of the Flink >>>>> cluster, >>>>> > > then >>>>> > > > > the >>>>> > > > > > > >> > attacker >>>>> > > > > > > >> > >>> can >>>>> > > > > > > >> > >>> >> >>> generate a new certificate, sign it and >>>>> then talk >>>>> > > to >>>>> > > > > the >>>>> > > > > > > >> Flink >>>>> > > > > > > >> > >>> >> cluster if >>>>> > > > > > > >> > >>> >> >>> he has access to the node on which the REST >>>>> > > endpoint >>>>> > > > > > runs? >>>>> > > > > > > My >>>>> > > > > > > >> > >>> >> understanding >>>>> > > > > > > >> > >>> >> >>> was that you need the corresponding >>>>> private key >>>>> > > which >>>>> > > > > in >>>>> > > > > > my >>>>> > > > > > > >> > >>> proposed >>>>> > > > > > > >> > >>> >> setup >>>>> > > > > > > >> > >>> >> >>> would be under the control of the operator >>>>> as >>>>> > well >>>>> > > > > (e.g. >>>>> > > > > > > >> stored >>>>> > > > > > > >> > >>> in a >>>>> > > > > > > >> > >>> >> >>> keystore on the same machine but guarded >>>>> by some >>>>> > > > > secret). >>>>> > > > > > > >> That >>>>> > > > > > > >> > way >>>>> > > > > > > >> > >>> >> (if I am >>>>> > > > > > > >> > >>> >> >>> not mistaken), only the entity which has >>>>> access >>>>> > to >>>>> > > > the >>>>> > > > > > > >> keystore >>>>> > > > > > > >> > is >>>>> > > > > > > >> > >>> >> able to >>>>> > > > > > > >> > >>> >> >>> talk to the Flink cluster. >>>>> > > > > > > >> > >>> >> >>> >>>>> > > > > > > >> > >>> >> >>> Maybe we are also getting our wires >>>>> crossed here >>>>> > > and >>>>> > > > > are >>>>> > > > > > > >> talking >>>>> > > > > > > >> > >>> about >>>>> > > > > > > >> > >>> >> >>> different things. >>>>> > > > > > > >> > >>> >> >>> >>>>> > > > > > > >> > >>> >> >>> Thanks for listing the pros and cons of >>>>> Kerberos. >>>>> > > > > > > Concerning >>>>> > > > > > > >> > what >>>>> > > > > > > >> > >>> >> other >>>>> > > > > > > >> > >>> >> >>> authentication mechanisms are used in the >>>>> > > industry, I >>>>> > > > > am >>>>> > > > > > > not >>>>> > > > > > > >> > 100% >>>>> > > > > > > >> > >>> >> sure. >>>>> > > > > > > >> > >>> >> >>> >>>>> > > > > > > >> > >>> >> >>> Cheers, >>>>> > > > > > > >> > >>> >> >>> Till >>>>> > > > > > > >> > >>> >> >>> >>>>> > > > > > > >> > >>> >> >>> On Fri, Jun 4, 2021 at 11:09 AM Gabor >>>>> Somogyi < >>>>> > > > > > > >> > >>> >> gabor.g.somo...@gmail.com> >>>>> > > > > > > >> > >>> >> >>> wrote: >>>>> > > > > > > >> > >>> >> >>> >>>>> > > > > > > >> > >>> >> >>>> > I did not mean for the user to sign its >>>>> own >>>>> > > > > > certificates >>>>> > > > > > > >> but >>>>> > > > > > > >> > >>> for >>>>> > > > > > > >> > >>> >> the >>>>> > > > > > > >> > >>> >> >>>> operator of the cluster. Once the user >>>>> request >>>>> > > hits >>>>> > > > > the >>>>> > > > > > > >> proxy, >>>>> > > > > > > >> > it >>>>> > > > > > > >> > >>> >> should no >>>>> > > > > > > >> > >>> >> >>>> longer be under his control. I think I do >>>>> not >>>>> > > fully >>>>> > > > > > > >> understand >>>>> > > > > > > >> > >>> yet >>>>> > > > > > > >> > >>> >> why this >>>>> > > > > > > >> > >>> >> >>>> would not work. >>>>> > > > > > > >> > >>> >> >>>> I said it's not solving the authentication >>>>> > problem >>>>> > > > > over >>>>> > > > > > > any >>>>> > > > > > > >> > >>> proxy. >>>>> > > > > > > >> > >>> >> Even >>>>> > > > > > > >> > >>> >> >>>> if the operator is signing the >>>>> certificate one >>>>> > can >>>>> > > > > have >>>>> > > > > > > >> access >>>>> > > > > > > >> > >>> to an >>>>> > > > > > > >> > >>> >> >>>> internal node. >>>>> > > > > > > >> > >>> >> >>>> Such case anybody can craft certificates >>>>> which >>>>> > is >>>>> > > > > > accepted >>>>> > > > > > > >> by >>>>> > > > > > > >> > the >>>>> > > > > > > >> > >>> >> >>>> server. When it's accepted a bad guy can >>>>> cancel >>>>> > > jobs >>>>> > > > > > > causing >>>>> > > > > > > >> > huge >>>>> > > > > > > >> > >>> >> impacts. >>>>> > > > > > > >> > >>> >> >>>> >>>>> > > > > > > >> > >>> >> >>>> > Also, I am missing a bit the comparison >>>>> of >>>>> > > > Kerberos >>>>> > > > > to >>>>> > > > > > > >> other >>>>> > > > > > > >> > >>> >> >>>> authentication mechanisms and why they >>>>> were >>>>> > > rejected >>>>> > > > > in >>>>> > > > > > > >> favour >>>>> > > > > > > >> > of >>>>> > > > > > > >> > >>> >> Kerberos. >>>>> > > > > > > >> > >>> >> >>>> PROS: >>>>> > > > > > > >> > >>> >> >>>> * Since it's not depending on cloud >>>>> provider >>>>> > > and/or >>>>> > > > > k8s >>>>> > > > > > or >>>>> > > > > > > >> > >>> bare-metal >>>>> > > > > > > >> > >>> >> >>>> etc. deployment it's the biggest plus >>>>> > > > > > > >> > >>> >> >>>> * Centralized with tools and no need to >>>>> write >>>>> > tons >>>>> > > > of >>>>> > > > > > > tools >>>>> > > > > > > >> > >>> around >>>>> > > > > > > >> > >>> >> >>>> * There are clients/tools on almost all >>>>> OS-es >>>>> > and >>>>> > > > > > several >>>>> > > > > > > >> > >>> languages >>>>> > > > > > > >> > >>> >> >>>> * Super huge users are using it for years >>>>> in >>>>> > > > > production >>>>> > > > > > > w/o >>>>> > > > > > > >> > huge >>>>> > > > > > > >> > >>> >> issues >>>>> > > > > > > >> > >>> >> >>>> * Provides cross-realm trust possibility >>>>> amongst >>>>> > > > other >>>>> > > > > > > >> features >>>>> > > > > > > >> > >>> >> >>>> * Several open source components using it >>>>> which >>>>> > > > could >>>>> > > > > > > >> increase >>>>> > > > > > > >> > >>> >> >>>> compatibility >>>>> > > > > > > >> > >>> >> >>>> >>>>> > > > > > > >> > >>> >> >>>> CONS: >>>>> > > > > > > >> > >>> >> >>>> * Not everybody using kerberos >>>>> > > > > > > >> > >>> >> >>>> * It would increase the code footprint >>>>> but this >>>>> > is >>>>> > > > > true >>>>> > > > > > > for >>>>> > > > > > > >> > many >>>>> > > > > > > >> > >>> >> >>>> features (as a side note I'm here to >>>>> maintain >>>>> > it) >>>>> > > > > > > >> > >>> >> >>>> >>>>> > > > > > > >> > >>> >> >>>> Feel free to add your points because it >>>>> only >>>>> > > > > represents >>>>> > > > > > a >>>>> > > > > > > >> > single >>>>> > > > > > > >> > >>> >> >>>> viewpoint. >>>>> > > > > > > >> > >>> >> >>>> Also if you have any better option for >>>>> strong >>>>> > > > > > > authentication >>>>> > > > > > > >> > >>> please >>>>> > > > > > > >> > >>> >> >>>> share it and we can consider the >>>>> pros/cons here. >>>>> > > > > > > >> > >>> >> >>>> >>>>> > > > > > > >> > >>> >> >>>> BR, >>>>> > > > > > > >> > >>> >> >>>> G >>>>> > > > > > > >> > >>> >> >>>> >>>>> > > > > > > >> > >>> >> >>>> >>>>> > > > > > > >> > >>> >> >>>> On Fri, Jun 4, 2021 at 10:32 AM Till >>>>> Rohrmann < >>>>> > > > > > > >> > >>> trohrm...@apache.org> >>>>> > > > > > > >> > >>> >> >>>> wrote: >>>>> > > > > > > >> > >>> >> >>>> >>>>> > > > > > > >> > >>> >> >>>>> I did not mean for the user to sign its >>>>> own >>>>> > > > > > certificates >>>>> > > > > > > >> but >>>>> > > > > > > >> > >>> for the >>>>> > > > > > > >> > >>> >> >>>>> operator of the cluster. Once the user >>>>> request >>>>> > > hits >>>>> > > > > the >>>>> > > > > > > >> proxy, >>>>> > > > > > > >> > >>> it >>>>> > > > > > > >> > >>> >> should no >>>>> > > > > > > >> > >>> >> >>>>> longer be under his control. I think I >>>>> do not >>>>> > > fully >>>>> > > > > > > >> understand >>>>> > > > > > > >> > >>> yet >>>>> > > > > > > >> > >>> >> why this >>>>> > > > > > > >> > >>> >> >>>>> would not work. >>>>> > > > > > > >> > >>> >> >>>>> >>>>> > > > > > > >> > >>> >> >>>>> What I would like to avoid is to add more >>>>> > > > complexity >>>>> > > > > > into >>>>> > > > > > > >> > Flink >>>>> > > > > > > >> > >>> if >>>>> > > > > > > >> > >>> >> >>>>> there is an easy solution which fulfills >>>>> the >>>>> > > > > > > requirements. >>>>> > > > > > > >> > >>> That's >>>>> > > > > > > >> > >>> >> why I >>>>> > > > > > > >> > >>> >> >>>>> would like to exercise thoroughly >>>>> through the >>>>> > > > > different >>>>> > > > > > > >> > >>> >> alternatives. Also, >>>>> > > > > > > >> > >>> >> >>>>> I am missing a bit the comparison of >>>>> Kerberos >>>>> > to >>>>> > > > > other >>>>> > > > > > > >> > >>> >> authentication >>>>> > > > > > > >> > >>> >> >>>>> mechanisms and why they were rejected in >>>>> favour >>>>> > > of >>>>> > > > > > > >> Kerberos. >>>>> > > > > > > >> > >>> >> >>>>> >>>>> > > > > > > >> > >>> >> >>>>> Cheers, >>>>> > > > > > > >> > >>> >> >>>>> Till >>>>> > > > > > > >> > >>> >> >>>>> >>>>> > > > > > > >> > >>> >> >>>>> On Fri, Jun 4, 2021 at 10:26 AM Gyula >>>>> Fóra < >>>>> > > > > > > >> gyf...@apache.org >>>>> > > > > > > >> > > >>>>> > > > > > > >> > >>> >> wrote: >>>>> > > > > > > >> > >>> >> >>>>> >>>>> > > > > > > >> > >>> >> >>>>>> Hi! >>>>> > > > > > > >> > >>> >> >>>>>> >>>>> > > > > > > >> > >>> >> >>>>>> I think there might be possible >>>>> alternatives >>>>> > but >>>>> > > > it >>>>> > > > > > > seems >>>>> > > > > > > >> > >>> Kerberos >>>>> > > > > > > >> > >>> >> on >>>>> > > > > > > >> > >>> >> >>>>>> the rest endpoint ticks all the right >>>>> boxes >>>>> > and >>>>> > > > > > > provides a >>>>> > > > > > > >> > >>> super >>>>> > > > > > > >> > >>> >> clean and >>>>> > > > > > > >> > >>> >> >>>>>> simple solution for strong >>>>> authentication. >>>>> > > > > > > >> > >>> >> >>>>>> >>>>> > > > > > > >> > >>> >> >>>>>> I wouldn’t even consider sidecar >>>>> proxies etc >>>>> > if >>>>> > > we >>>>> > > > > can >>>>> > > > > > > >> solve >>>>> > > > > > > >> > >>> it in >>>>> > > > > > > >> > >>> >> >>>>>> such a simple way as proposed by G. >>>>> > > > > > > >> > >>> >> >>>>>> >>>>> > > > > > > >> > >>> >> >>>>>> Cheers >>>>> > > > > > > >> > >>> >> >>>>>> Gyula >>>>> > > > > > > >> > >>> >> >>>>>> >>>>> > > > > > > >> > >>> >> >>>>>> On Fri, 4 Jun 2021 at 10:03, Till >>>>> Rohrmann < >>>>> > > > > > > >> > >>> trohrm...@apache.org> >>>>> > > > > > > >> > >>> >> >>>>>> wrote: >>>>> > > > > > > >> > >>> >> >>>>>> >>>>> > > > > > > >> > >>> >> >>>>>>> I am not saying that we shouldn't add a >>>>> > strong >>>>> > > > > > > >> > authentication >>>>> > > > > > > >> > >>> >> >>>>>>> mechanism if there are good reasons >>>>> for it. I >>>>> > > > > > primarily >>>>> > > > > > > >> > would >>>>> > > > > > > >> > >>> >> like to >>>>> > > > > > > >> > >>> >> >>>>>>> understand the context a bit better in >>>>> order >>>>> > to >>>>> > > > > give >>>>> > > > > > > >> > qualified >>>>> > > > > > > >> > >>> >> feedback and >>>>> > > > > > > >> > >>> >> >>>>>>> come to a good decision. In order to do >>>>> > this, I >>>>> > > > > have >>>>> > > > > > > the >>>>> > > > > > > >> > >>> feeling >>>>> > > > > > > >> > >>> >> that we >>>>> > > > > > > >> > >>> >> >>>>>>> haven't fully considered all available >>>>> > options >>>>> > > > > which >>>>> > > > > > > are >>>>> > > > > > > >> on >>>>> > > > > > > >> > >>> the >>>>> > > > > > > >> > >>> >> table, tbh. >>>>> > > > > > > >> > >>> >> >>>>>>> >>>>> > > > > > > >> > >>> >> >>>>>>> Does the problem of certificate expiry >>>>> also >>>>> > > apply >>>>> > > > > for >>>>> > > > > > > >> > >>> self-signed >>>>> > > > > > > >> > >>> >> >>>>>>> certificates? If yes, then this should >>>>> then >>>>> > > also >>>>> > > > > be a >>>>> > > > > > > >> > problem >>>>> > > > > > > >> > >>> for >>>>> > > > > > > >> > >>> >> the >>>>> > > > > > > >> > >>> >> >>>>>>> internal encryption of Flink's >>>>> communication. >>>>> > > If >>>>> > > > > not, >>>>> > > > > > > >> then >>>>> > > > > > > >> > one >>>>> > > > > > > >> > >>> >> could use >>>>> > > > > > > >> > >>> >> >>>>>>> self-signed certificates with a longer >>>>> > validity >>>>> > > > to >>>>> > > > > > > solve >>>>> > > > > > > >> the >>>>> > > > > > > >> > >>> >> mentioned >>>>> > > > > > > >> > >>> >> >>>>>>> issue. >>>>> > > > > > > >> > >>> >> >>>>>>> >>>>> > > > > > > >> > >>> >> >>>>>>> I think you can set up Flink in such a >>>>> way >>>>> > that >>>>> > > > you >>>>> > > > > > > don't >>>>> > > > > > > >> > >>> have to >>>>> > > > > > > >> > >>> >> >>>>>>> handle all the different certificates. >>>>> For >>>>> > > > example, >>>>> > > > > > you >>>>> > > > > > > >> > could >>>>> > > > > > > >> > >>> >> deploy Flink >>>>> > > > > > > >> > >>> >> >>>>>>> with a "sidecar proxy" which is >>>>> responsible >>>>> > for >>>>> > > > the >>>>> > > > > > > >> > >>> >> authentication using an >>>>> > > > > > > >> > >>> >> >>>>>>> arbitrary method (e.g. Kerberos) and >>>>> then >>>>> > bind >>>>> > > > the >>>>> > > > > > REST >>>>> > > > > > > >> > >>> endpoint >>>>> > > > > > > >> > >>> >> to a local >>>>> > > > > > > >> > >>> >> >>>>>>> network interface. That way, the REST >>>>> > endpoint >>>>> > > > > would >>>>> > > > > > > >> only be >>>>> > > > > > > >> > >>> >> available >>>>> > > > > > > >> > >>> >> >>>>>>> through the sidecar proxy. >>>>> Additionally, one >>>>> > > > could >>>>> > > > > > > enable >>>>> > > > > > > >> > SSL >>>>> > > > > > > >> > >>> for >>>>> > > > > > > >> > >>> >> this >>>>> > > > > > > >> > >>> >> >>>>>>> communication. Would this be a >>>>> solution for >>>>> > the >>>>> > > > > > > problem? >>>>> > > > > > > >> > >>> >> >>>>>>> >>>>> > > > > > > >> > >>> >> >>>>>>> Cheers, >>>>> > > > > > > >> > >>> >> >>>>>>> Till >>>>> > > > > > > >> > >>> >> >>>>>>> >>>>> > > > > > > >> > >>> >> >>>>>>> On Thu, Jun 3, 2021 at 10:46 PM Márton >>>>> > Balassi >>>>> > > < >>>>> > > > > > > >> > >>> >> >>>>>>> balassi.mar...@gmail.com> wrote: >>>>> > > > > > > >> > >>> >> >>>>>>> >>>>> > > > > > > >> > >>> >> >>>>>>>> That is an interesting idea, Till. >>>>> > > > > > > >> > >>> >> >>>>>>>> >>>>> > > > > > > >> > >>> >> >>>>>>>> The main issue with it is that TLS >>>>> > > certificates >>>>> > > > > have >>>>> > > > > > > an >>>>> > > > > > > >> > >>> >> expiration >>>>> > > > > > > >> > >>> >> >>>>>>>> time, usually they get approved for a >>>>> couple >>>>> > > > > years. >>>>> > > > > > > >> Forcing >>>>> > > > > > > >> > >>> our >>>>> > > > > > > >> > >>> >> users to >>>>> > > > > > > >> > >>> >> >>>>>>>> restart jobs to reprovision TLS >>>>> certificates >>>>> > > > would >>>>> > > > > > be >>>>> > > > > > > >> weird >>>>> > > > > > > >> > >>> when >>>>> > > > > > > >> > >>> >> we could >>>>> > > > > > > >> > >>> >> >>>>>>>> just implement a single proper strong >>>>> > > > > authentication >>>>> > > > > > > >> > >>> mechanism >>>>> > > > > > > >> > >>> >> instead in a >>>>> > > > > > > >> > >>> >> >>>>>>>> couple hundred lines of code. :-) >>>>> > > > > > > >> > >>> >> >>>>>>>> >>>>> > > > > > > >> > >>> >> >>>>>>>> In many cases it is also impractical >>>>> to go >>>>> > the >>>>> > > > TLS >>>>> > > > > > > >> mutual >>>>> > > > > > > >> > >>> route, >>>>> > > > > > > >> > >>> >> >>>>>>>> because the Flink Dashboard can end >>>>> up on >>>>> > any >>>>> > > > node >>>>> > > > > > in >>>>> > > > > > > >> the >>>>> > > > > > > >> > >>> >> k8s/Yarn cluster >>>>> > > > > > > >> > >>> >> >>>>>>>> which means that we need a >>>>> certificate per >>>>> > > node >>>>> > > > > (due >>>>> > > > > > > to >>>>> > > > > > > >> the >>>>> > > > > > > >> > >>> >> mutual auth), >>>>> > > > > > > >> > >>> >> >>>>>>>> but if we also want to protect the >>>>> private >>>>> > key >>>>> > > > of >>>>> > > > > > > these >>>>> > > > > > > >> > from >>>>> > > > > > > >> > >>> >> users >>>>> > > > > > > >> > >>> >> >>>>>>>> accidentally or intentionally leaking >>>>> them >>>>> > > then >>>>> > > > we >>>>> > > > > > > need >>>>> > > > > > > >> > this >>>>> > > > > > > >> > >>> per >>>>> > > > > > > >> > >>> >> user. As >>>>> > > > > > > >> > >>> >> >>>>>>>> in we end up managing user*machine >>>>> number >>>>> > > > > > certificates >>>>> > > > > > > >> and >>>>> > > > > > > >> > >>> >> having to renew >>>>> > > > > > > >> > >>> >> >>>>>>>> them periodically, which albeit >>>>> automatable >>>>> > is >>>>> > > > > > > >> > unfortunately >>>>> > > > > > > >> > >>> not >>>>> > > > > > > >> > >>> >> yet >>>>> > > > > > > >> > >>> >> >>>>>>>> automated in all large organizations. >>>>> > > > > > > >> > >>> >> >>>>>>>> >>>>> > > > > > > >> > >>> >> >>>>>>>> I fully agree that TLS certificate >>>>> mutual >>>>> > > > > > > authentication >>>>> > > > > > > >> > has >>>>> > > > > > > >> > >>> its >>>>> > > > > > > >> > >>> >> >>>>>>>> nice properties, especially at very >>>>> large >>>>> > > > > (multiple >>>>> > > > > > > >> > thousand >>>>> > > > > > > >> > >>> >> node) clusters >>>>> > > > > > > >> > >>> >> >>>>>>>> - but it has its own challenges too. >>>>> Thanks >>>>> > > for >>>>> > > > > > > >> bringing it >>>>> > > > > > > >> > >>> up. >>>>> > > > > > > >> > >>> >> >>>>>>>> >>>>> > > > > > > >> > >>> >> >>>>>>>> Happy to have this added to the >>>>> rejected >>>>> > > > > alternative >>>>> > > > > > > >> list >>>>> > > > > > > >> > so >>>>> > > > > > > >> > >>> that >>>>> > > > > > > >> > >>> >> >>>>>>>> we have the full picture documented. >>>>> > > > > > > >> > >>> >> >>>>>>>> >>>>> > > > > > > >> > >>> >> >>>>>>>> On Thu, Jun 3, 2021 at 5:52 PM Till >>>>> > Rohrmann < >>>>> > > > > > > >> > >>> >> trohrm...@apache.org> >>>>> > > > > > > >> > >>> >> >>>>>>>> wrote: >>>>> > > > > > > >> > >>> >> >>>>>>>> >>>>> > > > > > > >> > >>> >> >>>>>>>>> I guess the idea would then be to >>>>> let the >>>>> > > proxy >>>>> > > > > do >>>>> > > > > > > the >>>>> > > > > > > >> > >>> >> >>>>>>>>> authentication job and only forward >>>>> the >>>>> > > request >>>>> > > > > via >>>>> > > > > > > an >>>>> > > > > > > >> SSL >>>>> > > > > > > >> > >>> >> mutually >>>>> > > > > > > >> > >>> >> >>>>>>>>> encrypted connection to the Flink >>>>> cluster. >>>>> > > > Would >>>>> > > > > > this >>>>> > > > > > > >> be >>>>> > > > > > > >> > >>> >> possible? The >>>>> > > > > > > >> > >>> >> >>>>>>>>> beauty of this setup is in my >>>>> opinion that >>>>> > > this >>>>> > > > > > setup >>>>> > > > > > > >> > should >>>>> > > > > > > >> > >>> >> work with all >>>>> > > > > > > >> > >>> >> >>>>>>>>> kinds of authentication mechanisms. >>>>> > > > > > > >> > >>> >> >>>>>>>>> >>>>> > > > > > > >> > >>> >> >>>>>>>>> Cheers, >>>>> > > > > > > >> > >>> >> >>>>>>>>> Till >>>>> > > > > > > >> > >>> >> >>>>>>>>> >>>>> > > > > > > >> > >>> >> >>>>>>>>> On Thu, Jun 3, 2021 at 3:12 PM Gabor >>>>> > Somogyi >>>>> > > < >>>>> > > > > > > >> > >>> >> >>>>>>>>> gabor.g.somo...@gmail.com> wrote: >>>>> > > > > > > >> > >>> >> >>>>>>>>> >>>>> > > > > > > >> > >>> >> >>>>>>>>>> Thanks for giving options to fulfil >>>>> the >>>>> > > need. >>>>> > > > > > > >> > >>> >> >>>>>>>>>> >>>>> > > > > > > >> > >>> >> >>>>>>>>>> Users are looking for a solution >>>>> where >>>>> > users >>>>> > > > can >>>>> > > > > > be >>>>> > > > > > > >> > >>> identified >>>>> > > > > > > >> > >>> >> on >>>>> > > > > > > >> > >>> >> >>>>>>>>>> the whole cluster and restrict >>>>> access to >>>>> > > > > > > >> > resources/actions. >>>>> > > > > > > >> > >>> >> >>>>>>>>>> A good example for such an action is >>>>> > > > cancelling >>>>> > > > > > > other >>>>> > > > > > > >> > users >>>>> > > > > > > >> > >>> >> >>>>>>>>>> running jobs. >>>>> > > > > > > >> > >>> >> >>>>>>>>>> >>>>> > > > > > > >> > >>> >> >>>>>>>>>> * SSL does provide mutual >>>>> authentication >>>>> > but >>>>> > > > > when >>>>> > > > > > > >> > >>> >> authentication >>>>> > > > > > > >> > >>> >> >>>>>>>>>> passed there is no user based on >>>>> > > restrictions >>>>> > > > > can >>>>> > > > > > be >>>>> > > > > > > >> > made. >>>>> > > > > > > >> > >>> >> >>>>>>>>>> * The less problematic part is that >>>>> > > > > > > >> > generating/maintaining >>>>> > > > > > > >> > >>> >> short >>>>> > > > > > > >> > >>> >> >>>>>>>>>> time valid certificates would be a >>>>> hard >>>>> > > > (that's >>>>> > > > > > the >>>>> > > > > > > >> > reason >>>>> > > > > > > >> > >>> KDC >>>>> > > > > > > >> > >>> >> like servers >>>>> > > > > > > >> > >>> >> >>>>>>>>>> exist). >>>>> > > > > > > >> > >>> >> >>>>>>>>>> Having long time valid certificates >>>>> would >>>>> > > > widen >>>>> > > > > > the >>>>> > > > > > > >> > attack >>>>> > > > > > > >> > >>> >> >>>>>>>>>> surface but since the first concern >>>>> is >>>>> > there >>>>> > > > > this >>>>> > > > > > is >>>>> > > > > > > >> > just a >>>>> > > > > > > >> > >>> >> cosmetic issue. >>>>> > > > > > > >> > >>> >> >>>>>>>>>> >>>>> > > > > > > >> > >>> >> >>>>>>>>>> All in all using TLS certificates >>>>> is not >>>>> > > > > > sufficient >>>>> > > > > > > in >>>>> > > > > > > >> > >>> these >>>>> > > > > > > >> > >>> >> >>>>>>>>>> environments unfortunately. >>>>> > > > > > > >> > >>> >> >>>>>>>>>> >>>>> > > > > > > >> > >>> >> >>>>>>>>>> BR, >>>>> > > > > > > >> > >>> >> >>>>>>>>>> G >>>>> > > > > > > >> > >>> >> >>>>>>>>>> >>>>> > > > > > > >> > >>> >> >>>>>>>>>> >>>>> > > > > > > >> > >>> >> >>>>>>>>>> On Thu, Jun 3, 2021 at 12:49 PM Till >>>>> > > Rohrmann >>>>> > > > < >>>>> > > > > > > >> > >>> >> >>>>>>>>>> trohrm...@apache.org> wrote: >>>>> > > > > > > >> > >>> >> >>>>>>>>>> >>>>> > > > > > > >> > >>> >> >>>>>>>>>>> Thanks for the information Gabor. >>>>> If it >>>>> > is >>>>> > > > > about >>>>> > > > > > > >> > securing >>>>> > > > > > > >> > >>> the >>>>> > > > > > > >> > >>> >> >>>>>>>>>>> communication between the REST >>>>> client and >>>>> > > the >>>>> > > > > > REST >>>>> > > > > > > >> > server, >>>>> > > > > > > >> > >>> >> then Flink >>>>> > > > > > > >> > >>> >> >>>>>>>>>>> already supports enabling mutual >>>>> SSL >>>>> > > > > > authentication >>>>> > > > > > > >> [1]. >>>>> > > > > > > >> > >>> >> Would this be >>>>> > > > > > > >> > >>> >> >>>>>>>>>>> enough to secure the communication >>>>> and to >>>>> > > > pass >>>>> > > > > an >>>>> > > > > > > >> audit? >>>>> > > > > > > >> > >>> >> >>>>>>>>>>> >>>>> > > > > > > >> > >>> >> >>>>>>>>>>> [1] >>>>> > > > > > > >> > >>> >> >>>>>>>>>>> >>>>> > > > > > > >> > >>> >> >>>>> > > > > > > >> > >>> >>>>> > > > > > > >> > >>>>> > > > > > > >> >>>>> > > > > > > >>>>> > > > > > >>>>> > > > > >>>>> > > > >>>>> > > >>>>> > >>>>> https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/security/security-ssl/#external--rest-connectivity >>>>> > > > > > > >> > >>> >> >>>>>>>>>>> >>>>> > > > > > > >> > >>> >> >>>>>>>>>>> Cheers, >>>>> > > > > > > >> > >>> >> >>>>>>>>>>> Till >>>>> > > > > > > >> > >>> >> >>>>>>>>>>> >>>>> > > > > > > >> > >>> >> >>>>>>>>>>> On Thu, Jun 3, 2021 at 10:33 AM >>>>> Gabor >>>>> > > > Somogyi < >>>>> > > > > > > >> > >>> >> >>>>>>>>>>> gabor.g.somo...@gmail.com> wrote: >>>>> > > > > > > >> > >>> >> >>>>>>>>>>> >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> Hi Till, >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> Since I'm working in security >>>>> area 10+ >>>>> > > years >>>>> > > > > let >>>>> > > > > > > me >>>>> > > > > > > >> > >>> share my >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> thought. >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> I would like to emphasise there >>>>> are >>>>> > > experts >>>>> > > > > > better >>>>> > > > > > > >> than >>>>> > > > > > > >> > >>> me >>>>> > > > > > > >> > >>> >> but >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> I have some >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> basics. >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> The discussion is open and not >>>>> trying to >>>>> > > > tell >>>>> > > > > > > alone >>>>> > > > > > > >> > >>> things... >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> > I mean if an attacker can get >>>>> access >>>>> > to >>>>> > > > one >>>>> > > > > of >>>>> > > > > > > the >>>>> > > > > > > >> > >>> >> machines, >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> then it >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> should also be possible to obtain >>>>> the >>>>> > > right >>>>> > > > > > > Kerberos >>>>> > > > > > > >> > >>> token. >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> Not necessarily. For example if >>>>> one gets >>>>> > > > > access >>>>> > > > > > > to a >>>>> > > > > > > >> > >>> specific >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> user's >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> credentials then it's not >>>>> possible to >>>>> > > > > compromise >>>>> > > > > > > >> other >>>>> > > > > > > >> > >>> user's >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> jobs, data, >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> etc... >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> Security is like an onion, the >>>>> more >>>>> > layers >>>>> > > > has >>>>> > > > > > > been >>>>> > > > > > > >> > >>> added the >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> more time an >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> attacker needs to proceed. >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> At the end of the day if one is >>>>> in, then >>>>> > > > most >>>>> > > > > > > >> probably >>>>> > > > > > > >> > >>> can >>>>> > > > > > > >> > >>> >> find >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> the way but >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> this time is normally enough to >>>>> > sysadmins >>>>> > > or >>>>> > > > > > > >> security >>>>> > > > > > > >> > >>> >> experts to >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> close down the system and >>>>> minimize the >>>>> > > > damage. >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> The other thing is that all >>>>> tokens has a >>>>> > > > > timeout >>>>> > > > > > > >> and if >>>>> > > > > > > >> > >>> the >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> token is >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> invalid then the attacker can't >>>>> proceed >>>>> > > > > further. >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> > Is Kerberos also the standard >>>>> > > > authentication >>>>> > > > > > > >> protocol >>>>> > > > > > > >> > >>> for >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> Kubernetes >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> deployments? >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> Kerberos is an industry standard >>>>> which >>>>> > is >>>>> > > > > > > >> > >>> cloud/deployment >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> agnostic and it >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> can be used in any deployments >>>>> including >>>>> > > > k8s. >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> The main intention is to use >>>>> kerberos in >>>>> > > k8s >>>>> > > > > > > >> > deployments >>>>> > > > > > > >> > >>> too >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> since we're >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> going this direction as well. >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> Please see how Spark does this: >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>>> > > > > > > >> > >>> >> >>>>> > > > > > > >> > >>> >>>>> > > > > > > >> > >>>>> > > > > > > >> >>>>> > > > > > > >>>>> > > > > > >>>>> > > > > >>>>> > > > >>>>> > > >>>>> > >>>>> https://spark.apache.org/docs/latest/security.html#secure-interaction-with-kubernetes >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> Last but not least the most >>>>> important >>>>> > > reason >>>>> > > > > to >>>>> > > > > > > add >>>>> > > > > > > >> at >>>>> > > > > > > >> > >>> least >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> one strong >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> authentication is that we have >>>>> users who >>>>> > > has >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> hard requirements on this. >>>>> They're doing >>>>> > > > > > security >>>>> > > > > > > >> > audits >>>>> > > > > > > >> > >>> and >>>>> > > > > > > >> > >>> >> if >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> they fail >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> then it's deal breaking. >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> That is why we have added >>>>> kerberos at >>>>> > the >>>>> > > > > first >>>>> > > > > > > >> place. >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> Unfortunately we >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> can't name them in this public >>>>> list, >>>>> > > however >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> the customers who specifically >>>>> asked for >>>>> > > > this >>>>> > > > > > were >>>>> > > > > > > >> > >>> mainly in >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> the banking >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> and telco sector. >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> BR, >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> G >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> On Thu, Jun 3, 2021 at 9:20 AM >>>>> Till >>>>> > > > Rohrmann < >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> trohrm...@apache.org> wrote: >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> > Thanks for updating the document >>>>> > Márton. >>>>> > > > Why >>>>> > > > > > is >>>>> > > > > > > it >>>>> > > > > > > >> > that >>>>> > > > > > > >> > >>> >> banks >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> will >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> > consider it more secure if >>>>> Flink comes >>>>> > > > with >>>>> > > > > > > >> Kerberos >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> authentication >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> > (assuming a properly secured >>>>> setup)? I >>>>> > > > mean >>>>> > > > > if >>>>> > > > > > > an >>>>> > > > > > > >> > >>> attacker >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> can get access >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> > to one of the machines, then it >>>>> should >>>>> > > > also >>>>> > > > > be >>>>> > > > > > > >> > >>> possible to >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> obtain the right >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> > Kerberos token. >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> > >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> > I am not an authentication >>>>> expert and >>>>> > > > that's >>>>> > > > > > > why I >>>>> > > > > > > >> > >>> wanted >>>>> > > > > > > >> > >>> >> to >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> ask what are >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> > other authentication protocols >>>>> other >>>>> > > than >>>>> > > > > > > >> Kerberos? >>>>> > > > > > > >> > >>> Why did >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> we select >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> > Kerberos and not any other >>>>> > > authentication >>>>> > > > > > > >> protocol? >>>>> > > > > > > >> > >>> Maybe >>>>> > > > > > > >> > >>> >> you >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> can list the >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> > pros and cons for the different >>>>> > > protocols. >>>>> > > > > Is >>>>> > > > > > > >> > Kerberos >>>>> > > > > > > >> > >>> also >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> the standard >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> > authentication protocol for >>>>> Kubernetes >>>>> > > > > > > >> deployments? >>>>> > > > > > > >> > If >>>>> > > > > > > >> > >>> not, >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> what would be >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> > the answer when deploying on >>>>> K8s? >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> > >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> > Cheers, >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> > Till >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> > >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> > On Wed, Jun 2, 2021 at 12:07 PM >>>>> Gabor >>>>> > > > > Somogyi >>>>> > > > > > < >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> gabor.g.somo...@gmail.com> >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> > wrote: >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> > >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >> Hi team, >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >> >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >> Happy to be here and hope I can >>>>> > provide >>>>> > > > > > quality >>>>> > > > > > > >> > >>> additions >>>>> > > > > > > >> > >>> >> in >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> the future. >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >> >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >> Thank you all for helpful the >>>>> > > > suggestions! >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >> Considering them the FLIP has >>>>> been >>>>> > > > modified >>>>> > > > > > and >>>>> > > > > > > >> the >>>>> > > > > > > >> > >>> work >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> continues on the >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >> already existing Jira. >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >> >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >> BR, >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >> G >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >> >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >> >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >> On Wed, Jun 2, 2021 at 11:23 AM >>>>> > Márton >>>>> > > > > > Balassi >>>>> > > > > > > < >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> balassi.mar...@gmail.com> >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >> wrote: >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >> >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>> Thanks, Chesney - I totally >>>>> missed >>>>> > > that. >>>>> > > > > > > >> Answered >>>>> > > > > > > >> > on >>>>> > > > > > > >> > >>> the >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> ticket too, let >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>> us continue there then. >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>> >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>> Till, I agree that we should >>>>> keep >>>>> > this >>>>> > > > > > > codepath >>>>> > > > > > > >> as >>>>> > > > > > > >> > >>> slim >>>>> > > > > > > >> > >>> >> as >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> possible. It >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>> is an important design >>>>> decision that >>>>> > > we >>>>> > > > > aim >>>>> > > > > > to >>>>> > > > > > > >> keep >>>>> > > > > > > >> > >>> the >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> list of >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>> authentication protocols to a >>>>> > minimum. >>>>> > > > We >>>>> > > > > > > >> believe >>>>> > > > > > > >> > >>> that >>>>> > > > > > > >> > >>> >> this >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> should not be a >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>> primary concern of Flink and a >>>>> > trusted >>>>> > > > > proxy >>>>> > > > > > > >> > service >>>>> > > > > > > >> > >>> (for >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> example Apache >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>> Knox) should be used to >>>>> enable a >>>>> > > > multitude >>>>> > > > > > of >>>>> > > > > > > >> > enduser >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> authentication >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>> mechanisms. The bare minimum >>>>> of >>>>> > > > > > authentication >>>>> > > > > > > >> > >>> mechanisms >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> to support >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>> consequently consist of a >>>>> single >>>>> > > strong >>>>> > > > > > > >> > >>> authentication >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> protocol for which >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>> Kerberos is the enterprise >>>>> solution >>>>> > > and >>>>> > > > > HTTP >>>>> > > > > > > >> Basic >>>>> > > > > > > >> > >>> >> primary >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> for development >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>> and light-weight scenarios. >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>> >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>> Added the above wording to >>>>> G's doc. >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>> >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>> >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>>> > > > > > > >> > >>> >> >>>>> > > > > > > >> > >>> >>>>> > > > > > > >> > >>>>> > > > > > > >> >>>>> > > > > > > >>>>> > > > > > >>>>> > > > > >>>>> > > > >>>>> > > >>>>> > >>>>> https://docs.google.com/document/d/1NMPeJ9H0G49TGy3AzTVVJVKmYC0okwOtqLTSPnGqzHw/edit >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>> >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>> >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>> >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>> On Tue, Jun 1, 2021 at 11:47 >>>>> AM >>>>> > > Chesnay >>>>> > > > > > > >> Schepler < >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> ches...@apache.org> >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>> wrote: >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>> >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> There's a related effort: >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> >>>>> > > > > > > >> https://issues.apache.org/jira/browse/FLINK-21108 >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> On 6/1/2021 10:14 AM, Till >>>>> Rohrmann >>>>> > > > > wrote: >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> > Hi Gabor, welcome to the >>>>> Flink >>>>> > > > > community! >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> > >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> > Thanks for sharing this >>>>> proposal >>>>> > > with >>>>> > > > > the >>>>> > > > > > > >> > >>> community >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> Márton. In >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> general, I >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> > agree that authentication >>>>> is >>>>> > > missing >>>>> > > > > and >>>>> > > > > > > that >>>>> > > > > > > >> > >>> this is >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> required for >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> using >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> > Flink within an >>>>> enterprise. The >>>>> > > > thing I >>>>> > > > > > am >>>>> > > > > > > >> > >>> wondering >>>>> > > > > > > >> > >>> >> is >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> whether this >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> > feature strictly needs to >>>>> be >>>>> > > > > implemented >>>>> > > > > > > >> inside >>>>> > > > > > > >> > of >>>>> > > > > > > >> > >>> >> Flink >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> or whether a >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> proxy >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> > setup could do the job? >>>>> Have you >>>>> > > > > > considered >>>>> > > > > > > >> this >>>>> > > > > > > >> > >>> >> option? >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> If yes, then >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> it >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> > would be good to list it >>>>> under >>>>> > the >>>>> > > > > point >>>>> > > > > > of >>>>> > > > > > > >> > >>> rejected >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> alternatives. >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> > >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> > I do see the benefit of >>>>> > > implementing >>>>> > > > > this >>>>> > > > > > > >> > feature >>>>> > > > > > > >> > >>> >> inside >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> of Flink if >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> many >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> > users need it. If not, >>>>> then it >>>>> > > might >>>>> > > > be >>>>> > > > > > > >> easier >>>>> > > > > > > >> > >>> for the >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> project to not >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> > increase the surface area >>>>> since >>>>> > it >>>>> > > > > makes >>>>> > > > > > > the >>>>> > > > > > > >> > >>> overall >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> maintenance >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> harder. >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> > >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> > Cheers, >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> > Till >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> > >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> > On Mon, May 31, 2021 at >>>>> 4:57 PM >>>>> > > > Márton >>>>> > > > > > > >> Balassi < >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> mbala...@apache.org> >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> wrote: >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> > >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> >> Hi team, >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> >> >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> >> Firstly I would like to >>>>> > introduce >>>>> > > > > Gabor >>>>> > > > > > > or G >>>>> > > > > > > >> > [1] >>>>> > > > > > > >> > >>> for >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> short to the >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> >> community, he is a Spark >>>>> > committer >>>>> > > > who >>>>> > > > > > has >>>>> > > > > > > >> > >>> recently >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> transitioned to >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> the >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> >> Flink Engineering team at >>>>> > Cloudera >>>>> > > > and >>>>> > > > > > is >>>>> > > > > > > >> > looking >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> forward to >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> contributing >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> >> to Apache Flink. >>>>> Previously G >>>>> > > > > primarily >>>>> > > > > > > >> focused >>>>> > > > > > > >> > >>> on >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> Spark Streaming >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> and >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> >> security. >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> >> >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> >> Based on requests from our >>>>> > > > customers G >>>>> > > > > > has >>>>> > > > > > > >> > >>> >> implemented >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> Kerberos and >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> HTTP >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> >> Basic Authentication for >>>>> the >>>>> > Flink >>>>> > > > > > > Dashboard >>>>> > > > > > > >> > and >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> HistoryServer. >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> Previously >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> >> lacked an authentication >>>>> story. >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> >> >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> >> We are looking to >>>>> contribute >>>>> > this >>>>> > > > > > > >> functionality >>>>> > > > > > > >> > >>> back >>>>> > > > > > > >> > >>> >> to >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> the >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> community, we >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> >> believe that given Flink's >>>>> > > maturity >>>>> > > > > > there >>>>> > > > > > > >> > should >>>>> > > > > > > >> > >>> be a >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> common code >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> solution >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> >> for this general pattern. >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> >> >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> >> We are looking forward to >>>>> your >>>>> > > > > feedback >>>>> > > > > > on >>>>> > > > > > > >> G's >>>>> > > > > > > >> > >>> >> design. >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> [2] >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> >> >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> >> [1] >>>>> http://gaborsomogyi.com/ >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> >> [2] >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> >> >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> >> >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>>> > > > > > > >> > >>> >> >>>>> > > > > > > >> > >>> >>>>> > > > > > > >> > >>>>> > > > > > > >> >>>>> > > > > > > >>>>> > > > > > >>>>> > > > > >>>>> > > > >>>>> > > >>>>> > >>>>> https://docs.google.com/document/d/1NMPeJ9H0G49TGy3AzTVVJVKmYC0okwOtqLTSPnGqzHw/edit >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> >> >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>> >>>>> > > > > > > >> > >>> >> >>>>>>>>>>>> >>>>> > > > > > > >> > >>> >> >>>>>>>>>>> >>>>> > > > > > > >> > >>> >> >>>>> > > > > > > >> > >>> > >>>>> > > > > > > >> > >>> > >>>>> > > > > > > >> > >>> > -- >>>>> > > > > > > >> > >>> > >>>>> > > > > > > >> > >>> > Konstantin Knauf >>>>> > > > > > > >> > >>> > >>>>> > > > > > > >> > >>> > https://twitter.com/snntrable >>>>> > > > > > > >> > >>> > >>>>> > > > > > > >> > >>> > https://github.com/knaufk >>>>> > > > > > > >> > >>> > >>>>> > > > > > > >> > >>> >>>>> > > > > > > >> > >> >>>>> > > > > > > >> > >> >>>>> > > > > > > >> > >> -- >>>>> > > > > > > >> > >> >>>>> > > > > > > >> > >> Konstantin Knauf >>>>> > > > > > > >> > >> >>>>> > > > > > > >> > >> https://twitter.com/snntrable >>>>> > > > > > > >> > >> >>>>> > > > > > > >> > >> https://github.com/knaufk >>>>> > > > > > > >> > >> >>>>> > > > > > > >> > > >>>>> > > > > > > >> > >>>>> > > > > > > >> >>>>> > > > > > > > >>>>> > > > > > > >>>>> > > > > > >>>>> > > > > >>>>> > > > >>>>> > > > >>>>> > > > -- >>>>> > > > >>>>> > > > Konstantin Knauf >>>>> > > > >>>>> > > > https://twitter.com/snntrable >>>>> > > > >>>>> > > > https://github.com/knaufk >>>>> > > > >>>>> > > >>>>> > >>>>> >>>>
