Thank you, team. Then based on this agreement we are moving the proposal to
the wiki and opening the PR soon.
On Thu, Jul 1, 2021 at 12:28 AM Austin Cawley-Edwards <
austin.caw...@gmail.com> wrote:
> >
> > * Even if there is a major breaking version, Flink releases major
> versions
> > too where
>
> * Even if there is a major breaking version, Flink releases major versions
> too where it could be added
> Netty framework locking is true but AFAIK there was a discussion to
> rewrite the Netty stuff to a more sexy thing but there was no agreement to
> do that.
>
Flink major releases seem to
Answered here because the text started to be crowded.
> It also locks Flink into the current major version of Netty (and the
Netty framework itself) for the foreseeable future.
It's not doing any Netty version locking because:
* Netty not necessarily will add breaking changes in major versions,
Small correction:
I am *not *saying we should not do this, perhaps this is the best solution
to finding a good compromise here, but I am trying to discover +
acknowledge the full implications of this proposal so they can be
discussed.
Sorry :)
On Wed, Jun 30, 2021 at 9:53 AM Austin
Hi Gabor,
Thanks for your answers. I appreciate the explanations. Please see my
responses + further questions below.
* What stability semantics do you envision for this API?
>>
> As I foresee the API will be as stable as Netty API. Since there is
> guarantee on no breaking changes between minor
Hi Austin,
Please see my answers embedded down below.
BR,
G
On Tue, Jun 29, 2021 at 9:59 PM Austin Cawley-Edwards <
austin.caw...@gmail.com> wrote:
> Hi all,
>
> Thanks for the updated proposal. I have a few questions about the API,
> please see below.
>
> * What stability semantics do you
Hi all,
Thanks for the updated proposal. I have a few questions about the API,
please see below.
* What stability semantics do you envision for this API?
* Does Flink expose dependencies’ APIs in other places? Since this exposes
the Netty API, will this make it difficult to upgrade Netty?
* I
Hi all,
I commend Konstantin and Till when it comes to standing up for the
community values.
Based on your feedback we are withdrawing the original proposal and
attaching a more general custom netty handler API proposal [1] written by
G. The change necessary to the Flink repository is
Hi all,
Thanks, Konstantin and Till, for guiding the discussion.
I was not aware of the results of the call with Konstantin and was
attempting to resolve the unanswered questions before more, potentially
fruitless, work was done.
I am also looking forward to the coming proposal, as well as
Hi everyone,
I do like the idea of keeping the actual change outside of Flink but to
enable Flink to support such a use case (different authentication
mechanisms). I think this is a good compromise for the community that
combines long-term maintainability with support for new use-cases. I am
Hi everyone,
First, Marton and I had a brief conversation yesterday offline and
discussed exploring the approach of exposing the authentication
functionality via an API. So, I am looking forward to your proposal in that
direction. The benefit of such a solution would be that it is extensible
for
Hi Austin,
Thank you for your thoughts. This is where this discussion stops. This
email thread already contains more characters than the implementation and
what is needed for the next 20 years of maintenance.
It is great that you have a view on modern solutions and thank you for
offering your
Hi Gabor + Marton,
I don't believe that the issue with this proposal is the specific mechanism
proposed (Kerberos), but rather that it is not the level to implement it at
(Flink). I'm just one voice, so please take this with a grain of salt.
In the other solutions previously noted there is no
Hi team,
Thank you for your input. Based on this discussion I agree with G that
selecting and standardizing on a specific strong authentication mechanism
is more challenging than the whole rest of the scope of this authentication
story. :-) I suggest that G and I go back to the drawing board and
Hi All,
We see that adding any kind of specific authentication raises more
questions than answers.
What would be if a generic API would be added without any real
authentication logic?
That way every provider can add its own protocol implementation as
additional jar.
BR,
G
On Thu, Jun 17, 2021
Hi all,
Sorry to be joining the conversation late. I'm also on the side of
Konstantin, generally, in that this seems to not be a core goal of Flink as
a project and adds a maintenance burden.
Would another con of Kerberos be that is likely a fading project in terms
of network security? (serious
I left some comments in the Google document. It would be great if
someone from the community with security experience could also take a look
at it. Maybe Eron you have an opinion on the topic.
Cheers,
Till
On Thu, Jun 17, 2021 at 6:57 PM Till Rohrmann wrote:
> Hi Gabor,
>
> I haven't found
Hi Gabor,
I haven't found time to look into the updated FLIP yet. I'll try to do it
asap.
Cheers,
Till
On Wed, Jun 16, 2021 at 9:35 PM Konstantin Knauf wrote:
> Hi Gabor,
>
> > However representing Kerberos as completely new feature is not true
> because
> it's already in since Flink makes
Hi Gabor,
> However representing Kerberos as completely new feature is not true
because
it's already in since Flink makes authentication at least with HDFS and
Hbase through Kerberos.
True, that is one way to look at it, but there are differences, too:
Control Plane vs Data Plane, Core vs
Hi Konstantin,
Thanks for the response. Related new feature introduction in case of Basic
auth I tend to agree, anything else can be chosen.
However representing Kerberos as completely new feature is not true because
it's already in since Flink makes authentication at least with HDFS and
Hbase
Hi everyone,
sorry for joining late and thanks for the insightful discussion.
In general, I'd personally prefer not to increase the surface area of
Apache Flink unless there is a good reason. It seems we all agree that
authx is not part of the core value proposition of Apache Flink, so if we
can
Hi Till,
Did you have the chance to take a look at the doc? Not yet seen any update.
BR,
G
On Wed, Jun 9, 2021 at 1:43 PM Till Rohrmann wrote:
> Thanks for the update Gabor. I'll take a look and respond in the document.
>
> Cheers,
> Till
>
> On Wed, Jun 9, 2021 at 12:59 PM Gabor Somogyi
>
Thanks for the update Gabor. I'll take a look and respond in the document.
Cheers,
Till
On Wed, Jun 9, 2021 at 12:59 PM Gabor Somogyi
wrote:
> Hi Till,
>
> Your proxy suggestion has been considered in-depth and updated the FLIP
> accordingly.
> We've considered 2 proxy implementation (Nginx
Hi Till,
Your proxy suggestion has been considered in-depth and updated the FLIP
accordingly.
We've considered 2 proxy implementation (Nginx and Squid) but according to
our analysis and testing it's not suitable for the mentioned use-cases.
Please take a look at the rejected alternatives for
As I've said I am not a security expert and that's why I have to ask for
clarification, Gabor. You are saying that if we configure a truststore for
the REST endpoint with a single trusted certificate which has been
generated by the operator of the Flink cluster, then the attacker can
generate a
> I did not mean for the user to sign its own certificates but for the
operator of the cluster. Once the user request hits the proxy, it should no
longer be under his control. I think I do not fully understand yet why this
would not work.
I said it's not solving the authentication problem over any
I did not mean for the user to sign its own certificates but for the
operator of the cluster. Once the user request hits the proxy, it should no
longer be under his control. I think I do not fully understand yet why this
would not work.
What I would like to avoid is to add more complexity into
Hi!
I think there might be possible alternatives but it seems Kerberos on the
rest endpoint ticks all the right boxes and provides a super clean and
simple solution for strong authentication.
I wouldn’t even consider sidecar proxies etc if we can solve it in such a
simple way as proposed by G.
Till, thanks for investing time in giving further options.
Marci, thanks for summarizing the use-case point of view.
We've arrived back to one of the original problems. Namely if an attacker
gets access to a node it's possible to cancel other user's jobs (and more
can be done).
Self signed
I am not saying that we shouldn't add a strong authentication mechanism if
there are good reasons for it. I primarily would like to understand the
context a bit better in order to give qualified feedback and come to a good
decision. In order to do this, I have the feeling that we haven't fully
That is an interesting idea, Till.
The main issue with it is that TLS certificates have an expiration time,
usually they get approved for a couple years. Forcing our users to restart
jobs to reprovision TLS certificates would be weird when we could just
implement a single proper strong
I guess the idea would then be to let the proxy do the authentication job
and only forward the request via an SSL mutually encrypted connection to
the Flink cluster. Would this be possible? The beauty of this setup is in
my opinion that this setup should work with all kinds of authentication
Thanks for giving options to fulfil the need.
Users are looking for a solution where users can be identified on the whole
cluster and restrict access to resources/actions.
A good example for such an action is cancelling other users running jobs.
* SSL does provide mutual authentication but when
Thanks for the information Gabor. If it is about securing the
communication between the REST client and the REST server, then Flink
already supports enabling mutual SSL authentication [1]. Would this be
enough to secure the communication and to pass an audit?
[1]
Hi Till,
Since I'm working in security area 10+ years let me share my thought.
I would like to emphasise there are experts better than me but I have some
basics.
The discussion is open and not trying to tell alone things...
> I mean if an attacker can get access to one of the machines, then it
Thanks for updating the document Márton. Why is it that banks will consider
it more secure if Flink comes with Kerberos authentication (assuming a
properly secured setup)? I mean if an attacker can get access to one of the
machines, then it should also be possible to obtain the right Kerberos
Hi team,
Happy to be here and hope I can provide quality additions in the future.
Thank you all for helpful the suggestions!
Considering them the FLIP has been modified and the work continues on the
already existing Jira.
BR,
G
On Wed, Jun 2, 2021 at 11:23 AM Márton Balassi
wrote:
> Thanks,
Thanks, Chesney - I totally missed that. Answered on the ticket too, let us
continue there then.
Till, I agree that we should keep this codepath as slim as possible. It is
an important design decision that we aim to keep the list of authentication
protocols to a minimum. We believe that this
There's a related effort: https://issues.apache.org/jira/browse/FLINK-21108
On 6/1/2021 10:14 AM, Till Rohrmann wrote:
Hi Gabor, welcome to the Flink community!
Thanks for sharing this proposal with the community Márton. In general, I
agree that authentication is missing and that this is
Hi Gabor, welcome to the Flink community!
Thanks for sharing this proposal with the community Márton. In general, I
agree that authentication is missing and that this is required for using
Flink within an enterprise. The thing I am wondering is whether this
feature strictly needs to be
Hi team,
Firstly I would like to introduce Gabor or G [1] for short to the
community, he is a Spark committer who has recently transitioned to the
Flink Engineering team at Cloudera and is looking forward to contributing
to Apache Flink. Previously G primarily focused on Spark Streaming and
41 matches
Mail list logo
