Re: [DISCUSS] Secure Hadoop without Kerberos

2020-05-26 Thread Eric Yang
This sounds promising and really fantastic news. We look forward to this feature, and let us know what we can do to help. Thanks regards, Eric On Tue, May 26, 2020 at 10:55 AM Daryn Sharp wrote: > There’s a few too many issues being mixed here. > > > We aren’t very far from having OIDC suppor

Re: [DISCUSS] Secure Hadoop without Kerberos

2020-05-26 Thread Daryn Sharp
There’s a few too many issues being mixed here. We aren’t very far from having OIDC support. The pre-requisite RPC/TLS & RPC/mTLS recently completed rollout to our entire production grid. Majority of the past year was spent shaking out bugs and ensuring 100% compatibility. There are a few rough

Re: [DISCUSS] Secure Hadoop without Kerberos

2020-05-21 Thread Eric Yang
Hi Steve, Thank you for sharing the work done for Amazon STS token to work with s3a connector. This works for direct HDFS to S3 bucket interaction. Your statement is also spot on for containers running in YARN has no mechanism to update the triple of session credentials. If I am not mistaken, Am

Re: [DISCUSS] Secure Hadoop without Kerberos

2020-05-21 Thread Vipin Rathor
Hi Eric, Thanks for starting this discussion. Kerberos was developed decade before web development becomes popular. > There are some Kerberos limitations which does not work well in Hadoop. > Sure, Kerberos was developed long before the web but it was selected as de facto authentication mechanism

Re: [DISCUSS] Secure Hadoop without Kerberos

2020-05-21 Thread Steve Loughran
On Wed, 6 May 2020 at 23:32, Eric Yang wrote: > Hi all, > > > 4. Passing different form of tokens does not work well with cloud provider > security mechanism. For example, passing AWS sts token for S3 bucket. > There is no renewal mechanism, nor good way to identify when the token > would expir

Re: [DISCUSS] Secure Hadoop without Kerberos

2020-05-20 Thread Eric Yang
See my comments inline: On Wed, May 20, 2020 at 4:50 PM Rajive Chittajallu wrote: > On Wed, May 20, 2020 at 1:47 PM Eric Yang wrote: > > > >> > Kerberos was developed decade before web development becomes popular. > >> > There are some Kerberos limitations which does not work well in > Hadoop.

Re: [DISCUSS] Secure Hadoop without Kerberos

2020-05-20 Thread Rajive Chittajallu
On Wed, May 20, 2020 at 1:47 PM Eric Yang wrote: > >> > Kerberos was developed decade before web development becomes popular. >> > There are some Kerberos limitations which does not work well in Hadoop. A >> > few examples of corner cases: >> >> Microsoft Active Directory, which is extensively us

Re: [DISCUSS] Secure Hadoop without Kerberos

2020-05-20 Thread Rajive Chittajallu
On Wed, May 6, 2020 at 3:32 PM Eric Yang wrote: > > Hi all, > > Kerberos was developed decade before web development becomes popular. > There are some Kerberos limitations which does not work well in Hadoop. A > few examples of corner cases: Microsoft Active Directory, which is extensively used

Re: [EXTERNAL] Re: [DISCUSS] Secure Hadoop without Kerberos

2020-05-20 Thread Craig . Condit
ot be possible. Craig Condit From: Eric Yang Sent: Wednesday, May 20, 2020 1:57 PM To: Akira Ajisaka Cc: Hadoop Common Subject: [EXTERNAL] Re: [DISCUSS] Secure Hadoop without Kerberos Hi Akira, Thank you for the information. Knox plays a main role in reverse prox

Re: [DISCUSS] Secure Hadoop without Kerberos

2020-05-20 Thread Eric Yang
Hi Akira, Thank you for the information. Knox plays a main role in reverse proxy for Hadoop cluster. I understand the importance to keep Knox running to centralize audit log for ingress into the cluster. Other reverse proxy solution like Nginx are more feature rich for caching static contents a

Re: [DISCUSS] Secure Hadoop without Kerberos

2020-05-19 Thread Akira Ajisaka
Hi Eric, thank you for starting the discussion. I'm interested in OpenID Connect (OIDC) integration. In addition to the benefits (security, cloud native), operating costs may be reduced in some companies. We have our company-wide OIDC provider and enable SSO for Hadoop Web UIs via Knox + OIDC in

[DISCUSS] Secure Hadoop without Kerberos

2020-05-06 Thread Eric Yang
Hi all, Kerberos was developed decade before web development becomes popular. There are some Kerberos limitations which does not work well in Hadoop. A few examples of corner cases: 1. Kerberos principal doesn't encode port number, it is difficult to know if the principal is coming from an autho