Yes, Kevin, that seems to be missing from the pull request. Also, the parent pom doesn't have all of the changes needed to add a new module.
See the section in the article called "Root Level Pom.xml" and the section below that for the "Gateway Release Module Pom.xml" https://github.com/lmccay/gateway-provider-security-pseudo/blob/master/AddingFederationProvider.md#root-level-pomxml On Thu, Nov 12, 2015 at 8:04 AM, Kevin Minder <kevin.min...@hortonworks.com> wrote: > I think I know what it probably is. The gateway-release module pom.xml is > what actually defines the classpath of the final packaged server. If you > take a look at that you will see it references all of the providers that > are intended to be included by default. Try adding yours to the > dependencies. > > > > > On 11/12/15, 7:49 AM, "larry mccay" <lmc...@apache.org> wrote: > > >We really need to fix this behavior... > > > >It seems you traded in your random authentication provider for a random > >federation provider. :/ > > > >This is most likely due to not having your new provider on the classpath > >for the serviceLoader to find. > > > >Just to try and articulate how it works... > > > >The deployment machinery loads all of the federation (whichever role) > >providers contributors that is finds on the classpath - via the > > >src/main/resources/META-INF/services/org.apache.hadoop.gateway.deploy.ProviderDeploymentContributor > >- and interrogates each for its name to match to the name in the topology. > >Once it finds a match it allows the contributor to "contribute" to the > >creation of the topology specific webapp. Part of which includes adding > >filters to the gateway.xml. > > > >We currently have an issue wherein, given a name that doesn't match any > >found on the classpath, a random provider is returned. This needs to be > >fixed to fail deployment with appropriate error messages. > > > >Now, this usually comes down to one of 2 things: > > > >1. there is no > src/main/resources/META-INF/services/org.apache.hadoop.gateway.deploy.ProviderDeploymentContributor > >file in the provider or it contains the classname of some other > contributor > >2. the provider itself isn't on the gateway classpath. I see that your > >service file seems to be defined properly - so it is probably not on the > >classpath. Did you add the new provider to the parent pom? The article > >should provide all you need to do so. > > > >I will file a JIRA to fix the random provider issue. > > > >On Thu, Nov 12, 2015 at 7:18 AM, Jérôme LELEU <lel...@gmail.com> wrote: > > > >> Hi, > >> > >> By changing the role to the appropriate value, the error is gone, but I > >> have a new problem: > >> > >> *Caused by: javax.servlet.ServletException: Required authentication > >> provider URL is missing.* > >> * at > >> > >> > org.apache.hadoop.gateway.provider.federation.jwt.filter.SSOCookieFederationFilter.init(SSOCookieFederationFilter.java:90)* > >> > >> My sandbox.xml file only references the pac4j provider as well as the > >> Default (role: identity-assertion) and static (role: hostmap) ones. > >> > >> Though, the SSOCookieFederationFilter is defined in the gateway.xml > file of > >> my sandbox deployment. Why do I need it? What's the expected > authentication > >> provider url? How this gatweay.xml is generated? > >> > >> Thanks. > >> Best regards, > >> Jérôme > >> > >> > >> > >> > >> > >> 2015-11-10 17:36 GMT+01:00 Kevin Minder <kevin.min...@hortonworks.com>: > >> > >> > To be a bit more explicit your Pac4jFederationProviderContributor > >> declares > >> > itself to be of role “federation” but when you used it in the > sandbox.xml > >> > topology file you were attempting to configure it as an provider with > >> role > >> > authentication. > >> > > >> > > >> > > >> > > >> > On 11/10/15, 11:32 AM, "larry mccay" <lmc...@apache.org> wrote: > >> > > >> > >Let's get you past the error that you are getting... > >> > > > >> > >It seems as though you have change the topology for pac4j but didn't > >> > change > >> > >the role to "federation". > >> > >It looks like it is grabbing a random authentication provider and > seems > >> to > >> > >be the hadoop-auth provider. > >> > > > >> > >Change that to federation and see if that helps. > >> > > > >> > >On Tue, Nov 10, 2015 at 11:21 AM, larry mccay <lmc...@apache.org> > >> wrote: > >> > > > >> > >> Hi Jérôme - > >> > >> > >> > >> Happy to see you here! > >> > >> I apologize for having missed your note on the list a few days ago. > >> > >> > >> > >> Glad to see that the article was helpful and it seems like you are > >> > making > >> > >> great progress. > >> > >> > >> > >> Let me dig into your note a bit deeper and answer your questions. > >> > >> > >> > >> Welcome! > >> > >> > >> > >> --larry > >> > >> > >> > >> On Tue, Nov 10, 2015 at 11:14 AM, Jérôme LELEU <lel...@gmail.com> > >> > wrote: > >> > >> > >> > >>> Hi, > >> > >>> > >> > >>> Perfect timing ;-) I have started implemented the pac4j support in > >> > Knox, > >> > >>> it's a very basic attempt: > >> https://github.com/apache/knox/pull/2/files > >> > >>> > >> > >>> This implementation reuses the pac4j implementation for J2E. I > have a > >> > >>> ProviderDeploymentContributorBase which registers two filters: > >> > >>> - the first one is a dispatcher filter which uses the j2e-pac4j > >> > >>> CallbackFilter on the /callabck url (it finishes the > authentication > >> > process > >> > >>> after a successful authentication at some identity provider) and > uses > >> > the > >> > >>> j2e-pac4j RequiresAuthenticationFilter otherwise (redirection to > the > >> > >>> identity provider to start the authentication when the user is not > >> > >>> authenticated) > >> > >>> - the second one is an identity adapter which gets the current > >> > >>> authenticated user from the pac4j point of view and populates the > J2E > >> > >>> security context. > >> > >>> > >> > >>> > >> > >>> * I have the following error: > >> > >>> > >> > >>> 2015-11-10 16:30:22,347 ERROR hadoop.gateway > >> > >>> (GatewayFilter.java:doFilter(135)) - Gateway processing failed: > >> > >>> javax.servlet.ServletException: javax.servlet.ServletException: > >> > >>> Authentication type must be specified: simple|kerberos|<class> > >> > >>> javax.servlet.ServletException: javax.servlet.ServletException: > >> > >>> Authentication type must be specified: simple|kerberos|<class> > >> > >>> at > >> > >>> > >> > > >> > org.apache.hadoop.gateway.GatewayFilter$Holder.getInstance(GatewayFilter.java:354) > >> > >>> > >> > >>> Notice I have changed the sandbox.xml file to use pac4j instead of > >> > Shiro, > >> > >>> maybe it's not the right way: > >> > >>> > >> > > >> > https://github.com/apache/knox/pull/2/files#diff-128ca80baa2dfd6c2d25de7b8160b9d9R24 > >> > >>> > >> > >>> Any idea of the problem? > >> > >>> > >> > >>> > >> > >>> * Am I understanding webflows correctly? > >> > >>> > >> > >>> I use the default request to: curl -ivk " > >> > >>> > https://localhost:8443/gateway/sandbox/webhdfs/v1/tmp?op=LISTSTATUS"; > >> > >>> > >> > >>> It looks like a REST request, though I'm expecting the following > >> > webflow: > >> > >>> redirection of the user to the identity provider, successful > login, > >> > >>> redirection back to the /callback url of the gateway: do I get it > >> > right? > >> > >>> > >> > >>> > >> > >>> * The callback url > >> > >>> > >> > >>> I currently do a check expecting the last part of the url to end > with > >> > >>> /callback, but does the gateway will receive requests on this url? > >> Do I > >> > >>> have to do something special? > >> > >>> > >> > >>> It's highly recommended to only have one callback url as this url > is > >> > >>> defined on the identity provider side. > >> > >>> > >> > >>> > >> > >>> * How to pass configuration? > >> > >>> > >> > >>> Currently, I hardcoded a FacebookClient for Facebook > authentication, > >> > but > >> > >>> we should be able to pass the appropriate client like Facebook or > >> SAML. > >> > >>> Basically, we could do that using filter properties: facebook.key > + > >> > >>> facebook.secret means we use Facebook authentication with the > >> > appropriate > >> > >>> key and secret for example. Any recommendation? > >> > >>> > >> > >>> > >> > >>> * Can I use the J2E session to store the requested protected url > and > >> > >>> other information? > >> > >>> > >> > >>> Or do I need to do something special? > >> > >>> > >> > >>> > >> > >>> Thanks. > >> > >>> Best regards, > >> > >>> Jérôme > >> > >>> > >> > >>> 2015-11-10 17:03 GMT+01:00 Kevin Minder < > >> kevin.min...@hortonworks.com > >> > >: > >> > >>> > >> > >>>> Hi Jerome, > >> > >>>> This sounds very exciting and is just the sort of thing the open > >> > >>>> architecture of Knox is intended to allow. Larry and I have > looked > >> at > >> > >>>> pac4j a number of times thinking that it would be a good fit and > >> would > >> > >>>> provide a great feature set to the Hadoop community. > >> > >>>> Kevin. > >> > >>>> > >> > >>>> > >> > >>>> > >> > >>>> On 11/6/15, 3:31 AM, "Jérôme LELEU" <lel...@gmail.com> wrote: > >> > >>>> > >> > >>>> >Hi, > >> > >>>> > > >> > >>>> >Let's open this private discussion on the Knox dev mailing list. > >> > >>>> > > >> > >>>> >I'm Jerome Leleu and the creator of pac4j ( > >> > >>>> https://github.com/pac4j/pac4j), > >> > >>>> >a security engine for Java with many implementations for J2E, > Play, > >> > >>>> Spring, > >> > >>>> >Vertx, Ratpack... The idea is to offer something as powerful as > >> > Spring > >> > >>>> >Security, but a lot easier and for all Java frameworks / tools > and > >> > all > >> > >>>> >authentication mechanisms. > >> > >>>> > > >> > >>>> >Two years ago, we had a discussion with Larry on how pac4j > could be > >> > >>>> used in > >> > >>>> >Knox. > >> > >>>> > > >> > >>>> >Meanwhile, both projects have grown up and I'm back to try to > see > >> if > >> > >>>> pac4j > >> > >>>> >can help Knox in terms of security. > >> > >>>> > > >> > >>>> >----- > >> > >>>> > > >> > >>>> >In its latest version, pac4j can be used to build a full > security > >> > >>>> library, > >> > >>>> >not only focused on delegated authentication (Facebook, SAML). > So > >> we > >> > >>>> have > >> > >>>> >the concepts of direct clients (a client is an authentication > >> > >>>> mechanism): > >> > >>>> >basic auth, header, token... and indirect clients: SAML, > Facebook, > >> > >>>> OpenID, > >> > >>>> >CAS... > >> > >>>> >It seems to be very close to the Authentication and Federation > >> > Providers > >> > >>>> >concepts from Knox. > >> > >>>> > > >> > >>>> >The provided article is great and as it's focused on > authentication > >> > >>>> with a > >> > >>>> >real example, it's easier to start with it. > >> > >>>> > > >> > >>>> >Thanks for your answers. > >> > >>>> > > >> > >>>> >Like Shiro, pac4j has LDAP support and caching can be done. The > >> only > >> > >>>> >difference is that pac4j relies on ldaptive ( > >> > http://www.ldaptive.org/). > >> > >>>> But > >> > >>>> >I think you're right: it would be better to start by > implementing > >> the > >> > >>>> >Federation part which might be the most expected feature for > Knox > >> > users > >> > >>>> (a > >> > >>>> >bit like buji-pac4j: https://github.com/bujiio/buji-pac4j is > for > >> > >>>> Shiro) and > >> > >>>> >see if it's worth the work to have an authentication provider > with > >> > >>>> pac4j in > >> > >>>> >addition to the one of Shiro. > >> > >>>> > > >> > >>>> >I plan to start working on that next week. Do you accept pull > >> > requests > >> > >>>> on > >> > >>>> >Github (it would make discussion easier on source code)? > >> > >>>> > > >> > >>>> >Any feedback will be appreciated. > >> > >>>> > > >> > >>>> >Thanks. > >> > >>>> >Best regards, > >> > >>>> >Jérôme > >> > >>>> > > >> > >>>> > > >> > >>>> > > >> > >>>> > > >> > >>>> > > >> > >>>> > > >> > >>>> >2015-11-05 17:13 GMT+01:00 larry mccay: > >> > >>>> > > >> > >>>> >> Hi Jérôme - > >> > >>>> >> > >> > >>>> >> Contributions coming from the community are owned and > maintained > >> by > >> > >>>> the > >> > >>>> >> entire community. > >> > >>>> >> You would certainly not be expected to do all work on it and > at > >> the > >> > >>>> same > >> > >>>> >> time any assistance and upgrade messages that you provided > would > >> be > >> > >>>> awesome. > >> > >>>> >> > >> > >>>> >> As for an empty module for pac4j, maybe the following example > >> > project > >> > >>>> >> would be helpful to you: > >> > >>>> >> > >> > >>>> >> https://github.com/lmccay/gateway-provider-security-pseudo > >> > >>>> >> > >> > >>>> >> It is in support of an article to illustrate how to go about > >> > adding a > >> > >>>> >> authentication or federation provider - as you would be doing: > >> > >>>> >> > >> > >>>> >> > >> > >>>> > >> > > >> > https://github.com/lmccay/gateway-provider-security-pseudo/blob/master/AddingFederationProvider.md > >> > >>>> >> > >> > >>>> >> That may be easier to consume than the entire developers > guide. > >> > >>>> >> That said, it would be great if you could describe what makes > the > >> > dev > >> > >>>> >> guide too much - if that is the case. > >> > >>>> >> > >> > >>>> >> I'll take a stab at your questions here but we should move > more > >> > >>>> detailed > >> > >>>> >> discussions on the topic to the dev@ list in order to keep > this > >> in > >> > >>>> the > >> > >>>> >> open and provide insights to others: > >> > >>>> >> > >> > >>>> >> 1) What do I do with the authenticated user: where / how do I > >> fill > >> > >>>> this > >> > >>>> >> identity in Knox? > >> > >>>> >> > >> > >>>> >> We normalize the authentication event into a Java subject to > >> > >>>> represent the > >> > >>>> >> user, groups and impersonated user as appropriate. > >> > >>>> >> In cases - like shiro - where we don't have access to the > actual > >> > >>>> >> authenticating code and we need to normalize the provider > >> specific > >> > >>>> security > >> > >>>> >> context, we add another filter. You can look at the shiro > >> provider > >> > >>>> for an > >> > >>>> >> example. The code inside the simple example provider in the > >> article > >> > >>>> also > >> > >>>> >> shows what is expected at least in terms of the > PrimaryPrincipal. > >> > >>>> >> > >> > >>>> >> 2) How to handle errors: not authenticated, not authorized? > >> > >>>> >> > >> > >>>> >> We aren't intrusive on what should be done by a provider here. > >> > >>>> >> Generally, authentication failures result in a 401 and > >> > authorizations > >> > >>>> in a > >> > >>>> >> 403 and they are usually sent by the provider. > >> > >>>> >> > >> > >>>> >> 3) How to handle redirections to an external provider? > >> > >>>> >> > >> > >>>> >> Redirects are sent by the providers themselves with a 307 or > 302. > >> > >>>> >> You can look at the SSOCookieFederationFilter in the jwt > module > >> as > >> > an > >> > >>>> >> example if you like. > >> > >>>> >> > >> > >>>> >> 4) Should the pac4j filter also handle authorizations (pac4j > can > >> do > >> > >>>> that)? > >> > >>>> >> > >> > >>>> >> This is an interesting question. > >> > >>>> >> Knox provides a separate slot in the provider chain for > >> > authorization. > >> > >>>> >> It is there because of the ability for identity assertion > >> providers > >> > >>>> to map > >> > >>>> >> principals in order to impersonate another user or map > external > >> > >>>> usernames > >> > >>>> >> to those used inside of hadoop clusters. So, the authorization > >> > >>>> decisions > >> > >>>> >> are made based on those identities. > >> > >>>> >> > >> > >>>> >> If we can add the authorization filter as an authorization > >> provider > >> > >>>> rather > >> > >>>> >> than part of the authentication provider than that might make > >> > sense. > >> > >>>> >> Although, the security context is normalized as Knox expects > not > >> as > >> > >>>> pac4j > >> > >>>> >> authorization probably does. Now, if there is real value in > doing > >> > so, > >> > >>>> the > >> > >>>> >> pac4j Subject adaptor could stuff additional context into the > >> > Subject > >> > >>>> or > >> > >>>> >> the request that can be accessed later by the pac4j > authorization > >> > >>>> provider. > >> > >>>> >> > >> > >>>> >> So, it can be done - the question is whether there is > compelling > >> > >>>> reason to > >> > >>>> >> do so. > >> > >>>> >> > >> > >>>> >> 5) How to test? > >> > >>>> >> > >> > >>>> >> This can be challenging especially for external > authentications > >> and > >> > >>>> web > >> > >>>> >> app flows. > >> > >>>> >> Generally, unit testing as much as possible is required. > >> > >>>> >> We would want to make sure that the security context > >> normalization > >> > >>>> happens > >> > >>>> >> as expected, etc. > >> > >>>> >> > >> > >>>> >> Additional considerations: > >> > >>>> >> > >> > >>>> >> 1. Our Shiro provider does group lookup and caching of the > >> > >>>> authentication > >> > >>>> >> event for optimizing interactions with authentication servers, > >> > LDAP, > >> > >>>> AD, > >> > >>>> >> etc. We wouldn't want to lose these. Doing so would be a > >> > non-starter > >> > >>>> for > >> > >>>> >> most folks replacing what Shiro is used for today. > >> > >>>> >> 2. It is probably best to concentrate on the gaps that pac4j > can > >> > fill > >> > >>>> for > >> > >>>> >> openid, oauth, etc for the initial contribution and if the > >> usecases > >> > >>>> handled > >> > >>>> >> by Shiro can be done better, easier, simpler then we can > >> consider a > >> > >>>> >> migration path. > >> > >>>> >> > >> > >>>> >> BTW - we have a couple very interesting possibilities for > strong > >> > and > >> > >>>> >> multi-factor authentication that would be enabled by OpenID > >> > >>>> immediately. > >> > >>>> >> > >> > >>>> >> Hope this isn't too long a response and that it is helpful for > >> you. > >> > >>>> >> > >> > >>>> >> Let's determine what an initial contribution would be for and > >> > bring a > >> > >>>> >> proposal to the dev@ list and/or file a jira for the > >> integration. > >> > >>>> >> This would be great for adding features and growing the > >> community - > >> > >>>> so we > >> > >>>> >> are all for it! > >> > >>>> >> > >> > >>>> >> thanks, > >> > >>>> >> > >> > >>>> >> --larry > >> > >>>> >> > >> > >>>> >> > >> > >>>> >> On Thu, Nov 5, 2015 at 9:43 AM, Jérôme LELEU < > lel...@gmail.com> > >> > >>>> wrote: > >> > >>>> >> > >> > >>>> >>> Hi, > >> > >>>> >>> > >> > >>>> >>> I see that Shiro is already used for basic auth and LDAP > >> > >>>> authentication > >> > >>>> >>> and Picketlink for SAML. pac4j v1.8 can now handle both cases > >> and > >> > >>>> even > >> > >>>> >>> more. So I think we could create a > >> gateway-provider-security-pac4j > >> > >>>> >>> supporting all authentication mechanisms and not only OpenID. > >> > >>>> >>> > >> > >>>> >>> pac4j implementations generally work with two filters: one to > >> > >>>> protect a > >> > >>>> >>> resource and play direct authentication like basic auth (and > >> check > >> > >>>> >>> authorizations) and a callback filter to finish remote > >> > >>>> authentication like > >> > >>>> >>> Facebook, SAML, OpenID. The easiest one is the j2e-pac4j > with: > >> > >>>> >>> > >> > >>>> > >> > > >> > https://github.com/pac4j/j2e-pac4j/blob/master/src/main/java/org/pac4j/j2e/filter/RequiresAuthenticationFilter.java#L96 > >> > >>>> >>> and > >> > >>>> >>> > >> > >>>> > >> > > >> > https://github.com/pac4j/j2e-pac4j/blob/master/src/main/java/org/pac4j/j2e/filter/CallbackFilter.java#L65 > >> > >>>> >>> > >> > >>>> >>> That said, I have already many pac4j implementations to > handle > >> so > >> > I'm > >> > >>>> >>> pushing communities to maintain on their own their pac4j > >> > >>>> implementations: > >> > >>>> >>> can we imagine having an official: > >> gateway-provider-security-pac4j > >> > >>>> module > >> > >>>> >>> maintained by the Knox community with my help of course? > This is > >> > the > >> > >>>> kind > >> > >>>> >>> of message I deliver to communities when a new version of > pac4j > >> is > >> > >>>> >>> available so that they can upgrade: > >> > >>>> >>> https://github.com/ratpack/ratpack/issues/819 > >> > >>>> >>> > >> > >>>> >>> I read: > >> > >>>> > https://knox.apache.org/books/knox-0-6-0/dev-guide.html#Providers, > >> > >>>> >>> it might be easy for Knox contributors, but it's a bit hard > for > >> me > >> > >>>> to get > >> > >>>> >>> in everything: can you or someone in the Knox community > provides > >> > me > >> > >>>> an > >> > >>>> >>> empty gateway-provider-security-pac4j module and the default > >> > >>>> expectations > >> > >>>> >>> from a Knox point of view? > >> > >>>> >>> My main questions: > >> > >>>> >>> 1) What do I do with the authenticated user: where / how do I > >> fill > >> > >>>> this > >> > >>>> >>> identity in Knox? > >> > >>>> >>> 2) How to handle errors: not authenticated, not authorized? > >> > >>>> >>> 3) How to handle redirections to an external provider? > >> > >>>> >>> 4) Should the pac4j filter also handle authorizations (pac4j > can > >> > do > >> > >>>> that)? > >> > >>>> >>> 5) How to test? > >> > >>>> >>> > >> > >>>> >>> Thanks. > >> > >>>> >>> Best regards, > >> > >>>> >>> Jérôme > >> > >>>> >>> > >> > >>>> >>> > >> > >>>> >>> > >> > >>>> >>> > >> > >>>> >>> 2015-11-03 16:58 GMT+01:00 larry mccay: > >> > >>>> >>> > >> > >>>> >>>> Hi Jérôme - > >> > >>>> >>>> > >> > >>>> >>>> We were evaluating buji-pac4j for SSO with SAML and ended up > >> > going > >> > >>>> with > >> > >>>> >>>> Picketlink at the time. > >> > >>>> >>>> That said, we do have a pluggable architecture that would > allow > >> > for > >> > >>>> a > >> > >>>> >>>> pac4j provider as well. > >> > >>>> >>>> > >> > >>>> >>>> If you are interested in contributing such a provider that > >> would > >> > >>>> >>>> certainly be welcome. > >> > >>>> >>>> There is someone in the community working on an openid > provider > >> > but > >> > >>>> >>>> perhaps pac4j would be the way to go there? > >> > >>>> >>>> > >> > >>>> >>>> thanks, > >> > >>>> >>>> > >> > >>>> >>>> --larry > >> > >>>> >>>> > >> > >>>> >>>> > >> > >>>> > >> > >>> > >> > >>> > >> > >> > >> > > >> >
