We really need to fix this behavior... It seems you traded in your random authentication provider for a random federation provider. :/
This is most likely due to not having your new provider on the classpath for the serviceLoader to find. Just to try and articulate how it works... The deployment machinery loads all of the federation (whichever role) providers contributors that is finds on the classpath - via the src/main/resources/META-INF/services/org.apache.hadoop.gateway.deploy.ProviderDeploymentContributor - and interrogates each for its name to match to the name in the topology. Once it finds a match it allows the contributor to "contribute" to the creation of the topology specific webapp. Part of which includes adding filters to the gateway.xml. We currently have an issue wherein, given a name that doesn't match any found on the classpath, a random provider is returned. This needs to be fixed to fail deployment with appropriate error messages. Now, this usually comes down to one of 2 things: 1. there is no src/main/resources/META-INF/services/org.apache.hadoop.gateway.deploy.ProviderDeploymentContributor file in the provider or it contains the classname of some other contributor 2. the provider itself isn't on the gateway classpath. I see that your service file seems to be defined properly - so it is probably not on the classpath. Did you add the new provider to the parent pom? The article should provide all you need to do so. I will file a JIRA to fix the random provider issue. On Thu, Nov 12, 2015 at 7:18 AM, Jérôme LELEU <[email protected]> wrote: > Hi, > > By changing the role to the appropriate value, the error is gone, but I > have a new problem: > > *Caused by: javax.servlet.ServletException: Required authentication > provider URL is missing.* > * at > > org.apache.hadoop.gateway.provider.federation.jwt.filter.SSOCookieFederationFilter.init(SSOCookieFederationFilter.java:90)* > > My sandbox.xml file only references the pac4j provider as well as the > Default (role: identity-assertion) and static (role: hostmap) ones. > > Though, the SSOCookieFederationFilter is defined in the gateway.xml file of > my sandbox deployment. Why do I need it? What's the expected authentication > provider url? How this gatweay.xml is generated? > > Thanks. > Best regards, > Jérôme > > > > > > 2015-11-10 17:36 GMT+01:00 Kevin Minder <[email protected]>: > > > To be a bit more explicit your Pac4jFederationProviderContributor > declares > > itself to be of role “federation” but when you used it in the sandbox.xml > > topology file you were attempting to configure it as an provider with > role > > authentication. > > > > > > > > > > On 11/10/15, 11:32 AM, "larry mccay" <[email protected]> wrote: > > > > >Let's get you past the error that you are getting... > > > > > >It seems as though you have change the topology for pac4j but didn't > > change > > >the role to "federation". > > >It looks like it is grabbing a random authentication provider and seems > to > > >be the hadoop-auth provider. > > > > > >Change that to federation and see if that helps. > > > > > >On Tue, Nov 10, 2015 at 11:21 AM, larry mccay <[email protected]> > wrote: > > > > > >> Hi Jérôme - > > >> > > >> Happy to see you here! > > >> I apologize for having missed your note on the list a few days ago. > > >> > > >> Glad to see that the article was helpful and it seems like you are > > making > > >> great progress. > > >> > > >> Let me dig into your note a bit deeper and answer your questions. > > >> > > >> Welcome! > > >> > > >> --larry > > >> > > >> On Tue, Nov 10, 2015 at 11:14 AM, Jérôme LELEU <[email protected]> > > wrote: > > >> > > >>> Hi, > > >>> > > >>> Perfect timing ;-) I have started implemented the pac4j support in > > Knox, > > >>> it's a very basic attempt: > https://github.com/apache/knox/pull/2/files > > >>> > > >>> This implementation reuses the pac4j implementation for J2E. I have a > > >>> ProviderDeploymentContributorBase which registers two filters: > > >>> - the first one is a dispatcher filter which uses the j2e-pac4j > > >>> CallbackFilter on the /callabck url (it finishes the authentication > > process > > >>> after a successful authentication at some identity provider) and uses > > the > > >>> j2e-pac4j RequiresAuthenticationFilter otherwise (redirection to the > > >>> identity provider to start the authentication when the user is not > > >>> authenticated) > > >>> - the second one is an identity adapter which gets the current > > >>> authenticated user from the pac4j point of view and populates the J2E > > >>> security context. > > >>> > > >>> > > >>> * I have the following error: > > >>> > > >>> 2015-11-10 16:30:22,347 ERROR hadoop.gateway > > >>> (GatewayFilter.java:doFilter(135)) - Gateway processing failed: > > >>> javax.servlet.ServletException: javax.servlet.ServletException: > > >>> Authentication type must be specified: simple|kerberos|<class> > > >>> javax.servlet.ServletException: javax.servlet.ServletException: > > >>> Authentication type must be specified: simple|kerberos|<class> > > >>> at > > >>> > > > org.apache.hadoop.gateway.GatewayFilter$Holder.getInstance(GatewayFilter.java:354) > > >>> > > >>> Notice I have changed the sandbox.xml file to use pac4j instead of > > Shiro, > > >>> maybe it's not the right way: > > >>> > > > https://github.com/apache/knox/pull/2/files#diff-128ca80baa2dfd6c2d25de7b8160b9d9R24 > > >>> > > >>> Any idea of the problem? > > >>> > > >>> > > >>> * Am I understanding webflows correctly? > > >>> > > >>> I use the default request to: curl -ivk " > > >>> https://localhost:8443/gateway/sandbox/webhdfs/v1/tmp?op=LISTSTATUS"; > > >>> > > >>> It looks like a REST request, though I'm expecting the following > > webflow: > > >>> redirection of the user to the identity provider, successful login, > > >>> redirection back to the /callback url of the gateway: do I get it > > right? > > >>> > > >>> > > >>> * The callback url > > >>> > > >>> I currently do a check expecting the last part of the url to end with > > >>> /callback, but does the gateway will receive requests on this url? > Do I > > >>> have to do something special? > > >>> > > >>> It's highly recommended to only have one callback url as this url is > > >>> defined on the identity provider side. > > >>> > > >>> > > >>> * How to pass configuration? > > >>> > > >>> Currently, I hardcoded a FacebookClient for Facebook authentication, > > but > > >>> we should be able to pass the appropriate client like Facebook or > SAML. > > >>> Basically, we could do that using filter properties: facebook.key + > > >>> facebook.secret means we use Facebook authentication with the > > appropriate > > >>> key and secret for example. Any recommendation? > > >>> > > >>> > > >>> * Can I use the J2E session to store the requested protected url and > > >>> other information? > > >>> > > >>> Or do I need to do something special? > > >>> > > >>> > > >>> Thanks. > > >>> Best regards, > > >>> Jérôme > > >>> > > >>> 2015-11-10 17:03 GMT+01:00 Kevin Minder < > [email protected] > > >: > > >>> > > >>>> Hi Jerome, > > >>>> This sounds very exciting and is just the sort of thing the open > > >>>> architecture of Knox is intended to allow. Larry and I have looked > at > > >>>> pac4j a number of times thinking that it would be a good fit and > would > > >>>> provide a great feature set to the Hadoop community. > > >>>> Kevin. > > >>>> > > >>>> > > >>>> > > >>>> On 11/6/15, 3:31 AM, "Jérôme LELEU" <[email protected]> wrote: > > >>>> > > >>>> >Hi, > > >>>> > > > >>>> >Let's open this private discussion on the Knox dev mailing list. > > >>>> > > > >>>> >I'm Jerome Leleu and the creator of pac4j ( > > >>>> https://github.com/pac4j/pac4j), > > >>>> >a security engine for Java with many implementations for J2E, Play, > > >>>> Spring, > > >>>> >Vertx, Ratpack... The idea is to offer something as powerful as > > Spring > > >>>> >Security, but a lot easier and for all Java frameworks / tools and > > all > > >>>> >authentication mechanisms. > > >>>> > > > >>>> >Two years ago, we had a discussion with Larry on how pac4j could be > > >>>> used in > > >>>> >Knox. > > >>>> > > > >>>> >Meanwhile, both projects have grown up and I'm back to try to see > if > > >>>> pac4j > > >>>> >can help Knox in terms of security. > > >>>> > > > >>>> >----- > > >>>> > > > >>>> >In its latest version, pac4j can be used to build a full security > > >>>> library, > > >>>> >not only focused on delegated authentication (Facebook, SAML). So > we > > >>>> have > > >>>> >the concepts of direct clients (a client is an authentication > > >>>> mechanism): > > >>>> >basic auth, header, token... and indirect clients: SAML, Facebook, > > >>>> OpenID, > > >>>> >CAS... > > >>>> >It seems to be very close to the Authentication and Federation > > Providers > > >>>> >concepts from Knox. > > >>>> > > > >>>> >The provided article is great and as it's focused on authentication > > >>>> with a > > >>>> >real example, it's easier to start with it. > > >>>> > > > >>>> >Thanks for your answers. > > >>>> > > > >>>> >Like Shiro, pac4j has LDAP support and caching can be done. The > only > > >>>> >difference is that pac4j relies on ldaptive ( > > http://www.ldaptive.org/). > > >>>> But > > >>>> >I think you're right: it would be better to start by implementing > the > > >>>> >Federation part which might be the most expected feature for Knox > > users > > >>>> (a > > >>>> >bit like buji-pac4j: https://github.com/bujiio/buji-pac4j is for > > >>>> Shiro) and > > >>>> >see if it's worth the work to have an authentication provider with > > >>>> pac4j in > > >>>> >addition to the one of Shiro. > > >>>> > > > >>>> >I plan to start working on that next week. Do you accept pull > > requests > > >>>> on > > >>>> >Github (it would make discussion easier on source code)? > > >>>> > > > >>>> >Any feedback will be appreciated. > > >>>> > > > >>>> >Thanks. > > >>>> >Best regards, > > >>>> >Jérôme > > >>>> > > > >>>> > > > >>>> > > > >>>> > > > >>>> > > > >>>> > > > >>>> >2015-11-05 17:13 GMT+01:00 larry mccay: > > >>>> > > > >>>> >> Hi Jérôme - > > >>>> >> > > >>>> >> Contributions coming from the community are owned and maintained > by > > >>>> the > > >>>> >> entire community. > > >>>> >> You would certainly not be expected to do all work on it and at > the > > >>>> same > > >>>> >> time any assistance and upgrade messages that you provided would > be > > >>>> awesome. > > >>>> >> > > >>>> >> As for an empty module for pac4j, maybe the following example > > project > > >>>> >> would be helpful to you: > > >>>> >> > > >>>> >> https://github.com/lmccay/gateway-provider-security-pseudo > > >>>> >> > > >>>> >> It is in support of an article to illustrate how to go about > > adding a > > >>>> >> authentication or federation provider - as you would be doing: > > >>>> >> > > >>>> >> > > >>>> > > > https://github.com/lmccay/gateway-provider-security-pseudo/blob/master/AddingFederationProvider.md > > >>>> >> > > >>>> >> That may be easier to consume than the entire developers guide. > > >>>> >> That said, it would be great if you could describe what makes the > > dev > > >>>> >> guide too much - if that is the case. > > >>>> >> > > >>>> >> I'll take a stab at your questions here but we should move more > > >>>> detailed > > >>>> >> discussions on the topic to the dev@ list in order to keep this > in > > >>>> the > > >>>> >> open and provide insights to others: > > >>>> >> > > >>>> >> 1) What do I do with the authenticated user: where / how do I > fill > > >>>> this > > >>>> >> identity in Knox? > > >>>> >> > > >>>> >> We normalize the authentication event into a Java subject to > > >>>> represent the > > >>>> >> user, groups and impersonated user as appropriate. > > >>>> >> In cases - like shiro - where we don't have access to the actual > > >>>> >> authenticating code and we need to normalize the provider > specific > > >>>> security > > >>>> >> context, we add another filter. You can look at the shiro > provider > > >>>> for an > > >>>> >> example. The code inside the simple example provider in the > article > > >>>> also > > >>>> >> shows what is expected at least in terms of the PrimaryPrincipal. > > >>>> >> > > >>>> >> 2) How to handle errors: not authenticated, not authorized? > > >>>> >> > > >>>> >> We aren't intrusive on what should be done by a provider here. > > >>>> >> Generally, authentication failures result in a 401 and > > authorizations > > >>>> in a > > >>>> >> 403 and they are usually sent by the provider. > > >>>> >> > > >>>> >> 3) How to handle redirections to an external provider? > > >>>> >> > > >>>> >> Redirects are sent by the providers themselves with a 307 or 302. > > >>>> >> You can look at the SSOCookieFederationFilter in the jwt module > as > > an > > >>>> >> example if you like. > > >>>> >> > > >>>> >> 4) Should the pac4j filter also handle authorizations (pac4j can > do > > >>>> that)? > > >>>> >> > > >>>> >> This is an interesting question. > > >>>> >> Knox provides a separate slot in the provider chain for > > authorization. > > >>>> >> It is there because of the ability for identity assertion > providers > > >>>> to map > > >>>> >> principals in order to impersonate another user or map external > > >>>> usernames > > >>>> >> to those used inside of hadoop clusters. So, the authorization > > >>>> decisions > > >>>> >> are made based on those identities. > > >>>> >> > > >>>> >> If we can add the authorization filter as an authorization > provider > > >>>> rather > > >>>> >> than part of the authentication provider than that might make > > sense. > > >>>> >> Although, the security context is normalized as Knox expects not > as > > >>>> pac4j > > >>>> >> authorization probably does. Now, if there is real value in doing > > so, > > >>>> the > > >>>> >> pac4j Subject adaptor could stuff additional context into the > > Subject > > >>>> or > > >>>> >> the request that can be accessed later by the pac4j authorization > > >>>> provider. > > >>>> >> > > >>>> >> So, it can be done - the question is whether there is compelling > > >>>> reason to > > >>>> >> do so. > > >>>> >> > > >>>> >> 5) How to test? > > >>>> >> > > >>>> >> This can be challenging especially for external authentications > and > > >>>> web > > >>>> >> app flows. > > >>>> >> Generally, unit testing as much as possible is required. > > >>>> >> We would want to make sure that the security context > normalization > > >>>> happens > > >>>> >> as expected, etc. > > >>>> >> > > >>>> >> Additional considerations: > > >>>> >> > > >>>> >> 1. Our Shiro provider does group lookup and caching of the > > >>>> authentication > > >>>> >> event for optimizing interactions with authentication servers, > > LDAP, > > >>>> AD, > > >>>> >> etc. We wouldn't want to lose these. Doing so would be a > > non-starter > > >>>> for > > >>>> >> most folks replacing what Shiro is used for today. > > >>>> >> 2. It is probably best to concentrate on the gaps that pac4j can > > fill > > >>>> for > > >>>> >> openid, oauth, etc for the initial contribution and if the > usecases > > >>>> handled > > >>>> >> by Shiro can be done better, easier, simpler then we can > consider a > > >>>> >> migration path. > > >>>> >> > > >>>> >> BTW - we have a couple very interesting possibilities for strong > > and > > >>>> >> multi-factor authentication that would be enabled by OpenID > > >>>> immediately. > > >>>> >> > > >>>> >> Hope this isn't too long a response and that it is helpful for > you. > > >>>> >> > > >>>> >> Let's determine what an initial contribution would be for and > > bring a > > >>>> >> proposal to the dev@ list and/or file a jira for the > integration. > > >>>> >> This would be great for adding features and growing the > community - > > >>>> so we > > >>>> >> are all for it! > > >>>> >> > > >>>> >> thanks, > > >>>> >> > > >>>> >> --larry > > >>>> >> > > >>>> >> > > >>>> >> On Thu, Nov 5, 2015 at 9:43 AM, Jérôme LELEU <[email protected]> > > >>>> wrote: > > >>>> >> > > >>>> >>> Hi, > > >>>> >>> > > >>>> >>> I see that Shiro is already used for basic auth and LDAP > > >>>> authentication > > >>>> >>> and Picketlink for SAML. pac4j v1.8 can now handle both cases > and > > >>>> even > > >>>> >>> more. So I think we could create a > gateway-provider-security-pac4j > > >>>> >>> supporting all authentication mechanisms and not only OpenID. > > >>>> >>> > > >>>> >>> pac4j implementations generally work with two filters: one to > > >>>> protect a > > >>>> >>> resource and play direct authentication like basic auth (and > check > > >>>> >>> authorizations) and a callback filter to finish remote > > >>>> authentication like > > >>>> >>> Facebook, SAML, OpenID. The easiest one is the j2e-pac4j with: > > >>>> >>> > > >>>> > > > https://github.com/pac4j/j2e-pac4j/blob/master/src/main/java/org/pac4j/j2e/filter/RequiresAuthenticationFilter.java#L96 > > >>>> >>> and > > >>>> >>> > > >>>> > > > https://github.com/pac4j/j2e-pac4j/blob/master/src/main/java/org/pac4j/j2e/filter/CallbackFilter.java#L65 > > >>>> >>> > > >>>> >>> That said, I have already many pac4j implementations to handle > so > > I'm > > >>>> >>> pushing communities to maintain on their own their pac4j > > >>>> implementations: > > >>>> >>> can we imagine having an official: > gateway-provider-security-pac4j > > >>>> module > > >>>> >>> maintained by the Knox community with my help of course? This is > > the > > >>>> kind > > >>>> >>> of message I deliver to communities when a new version of pac4j > is > > >>>> >>> available so that they can upgrade: > > >>>> >>> https://github.com/ratpack/ratpack/issues/819 > > >>>> >>> > > >>>> >>> I read: > > >>>> https://knox.apache.org/books/knox-0-6-0/dev-guide.html#Providers, > > >>>> >>> it might be easy for Knox contributors, but it's a bit hard for > me > > >>>> to get > > >>>> >>> in everything: can you or someone in the Knox community provides > > me > > >>>> an > > >>>> >>> empty gateway-provider-security-pac4j module and the default > > >>>> expectations > > >>>> >>> from a Knox point of view? > > >>>> >>> My main questions: > > >>>> >>> 1) What do I do with the authenticated user: where / how do I > fill > > >>>> this > > >>>> >>> identity in Knox? > > >>>> >>> 2) How to handle errors: not authenticated, not authorized? > > >>>> >>> 3) How to handle redirections to an external provider? > > >>>> >>> 4) Should the pac4j filter also handle authorizations (pac4j can > > do > > >>>> that)? > > >>>> >>> 5) How to test? > > >>>> >>> > > >>>> >>> Thanks. > > >>>> >>> Best regards, > > >>>> >>> Jérôme > > >>>> >>> > > >>>> >>> > > >>>> >>> > > >>>> >>> > > >>>> >>> 2015-11-03 16:58 GMT+01:00 larry mccay: > > >>>> >>> > > >>>> >>>> Hi Jérôme - > > >>>> >>>> > > >>>> >>>> We were evaluating buji-pac4j for SSO with SAML and ended up > > going > > >>>> with > > >>>> >>>> Picketlink at the time. > > >>>> >>>> That said, we do have a pluggable architecture that would allow > > for > > >>>> a > > >>>> >>>> pac4j provider as well. > > >>>> >>>> > > >>>> >>>> If you are interested in contributing such a provider that > would > > >>>> >>>> certainly be welcome. > > >>>> >>>> There is someone in the community working on an openid provider > > but > > >>>> >>>> perhaps pac4j would be the way to go there? > > >>>> >>>> > > >>>> >>>> thanks, > > >>>> >>>> > > >>>> >>>> --larry > > >>>> >>>> > > >>>> >>>> > > >>>> > > >>> > > >>> > > >> > > >
