Hi, I'm looking forward to read the new documentation you'll write on the REST API support.
About the 404, I'm not sure to understand why you want me to remove "idp/" from the url as the KnoxSSO service is defined in the idp.xml topology: does its url not have a /idp? About the OpenID Connect support, I added it, it's now in pac4j: https://github.com/pac4j/pac4j/blob/master/pac4j-config/src/main/java/org/pac4j/config/client/ConfigPropertiesFactory.java#L60 Thanks. Best regards, Jérôme 2015-11-18 14:35 GMT+01:00 larry mccay <larry.mc...@gmail.com>: > Hi Jérôme - > > Great progress! > You have the flow down perfectly. > > You should be also aware that the SSOCookieProvider is a mechanism for > consuming the SSO cookie for the REST APIs. > There are also integrations for the various Hadoop UIs - most components > within the core Hadoop common project have a UI - ie. NameNode, DataNode, > YARN ResourceManager, etc. Additionally, ecosystem components such as > Oozie, HBase, Hive, Falcon, Atlas, Ambari, Ranger, etc all have UIs and are > adding or already have support for the SSO cookie being created in KnoxSSO. > Many of these are able to leverage the support added to Hadoop common and > get it largely for free. > > REST API support is interesting and immediately relevant to Knox but may > actually be secondary to the WebSSO flow for the UIs. > When deployments do want to provide REST API support SSOCookieProvider > enables CORS for allowing javascript API calls from browsers in other > domains to access Hadoop resources. It is also great for development > scenarios like this where we need some consumer to test integrations. > > I am in the process of writing up more docs on these very aspects of > KnoxSSO and JWT cookie support across Hadoop. > > The 404 is related to the configured provider url on the SSOCookieProvider > - remove the "idp/" in the URL. > > That used to be the correct URL but we have since simplified it for the > time being to just be knoxsso. > There is a thread [1] on the dev list that describes that decision and what > we should consider moving forward. > > We will have generic OpenId Connect support as well? This is something that > would be of immediate value. > > [1] > > http://mail-archives.apache.org/mod_mbox/knox-dev/201511.mbox/%3CCACRbFygkWLJ-tFW612ha_Cr82u%2B41PwDqGx_TWbijbP0g34g%3DQ%40mail.gmail.com%3E > > Thanks! > > --larry > > On Wed, Nov 18, 2015 at 6:11 AM, Jérôme LELEU <lel...@gmail.com> wrote: > > > Hi, > > > > I've read the documentation and things seem fairly clear to me. Let me > > share my understanding and progress: > > > > - the KnoxSSO service is mandatory and the > gateway-provider-security-pac4j > > won't work without it: the KnoxSSO service is responsible for creating a > > hadoop-jwt cookie from the current authenticated user (J2E subject). > > - the SSOCookieProvider is also mandatory to protect services and request > > the hadoop-jwt cookie to exist by forwarding to the KnoxSSO endpoint. > > > > I have created two gateways: one for regular Hadoop services which is > > protected by the SSOCookieProvider -> > > > > > https://github.com/apache/knox/pull/2/files#diff-128ca80baa2dfd6c2d25de7b8160b9d9R23 > > the other one for the KnoxSSO service protected by pac4j -> > > > > > https://github.com/apache/knox/pull/2/files#diff-4ea9a9a5ee5968f29982478512a63c54R23 > > > > Here is my webflow : > > > > $ curl -ivk " > > https://127.0.0.1:8443/gateway/sandbox/webhdfs/v1/tmp?op=LISTSTATUS"; > > 302 Found > > Location: > > > > > https://127.0.0.1:8443/gateway/idp/knoxsso/api/v1?originalUrl=https://127.0.0.1:8443/gateway/sandbox/webhdfs/v1/tmp?op=LISTSTATUS > > > > > > Calling a regular Hadoop service through the gateway, it is redirected to > > the KnoxSSO endpoint (as no hadoop-jwt cookie exists) > > > > $ curl -ivk " > > > > > https://127.0.0.1:8443/gateway/idp/knoxsso/api/v1?originalUrl=https://127.0.0.1:8443/gateway/sandbox/webhdfs/v1/tmp?op=LISTSTATUS > > " > > 302 Found > > Set-Cookie: pac4j.session.pac4jRequestedUrl=" > > > > > https://127.0.0.1:8443/gateway/idp/knoxsso/api/v1?originalUrl=https://127.0.0.1:8443/gateway/sandbox/webhdfs/v1/tmp?op=LISTSTATUS > > " > > Location: > > > > > https://casserverpac4j.herokuapp.com?service=https%3A%2F%2F127.0.0.1%3A8443%2Fgateway%2Fidp%2Fknoxsso%2Fapi%2Fv1%3Fclient_name%3DCasClient > > > > > > Calling the KnosSSO endpoint, the originally requested url is saved into > a > > cookie (the naive implementation I currently use) and it is redirected > to a > > CAS server login for authentication (my internal configuration). > > > > After login at the CAS server, the user is redirected back to the > callback > > url, which is the KnoxSSO endpoint. > > > > $ curl -ivk --cookie "pac4j.session.pac4jRequestedUrl= > > > > > https://127.0.0.1:8443/gateway/idp/knoxsso/api/v1?originalUrl=https://127.0.0.1:8443/gateway/sandbox/webhbhdfs/v1/tmp?op=LISTSTATUS > > " > > " > > > > > https://127.0.0.1:8443/gateway/idp/knoxsso/api/v1?client_name=CasClient&ticket=ST-2-LQ2QLLEdAhBmrCPE1MZA-cas01.example.org > > " > > 302 Found > > Set-Cookie: pac4j.session.pac4jUserProfile=CasProfile#jleleu > > Location: > > > > > https://127.0.0.1:8443/gateway/idp/knoxsso/api/v1?originalUrl=https://127.0.0.1:8443/gateway/sandbox/webhdfs/v1/tmp?op=LISTSTATUS > > > > > > The fact it's a callback (to finish the login process) is now based on > the* > > client_name* parameter (instead of a /callback url). The authentication > is > > finished by validating the CAS service ticket and a user profile is > created > > into session and saved in a cookie. It is redirected to the originally > > requested url. > > > > $ curl -ivk --cookie "pac4j.session.pac4jUserProfile=CasProfile#jleleu" " > > > > > https://127.0.0.1:8443/gateway/idp/knoxsso/api/v1?originalUrl=https://127.0.0.1:8443/gateway/sandbox/webhdfs/v1/tmp?op=LISTSTATUS > > " > > 404 > > > > > > The call to the KnoxSSO endpoint triggers the pac4j filter, which gets > the > > user profile from the cookie, calls the identity adapter and sets the J2E > > subject with the pac4j user profile. Here I get a 404 I don't understand > > and on which I'm currently investigating. > > > > So the callback url issue is solved: the KnoxSSO endpoint is the static > url > > used for callback, for any identity provider and the *client_name* > > parameter is the way to know if it is a callback or not. > > > > To save the data in session, I used non-secure cookies for now, it's my > > naive implementation. It currently doesn't work properly because as we > rely > > on j2e-pac4j, response.sendRedirect occurs and after that, nothing can be > > done on the response. So I need to change pac4j v1.8.1-SNAPSHOT and > > j2e-pac4j v1.2.1-SNAPSHOT to be able to plugin a specific session storage > > mechanism (the need has arisen for Play and Shiro as well) -> > > https://github.com/pac4j/pac4j/issues/359 > > > > As for configuration, we said that the client configuration will be > defined > > based on properties. As the same mechanism is available for the CAS > server, > > this capability has been moved to pac4j -> > > https://github.com/pac4j/pac4j/issues/357 > > It means that currently, Facebook, Twitter, a SAML IdP and a CAS server > can > > be created via properties: if you need other identity providers, please > > speak up so that I can integrate them. > > > > So I have some more work to do in pac4j and j2e-pac4j before being able > to > > go further with Knox. I'll keep you posted. > > > > Thanks. > > Best regards, > > Jérôme > > > > > > > > > > 2015-11-14 19:03 GMT+01:00 larry mccay <lmc...@apache.org>: > > > > > Here is a *very* early version of an integration guide for KnoxSSO: > > > > > > http://knox.apache.org/books/knox-0-7-0/knoxsso_integration.html > > > > > > It will walk you through setting up an environment to for using KnoxSSO > > to > > > secure the REST APIs using KnoxSSO cookies and a webflow. > > > The initial environment uses simple browser HTTP basic authentication > to > > > protect the WebSSO service which hands out the cookies. > > > This allows us to ensure that KnoxSSO is setup properly without > > introducing > > > a 3rd party provider into the mix first. > > > > > > Once you have it working with Basic Auth then you can switch to your > > pac4j > > > provider instead of Shiro to protect the websso service and we can > start > > to > > > work through the integration of your provider. > > > > > > Please understand that this is still a document that is very much a > work > > in > > > progress and has content, formatting and other issues. > > > It should serve as a general guide for you in the short term though. > > > > > > Hope it is useful for you! > > > > > > > > > > > > On Fri, Nov 13, 2015 at 9:51 AM, Kevin Minder < > > > kevin.min...@hortonworks.com> > > > wrote: > > > > > > > I’ll add a bit to the logging answer. I created that for several > > > reasons. > > > > 1) Localization as Larry mentions. > > > > 2) It abstract from actual logging provider and you can see for > example > > > > gateway-i18n-logging-sl4j as an alternate integration. > > > > 3) Another important aspect is centralization. It centralizes all > > > aspects > > > > of a given message. For example the level for a given message is > kept > > > with > > > > the message to hopefully prevent the message from being duplicated > with > > > > potentially different levels. It is also intended to support the > > notion > > > of > > > > Ids and potentially cause/action annotations which are very common as > > > > products mature. > > > > 4) Lastly and possibly the most interesting/important from a > developer > > > > perspective is traceability. Once you find the log method for a > given > > > > message you can use the IDE to find all of the palaces where a given > > > > message is used. You can also have the IDE tell you if the message > > isn’t > > > > used anymore. > > > > > > > > All that being said the external components we integrate with > certainly > > > > don’t use it. So from that perspective, direct use of log4j if that > is > > > > your choice isn’t going to cause any major problems. > > > > > > > > > > > > > > > > > > > > > > > > On 11/13/15, 8:40 AM, "larry mccay" <lmc...@apache.org> wrote: > > > > > > > > >Hi Jérôme - > > > > > > > > > >Great questions and I'm so glad that you are here to ask them. > > > > > > > > > >I think that the set of documentation that I am working on for a > > KnoxSSO > > > > >integration guide will probably answer most of your questions. > > > > >I need to spend a bit more time thinking about the some of it so > that > > I > > > > can > > > > >incorporate that information as well. > > > > > > > > > >The cookie approach is what we use in KnoxSSO. Cookies obviously > have > > > some > > > > >disadvantages but is probably the best alternative for KnoxSSO. > > > > >Something that we need to consider is that KnoxSSO is already > keeping > > > > track > > > > >of the original URL for redirecting back to the requested resource. > > > > >We have to reconcile whether that is separate from the callback > needs > > or > > > > >actually the same thing. > > > > >Having a little trouble wrapping my head around it right now - must > > have > > > > >more coffee. :) > > > > > > > > > >The Messages layer allows for the localization of the actual > messages > > by > > > > >decoupling the messages from the actual runtime code. > > > > > > > > > >I will try and get some docs published as quickly as possible for > > > KnoxSSO > > > > >integration. > > > > > > > > > >thanks, > > > > > > > > > >--larry > > > > > > > > > >On Fri, Nov 13, 2015 at 3:27 AM, Jérôme LELEU <lel...@gmail.com> > > wrote: > > > > > > > > > >> Hi, > > > > >> > > > > >> 1) Webflow: I think I get the idea with the KnoxSSO service: how > > can I > > > > test > > > > >> everything to ensure pac4j works correctly with it and will be > > usable > > > in > > > > >> Hadoop UIs? > > > > >> > > > > >> > > > > >> 2) Callback url: > > > > >> > > > > >> For indirect clients, pac4j is designed to be called on any url, > to > > > save > > > > >> it, to call the identity provider providing a static callback url, > > to > > > be > > > > >> called back on this static callback url and to restore the > > originally > > > > >> requested url. > > > > >> > > > > >> From point 1), I conclude that pac4j will always be used with the > > > > KnoxSSO > > > > >> service (for indirect clients): will KnoxSSO call the pac4j > provider > > > > always > > > > >> on the same url? Can you describe your first flow with urls? > > > > >> > > > > >> Because in that case, I could find a solution where I check for > the > > > > >> client_name parameter to define if it's a callback or a first > call. > > > > >> > > > > >> Otherwise, I will need a specific pac4j service. > > > > >> > > > > >> > > > > >> 3) Configuration: no problem here, I will use the AliasService. > > > > >> > > > > >> > > > > >> 4) Web session: I really need to put some information in session > and > > > > check > > > > >> them on callback: I think I can take the data put in session, save > > > them > > > > in > > > > >> a cookie and restore them before the callback. > > > > >> > > > > >> Is it the recommended solution? Can I re-use the components > > available > > > > for > > > > >> JWT tokens and cookies? And how? > > > > >> > > > > >> > > > > >> 5) Logs: I see in every descriptor the use of Messages and > > > > MessagesFactory: > > > > >> I can't use log4j directly, can I? What's the expected benefits > > using > > > > this > > > > >> Messages layer? > > > > >> > > > > >> > > > > >> Thanks. > > > > >> Best regards, > > > > >> Jérôme > > > > >> > > > > >> > > > > >> > > > > >> > > > > >> > > > > >> 2015-11-12 18:14 GMT+01:00 larry mccay <lmc...@apache.org>: > > > > >> > > > > >> > Terrific! > > > > >> > > > > > >> > Okay... original questions: > > > > >> > > > > > >> > 1) Webflow: is it normal to get a redirection to Facebook or > > another > > > > IdP > > > > >> > when I made the request (for such a configuration of course)? Is > > > this > > > > >> > request meant to happen in a browser? (because it's the use case > > it > > > > was > > > > >> > made for) > > > > >> > > > > > >> > It is normal. > > > > >> > The usecases in which this mechanism would be used would involve > > the > > > > >> > KnoxSSO service/feature. > > > > >> > What we are doing with that is providing a normalization of > > > > >> authentication > > > > >> > mechanisms by federating each auth event through a JWT token > that > > > gets > > > > >> set > > > > >> > as a cookie. > > > > >> > These usecases allow us to provide WebSSO flows for Hadoop UIs > and > > > > other > > > > >> > applications and those applications to only ever have to know > > about > > > > >> KnoxSSO > > > > >> > tokens. > > > > >> > > > > > >> > What is slightly less normal is that the flow needs to have > > KnoxSSO > > > in > > > > >> the > > > > >> > middle. > > > > >> > For instance: > > > > >> > 1. a browser requests a resource from a KnoxSSO participating > > > service > > > > >> > provider > > > > >> > 2. the service provider redirects the browser to KnoxSSO since > > there > > > > is > > > > >> no > > > > >> > cookie > > > > >> > 3. KnoxSSO is protected with pac4j provider and redirects the > > > browser > > > > to > > > > >> FB > > > > >> > 4. user authenticates to FB > > > > >> > 5. pac4j accepts the FB authentication and normalizes the > security > > > > >> context > > > > >> > to a J2E Subject > > > > >> > 6. WebSSO service then creates a signed JWT token, sets the > cookie > > > and > > > > >> > redirects to the originally requested resource > > > > >> > 7. originally requested resource finds the hadoop-jwt cookie, > > > > validates > > > > >> the > > > > >> > token and verifies the signature and grants access to the > resource > > > > >> > > > > > >> > 2) Callback url: > https://localhost:8443/gateway/sandbox/callback > > > > >> currently > > > > >> > returns a 404, so I must declare this url somewhere to be taken > > into > > > > >> > account and pass to the pac4j filter. It would be even better if > > it > > > > could > > > > >> > be done by the pac4j deployment contributor. > > > > >> > > > > > >> > How and where to do that? > > > > >> > > > > > >> > This is an interesting question and one that may have multiple > > > > solutions: > > > > >> > > > > > >> > * we may be able to add the callback filter as part of the > > original > > > > >> > provider and just point them to the same URL. This would require > > the > > > > >> > ability to chain them where one knows to not handle a request > that > > > is > > > > >> > really meant for the other. > > > > >> > * If we truly need a separate URL the way you depict it there > then > > > we > > > > >> will > > > > >> > need to add a new service which would be unfortunate because it > > > > couples > > > > >> the > > > > >> > use of a certain provider to the use of a particular service. > > > > >> > > > > > >> > 3) Configuration > > > > >> > > > > > >> > Currently, I hardcoded a FacebookClient for Facebook > > authentication, > > > > but > > > > >> we > > > > >> > should be able to pass the appropriate client like Facebook or > > SAML. > > > > >> > Basically, we could do that using filter properties: > facebook.key > > + > > > > >> > facebook.secret means we use Facebook authentication with the > > > > appropriate > > > > >> > key and secret for example. Any recommendation? > > > > >> > > > > > >> > This is fine. You will add your properties to the provider > > > properties > > > > in > > > > >> > the topology for the pac4j provider. > > > > >> > You can find example code in the article to blindly add all > > > > properties to > > > > >> > the filter init params. > > > > >> > > > > > >> > Incidentally, the storage of credentials directly in config > files > > > > should > > > > >> be > > > > >> > avoided. > > > > >> > See the use of the AliasService where such credentials can be > > > > persisted > > > > >> in > > > > >> > a credential store and resolved at runtime. > > > > >> > I will find examples of this for you - we can go forward with > > config > > > > >> based > > > > >> > values but will need to fix that later. > > > > >> > > > > > >> > 4) Web session: I seem to be able to use the web session to sore > > > data. > > > > >> > Can you confirm that point? > > > > >> > > > > > >> > This is probably not going to work. > > > > >> > Keep in mind that there may be a cluster of Knox instances > running > > > and > > > > >> the > > > > >> > session is not replicated across all instances. > > > > >> > Additionally, keep in mind that your provider will only be > engaged > > > on > > > > the > > > > >> > first request to KnoxSSO and then not until the token/cookie > > > expires. > > > > >> > > > > > >> > > > > > >> > > > > > >> > > > > > > > > > >
