It seems to me that we should have very simple config for the endpoints and other deployment specific config added to a shiro.ini that has all the boilerplate stuff in it. Then you can make the rewrite rules stay in there as well.
Which, btw, is the idea behind topology.xml - it is a simple deployment specific descriptor that gets ingested and *compiled* into effective deployment configuration. Today that is gateway.xml and what you are proposing is shiro.ini. On Fri, Oct 25, 2013 at 12:25 PM, Dilli Arumugam <[email protected]>wrote: > Larry, > > I would go with specifying the end points of hadoop services in > knox.ini(=shiro.ini) and leave the rewrite rules in its own file. > > Rewrite rules are not going to be cluster specific and admins typically > would not have to tinker with them. > > Thanks > Dilli > > > On Fri, Oct 25, 2013 at 9:03 AM, Dilli Arumugam > <[email protected]>wrote: > > > Thanks Larry for the review and comments. > > > > The proposed knox.ini is meant to be edited by Knox system administrator > > until we get to have a admin console. This is not meant to be edited by > end > > user. > > > > To integrate additional back end hadoop service, following pieces of > > information have to come from pointers from knox.ini to external > > configuration files. > > > > 1. end point of the new hadoop servie back end > > 2. rewriting rules for the new service > > > > These can be specified in one file or split between 2 files with one of > > them being topology file or its equivalent. > > I have not shown all those details in the current shiro.ini. > > But, you would get the idea. > > > > Dilli > > > > > > On Fri, Oct 25, 2013 at 8:50 AM, larry mccay <[email protected]> wrote: > > > >> Hi Dilli - > >> > >> Good work here! > >> > >> Question: > >> 1. Is this complicated ini hand edited by the enduser in order to > deploy a > >> given topology or have you changed the shiro contributor to write this > out > >> from a much more understandable config within the topology? > >> > >> 2. Where are the services configured for a given topology? > >> 3. Did you rename all the existing filters to start with knox? > >> > >> A couple observations: > >> > >> 1. removing the use of gateway.xml means that we *require* Shiro for our > >> servlet filter chaining > >> 2. While comparing this to the current format of gateway.xml I can agree > >> that this is easier to understand - however, I don't actually find ini > >> format simpler to comprehend and administer than XML or JSON. In fact, > >> comparing this to topology.xml which is all you really need to do for > >> admin, I think the topology file is very easy to understand and > >> administer. > >> 3. moving the rewrite rules out to not require coding is good but should > >> probably not require that they be in this format and file > >> 4. Someone that wants to leverage their own or another third party > >> solution > >> is stuck squeezing it into Shiro > >> 5. This would violate a principle of mine of "Is-a" vs "Has-a" - Knox is > >> not a Shiro provider it can however *have* a Shiro provider > >> 6. End users should never have to see rewrite rules and maybe a lot of > >> what > >> is in that ini. Principal mapping, authz policy, etc will eventually > come > >> from Zookeeper which should remove their need to be in this file for > those > >> things. Maybe we can even get the LDAP config separately from there as > >> well > >> - though I still think it is best provided in a topology file with a > much > >> smaller config requirements and get pushed into shiro.ini. Then the user > >> may never need to see - or at least open - this file. > >> > >> I continue to believe that we are best served by enabling a deployment > >> that > >> looks like this and encouraging the use of it as the preferred solution. > >> However, we should continue to support the use of arbitrary providers > >> through the use of gateway.xml. In most cases, gateway.xml can be > >> leveraged > >> to only include the Shiro provider. In addition, all knowledge of Shiro > >> specifics should remain inside of that provider. If it leaks out then we > >> are violating the separation of Is-a vs Has-a again. > >> > >> This also allows us the flexibility to add something before or after > that > >> provider for things that we find are not easily or appropriately done > >> inside the security provider. > >> > >> "I think we can rename shiro.ini as knox.ini to make it explicit this is > >> more about knox configuration than shiro library configuration." > >> > >> While I can understand the thinking of renaming the shiro.ini file to > >> knox.ini, also violates the Is-a vs Has-a principle for me. We are not a > >> shiro security provider extension. We are a REST API Gateway for Hadoop > >> that has security providers - one of which is Shiro. > >> > >> "We are using shiro config file mechanism as simple, lightweight > depenency > >> injection." > >> > >> Where is there dependency injection here? If you are referring to the > >> configuration of the filter chain that is not really injection though it > >> is > >> an admittedly simpler mechanism than the current state of the > contributors > >> and our deployment machinery - which could be refactored and improved. > >> > >> "This does not really tie us to using only Shiro authentication > mechanism > >> or > >> authorization mechanisms. > >> > >> We have the choice of writing all our authentication or authorization in > >> our own servlet filters or leverage filters from Shiro library, Realms > >> from > >> Shiro library or write your own Shiro filters or Shiro realms." > >> > >> The only change that this proposal makes to our flexibility is the > >> inability to take Shiro out of the picture. > >> We could have leveraged Shiro to do all these things with the current > >> design as well. > >> BUT, we have control over our own interception channel in our chains > >> today. > >> This proposal takes a specific *security provider's* proprietary filter > >> chain and uses it as the only interceptor channel for the entire server. > >> Just because they also had a need for one - for their security work - > >> doesn't mean that we shouldn't have our own. The tail should not wag the > >> dog. Shiro is not an application or server framework it is a security > >> provider. > >> > >> "In most of the cases, when we want to integrate with a new Hadoop back > >> end > >> service, all that we have to do is specify the path to a file having > >> rewrite rules for the service. Rest of the things in > knox.ini(=shiro.ini) > >> would remain same." > >> > >> This isn't really very clear to me. Do we still have services in a > >> topology.xml file? > >> I don't see where you specify the url for the backend services - so I > >> assume there is still a topology file. > >> How do we know where to dispatch the requests to? > >> > >> I do agree that this is a very good step forward in defining a preferred > >> security provider model for Knox and that we should continue down this > >> road. We just need to do so carefully. If, over time, we have found no > >> reason to use the existing solution that we can consider removing it > >> entirely. > >> > >> Thanks for all this work and insight, Dilli! > >> > >> --larry > >> > >> > >> On Fri, Oct 25, 2013 at 10:27 AM, Dilli Arumugam > >> <[email protected]>wrote: > >> > >> > Have been trying out a few ideas with Knox and managed to get Knox > >> running > >> > with > >> > > >> > 1. WEB-INF/gateway.xml is completely removed > >> > 2. WEB-INF/web.xml declares only shiro filter and a defautl servlet > >> > 2. all filters are defined and injected using WEB-INF/shiro.ini > >> > > >> > To me that looks much simpler to comprehend and administer. > >> > Agreed, this could be subjective. > >> > Hence, seeking comments from community. > >> > > >> > Pasting web.xml and shiro.ini inline. > >> > > >> > web.xml > >> > -------------- > >> > > >> > <?xml version="1.0" encoding="UTF-8" standalone="yes"?> > >> > <web-app xmlns="http://java.sun.com/xml/ns/javaee"; xmlns:xsi=" > >> > http://www.w3.org/2001/XMLSchema-instance"; xsi:schemaLocation=" > >> > http://java.sun.com/xml/ns/javaee > >> > http://java.sun.com/xml/ns/javaee/web-app_3_0.xsd";> > >> > > >> > <listener> > >> > > >> > > >> > > >> > <listener-class>org.apache.shiro.web.env.EnvironmentLoaderListener</listener-class> > >> > </listener> > >> > > >> > <listener> > >> > > >> > > >> > > >> > <listener-class>org.apache.hadoop.gateway.services.GatewayServicesContextListener</listener-class> > >> > </listener> > >> > > >> > <listener> > >> > > >> > > >> > > >> > <listener-class>org.apache.hadoop.gateway.filter.rewrite.api.UrlRewriteServletContextListener</listener-class> > >> > </listener> > >> > > >> > <context-param> > >> > <param-name>rewriteDescriptorLocation</param-name> > >> > <param-value>/WEB-INF/rewrite.xml</param-value> > >> > </context-param> > >> > > >> > <filter> > >> > <filter-name>ShiroFilter</filter-name> > >> > > >> <filter-class>org.apache.shiro.web.servlet.ShiroFilter</filter-class> > >> > </filter> > >> > > >> > <filter-mapping> > >> > <filter-name>ShiroFilter</filter-name> > >> > <url-pattern>/*</url-pattern> > >> > </filter-mapping> > >> > > >> > <session-config> > >> > <session-timeout>30</session-timeout> > >> > </session-config> > >> > > >> > <servlet> > >> > <servlet-name>errorservlet</servlet-name> > >> > > >> > > >> > > >> > <servlet-class>org.apache.hadoop.gateway.filter.KnoxErrorServlet</servlet-class> > >> > </servlet> > >> > > >> > <servlet-mapping> > >> > <servlet-name>errorservlet</servlet-name> > >> > <url-pattern>/*</url-pattern> > >> > </servlet-mapping> > >> > > >> > </web-app> > >> > > >> > > >> > shiro.ini > >> > ------------- > >> > > >> > [main] > >> > > >> > # define ldapRealm > >> > ldapRealm=org.apache.shiro.realm.ldap.JndiLdapRealm > >> > ldapRealm.contextFactory.authenticationMechanism=simple > >> > ldapRealm.contextFactory.url=ldap://localhost:33389 > >> > ldapRealm.userDnTemplate=uid={0},ou=people,dc=hadoop,dc=apache,dc=org > >> > > >> > > >> > # define filter: knoxResponseCookieFilter > >> > knoxResponseCookieFilter = > >> > org.apache.hadoop.gateway.filter.ResponseCookieFilter > >> > knoxResponseCookieFilter.enabled = true > >> > knoxResponseCookieFilter.filterHeaders = rememberMe, > hadoop.auth.cookie > >> > > >> > # define filter: knoxPrincipalMapper > >> > knoxPrincipalMapper = > >> org.apache.hadoop.gateway.filter.KnoxPrincipalMapper > >> > knoxPrincipalMapper.enabled = true > >> > knoxPrincipalMapper.userToUserMap = bob:guest, jon:bob > >> > knoxPrincipalMapper.userToGroupMap = *:users, bob:admin > >> > > >> > # define filter: knoxIPTracker > >> > knoxIPTracker =org.apache.hadoop.gateway.filter.KnoxIPTracker > >> > knoxIPTracker.enabled = true > >> > > >> > # define filter: knoxAclAuthzFilter > >> > knoxAclAuthzFilter = > org.apache.hadoop.gateway.filter.KnoxAclAuthzFilter > >> > knoxAclAuthzFilter.enabled = true > >> > knoxAclAuthzFilter.globalGroupAclMode = OR > >> > knoxAclAuthzFilter.serviceGroupAclModeMap = /webhdfs/:OR, > /templeton/:OR > >> > knoxAclAuthzFilter.serviceAclMap = /webhdfs/:user1 user2; users admin; > >> > 127.*.*.*, /templeton/:user11 user12; users admin > >> > > >> > # define filter: javaSubjectMapper > >> > javaSubjectMapper = > >> > org.apache.hadoop.gateway.filter.PostAuthenticationFilter > >> > > >> > # define filter: knoxIdentityAsserter > >> > knoxIdentityAsserter = > >> > org.apache.hadoop.gateway.filter.KnoxIdentityAsserter > >> > knoxIdentityAsserter.enabled = true > >> > > >> > # define filter: knoxUrlRewriter > >> > knoxUrlRewriter = > >> > org.apache.hadoop.gateway.filter.rewrite.api.UrlRewriteServletFilter > >> > > >> > knoxUrlRewriter.requestUrlMap = > >> > /webhdfs/v1/:WEBHDFS/webhdfs/inbound/namenode/root, > >> > /webhdfs/v1/[^~]+:WEBHDFS/webhdfs/inbound/namenode/file, > >> > /webhdfs/v1/~:WEBHDFS/webhdfs/inbound/namenode/home, > >> > /webhdfs/v1/~/.*:WEBHDFS/webhdfs/inbound/namenode/home/file, > >> > /webhdfs/data/v1/.+:WEBHDFS/webhdfs/inbound/datanode > >> > > >> > knoxUrlRewriter.requestBodyMap = /oozie/:OOZIE/oozie/configuration, > >> > /oozie/v1/.*:OOZIE/oozie/configuration, > >> > /oozie/v2/.*:OOZIE/oozie/configuration > >> > > >> > knoxUrlRewriter.responseHeadersMap = > >> > /webhdfs/v1/[^~]*:WEBHDFS/webhdfs/outbound/namenode/headers, > >> > /webhdfs/v1/~/:WEBHDFS/webhdfs/outbound/namenode/headers, > >> > /hbase/:WEBHBASE/webhbase/headers/outbound, > >> > /hbase/[~/]*:WEBHBASE/webhbase/headers/outbound, > >> > /hbase/status/cluster:WEBHBASE/webhbase/status/outbound, > >> > /hbase/[^/]*/regions:WEBHBASE/webhbase/regions/outbound > >> > > >> > # define filter: knoxHttpDispatcher > >> > knoxHttpDispatcher = > >> org.apache.hadoop.gateway.dispatch.HttpClientDispatch > >> > knoxHttpDispatcher.replayBufferSizeMap = webhdfs:4, templeton:8, > oozie:4 > >> > > >> > [urls] > >> > # you could choose to have a different chain of filter for different > url > >> > patterns > >> > # so far Knox did not need it > >> > /** = knoxResponseCookieFilter, authcBasic, knoxPrincipalMapper, > >> > knoxIPTracker, knoxAclAuthzFilter, javaSubjectMapper, > >> knoxIdentityAsserter, > >> > knoxUrlRewriter, knoxHttpDispatcher > >> > > >> > # end of shiro.ini > >> > > >> > I think we can rename shiro.ini as knox.ini to make it explicit this > is > >> > more about knox configuration than shiro library configuration. > >> > > >> > > >> > We are using shiro config file mechanism as simple, lightweight > >> depenency > >> > injection. > >> > > >> > This does not really tie us to using only Shiro authentication > >> mechanism or > >> > authorization mechanisms. > >> > > >> > We have the choice of writing all our authentication or authorization > in > >> > our own servlet filters or leverage filters from Shiro library, Realms > >> from > >> > Shiro library or write your own Shiro filters or Shiro realms. > >> > > >> > In most of the cases, when we want to integrate with a new Hadoop back > >> end > >> > service, all that we have to do is specify the path to a file having > >> > rewrite rules for the service. Rest of the things in > >> knox.ini(=shiro.ini) > >> > would remain same. > >> > > >> > Please review and comment. > >> > > >> > > >> > Thanks > >> > Dilli > >> > > >> > -- > >> > CONFIDENTIALITY NOTICE > >> > NOTICE: This message is intended for the use of the individual or > >> entity to > >> > which it is addressed and may contain information that is > confidential, > >> > privileged and exempt from disclosure under applicable law. If the > >> reader > >> > of this message is not the intended recipient, you are hereby notified > >> that > >> > any printing, copying, dissemination, distribution, disclosure or > >> > forwarding of this communication is strictly prohibited. If you have > >> > received this communication in error, please contact the sender > >> immediately > >> > and delete it from your system. Thank You. > >> > > >> > > > > > > -- > CONFIDENTIALITY NOTICE > NOTICE: This message is intended for the use of the individual or entity to > which it is addressed and may contain information that is confidential, > privileged and exempt from disclosure under applicable law. If the reader > of this message is not the intended recipient, you are hereby notified that > any printing, copying, dissemination, distribution, disclosure or > forwarding of this communication is strictly prohibited. If you have > received this communication in error, please contact the sender immediately > and delete it from your system. Thank You. >
