On Fri, Oct 25, 2013 at 12:03 PM, Dilli Arumugam <[email protected]>wrote:
> Thanks Larry for the review and comments. > > The proposed knox.ini is meant to be edited by Knox system administrator > until we get to have a admin console. This is not meant to be edited by end > user. > Well, by end user here, I meant the same person that had to edit topology.xml. Topology.xml is much simpler than this file and most of what is in shiro.ini here should never need to be seen - even by the administrators. Giving the ability to mess with the order of the chain and the like is only asking for trouble and user problems. I think that we should limit the amount of time we have to support Shiro config file problems by keeping folks out as much as we can. > > To integrate additional back end hadoop service, following pieces of > information have to come from pointers from knox.ini to external > configuration files. > > 1. end point of the new hadoop servie back end > 2. rewriting rules for the new service > These can be specified in one file or split between 2 files with one of > them being topology file or its equivalent. > I have not shown all those details in the current shiro.ini. > But, you would get the idea. > But if you don't show all of that then it isn't showing the whole picture. What are you currently doing in your POC for the services? > > Dilli > > > On Fri, Oct 25, 2013 at 8:50 AM, larry mccay <[email protected]> wrote: > > > Hi Dilli - > > > > Good work here! > > > > Question: > > 1. Is this complicated ini hand edited by the enduser in order to deploy > a > > given topology or have you changed the shiro contributor to write this > out > > from a much more understandable config within the topology? > > > > 2. Where are the services configured for a given topology? > > 3. Did you rename all the existing filters to start with knox? > > > > A couple observations: > > > > 1. removing the use of gateway.xml means that we *require* Shiro for our > > servlet filter chaining > > 2. While comparing this to the current format of gateway.xml I can agree > > that this is easier to understand - however, I don't actually find ini > > format simpler to comprehend and administer than XML or JSON. In fact, > > comparing this to topology.xml which is all you really need to do for > > admin, I think the topology file is very easy to understand and > administer. > > 3. moving the rewrite rules out to not require coding is good but should > > probably not require that they be in this format and file > > 4. Someone that wants to leverage their own or another third party > solution > > is stuck squeezing it into Shiro > > 5. This would violate a principle of mine of "Is-a" vs "Has-a" - Knox is > > not a Shiro provider it can however *have* a Shiro provider > > 6. End users should never have to see rewrite rules and maybe a lot of > what > > is in that ini. Principal mapping, authz policy, etc will eventually come > > from Zookeeper which should remove their need to be in this file for > those > > things. Maybe we can even get the LDAP config separately from there as > well > > - though I still think it is best provided in a topology file with a much > > smaller config requirements and get pushed into shiro.ini. Then the user > > may never need to see - or at least open - this file. > > > > I continue to believe that we are best served by enabling a deployment > that > > looks like this and encouraging the use of it as the preferred solution. > > However, we should continue to support the use of arbitrary providers > > through the use of gateway.xml. In most cases, gateway.xml can be > leveraged > > to only include the Shiro provider. In addition, all knowledge of Shiro > > specifics should remain inside of that provider. If it leaks out then we > > are violating the separation of Is-a vs Has-a again. > > > > This also allows us the flexibility to add something before or after that > > provider for things that we find are not easily or appropriately done > > inside the security provider. > > > > "I think we can rename shiro.ini as knox.ini to make it explicit this is > > more about knox configuration than shiro library configuration." > > > > While I can understand the thinking of renaming the shiro.ini file to > > knox.ini, also violates the Is-a vs Has-a principle for me. We are not a > > shiro security provider extension. We are a REST API Gateway for Hadoop > > that has security providers - one of which is Shiro. > > > > "We are using shiro config file mechanism as simple, lightweight > depenency > > injection." > > > > Where is there dependency injection here? If you are referring to the > > configuration of the filter chain that is not really injection though it > is > > an admittedly simpler mechanism than the current state of the > contributors > > and our deployment machinery - which could be refactored and improved. > > > > "This does not really tie us to using only Shiro authentication mechanism > > or > > authorization mechanisms. > > > > We have the choice of writing all our authentication or authorization in > > our own servlet filters or leverage filters from Shiro library, Realms > from > > Shiro library or write your own Shiro filters or Shiro realms." > > > > The only change that this proposal makes to our flexibility is the > > inability to take Shiro out of the picture. > > We could have leveraged Shiro to do all these things with the current > > design as well. > > BUT, we have control over our own interception channel in our chains > today. > > This proposal takes a specific *security provider's* proprietary filter > > chain and uses it as the only interceptor channel for the entire server. > > Just because they also had a need for one - for their security work - > > doesn't mean that we shouldn't have our own. The tail should not wag the > > dog. Shiro is not an application or server framework it is a security > > provider. > > > > "In most of the cases, when we want to integrate with a new Hadoop back > end > > service, all that we have to do is specify the path to a file having > > rewrite rules for the service. Rest of the things in knox.ini(=shiro.ini) > > would remain same." > > > > This isn't really very clear to me. Do we still have services in a > > topology.xml file? > > I don't see where you specify the url for the backend services - so I > > assume there is still a topology file. > > How do we know where to dispatch the requests to? > > > > I do agree that this is a very good step forward in defining a preferred > > security provider model for Knox and that we should continue down this > > road. We just need to do so carefully. If, over time, we have found no > > reason to use the existing solution that we can consider removing it > > entirely. > > > > Thanks for all this work and insight, Dilli! > > > > --larry > > > > > > On Fri, Oct 25, 2013 at 10:27 AM, Dilli Arumugam > > <[email protected]>wrote: > > > > > Have been trying out a few ideas with Knox and managed to get Knox > > running > > > with > > > > > > 1. WEB-INF/gateway.xml is completely removed > > > 2. WEB-INF/web.xml declares only shiro filter and a defautl servlet > > > 2. all filters are defined and injected using WEB-INF/shiro.ini > > > > > > To me that looks much simpler to comprehend and administer. > > > Agreed, this could be subjective. > > > Hence, seeking comments from community. > > > > > > Pasting web.xml and shiro.ini inline. > > > > > > web.xml > > > -------------- > > > > > > <?xml version="1.0" encoding="UTF-8" standalone="yes"?> > > > <web-app xmlns="http://java.sun.com/xml/ns/javaee"; xmlns:xsi=" > > > http://www.w3.org/2001/XMLSchema-instance"; xsi:schemaLocation=" > > > http://java.sun.com/xml/ns/javaee > > > http://java.sun.com/xml/ns/javaee/web-app_3_0.xsd";> > > > > > > <listener> > > > > > > > > > > > > <listener-class>org.apache.shiro.web.env.EnvironmentLoaderListener</listener-class> > > > </listener> > > > > > > <listener> > > > > > > > > > > > > <listener-class>org.apache.hadoop.gateway.services.GatewayServicesContextListener</listener-class> > > > </listener> > > > > > > <listener> > > > > > > > > > > > > <listener-class>org.apache.hadoop.gateway.filter.rewrite.api.UrlRewriteServletContextListener</listener-class> > > > </listener> > > > > > > <context-param> > > > <param-name>rewriteDescriptorLocation</param-name> > > > <param-value>/WEB-INF/rewrite.xml</param-value> > > > </context-param> > > > > > > <filter> > > > <filter-name>ShiroFilter</filter-name> > > > > > <filter-class>org.apache.shiro.web.servlet.ShiroFilter</filter-class> > > > </filter> > > > > > > <filter-mapping> > > > <filter-name>ShiroFilter</filter-name> > > > <url-pattern>/*</url-pattern> > > > </filter-mapping> > > > > > > <session-config> > > > <session-timeout>30</session-timeout> > > > </session-config> > > > > > > <servlet> > > > <servlet-name>errorservlet</servlet-name> > > > > > > > > > > > > <servlet-class>org.apache.hadoop.gateway.filter.KnoxErrorServlet</servlet-class> > > > </servlet> > > > > > > <servlet-mapping> > > > <servlet-name>errorservlet</servlet-name> > > > <url-pattern>/*</url-pattern> > > > </servlet-mapping> > > > > > > </web-app> > > > > > > > > > shiro.ini > > > ------------- > > > > > > [main] > > > > > > # define ldapRealm > > > ldapRealm=org.apache.shiro.realm.ldap.JndiLdapRealm > > > ldapRealm.contextFactory.authenticationMechanism=simple > > > ldapRealm.contextFactory.url=ldap://localhost:33389 > > > ldapRealm.userDnTemplate=uid={0},ou=people,dc=hadoop,dc=apache,dc=org > > > > > > > > > # define filter: knoxResponseCookieFilter > > > knoxResponseCookieFilter = > > > org.apache.hadoop.gateway.filter.ResponseCookieFilter > > > knoxResponseCookieFilter.enabled = true > > > knoxResponseCookieFilter.filterHeaders = rememberMe, hadoop.auth.cookie > > > > > > # define filter: knoxPrincipalMapper > > > knoxPrincipalMapper = > > org.apache.hadoop.gateway.filter.KnoxPrincipalMapper > > > knoxPrincipalMapper.enabled = true > > > knoxPrincipalMapper.userToUserMap = bob:guest, jon:bob > > > knoxPrincipalMapper.userToGroupMap = *:users, bob:admin > > > > > > # define filter: knoxIPTracker > > > knoxIPTracker =org.apache.hadoop.gateway.filter.KnoxIPTracker > > > knoxIPTracker.enabled = true > > > > > > # define filter: knoxAclAuthzFilter > > > knoxAclAuthzFilter = > org.apache.hadoop.gateway.filter.KnoxAclAuthzFilter > > > knoxAclAuthzFilter.enabled = true > > > knoxAclAuthzFilter.globalGroupAclMode = OR > > > knoxAclAuthzFilter.serviceGroupAclModeMap = /webhdfs/:OR, > /templeton/:OR > > > knoxAclAuthzFilter.serviceAclMap = /webhdfs/:user1 user2; users admin; > > > 127.*.*.*, /templeton/:user11 user12; users admin > > > > > > # define filter: javaSubjectMapper > > > javaSubjectMapper = > > > org.apache.hadoop.gateway.filter.PostAuthenticationFilter > > > > > > # define filter: knoxIdentityAsserter > > > knoxIdentityAsserter = > > > org.apache.hadoop.gateway.filter.KnoxIdentityAsserter > > > knoxIdentityAsserter.enabled = true > > > > > > # define filter: knoxUrlRewriter > > > knoxUrlRewriter = > > > org.apache.hadoop.gateway.filter.rewrite.api.UrlRewriteServletFilter > > > > > > knoxUrlRewriter.requestUrlMap = > > > /webhdfs/v1/:WEBHDFS/webhdfs/inbound/namenode/root, > > > /webhdfs/v1/[^~]+:WEBHDFS/webhdfs/inbound/namenode/file, > > > /webhdfs/v1/~:WEBHDFS/webhdfs/inbound/namenode/home, > > > /webhdfs/v1/~/.*:WEBHDFS/webhdfs/inbound/namenode/home/file, > > > /webhdfs/data/v1/.+:WEBHDFS/webhdfs/inbound/datanode > > > > > > knoxUrlRewriter.requestBodyMap = /oozie/:OOZIE/oozie/configuration, > > > /oozie/v1/.*:OOZIE/oozie/configuration, > > > /oozie/v2/.*:OOZIE/oozie/configuration > > > > > > knoxUrlRewriter.responseHeadersMap = > > > /webhdfs/v1/[^~]*:WEBHDFS/webhdfs/outbound/namenode/headers, > > > /webhdfs/v1/~/:WEBHDFS/webhdfs/outbound/namenode/headers, > > > /hbase/:WEBHBASE/webhbase/headers/outbound, > > > /hbase/[~/]*:WEBHBASE/webhbase/headers/outbound, > > > /hbase/status/cluster:WEBHBASE/webhbase/status/outbound, > > > /hbase/[^/]*/regions:WEBHBASE/webhbase/regions/outbound > > > > > > # define filter: knoxHttpDispatcher > > > knoxHttpDispatcher = > > org.apache.hadoop.gateway.dispatch.HttpClientDispatch > > > knoxHttpDispatcher.replayBufferSizeMap = webhdfs:4, templeton:8, > oozie:4 > > > > > > [urls] > > > # you could choose to have a different chain of filter for different > url > > > patterns > > > # so far Knox did not need it > > > /** = knoxResponseCookieFilter, authcBasic, knoxPrincipalMapper, > > > knoxIPTracker, knoxAclAuthzFilter, javaSubjectMapper, > > knoxIdentityAsserter, > > > knoxUrlRewriter, knoxHttpDispatcher > > > > > > # end of shiro.ini > > > > > > I think we can rename shiro.ini as knox.ini to make it explicit this is > > > more about knox configuration than shiro library configuration. > > > > > > > > > We are using shiro config file mechanism as simple, lightweight > depenency > > > injection. > > > > > > This does not really tie us to using only Shiro authentication > mechanism > > or > > > authorization mechanisms. > > > > > > We have the choice of writing all our authentication or authorization > in > > > our own servlet filters or leverage filters from Shiro library, Realms > > from > > > Shiro library or write your own Shiro filters or Shiro realms. > > > > > > In most of the cases, when we want to integrate with a new Hadoop back > > end > > > service, all that we have to do is specify the path to a file having > > > rewrite rules for the service. Rest of the things in > knox.ini(=shiro.ini) > > > would remain same. > > > > > > Please review and comment. > > > > > > > > > Thanks > > > Dilli > > > > > > -- > > > CONFIDENTIALITY NOTICE > > > NOTICE: This message is intended for the use of the individual or > entity > > to > > > which it is addressed and may contain information that is confidential, > > > privileged and exempt from disclosure under applicable law. If the > reader > > > of this message is not the intended recipient, you are hereby notified > > that > > > any printing, copying, dissemination, distribution, disclosure or > > > forwarding of this communication is strictly prohibited. If you have > > > received this communication in error, please contact the sender > > immediately > > > and delete it from your system. Thank You. > > > > > > > -- > CONFIDENTIALITY NOTICE > NOTICE: This message is intended for the use of the individual or entity to > which it is addressed and may contain information that is confidential, > privileged and exempt from disclosure under applicable law. If the reader > of this message is not the intended recipient, you are hereby notified that > any printing, copying, dissemination, distribution, disclosure or > forwarding of this communication is strictly prohibited. If you have > received this communication in error, please contact the sender immediately > and delete it from your system. Thank You. >
