Hi Dilli - Good work here!
Question: 1. Is this complicated ini hand edited by the enduser in order to deploy a given topology or have you changed the shiro contributor to write this out from a much more understandable config within the topology? 2. Where are the services configured for a given topology? 3. Did you rename all the existing filters to start with knox? A couple observations: 1. removing the use of gateway.xml means that we *require* Shiro for our servlet filter chaining 2. While comparing this to the current format of gateway.xml I can agree that this is easier to understand - however, I don't actually find ini format simpler to comprehend and administer than XML or JSON. In fact, comparing this to topology.xml which is all you really need to do for admin, I think the topology file is very easy to understand and administer. 3. moving the rewrite rules out to not require coding is good but should probably not require that they be in this format and file 4. Someone that wants to leverage their own or another third party solution is stuck squeezing it into Shiro 5. This would violate a principle of mine of "Is-a" vs "Has-a" - Knox is not a Shiro provider it can however *have* a Shiro provider 6. End users should never have to see rewrite rules and maybe a lot of what is in that ini. Principal mapping, authz policy, etc will eventually come from Zookeeper which should remove their need to be in this file for those things. Maybe we can even get the LDAP config separately from there as well - though I still think it is best provided in a topology file with a much smaller config requirements and get pushed into shiro.ini. Then the user may never need to see - or at least open - this file. I continue to believe that we are best served by enabling a deployment that looks like this and encouraging the use of it as the preferred solution. However, we should continue to support the use of arbitrary providers through the use of gateway.xml. In most cases, gateway.xml can be leveraged to only include the Shiro provider. In addition, all knowledge of Shiro specifics should remain inside of that provider. If it leaks out then we are violating the separation of Is-a vs Has-a again. This also allows us the flexibility to add something before or after that provider for things that we find are not easily or appropriately done inside the security provider. "I think we can rename shiro.ini as knox.ini to make it explicit this is more about knox configuration than shiro library configuration." While I can understand the thinking of renaming the shiro.ini file to knox.ini, also violates the Is-a vs Has-a principle for me. We are not a shiro security provider extension. We are a REST API Gateway for Hadoop that has security providers - one of which is Shiro. "We are using shiro config file mechanism as simple, lightweight depenency injection." Where is there dependency injection here? If you are referring to the configuration of the filter chain that is not really injection though it is an admittedly simpler mechanism than the current state of the contributors and our deployment machinery - which could be refactored and improved. "This does not really tie us to using only Shiro authentication mechanism or authorization mechanisms. We have the choice of writing all our authentication or authorization in our own servlet filters or leverage filters from Shiro library, Realms from Shiro library or write your own Shiro filters or Shiro realms." The only change that this proposal makes to our flexibility is the inability to take Shiro out of the picture. We could have leveraged Shiro to do all these things with the current design as well. BUT, we have control over our own interception channel in our chains today. This proposal takes a specific *security provider's* proprietary filter chain and uses it as the only interceptor channel for the entire server. Just because they also had a need for one - for their security work - doesn't mean that we shouldn't have our own. The tail should not wag the dog. Shiro is not an application or server framework it is a security provider. "In most of the cases, when we want to integrate with a new Hadoop back end service, all that we have to do is specify the path to a file having rewrite rules for the service. Rest of the things in knox.ini(=shiro.ini) would remain same." This isn't really very clear to me. Do we still have services in a topology.xml file? I don't see where you specify the url for the backend services - so I assume there is still a topology file. How do we know where to dispatch the requests to? I do agree that this is a very good step forward in defining a preferred security provider model for Knox and that we should continue down this road. We just need to do so carefully. If, over time, we have found no reason to use the existing solution that we can consider removing it entirely. Thanks for all this work and insight, Dilli! --larry On Fri, Oct 25, 2013 at 10:27 AM, Dilli Arumugam <[email protected]>wrote: > Have been trying out a few ideas with Knox and managed to get Knox running > with > > 1. WEB-INF/gateway.xml is completely removed > 2. WEB-INF/web.xml declares only shiro filter and a defautl servlet > 2. all filters are defined and injected using WEB-INF/shiro.ini > > To me that looks much simpler to comprehend and administer. > Agreed, this could be subjective. > Hence, seeking comments from community. > > Pasting web.xml and shiro.ini inline. > > web.xml > -------------- > > <?xml version="1.0" encoding="UTF-8" standalone="yes"?> > <web-app xmlns="http://java.sun.com/xml/ns/javaee"; xmlns:xsi=" > http://www.w3.org/2001/XMLSchema-instance"; xsi:schemaLocation=" > http://java.sun.com/xml/ns/javaee > http://java.sun.com/xml/ns/javaee/web-app_3_0.xsd";> > > <listener> > > > <listener-class>org.apache.shiro.web.env.EnvironmentLoaderListener</listener-class> > </listener> > > <listener> > > > <listener-class>org.apache.hadoop.gateway.services.GatewayServicesContextListener</listener-class> > </listener> > > <listener> > > > <listener-class>org.apache.hadoop.gateway.filter.rewrite.api.UrlRewriteServletContextListener</listener-class> > </listener> > > <context-param> > <param-name>rewriteDescriptorLocation</param-name> > <param-value>/WEB-INF/rewrite.xml</param-value> > </context-param> > > <filter> > <filter-name>ShiroFilter</filter-name> > <filter-class>org.apache.shiro.web.servlet.ShiroFilter</filter-class> > </filter> > > <filter-mapping> > <filter-name>ShiroFilter</filter-name> > <url-pattern>/*</url-pattern> > </filter-mapping> > > <session-config> > <session-timeout>30</session-timeout> > </session-config> > > <servlet> > <servlet-name>errorservlet</servlet-name> > > > <servlet-class>org.apache.hadoop.gateway.filter.KnoxErrorServlet</servlet-class> > </servlet> > > <servlet-mapping> > <servlet-name>errorservlet</servlet-name> > <url-pattern>/*</url-pattern> > </servlet-mapping> > > </web-app> > > > shiro.ini > ------------- > > [main] > > # define ldapRealm > ldapRealm=org.apache.shiro.realm.ldap.JndiLdapRealm > ldapRealm.contextFactory.authenticationMechanism=simple > ldapRealm.contextFactory.url=ldap://localhost:33389 > ldapRealm.userDnTemplate=uid={0},ou=people,dc=hadoop,dc=apache,dc=org > > > # define filter: knoxResponseCookieFilter > knoxResponseCookieFilter = > org.apache.hadoop.gateway.filter.ResponseCookieFilter > knoxResponseCookieFilter.enabled = true > knoxResponseCookieFilter.filterHeaders = rememberMe, hadoop.auth.cookie > > # define filter: knoxPrincipalMapper > knoxPrincipalMapper = org.apache.hadoop.gateway.filter.KnoxPrincipalMapper > knoxPrincipalMapper.enabled = true > knoxPrincipalMapper.userToUserMap = bob:guest, jon:bob > knoxPrincipalMapper.userToGroupMap = *:users, bob:admin > > # define filter: knoxIPTracker > knoxIPTracker =org.apache.hadoop.gateway.filter.KnoxIPTracker > knoxIPTracker.enabled = true > > # define filter: knoxAclAuthzFilter > knoxAclAuthzFilter = org.apache.hadoop.gateway.filter.KnoxAclAuthzFilter > knoxAclAuthzFilter.enabled = true > knoxAclAuthzFilter.globalGroupAclMode = OR > knoxAclAuthzFilter.serviceGroupAclModeMap = /webhdfs/:OR, /templeton/:OR > knoxAclAuthzFilter.serviceAclMap = /webhdfs/:user1 user2; users admin; > 127.*.*.*, /templeton/:user11 user12; users admin > > # define filter: javaSubjectMapper > javaSubjectMapper = > org.apache.hadoop.gateway.filter.PostAuthenticationFilter > > # define filter: knoxIdentityAsserter > knoxIdentityAsserter = > org.apache.hadoop.gateway.filter.KnoxIdentityAsserter > knoxIdentityAsserter.enabled = true > > # define filter: knoxUrlRewriter > knoxUrlRewriter = > org.apache.hadoop.gateway.filter.rewrite.api.UrlRewriteServletFilter > > knoxUrlRewriter.requestUrlMap = > /webhdfs/v1/:WEBHDFS/webhdfs/inbound/namenode/root, > /webhdfs/v1/[^~]+:WEBHDFS/webhdfs/inbound/namenode/file, > /webhdfs/v1/~:WEBHDFS/webhdfs/inbound/namenode/home, > /webhdfs/v1/~/.*:WEBHDFS/webhdfs/inbound/namenode/home/file, > /webhdfs/data/v1/.+:WEBHDFS/webhdfs/inbound/datanode > > knoxUrlRewriter.requestBodyMap = /oozie/:OOZIE/oozie/configuration, > /oozie/v1/.*:OOZIE/oozie/configuration, > /oozie/v2/.*:OOZIE/oozie/configuration > > knoxUrlRewriter.responseHeadersMap = > /webhdfs/v1/[^~]*:WEBHDFS/webhdfs/outbound/namenode/headers, > /webhdfs/v1/~/:WEBHDFS/webhdfs/outbound/namenode/headers, > /hbase/:WEBHBASE/webhbase/headers/outbound, > /hbase/[~/]*:WEBHBASE/webhbase/headers/outbound, > /hbase/status/cluster:WEBHBASE/webhbase/status/outbound, > /hbase/[^/]*/regions:WEBHBASE/webhbase/regions/outbound > > # define filter: knoxHttpDispatcher > knoxHttpDispatcher = org.apache.hadoop.gateway.dispatch.HttpClientDispatch > knoxHttpDispatcher.replayBufferSizeMap = webhdfs:4, templeton:8, oozie:4 > > [urls] > # you could choose to have a different chain of filter for different url > patterns > # so far Knox did not need it > /** = knoxResponseCookieFilter, authcBasic, knoxPrincipalMapper, > knoxIPTracker, knoxAclAuthzFilter, javaSubjectMapper, knoxIdentityAsserter, > knoxUrlRewriter, knoxHttpDispatcher > > # end of shiro.ini > > I think we can rename shiro.ini as knox.ini to make it explicit this is > more about knox configuration than shiro library configuration. > > > We are using shiro config file mechanism as simple, lightweight depenency > injection. > > This does not really tie us to using only Shiro authentication mechanism or > authorization mechanisms. > > We have the choice of writing all our authentication or authorization in > our own servlet filters or leverage filters from Shiro library, Realms from > Shiro library or write your own Shiro filters or Shiro realms. > > In most of the cases, when we want to integrate with a new Hadoop back end > service, all that we have to do is specify the path to a file having > rewrite rules for the service. Rest of the things in knox.ini(=shiro.ini) > would remain same. > > Please review and comment. > > > Thanks > Dilli > > -- > CONFIDENTIALITY NOTICE > NOTICE: This message is intended for the use of the individual or entity to > which it is addressed and may contain information that is confidential, > privileged and exempt from disclosure under applicable law. If the reader > of this message is not the intended recipient, you are hereby notified that > any printing, copying, dissemination, distribution, disclosure or > forwarding of this communication is strictly prohibited. If you have > received this communication in error, please contact the sender immediately > and delete it from your system. Thank You. >
