Larry, I would go with specifying the end points of hadoop services in knox.ini(=shiro.ini) and leave the rewrite rules in its own file.
Rewrite rules are not going to be cluster specific and admins typically would not have to tinker with them. Thanks Dilli On Fri, Oct 25, 2013 at 9:03 AM, Dilli Arumugam <[email protected]>wrote: > Thanks Larry for the review and comments. > > The proposed knox.ini is meant to be edited by Knox system administrator > until we get to have a admin console. This is not meant to be edited by end > user. > > To integrate additional back end hadoop service, following pieces of > information have to come from pointers from knox.ini to external > configuration files. > > 1. end point of the new hadoop servie back end > 2. rewriting rules for the new service > > These can be specified in one file or split between 2 files with one of > them being topology file or its equivalent. > I have not shown all those details in the current shiro.ini. > But, you would get the idea. > > Dilli > > > On Fri, Oct 25, 2013 at 8:50 AM, larry mccay <[email protected]> wrote: > >> Hi Dilli - >> >> Good work here! >> >> Question: >> 1. Is this complicated ini hand edited by the enduser in order to deploy a >> given topology or have you changed the shiro contributor to write this out >> from a much more understandable config within the topology? >> >> 2. Where are the services configured for a given topology? >> 3. Did you rename all the existing filters to start with knox? >> >> A couple observations: >> >> 1. removing the use of gateway.xml means that we *require* Shiro for our >> servlet filter chaining >> 2. While comparing this to the current format of gateway.xml I can agree >> that this is easier to understand - however, I don't actually find ini >> format simpler to comprehend and administer than XML or JSON. In fact, >> comparing this to topology.xml which is all you really need to do for >> admin, I think the topology file is very easy to understand and >> administer. >> 3. moving the rewrite rules out to not require coding is good but should >> probably not require that they be in this format and file >> 4. Someone that wants to leverage their own or another third party >> solution >> is stuck squeezing it into Shiro >> 5. This would violate a principle of mine of "Is-a" vs "Has-a" - Knox is >> not a Shiro provider it can however *have* a Shiro provider >> 6. End users should never have to see rewrite rules and maybe a lot of >> what >> is in that ini. Principal mapping, authz policy, etc will eventually come >> from Zookeeper which should remove their need to be in this file for those >> things. Maybe we can even get the LDAP config separately from there as >> well >> - though I still think it is best provided in a topology file with a much >> smaller config requirements and get pushed into shiro.ini. Then the user >> may never need to see - or at least open - this file. >> >> I continue to believe that we are best served by enabling a deployment >> that >> looks like this and encouraging the use of it as the preferred solution. >> However, we should continue to support the use of arbitrary providers >> through the use of gateway.xml. In most cases, gateway.xml can be >> leveraged >> to only include the Shiro provider. In addition, all knowledge of Shiro >> specifics should remain inside of that provider. If it leaks out then we >> are violating the separation of Is-a vs Has-a again. >> >> This also allows us the flexibility to add something before or after that >> provider for things that we find are not easily or appropriately done >> inside the security provider. >> >> "I think we can rename shiro.ini as knox.ini to make it explicit this is >> more about knox configuration than shiro library configuration." >> >> While I can understand the thinking of renaming the shiro.ini file to >> knox.ini, also violates the Is-a vs Has-a principle for me. We are not a >> shiro security provider extension. We are a REST API Gateway for Hadoop >> that has security providers - one of which is Shiro. >> >> "We are using shiro config file mechanism as simple, lightweight depenency >> injection." >> >> Where is there dependency injection here? If you are referring to the >> configuration of the filter chain that is not really injection though it >> is >> an admittedly simpler mechanism than the current state of the contributors >> and our deployment machinery - which could be refactored and improved. >> >> "This does not really tie us to using only Shiro authentication mechanism >> or >> authorization mechanisms. >> >> We have the choice of writing all our authentication or authorization in >> our own servlet filters or leverage filters from Shiro library, Realms >> from >> Shiro library or write your own Shiro filters or Shiro realms." >> >> The only change that this proposal makes to our flexibility is the >> inability to take Shiro out of the picture. >> We could have leveraged Shiro to do all these things with the current >> design as well. >> BUT, we have control over our own interception channel in our chains >> today. >> This proposal takes a specific *security provider's* proprietary filter >> chain and uses it as the only interceptor channel for the entire server. >> Just because they also had a need for one - for their security work - >> doesn't mean that we shouldn't have our own. The tail should not wag the >> dog. Shiro is not an application or server framework it is a security >> provider. >> >> "In most of the cases, when we want to integrate with a new Hadoop back >> end >> service, all that we have to do is specify the path to a file having >> rewrite rules for the service. Rest of the things in knox.ini(=shiro.ini) >> would remain same." >> >> This isn't really very clear to me. Do we still have services in a >> topology.xml file? >> I don't see where you specify the url for the backend services - so I >> assume there is still a topology file. >> How do we know where to dispatch the requests to? >> >> I do agree that this is a very good step forward in defining a preferred >> security provider model for Knox and that we should continue down this >> road. We just need to do so carefully. If, over time, we have found no >> reason to use the existing solution that we can consider removing it >> entirely. >> >> Thanks for all this work and insight, Dilli! >> >> --larry >> >> >> On Fri, Oct 25, 2013 at 10:27 AM, Dilli Arumugam >> <[email protected]>wrote: >> >> > Have been trying out a few ideas with Knox and managed to get Knox >> running >> > with >> > >> > 1. WEB-INF/gateway.xml is completely removed >> > 2. WEB-INF/web.xml declares only shiro filter and a defautl servlet >> > 2. all filters are defined and injected using WEB-INF/shiro.ini >> > >> > To me that looks much simpler to comprehend and administer. >> > Agreed, this could be subjective. >> > Hence, seeking comments from community. >> > >> > Pasting web.xml and shiro.ini inline. >> > >> > web.xml >> > -------------- >> > >> > <?xml version="1.0" encoding="UTF-8" standalone="yes"?> >> > <web-app xmlns="http://java.sun.com/xml/ns/javaee"; xmlns:xsi=" >> > http://www.w3.org/2001/XMLSchema-instance"; xsi:schemaLocation=" >> > http://java.sun.com/xml/ns/javaee >> > http://java.sun.com/xml/ns/javaee/web-app_3_0.xsd";> >> > >> > <listener> >> > >> > >> > >> <listener-class>org.apache.shiro.web.env.EnvironmentLoaderListener</listener-class> >> > </listener> >> > >> > <listener> >> > >> > >> > >> <listener-class>org.apache.hadoop.gateway.services.GatewayServicesContextListener</listener-class> >> > </listener> >> > >> > <listener> >> > >> > >> > >> <listener-class>org.apache.hadoop.gateway.filter.rewrite.api.UrlRewriteServletContextListener</listener-class> >> > </listener> >> > >> > <context-param> >> > <param-name>rewriteDescriptorLocation</param-name> >> > <param-value>/WEB-INF/rewrite.xml</param-value> >> > </context-param> >> > >> > <filter> >> > <filter-name>ShiroFilter</filter-name> >> > >> <filter-class>org.apache.shiro.web.servlet.ShiroFilter</filter-class> >> > </filter> >> > >> > <filter-mapping> >> > <filter-name>ShiroFilter</filter-name> >> > <url-pattern>/*</url-pattern> >> > </filter-mapping> >> > >> > <session-config> >> > <session-timeout>30</session-timeout> >> > </session-config> >> > >> > <servlet> >> > <servlet-name>errorservlet</servlet-name> >> > >> > >> > >> <servlet-class>org.apache.hadoop.gateway.filter.KnoxErrorServlet</servlet-class> >> > </servlet> >> > >> > <servlet-mapping> >> > <servlet-name>errorservlet</servlet-name> >> > <url-pattern>/*</url-pattern> >> > </servlet-mapping> >> > >> > </web-app> >> > >> > >> > shiro.ini >> > ------------- >> > >> > [main] >> > >> > # define ldapRealm >> > ldapRealm=org.apache.shiro.realm.ldap.JndiLdapRealm >> > ldapRealm.contextFactory.authenticationMechanism=simple >> > ldapRealm.contextFactory.url=ldap://localhost:33389 >> > ldapRealm.userDnTemplate=uid={0},ou=people,dc=hadoop,dc=apache,dc=org >> > >> > >> > # define filter: knoxResponseCookieFilter >> > knoxResponseCookieFilter = >> > org.apache.hadoop.gateway.filter.ResponseCookieFilter >> > knoxResponseCookieFilter.enabled = true >> > knoxResponseCookieFilter.filterHeaders = rememberMe, hadoop.auth.cookie >> > >> > # define filter: knoxPrincipalMapper >> > knoxPrincipalMapper = >> org.apache.hadoop.gateway.filter.KnoxPrincipalMapper >> > knoxPrincipalMapper.enabled = true >> > knoxPrincipalMapper.userToUserMap = bob:guest, jon:bob >> > knoxPrincipalMapper.userToGroupMap = *:users, bob:admin >> > >> > # define filter: knoxIPTracker >> > knoxIPTracker =org.apache.hadoop.gateway.filter.KnoxIPTracker >> > knoxIPTracker.enabled = true >> > >> > # define filter: knoxAclAuthzFilter >> > knoxAclAuthzFilter = org.apache.hadoop.gateway.filter.KnoxAclAuthzFilter >> > knoxAclAuthzFilter.enabled = true >> > knoxAclAuthzFilter.globalGroupAclMode = OR >> > knoxAclAuthzFilter.serviceGroupAclModeMap = /webhdfs/:OR, /templeton/:OR >> > knoxAclAuthzFilter.serviceAclMap = /webhdfs/:user1 user2; users admin; >> > 127.*.*.*, /templeton/:user11 user12; users admin >> > >> > # define filter: javaSubjectMapper >> > javaSubjectMapper = >> > org.apache.hadoop.gateway.filter.PostAuthenticationFilter >> > >> > # define filter: knoxIdentityAsserter >> > knoxIdentityAsserter = >> > org.apache.hadoop.gateway.filter.KnoxIdentityAsserter >> > knoxIdentityAsserter.enabled = true >> > >> > # define filter: knoxUrlRewriter >> > knoxUrlRewriter = >> > org.apache.hadoop.gateway.filter.rewrite.api.UrlRewriteServletFilter >> > >> > knoxUrlRewriter.requestUrlMap = >> > /webhdfs/v1/:WEBHDFS/webhdfs/inbound/namenode/root, >> > /webhdfs/v1/[^~]+:WEBHDFS/webhdfs/inbound/namenode/file, >> > /webhdfs/v1/~:WEBHDFS/webhdfs/inbound/namenode/home, >> > /webhdfs/v1/~/.*:WEBHDFS/webhdfs/inbound/namenode/home/file, >> > /webhdfs/data/v1/.+:WEBHDFS/webhdfs/inbound/datanode >> > >> > knoxUrlRewriter.requestBodyMap = /oozie/:OOZIE/oozie/configuration, >> > /oozie/v1/.*:OOZIE/oozie/configuration, >> > /oozie/v2/.*:OOZIE/oozie/configuration >> > >> > knoxUrlRewriter.responseHeadersMap = >> > /webhdfs/v1/[^~]*:WEBHDFS/webhdfs/outbound/namenode/headers, >> > /webhdfs/v1/~/:WEBHDFS/webhdfs/outbound/namenode/headers, >> > /hbase/:WEBHBASE/webhbase/headers/outbound, >> > /hbase/[~/]*:WEBHBASE/webhbase/headers/outbound, >> > /hbase/status/cluster:WEBHBASE/webhbase/status/outbound, >> > /hbase/[^/]*/regions:WEBHBASE/webhbase/regions/outbound >> > >> > # define filter: knoxHttpDispatcher >> > knoxHttpDispatcher = >> org.apache.hadoop.gateway.dispatch.HttpClientDispatch >> > knoxHttpDispatcher.replayBufferSizeMap = webhdfs:4, templeton:8, oozie:4 >> > >> > [urls] >> > # you could choose to have a different chain of filter for different url >> > patterns >> > # so far Knox did not need it >> > /** = knoxResponseCookieFilter, authcBasic, knoxPrincipalMapper, >> > knoxIPTracker, knoxAclAuthzFilter, javaSubjectMapper, >> knoxIdentityAsserter, >> > knoxUrlRewriter, knoxHttpDispatcher >> > >> > # end of shiro.ini >> > >> > I think we can rename shiro.ini as knox.ini to make it explicit this is >> > more about knox configuration than shiro library configuration. >> > >> > >> > We are using shiro config file mechanism as simple, lightweight >> depenency >> > injection. >> > >> > This does not really tie us to using only Shiro authentication >> mechanism or >> > authorization mechanisms. >> > >> > We have the choice of writing all our authentication or authorization in >> > our own servlet filters or leverage filters from Shiro library, Realms >> from >> > Shiro library or write your own Shiro filters or Shiro realms. >> > >> > In most of the cases, when we want to integrate with a new Hadoop back >> end >> > service, all that we have to do is specify the path to a file having >> > rewrite rules for the service. Rest of the things in >> knox.ini(=shiro.ini) >> > would remain same. >> > >> > Please review and comment. >> > >> > >> > Thanks >> > Dilli >> > >> > -- >> > CONFIDENTIALITY NOTICE >> > NOTICE: This message is intended for the use of the individual or >> entity to >> > which it is addressed and may contain information that is confidential, >> > privileged and exempt from disclosure under applicable law. If the >> reader >> > of this message is not the intended recipient, you are hereby notified >> that >> > any printing, copying, dissemination, distribution, disclosure or >> > forwarding of this communication is strictly prohibited. If you have >> > received this communication in error, please contact the sender >> immediately >> > and delete it from your system. Thank You. >> > >> > > -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
