Thanks Larry. Might be thinking deja vu as well since this was opened as a Teradata/Hortonworks support ticket as well (if it got back to you that way). I'll follow KNOX-949 and see what I can do to help.
Kevin Risden On Tue, Jun 6, 2017 at 1:38 PM, larry mccay <[email protected]> wrote: > Hi Kevin - > > I thought that I responded to this thread but I don't see it here and > would really like to try and get to a resolution for the 0.13.0 release. > Since we are trying to close that down at the moment, this needs to be > dealt with with some urgency. > > Actually, I just found another thread and JIRA filed for this > https://issues.apache.org/jira/browse/KNOX-949 - must have been thinking > about my response there. > > I don't quite remember why KNOX-690 was added though. > Maybe someone else has some additional details. > > Let's work on this issue on KNOX-949 with the first priority being > determining whether this needs to block 0.13.0 release. > > thanks, > > --larry > > > On Fri, Jun 2, 2017 at 10:04 AM, Kevin Risden <[email protected]> > wrote: > >> I know the last email was long so wanted to boil down the question: >> >> Does the change from KNOX-690 make sense since it changes the URL >> meaning? Maybe there is a way to do template matching without modifying the >> URL that gets passed to the backend? >> >> Kevin Risden >> >> On Wed, May 24, 2017 at 5:21 PM, Kevin Risden <[email protected]> >> wrote: >> >>> Sorry for the long email but hopefully it provides enough detail to >>> understand the problem and if there is anything we can do to work around it >>> differently. >>> >>> >>> *Problem* >>> >>> >>> >>> With Knox 0.6.x (HDP 2.3), the Knox WebHBase call returns results >>> correctly. With Knox 0.9.x (HDP 2.5), the Knox WebHBase call returns a 404 >>> not found. If we hit WebHBase directly then there is no issue. >>> >>> >>> >>> An example call: >>> >>> >>> >>> curl -i -k -u USER 'https://HOST:8443/gateway/TOP >>> OLOGY/hbase/ns:table/%2frkpart1%2frkpart2' >>> >>> >>> >>> *Analysis* >>> >>> >>> >>> Looked closer at gateway-audit and noticed the dispatch urls were being >>> encoded differently between the two versions. >>> >>> >>> >>> Works – Knox 0.6.x (HDP 2.3) >>> >>> >>> >>> 17/05/23 16:54:13 ||7c4131fc-8638-4a1a-9228-d9a6 >>> 7a312a40|audit|WEBHBASE|USER|||dispatch|uri|http://HOST:8084 >>> /ns:table/%2frkpart1%2frkpart2?doAs=USER|success|Response >>> <http://HOST:8084/ns:table/%2frkpart1%2frkpart2?doAs=USER%7Csuccess%7CResponse> >>> status: 200 >>> >>> >>> >>> Doesn’t work – Knox 0.9.x (HDP 2.5) >>> >>> >>> >>> 17/05/23 17:23:13 ||4244f242-6694-40bb-914d-8dc7 >>> e222f074|audit|WEBHBASE|USER|||dispatch|uri|http://HOST:8084 >>> /ns%3Atable/rkpart1/rkpart2?doAs=USER|success|Response >>> <http://HOST:8084/ns%3Atable/rkpart1/rkpart2?doAs=USER%7Csuccess%7CResponse> >>> status: 404 >>> >>> >>> >>> The 404 is coming from WebHBase directly not being able to find the >>> split row key with the extra slash. The difference is that the %2f which is >>> a / is being decoded and then removed instead of being left as a %2f in the >>> URL. This changes the meaning of the url and causes issues for WebHBase on >>> the backend. >>> >>> >>> >>> At first the culprit seemed like https://issues.apache.org/jira >>> /browse/KNOX-709, but this wasn’t the case. Looks like KNOX-709 may >>> have been caused by KNOX-690. >>> >>> >>> >>> I pulled down a few Knox versions 0.8.0 and 0.9.0 and found that it did >>> not affect 0.8.0. I pulled down the code from >>> https://github.com/hortonworks/knox-release/tree/HDP-2.5.3.77-tag and >>> did a git bisect to find the offending commit using this test case: >>> https://gist.github.com/risdenk/afecc66d6fc0c9d665abd1ae5466f341. The >>> commit is https://git-wip-us.apache.org/repos/asf?p=knox.git;h=c28224c >>> and related JIRA is https://issues.apache.org/jira/browse/KNOX-690. >>> >>> >>> >>> *Resolution* >>> >>> >>> >>> I rebuilt Knox from https://github.com/hortonworks >>> /knox-release/tree/HDP-2.5.3.77-tag with the commit c28224c for that >>> reverted. The adjusted code is here: https://github.com/risdenk/kno >>> x-release/tree/hdp25_revert_KNOX-690. The change is only a single >>> commit https://github.com/risdenk/knox-release/commit/dc452126de99f >>> 6f1d15938f7294e95e3b7c89328 >>> >>> >>> >>> I rebuilt Knox with mvn -DskipTests package and copied the two affected >>> jars (gateway-provider-rewrite and gateway-util-urltemplate) to >>> /usr/hdp/current/knox-server/lib/ >>> >>> >>> >>> I moved the two old jars to /root. The affected jars were >>> >>> · gateway-provider-rewrite-0.9.0.2.5.3.0-37.jar >>> >>> · gateway-util-urltemplate-0.9.0.2.5.3.0-37.jar >>> >>> >>> >>> I then restarted Knox on hdpr05en02. This made the following curl call >>> work: >>> >>> >>> >>> curl -i -k -u USER 'https://KNOXHOST:8443/gateway >>> /TOPOLOGY/hbase/ns:table/%2frkpart1%2frkpart2' >>> >>> >>> >>> *Conclusion* >>> >>> >>> I'm not convinced that KNOX-690 is a good idea but it basically made it >>> so url encoded paths were checked by the templates/parser. URL encoding >>> should be left alone in many cases. Reverting the change from KNOX-690 >>> shouldn't affect us much more other than upgrades to HDP could break this. >>> I think we should really avoid using url encodable characters in the rowkey >>> especially for webhbase. / is a bad character to try to pass through >>> webservices. Since having the customer change rowkey design will be >>> painful, we will be using a workaround in the short term. >>> >>> Kevin Risden >>> >> >> >
