Hi Kevin - Have you been able to test this yet?
thanks, --larry On Sat, Jun 17, 2017 at 12:28 PM, larry mccay <[email protected]> wrote: > Hi Kevin - > > I pushed a fix for this. > Can you try a build from master? > > My HBase deployment went south and I ended up flip flopping on the > HBaseDispatch code - so not entirely sure it is addressed fully yet. > From what I can see in the audit log the double slash is being retained > but it is not encoded anymore. > > Hopefully it will work - since the audit log that you referenced in your > original description does not have the double slash, I am hopeful. > > Thanks, > > --larry > > On Thu, Jun 8, 2017 at 5:57 PM, larry mccay <[email protected]> wrote: > >> Hey Kevin - >> >> Would you mind trying the patch that I attached to KNOX-949? >> Hoping this satisfies this issue with HBase as well as KNOX-949 with >> WebHDFS *and* the original Ambari issue in KNOX-690. >> >> thanks! >> >> --larry >> >> On Tue, Jun 6, 2017 at 4:54 PM, Kevin Risden <[email protected]> >> wrote: >> >>> Thanks Larry. Might be thinking deja vu as well since this was opened as >>> a Teradata/Hortonworks support ticket as well (if it got back to you that >>> way). I'll follow KNOX-949 and see what I can do to help. >>> >>> Kevin Risden >>> >>> On Tue, Jun 6, 2017 at 1:38 PM, larry mccay <[email protected]> wrote: >>> >>>> Hi Kevin - >>>> >>>> I thought that I responded to this thread but I don't see it here and >>>> would really like to try and get to a resolution for the 0.13.0 release. >>>> Since we are trying to close that down at the moment, this needs to be >>>> dealt with with some urgency. >>>> >>>> Actually, I just found another thread and JIRA filed for this >>>> https://issues.apache.org/jira/browse/KNOX-949 - must have been >>>> thinking about my response there. >>>> >>>> I don't quite remember why KNOX-690 was added though. >>>> Maybe someone else has some additional details. >>>> >>>> Let's work on this issue on KNOX-949 with the first priority being >>>> determining whether this needs to block 0.13.0 release. >>>> >>>> thanks, >>>> >>>> --larry >>>> >>>> >>>> On Fri, Jun 2, 2017 at 10:04 AM, Kevin Risden <[email protected] >>>> > wrote: >>>> >>>>> I know the last email was long so wanted to boil down the question: >>>>> >>>>> Does the change from KNOX-690 make sense since it changes the URL >>>>> meaning? Maybe there is a way to do template matching without modifying >>>>> the >>>>> URL that gets passed to the backend? >>>>> >>>>> Kevin Risden >>>>> >>>>> On Wed, May 24, 2017 at 5:21 PM, Kevin Risden < >>>>> [email protected]> wrote: >>>>> >>>>>> Sorry for the long email but hopefully it provides enough detail to >>>>>> understand the problem and if there is anything we can do to work around >>>>>> it >>>>>> differently. >>>>>> >>>>>> >>>>>> *Problem* >>>>>> >>>>>> >>>>>> >>>>>> With Knox 0.6.x (HDP 2.3), the Knox WebHBase call returns results >>>>>> correctly. With Knox 0.9.x (HDP 2.5), the Knox WebHBase call returns a >>>>>> 404 >>>>>> not found. If we hit WebHBase directly then there is no issue. >>>>>> >>>>>> >>>>>> >>>>>> An example call: >>>>>> >>>>>> >>>>>> >>>>>> curl -i -k -u USER 'https://HOST:8443/gateway/TOP >>>>>> OLOGY/hbase/ns:table/%2frkpart1%2frkpart2' >>>>>> >>>>>> >>>>>> >>>>>> *Analysis* >>>>>> >>>>>> >>>>>> >>>>>> Looked closer at gateway-audit and noticed the dispatch urls were >>>>>> being encoded differently between the two versions. >>>>>> >>>>>> >>>>>> >>>>>> Works – Knox 0.6.x (HDP 2.3) >>>>>> >>>>>> >>>>>> >>>>>> 17/05/23 16:54:13 ||7c4131fc-8638-4a1a-9228-d9a6 >>>>>> 7a312a40|audit|WEBHBASE|USER|||dispatch|uri|http://HOST:8084 >>>>>> /ns:table/%2frkpart1%2frkpart2?doAs=USER|success|Response >>>>>> <http://HOST:8084/ns:table/%2frkpart1%2frkpart2?doAs=USER%7Csuccess%7CResponse> >>>>>> status: 200 >>>>>> >>>>>> >>>>>> >>>>>> Doesn’t work – Knox 0.9.x (HDP 2.5) >>>>>> >>>>>> >>>>>> >>>>>> 17/05/23 17:23:13 ||4244f242-6694-40bb-914d-8dc7 >>>>>> e222f074|audit|WEBHBASE|USER|||dispatch|uri|http://HOST:8084 >>>>>> /ns%3Atable/rkpart1/rkpart2?doAs=USER|success|Response >>>>>> <http://HOST:8084/ns%3Atable/rkpart1/rkpart2?doAs=USER%7Csuccess%7CResponse> >>>>>> status: 404 >>>>>> >>>>>> >>>>>> >>>>>> The 404 is coming from WebHBase directly not being able to find the >>>>>> split row key with the extra slash. The difference is that the %2f which >>>>>> is >>>>>> a / is being decoded and then removed instead of being left as a %2f in >>>>>> the >>>>>> URL. This changes the meaning of the url and causes issues for WebHBase >>>>>> on >>>>>> the backend. >>>>>> >>>>>> >>>>>> >>>>>> At first the culprit seemed like https://issues.apache.org/jira >>>>>> /browse/KNOX-709, but this wasn’t the case. Looks like KNOX-709 may >>>>>> have been caused by KNOX-690. >>>>>> >>>>>> >>>>>> >>>>>> I pulled down a few Knox versions 0.8.0 and 0.9.0 and found that it >>>>>> did not affect 0.8.0. I pulled down the code from >>>>>> https://github.com/hortonworks/knox-release/tree/HDP-2.5.3.77-tag >>>>>> and did a git bisect to find the offending commit using this test case: >>>>>> https://gist.github.com/risdenk/afecc66d6fc0c9d665abd1ae5466f341. >>>>>> The commit is https://git-wip-us.apache.org/ >>>>>> repos/asf?p=knox.git;h=c28224c and related JIRA is >>>>>> https://issues.apache.org/jira/browse/KNOX-690. >>>>>> >>>>>> >>>>>> >>>>>> *Resolution* >>>>>> >>>>>> >>>>>> >>>>>> I rebuilt Knox from https://github.com/hortonworks >>>>>> /knox-release/tree/HDP-2.5.3.77-tag with the commit c28224c for that >>>>>> reverted. The adjusted code is here: https://github.com/risdenk/kno >>>>>> x-release/tree/hdp25_revert_KNOX-690. The change is only a single >>>>>> commit https://github.com/risdenk/knox-release/commit/dc452126de99f >>>>>> 6f1d15938f7294e95e3b7c89328 >>>>>> >>>>>> >>>>>> >>>>>> I rebuilt Knox with mvn -DskipTests package and copied the two >>>>>> affected jars (gateway-provider-rewrite and gateway-util-urltemplate) to >>>>>> /usr/hdp/current/knox-server/lib/ >>>>>> >>>>>> >>>>>> >>>>>> I moved the two old jars to /root. The affected jars were >>>>>> >>>>>> · gateway-provider-rewrite-0.9.0.2.5.3.0-37.jar >>>>>> >>>>>> · gateway-util-urltemplate-0.9.0.2.5.3.0-37.jar >>>>>> >>>>>> >>>>>> >>>>>> I then restarted Knox on hdpr05en02. This made the following curl >>>>>> call work: >>>>>> >>>>>> >>>>>> >>>>>> curl -i -k -u USER 'https://KNOXHOST:8443/gateway >>>>>> /TOPOLOGY/hbase/ns:table/%2frkpart1%2frkpart2' >>>>>> >>>>>> >>>>>> >>>>>> *Conclusion* >>>>>> >>>>>> >>>>>> I'm not convinced that KNOX-690 is a good idea but it basically made >>>>>> it so url encoded paths were checked by the templates/parser. URL >>>>>> encoding >>>>>> should be left alone in many cases. Reverting the change from KNOX-690 >>>>>> shouldn't affect us much more other than upgrades to HDP could break >>>>>> this. >>>>>> I think we should really avoid using url encodable characters in the >>>>>> rowkey >>>>>> especially for webhbase. / is a bad character to try to pass through >>>>>> webservices. Since having the customer change rowkey design will be >>>>>> painful, we will be using a workaround in the short term. >>>>>> >>>>>> Kevin Risden >>>>>> >>>>> >>>>> >>>> >>> >> >
