No, I am waiting for the Knox dev team to provide a solution or suggestion to
try.
I followed all provided suggestions and nothing worked
Sent from my iPhone
> On Sep 25, 2018, at 4:41 PM, David Morin <morin.david....@gmail.com> wrote:
>
> I've faced to the same issue with the execution of Spark jobs on Yarn.
> 404 error with url that contains this: "amp%3B"
> Have you succeeded in resolving that ?
>
> Regards,
> David
>
>> Le ven. 7 sept. 2018 à 18:32, Theyaa Matti <theyaama...@gmail.com> a écrit :
>> I have made the necessary modifications to use my custom dispatcher. However
>> I am getting NPE from HttpServletRequest.getRequestURL() when the
>> queryString contains special characters like "amp%3B". In this case the
>> request object is null.
>>
>> Any suggestions?
>>
>> Regards,
>>
>>> On Thu, Sep 6, 2018 at 2:36 PM Sandeep Moré <moresand...@gmail.com> wrote:
>>> You can extend RMUIHaDispatch class and override getDispatchUrl() method
>>> from URLDecodingDispatch
>>>
>>> Best,
>>> Sandeep
>>>
>>>> On Thu, Sep 6, 2018 at 2:28 PM Theyaa Matti <theyaama...@gmail.com> wrote:
>>>> Yes I am using HA for resource manager and in that case what should I do
>>>> to the fix this issue?
>>>>
>>>> Best,
>>>>
>>>> Theyaa.
>>>>
>>>>
>>>>> On Thu, Sep 6, 2018 at 2:24 PM Sandeep Moré <moresand...@gmail.com> wrote:
>>>>> Are you using HA setup ? in which case
>>>>> org.apache.hadoop.gateway.rm.dispatch.RMUIHaDispatch dispatch will be
>>>>> used.
>>>>>
>>>>>> On Thu, Sep 6, 2018 at 2:19 PM Theyaa Matti <theyaama...@gmail.com>
>>>>>> wrote:
>>>>>> I created the class as follows:
>>>>>> package org.apache.knox.gateway.dispatch;
>>>>>>
>>>>>> import javax.servlet.http.HttpServletRequest;
>>>>>> import java.net.URI;
>>>>>> import java.net.URLDecoder;
>>>>>> import org.apache.hadoop.gateway.dispatch.DefaultDispatch;
>>>>>>
>>>>>> /**
>>>>>> * Dispatch which decodes the outgoing URLs (to services).
>>>>>> * This is useful in cases where the url is picked up
>>>>>> * from the query parameter and is already encoded.
>>>>>> *
>>>>>> * @since 1.1.0
>>>>>> */
>>>>>> public class URLDecodingDispatch extends DefaultDispatch {
>>>>>> public URLDecodingDispatch() {
>>>>>> super();
>>>>>> }
>>>>>>
>>>>>> @Override
>>>>>> public URI getDispatchUrl(final HttpServletRequest request) {
>>>>>> String decoded;
>>>>>>
>>>>>> try {
>>>>>> decoded =
>>>>>> URLDecoder.decode(request.getRequestURL().toString(), "UTF-8" );
>>>>>> } catch (final Exception e) {
>>>>>> /* fall back in case of exception */
>>>>>> decoded = request.getRequestURL().toString();
>>>>>> }
>>>>>>
>>>>>> final StringBuffer str = new StringBuffer(decoded);
>>>>>> final String query = request.getQueryString();
>>>>>> if ( query != null ) {
>>>>>> str.append('?');
>>>>>> str.append(query);
>>>>>> }
>>>>>> final URI url = URI.create(str.toString());
>>>>>> return url;
>>>>>> }
>>>>>> }
>>>>>>
>>>>>> packaged it into a jar and uploaded it to the knox lib dir. I can see
>>>>>> Knox loading the jar file at start:
>>>>>> resource.FileResource (FileResource.java:checkFileAlias(152)) - ALIAS
>>>>>> abs=/knox/bin/../lib/knox-1.0-SNAPSHOT.jar
>>>>>> can=/knox/lib/knox-1.0-SNAPSHOT.jar
>>>>>>
>>>>>> I modified the service.xml for yarnui as follows.
>>>>>> <dispatch
>>>>>> classname="org.apache.knox.gateway.dispatch.URLDecodingDispatch"
>>>>>> ha-classname="org.apache.hadoop.gateway.rm.dispatch.RMUIHaDispatch"/>
>>>>>>
>>>>>>
>>>>>> Deleted everything under deployments and restarted knox.
>>>>>>
>>>>>> I do not see any changes in the behavior and do not see that class being
>>>>>> called. Do you please know what I am missing?
>>>>>>
>>>>>> Regards,
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>> On Thu, Sep 6, 2018 at 9:38 AM Sandeep Moré <moresand...@gmail.com>
>>>>>>> wrote:
>>>>>>> URLDecodingDispatch extends DefaultDispatch so you should be there.
>>>>>>>
>>>>>>> Best,
>>>>>>> Sandeep
>>>>>>>
>>>>>>>> On Thu, Sep 6, 2018 at 9:32 AM Theyaa Matti <theyaama...@gmail.com>
>>>>>>>> wrote:
>>>>>>>> That should be a good idea, but I still have an issue with adding the
>>>>>>>> custom dispatch to the yarnui service.xml. I already have an existing
>>>>>>>> class there <dispatch
>>>>>>>> classname="org.apache.hadoop.gateway.dispatch.DefaultDispatch"
>>>>>>>> ha-classname="org.apache.hadoop.gateway.rm.dispatch.RMUIHaDispatch"/>
>>>>>>>>
>>>>>>>> Should I replace the DefaultDispatch or I can have more than one there?
>>>>>>>>
>>>>>>>> Regards,
>>>>>>>>
>>>>>>>>
>>>>>>>>> On Tue, Sep 4, 2018 at 10:14 PM Dhruv Goyal <777.dh...@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>> In that case you might have to build the jar yourself or preferably
>>>>>>>>> use your custom dispatch similar to URLDecodingDispatch.
>>>>>>>>>
>>>>>>>>> Regards
>>>>>>>>> Dhruv
>>>>>>>>>
>>>>>>>>>> On Wednesday, September 5, 2018, Theyaa Matti
>>>>>>>>>> <theyaama...@gmail.com> wrote:
>>>>>>>>>> Sorry 0.12.0
>>>>>>>>>>
>>>>>>>>>>> On Tue, Sep 4, 2018 at 2:30 PM Theyaa Matti <theyaama...@gmail.com>
>>>>>>>>>>> wrote:
>>>>>>>>>>> I am using Knox 1.12.0
>>>>>>>>>>>
>>>>>>>>>>>> On Tue, Sep 4, 2018 at 2:28 PM Sandeep Moré
>>>>>>>>>>>> <moresand...@gmail.com> wrote:
>>>>>>>>>>>> You can try using
>>>>>>>>>>>> "org.apache.knox.gateway.dispatch.URLDecodingDispatch" dispatch in
>>>>>>>>>>>> service.xml ( Knox 1.1.0)
>>>>>>>>>>>>
>>>>>>>>>>>> Best,
>>>>>>>>>>>> Sandeep
>>>>>>>>>>>>
>>>>>>>>>>>>> On Tue, Sep 4, 2018 at 2:09 PM Theyaa Matti
>>>>>>>>>>>>> <theyaama...@gmail.com> wrote:
>>>>>>>>>>>>> I am having issues with Knox encoding the following URL when
>>>>>>>>>>>>> parsing html content.
>>>>>>>>>>>>>
>>>>>>>>>>>>> /proxy/application_222233323323_0001/stages/stage?id=0&attempt=0
>>>>>>>>>>>>>
>>>>>>>>>>>>> This URL is generated by Yarn to track a running spark job. When
>>>>>>>>>>>>> applying regular rules to the above URL, Knox encodes the URL to:
>>>>>>>>>>>>>
>>>>>>>>>>>>> /proxy/application_222233323323_0001/stages/stage?amp%3Battempt=0&id=0
>>>>>>>>>>>>>
>>>>>>>>>>>>> Which makes the URL unusable and leads to a 404. To make the URL
>>>>>>>>>>>>> work I have to remove "amp%3B" and then it works.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Below are the filter and the rule I am using for this URL and I
>>>>>>>>>>>>> appreciate your help/comments for a remediation.
>>>>>>>>>>>>>
>>>>>>>>>>>>> <filter name="YARNUI/yarn/outbound/apps1">
>>>>>>>>>>>>> <content type="*/html">
>>>>>>>>>>>>> <apply path="/proxy/[^']+/stages/stage[^']*"
>>>>>>>>>>>>> rule="YARNUI/yarn/outbound/apps/history3"/>
>>>>>>>>>>>>> </content>
>>>>>>>>>>>>> </filter>
>>>>>>>>>>>>>
>>>>>>>>>>>>> <rule dir="OUT" name="YARNUI/yarn/outbound/apps/history3">
>>>>>>>>>>>>> <match pattern="/proxy/{*}/stages/stage?{**}"/>
>>>>>>>>>>>>> <rewrite
>>>>>>>>>>>>> template="{$frontend[url]}/yarn/proxy/{*}/stages/stage?{**}"/>
>>>>>>>>>>>>> </rule>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thank you for your help.
>>>>>>>>>>>>>
