I've faced to the same issue with the execution of Spark jobs on Yarn. 404 error with url that contains this: "amp%3B" Have you succeeded in resolving that ?
Regards, David Le ven. 7 sept. 2018 à 18:32, Theyaa Matti <[email protected]> a écrit : > I have made the necessary modifications to use my custom dispatcher. > However I am getting NPE from HttpServletRequest.getRequestURL() when the > queryString contains special characters like "amp%3B". In this case the > request object is null. > > Any suggestions? > > Regards, > > On Thu, Sep 6, 2018 at 2:36 PM Sandeep Moré <[email protected]> wrote: > >> You can extend RMUIHaDispatch class and override getDispatchUrl() method >> from URLDecodingDispatch >> >> Best, >> Sandeep >> >> On Thu, Sep 6, 2018 at 2:28 PM Theyaa Matti <[email protected]> >> wrote: >> >>> Yes I am using HA for resource manager and in that case what should I do >>> to the fix this issue? >>> >>> Best, >>> >>> Theyaa. >>> >>> >>> On Thu, Sep 6, 2018 at 2:24 PM Sandeep Moré <[email protected]> >>> wrote: >>> >>>> Are you using HA setup ? in which case org.apache. >>>> hadoop.gateway.rm.dispatch.RMUIHaDispatch dispatch will be used. >>>> >>>> On Thu, Sep 6, 2018 at 2:19 PM Theyaa Matti <[email protected]> >>>> wrote: >>>> >>>>> I created the class as follows: >>>>> >>>>> package org.apache.knox.gateway.dispatch; >>>>> >>>>> import javax.servlet.http.HttpServletRequest; >>>>> import java.net.URI; >>>>> import java.net.URLDecoder; >>>>> import org.apache.hadoop.gateway.dispatch.DefaultDispatch; >>>>> >>>>> /** >>>>> * Dispatch which decodes the outgoing URLs (to services). >>>>> * This is useful in cases where the url is picked up >>>>> * from the query parameter and is already encoded. >>>>> * >>>>> * @since 1.1.0 >>>>> */ >>>>> public class URLDecodingDispatch extends DefaultDispatch { >>>>> >>>>> public URLDecodingDispatch() { >>>>> super(); >>>>> } >>>>> >>>>> @Override >>>>> public URI getDispatchUrl(final HttpServletRequest request) { >>>>> String decoded; >>>>> >>>>> try { >>>>> decoded = >>>>> URLDecoder.decode(request.getRequestURL().toString(), "UTF-8" ); >>>>> } catch (final Exception e) { >>>>> /* fall back in case of exception */ >>>>> decoded = request.getRequestURL().toString(); >>>>> } >>>>> >>>>> final StringBuffer str = new StringBuffer(decoded); >>>>> final String query = request.getQueryString(); >>>>> if ( query != null ) { >>>>> str.append('?'); >>>>> str.append(query); >>>>> } >>>>> final URI url = URI.create(str.toString()); >>>>> return url; >>>>> } >>>>> } >>>>> >>>>> >>>>> packaged it into a jar and uploaded it to the knox lib dir. I can see >>>>> Knox loading the jar file at start: >>>>> resource.FileResource (FileResource.java:checkFileAlias(152)) - ALIAS >>>>> abs=/knox/bin/../lib/knox-1.0-SNAPSHOT.jar >>>>> can=/knox/lib/knox-1.0-SNAPSHOT.jar >>>>> >>>>> I modified the service.xml for yarnui as follows. >>>>> >>>>> <dispatch classname="org.apache.knox.gateway.dispatch.URLDecodingDispatch" >>>>> ha-classname="org.apache.hadoop.gateway.rm.dispatch.RMUIHaDispatch"/> >>>>> >>>>> >>>>> >>>>> Deleted everything under deployments and restarted knox. >>>>> >>>>> >>>>> I do not see any changes in the behavior and do not see that class >>>>> being called. Do you please know what I am missing? >>>>> >>>>> >>>>> Regards, >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> On Thu, Sep 6, 2018 at 9:38 AM Sandeep Moré <[email protected]> >>>>> wrote: >>>>> >>>>>> URLDecodingDispatch extends DefaultDispatch so you should be there. >>>>>> >>>>>> Best, >>>>>> Sandeep >>>>>> >>>>>> On Thu, Sep 6, 2018 at 9:32 AM Theyaa Matti <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> That should be a good idea, but I still have an issue with adding >>>>>>> the custom dispatch to the yarnui service.xml. I already have an >>>>>>> existing >>>>>>> class there <dispatch >>>>>>> classname="org.apache.hadoop.gateway.dispatch.DefaultDispatch" >>>>>>> ha-classname="org.apache.hadoop.gateway.rm.dispatch.RMUIHaDispatch" >>>>>>> /> >>>>>>> >>>>>>> Should I replace the DefaultDispatch or I can have more than one >>>>>>> there? >>>>>>> >>>>>>> Regards, >>>>>>> >>>>>>> >>>>>>> On Tue, Sep 4, 2018 at 10:14 PM Dhruv Goyal <[email protected]> >>>>>>> wrote: >>>>>>> >>>>>>>> In that case you might have to build the jar yourself or preferably >>>>>>>> use your custom dispatch similar to URLDecodingDispatch. >>>>>>>> >>>>>>>> Regards >>>>>>>> Dhruv >>>>>>>> >>>>>>>> On Wednesday, September 5, 2018, Theyaa Matti < >>>>>>>> [email protected]> wrote: >>>>>>>> >>>>>>>>> Sorry 0.12.0 >>>>>>>>> >>>>>>>>> On Tue, Sep 4, 2018 at 2:30 PM Theyaa Matti <[email protected]> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> I am using Knox 1.12.0 >>>>>>>>>> >>>>>>>>>> On Tue, Sep 4, 2018 at 2:28 PM Sandeep Moré < >>>>>>>>>> [email protected]> wrote: >>>>>>>>>> >>>>>>>>>>> You can try >>>>>>>>>>> using "org.apache.knox.gateway.dispatch.URLDecodingDispatch" >>>>>>>>>>> dispatch in >>>>>>>>>>> service.xml ( Knox 1.1.0) >>>>>>>>>>> >>>>>>>>>>> Best, >>>>>>>>>>> Sandeep >>>>>>>>>>> >>>>>>>>>>> On Tue, Sep 4, 2018 at 2:09 PM Theyaa Matti < >>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>> >>>>>>>>>>>> I am having issues with Knox encoding the following URL when >>>>>>>>>>>> parsing html content. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> /proxy/application_222233323323_0001/stages/stage?id=0&attempt=0 >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> This URL is generated by Yarn to track a running spark job. >>>>>>>>>>>> When applying regular rules to the above URL, Knox encodes the URL >>>>>>>>>>>> to: >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> /proxy/application_222233323323 >>>>>>>>>>>> _0001/stages/stage?amp%3Battempt=0&id=0 >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Which makes the URL unusable and leads to a 404. To make the >>>>>>>>>>>> URL work I have to remove "amp%3B" and then it works. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Below are the filter and the rule I am using for this URL and I >>>>>>>>>>>> appreciate your help/comments for a remediation. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> <filter name="YARNUI/yarn/outbound/apps1"> >>>>>>>>>>>> <content type="*/html"> >>>>>>>>>>>> <apply path="/proxy/[^']+/stages/stage[^']*" >>>>>>>>>>>> rule="YARNUI/yarn/outbound/apps/history3"/> >>>>>>>>>>>> </content> >>>>>>>>>>>> </filter> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> <rule dir="OUT" name="YARNUI/yarn/outbound/apps/history3"> >>>>>>>>>>>> <match pattern="/proxy/{*}/stages/stage?{**}"/> >>>>>>>>>>>> <rewrite >>>>>>>>>>>> template="{$frontend[url]}/yarn/proxy/{*}/stages/stage?{**}"/> >>>>>>>>>>>> </rule> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Thank you for your help. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>
