Thank you Sandeep for the quick reply. I created the new class and modified
services.xml for yarnui to point to it.
I could see the class being loaded in the logs but the url encoding is the
same, which tells me that the url is not passing through the new class.
Here is the dispatch tag from service.xml
<dispatch classname="com.gm.hadoop.knox.URLDecodingDispatch" ha-classname="
org.apache.knox.gateway.dispatch.RMHaURLDecodingDispatch"/>
and Here is the code for that classs
package org.apache.knox.gateway.dispatch;
import org.apache.hadoop.gateway.rm.dispatch.RMUIHaDispatch;
import javax.servlet.http.HttpServletRequest;
import java.net.URI;
import java.net.URLDecoder;
import javax.servlet.ServletException;
/**
* Dispatch which decodes the outgoing URLs (to services).
* This is useful in cases where the url is picked up
* from the query parameter and is already encoded.
*
* @since 1.1.0
*/
public class RMHaURLDecodingDispatch extends RMUIHaDispatch {
public RMHaURLDecodingDispatch() throws ServletException{
super();
}
@Override
public URI getDispatchUrl(final HttpServletRequest request) {
String decoded;
try {
decoded =
URLDecoder.decode(request.getRequestURL().toString(), "UTF-8" );
} catch (final Exception e) {
/* fall back in case of exception */
decoded = request.getRequestURL().toString();
}
final StringBuffer str = new StringBuffer(decoded);
final String query = request.getQueryString();
if ( query != null ) {
str.append('?');
str.append(query);
}
final URI url = URI.create(str.toString());
return url;
}
}
Regards,
On Thu, Sep 6, 2018 at 2:36 PM Sandeep Moré <moresand...@gmail.com> wrote:
> You can extend RMUIHaDispatch class and override getDispatchUrl() method
> from URLDecodingDispatch
>
> Best,
> Sandeep
>
> On Thu, Sep 6, 2018 at 2:28 PM Theyaa Matti <theyaama...@gmail.com> wrote:
>
>> Yes I am using HA for resource manager and in that case what should I do
>> to the fix this issue?
>>
>> Best,
>>
>> Theyaa.
>>
>>
>> On Thu, Sep 6, 2018 at 2:24 PM Sandeep Moré <moresand...@gmail.com>
>> wrote:
>>
>>> Are you using HA setup ? in which case org.apache.
>>> hadoop.gateway.rm.dispatch.RMUIHaDispatch dispatch will be used.
>>>
>>> On Thu, Sep 6, 2018 at 2:19 PM Theyaa Matti <theyaama...@gmail.com>
>>> wrote:
>>>
>>>> I created the class as follows:
>>>>
>>>> package org.apache.knox.gateway.dispatch;
>>>>
>>>> import javax.servlet.http.HttpServletRequest;
>>>> import java.net.URI;
>>>> import java.net.URLDecoder;
>>>> import org.apache.hadoop.gateway.dispatch.DefaultDispatch;
>>>>
>>>> /**
>>>> * Dispatch which decodes the outgoing URLs (to services).
>>>> * This is useful in cases where the url is picked up
>>>> * from the query parameter and is already encoded.
>>>> *
>>>> * @since 1.1.0
>>>> */
>>>> public class URLDecodingDispatch extends DefaultDispatch {
>>>>
>>>> public URLDecodingDispatch() {
>>>> super();
>>>> }
>>>>
>>>> @Override
>>>> public URI getDispatchUrl(final HttpServletRequest request) {
>>>> String decoded;
>>>>
>>>> try {
>>>> decoded =
>>>> URLDecoder.decode(request.getRequestURL().toString(), "UTF-8" );
>>>> } catch (final Exception e) {
>>>> /* fall back in case of exception */
>>>> decoded = request.getRequestURL().toString();
>>>> }
>>>>
>>>> final StringBuffer str = new StringBuffer(decoded);
>>>> final String query = request.getQueryString();
>>>> if ( query != null ) {
>>>> str.append('?');
>>>> str.append(query);
>>>> }
>>>> final URI url = URI.create(str.toString());
>>>> return url;
>>>> }
>>>> }
>>>>
>>>>
>>>> packaged it into a jar and uploaded it to the knox lib dir. I can see
>>>> Knox loading the jar file at start:
>>>> resource.FileResource (FileResource.java:checkFileAlias(152)) - ALIAS
>>>> abs=/knox/bin/../lib/knox-1.0-SNAPSHOT.jar
>>>> can=/knox/lib/knox-1.0-SNAPSHOT.jar
>>>>
>>>> I modified the service.xml for yarnui as follows.
>>>>
>>>> <dispatch classname="org.apache.knox.gateway.dispatch.URLDecodingDispatch"
>>>> ha-classname="org.apache.hadoop.gateway.rm.dispatch.RMUIHaDispatch"/>
>>>>
>>>>
>>>>
>>>> Deleted everything under deployments and restarted knox.
>>>>
>>>>
>>>> I do not see any changes in the behavior and do not see that class
>>>> being called. Do you please know what I am missing?
>>>>
>>>>
>>>> Regards,
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On Thu, Sep 6, 2018 at 9:38 AM Sandeep Moré <moresand...@gmail.com>
>>>> wrote:
>>>>
>>>>> URLDecodingDispatch extends DefaultDispatch so you should be there.
>>>>>
>>>>> Best,
>>>>> Sandeep
>>>>>
>>>>> On Thu, Sep 6, 2018 at 9:32 AM Theyaa Matti <theyaama...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> That should be a good idea, but I still have an issue with adding the
>>>>>> custom dispatch to the yarnui service.xml. I already have an existing
>>>>>> class
>>>>>> there <dispatch
>>>>>> classname="org.apache.hadoop.gateway.dispatch.DefaultDispatch"
>>>>>> ha-classname="org.apache.hadoop.gateway.rm.dispatch.RMUIHaDispatch"/>
>>>>>>
>>>>>> Should I replace the DefaultDispatch or I can have more than one
>>>>>> there?
>>>>>>
>>>>>> Regards,
>>>>>>
>>>>>>
>>>>>> On Tue, Sep 4, 2018 at 10:14 PM Dhruv Goyal <777.dh...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> In that case you might have to build the jar yourself or preferably
>>>>>>> use your custom dispatch similar to URLDecodingDispatch.
>>>>>>>
>>>>>>> Regards
>>>>>>> Dhruv
>>>>>>>
>>>>>>> On Wednesday, September 5, 2018, Theyaa Matti <theyaama...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Sorry 0.12.0
>>>>>>>>
>>>>>>>> On Tue, Sep 4, 2018 at 2:30 PM Theyaa Matti <theyaama...@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> I am using Knox 1.12.0
>>>>>>>>>
>>>>>>>>> On Tue, Sep 4, 2018 at 2:28 PM Sandeep Moré <moresand...@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> You can try
>>>>>>>>>> using "org.apache.knox.gateway.dispatch.URLDecodingDispatch"
>>>>>>>>>> dispatch in
>>>>>>>>>> service.xml ( Knox 1.1.0)
>>>>>>>>>>
>>>>>>>>>> Best,
>>>>>>>>>> Sandeep
>>>>>>>>>>
>>>>>>>>>> On Tue, Sep 4, 2018 at 2:09 PM Theyaa Matti <
>>>>>>>>>> theyaama...@gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> I am having issues with Knox encoding the following URL when
>>>>>>>>>>> parsing html content.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> /proxy/application_222233323323_0001/stages/stage?id=0&attempt=0
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> This URL is generated by Yarn to track a running spark job. When
>>>>>>>>>>> applying regular rules to the above URL, Knox encodes the URL to:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> /proxy/application_222233323323
>>>>>>>>>>> _0001/stages/stage?amp%3Battempt=0&id=0
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Which makes the URL unusable and leads to a 404. To make the URL
>>>>>>>>>>> work I have to remove "amp%3B" and then it works.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Below are the filter and the rule I am using for this URL and I
>>>>>>>>>>> appreciate your help/comments for a remediation.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> <filter name="YARNUI/yarn/outbound/apps1">
>>>>>>>>>>> <content type="*/html">
>>>>>>>>>>> <apply path="/proxy/[^']+/stages/stage[^']*"
>>>>>>>>>>> rule="YARNUI/yarn/outbound/apps/history3"/>
>>>>>>>>>>> </content>
>>>>>>>>>>> </filter>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> <rule dir="OUT" name="YARNUI/yarn/outbound/apps/history3">
>>>>>>>>>>> <match pattern="/proxy/{*}/stages/stage?{**}"/>
>>>>>>>>>>> <rewrite
>>>>>>>>>>> template="{$frontend[url]}/yarn/proxy/{*}/stages/stage?{**}"/>
>>>>>>>>>>> </rule>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Thank you for your help.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
