[
https://issues.apache.org/jira/browse/KNOX-1744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16776334#comment-16776334
]
Nicolas Colomer edited comment on KNOX-1744 at 2/24/19 6:09 PM:
----------------------------------------------------------------
Ok, I managed to find where those URLs in the UI come from.
It's an XHR call on:
{noformat}
https://knox/gateway/emr/sparkhistory/api/v1/applications/application_1549128656351_0631/1/allexecutors{noformat}
It returns the following JSON:
{noformat}
[
{
"id": "1",
...
"executorLogs": {
"stdout":
"http://emr-slave-1:8042/node/containerlogs/container_1549128656351_0631_01_000002/hadoop/stdout?start=-4096";,
"stderr":
"http://emr-slave-1:8042/node/containerlogs/container_1549128656351_0631_01_000002/hadoop/stderr?start=-4096";
}
},
{
"id": "driver",
...
"executorLogs": {
"stdout":
"http://emr-slave-2:8042/node/containerlogs/container_1549128656351_0631_01_000001/hadoop/stdout?start=-4096";,
"stderr":
"http://emr-slave-2:8042/node/containerlogs/container_1549128656351_0631_01_000001/hadoop/stderr?start=-4096";
}
}
]{noformat}
The URLs are extracted by some JS scripts and printed on the page.
I'd like to rewrite those URLs, by filtering on this JSON result and applying
dedicated rule on the *executorLogs.(stdout|stderr)* fields. I iterated but did
not succeed to make it run yet.
I ended with the following lines (inspired by
[yarnui/2.7.0/rewrite.xml|https://github.com/apache/knox/blob/master/gateway-service-definitions/src/main/resources/services/yarnui/2.7.0/rewrite.xml]
and
[yarn-rm/2.5.0/rewrite.xml|https://github.com/apache/knox/blob/master/gateway-service-definitions/src/main/resources/services/yarn-rm/2.5.0/rewrite.xml]
files), added in the end of the
[sparkhistoryui/2.3.0/rewrite.xml|https://github.com/apache/knox/blob/master/gateway-service-definitions/src/main/resources/services/sparkhistoryui/2.3.0/rewrite.xml]
file:
{noformat}
<!-- Yarn application containerlog outbound routing -->
<filter
name="SPARKHISTORYUI/sparkhistory/outbound/executors/containerlogs/filter">
<content type="*/json">
<apply path="$[*].executorLogs.stdout"
rule="SPARKHISTORYUI/sparkhistory/outbound/executors/containerlogs"/>
<apply path="$[*].executorLogs.stderr"
rule="SPARKHISTORYUI/sparkhistory/outbound/executors/containerlogs"/>
</content>
</filter>
<rule dir="OUT"
name="SPARKHISTORYUI/sparkhistory/outbound/executors/containerlogs">
<match pattern="{scheme}://{host}:{port}/node/containerlogs/{**}"/>
<rewrite
template="{$frontend[url]}/yarn/nodemanager/node/containerlogs/{**}?{scheme}?host={$hostmap(host)}?{port}"/>
</rule>{noformat}
cc [~anikam] [~smore]
was (Author: ncolomer):
Ok, I managed to find where those URLs in the UI come from.
It's an XHR call on:
{noformat}
https://knox/gateway/emr/sparkhistory/api/v1/applications/application_1549128656351_0631/1/allexecutors{noformat}
It returns the following JSON:
{noformat}
[
{
"id": "1",
...
"executorLogs": {
"stdout":
"http://emr-slave-1:8042/node/containerlogs/container_1549128656351_0631_01_000002/hadoop/stdout?start=-4096";,
"stderr":
"http://emr-slave-1:8042/node/containerlogs/container_1549128656351_0631_01_000002/hadoop/stderr?start=-4096";
}
},
{
"id": "driver",
...
"executorLogs": {
"stdout":
"http://emr-slave-2:8042/node/containerlogs/container_1549128656351_0631_01_000001/hadoop/stdout?start=-4096";,
"stderr":
"http://emr-slave-2:8042/node/containerlogs/container_1549128656351_0631_01_000001/hadoop/stderr?start=-4096";
}
}
]{noformat}
The URLs are extracted by some JS scripts and printed on the page.
I'd like to rewrite those URLs, by filtering on this JSON result and applying
dedicated rule on the *executorLogs.(stdout|stderr)* fields. I iterated but did
not succeed to make it run yet.
I ended with the following lines, added in the end of the
[sparkhistoryui/2.3.0/rewrite.xml|https://github.com/apache/knox/blob/master/gateway-service-definitions/src/main/resources/services/sparkhistoryui/2.3.0/rewrite.xml]
file:
{noformat}
<!-- Yarn application containerlog outbound routing -->
<filter
name="SPARKHISTORYUI/sparkhistory/outbound/executors/containerlogs/filter">
<content type="*/json">
<apply path="$[*].executorLogs.stdout"
rule="SPARKHISTORYUI/sparkhistory/outbound/executors/containerlogs"/>
<apply path="$[*].executorLogs.stderr"
rule="SPARKHISTORYUI/sparkhistory/outbound/executors/containerlogs"/>
</content>
</filter>
<rule dir="OUT"
name="SPARKHISTORYUI/sparkhistory/outbound/executors/containerlogs">
<match pattern="{scheme}://{host}:{port}/node/containerlogs/{**}"/>
<rewrite
template="{$frontend[url]}/yarn/nodemanager/node/containerlogs/{**}?{scheme}?host={$hostmap(host)}?{port}"/>
</rule>{noformat}
cc [~anikam] [~smore]
> [Spark History UI Service] Executor logs (stdout/stderr) links are broken
> --------------------------------------------------------------------------
>
> Key: KNOX-1744
> URL: https://issues.apache.org/jira/browse/KNOX-1744
> Project: Apache Knox
> Issue Type: Bug
> Affects Versions: 1.2.0
> Reporter: Nicolas Colomer
> Priority: Minor
> Attachments: image-2019-01-22-13-04-53-935.png,
> image-2019-01-22-13-11-55-412.png
>
>
> The executor logs (stdout/stderr) links from the Yarn UI Service all directly
> points to the slaves nodes.
> !image-2019-01-22-13-11-55-412.png!
> Eg. the link:
> {noformat}
> http://emr-slave-hostname:8042/node/containerlogs/container_1547901130703_0034_01_000001/hadoop/stdout?start=-4096{noformat}
> Should have been instead:
> {noformat}
> https://knox/gateway/cluster/yarn/nodemanager/node/containerlogs/container_1547901130703_0034_01_000001/hadoop?scheme=http&host=emr-slave-hostname&port=8042{noformat}
> And finally respect pattern:
> {noformat}
> https://knox/gateway/emr/yarn/nodemanager/node/containerlogs/{your container
> id}/hadoop?scheme=http&host={executor host}&port=8042 {noformat}
> I guess adding some voodoo in the Knox yarnui's
> [rewrite.xml|https://github.com/apache/knox/blob/master/gateway-service-definitions/src/main/resources/services/yarnui/2.7.0/rewrite.xml]
> file might be enough to fix this.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
