[jira] [Commented] (YARN-11154) Make router support proxy server.
[ https://issues.apache.org/jira/browse/YARN-11154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17756216#comment-17756216 ] ASF GitHub Bot commented on YARN-11154: --- slfan1989 commented on PR #5946: URL: https://github.com/apache/hadoop/pull/5946#issuecomment-1684698278 @zhengchenyu Thanks for your contribution! @goiri Thank you very much for helping to review the code! > Make router support proxy server. > - > > Key: YARN-11154 > URL: https://issues.apache.org/jira/browse/YARN-11154 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Affects Versions: 3.2.1 >Reporter: zhengchenyu >Assignee: zhengchenyu >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Attachments: YARN-11154.draft.patch > > > Detail message see: https://issues.apache.org/jira/browse/YARN-10775 and > YARN-10775-design-doc.001.pdf -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11154) Make router support proxy server.
[ https://issues.apache.org/jira/browse/YARN-11154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17756215#comment-17756215 ] ASF GitHub Bot commented on YARN-11154: --- slfan1989 merged PR #5946: URL: https://github.com/apache/hadoop/pull/5946 > Make router support proxy server. > - > > Key: YARN-11154 > URL: https://issues.apache.org/jira/browse/YARN-11154 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Affects Versions: 3.2.1 >Reporter: zhengchenyu >Assignee: zhengchenyu >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Attachments: YARN-11154.draft.patch > > > Detail message see: https://issues.apache.org/jira/browse/YARN-10775 and > YARN-10775-design-doc.001.pdf -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11154) Make router support proxy server.
[
https://issues.apache.org/jira/browse/YARN-11154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17754876#comment-17754876
]
ASF GitHub Bot commented on YARN-11154:
---
hadoop-yetus commented on PR #5946:
URL: https://github.com/apache/hadoop/pull/5946#issuecomment-1680002106
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 48s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 21s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 37m 57s | | trunk passed |
| +1 :green_heart: | compile | 8m 45s | | trunk passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | compile | 7m 16s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | checkstyle | 2m 1s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 28s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 28s | | trunk passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 2m 19s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 5m 19s | | trunk passed |
| +1 :green_heart: | shadedclient | 38m 20s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 26s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 39s | | the patch passed |
| +1 :green_heart: | compile | 7m 53s | | the patch passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javac | 7m 53s | | the patch passed |
| +1 :green_heart: | compile | 7m 15s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | javac | 7m 15s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 1m 51s | | the patch passed |
| +1 :green_heart: | mvnsite | 2m 23s | | the patch passed |
| +1 :green_heart: | javadoc | 2m 23s | | the patch passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 3m 3s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 5m 30s | | the patch passed |
| +1 :green_heart: | shadedclient | 39m 48s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 1m 5s | | hadoop-yarn-api in the patch
passed. |
| +1 :green_heart: | unit | 5m 27s | | hadoop-yarn-common in the patch
passed. |
| +1 :green_heart: | unit | 0m 34s | | hadoop-yarn-server-router in
the patch passed. |
| +1 :green_heart: | asflicense | 0m 49s | | The patch does not
generate ASF License warnings. |
| | | 207m 55s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5946/4/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5946 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint |
| uname | Linux e7369dac4792 4.15.0-212-generic #223-Ubuntu SMP Tue May 23
13:09:22 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 0d6adff94a09f16e94dec346f344df0f9f8f16cf |
| Default Java | Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5946/4/testReport/ |
| Max. process+thread count | 542 (vs. ulimit of 5500) |
[jira] [Commented] (YARN-11154) Make router support proxy server.
[
https://issues.apache.org/jira/browse/YARN-11154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17754873#comment-17754873
]
ASF GitHub Bot commented on YARN-11154:
---
hadoop-yetus commented on PR #5946:
URL: https://github.com/apache/hadoop/pull/5946#issuecomment-1679992539
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 52s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 41s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 32m 29s | | trunk passed |
| +1 :green_heart: | compile | 7m 46s | | trunk passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | compile | 7m 16s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | checkstyle | 2m 4s | | trunk passed |
| +1 :green_heart: | mvnsite | 3m 2s | | trunk passed |
| +1 :green_heart: | javadoc | 3m 7s | | trunk passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 2m 54s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 5m 26s | | trunk passed |
| +1 :green_heart: | shadedclient | 34m 57s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 33s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 44s | | the patch passed |
| +1 :green_heart: | compile | 7m 2s | | the patch passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javac | 7m 2s | | the patch passed |
| +1 :green_heart: | compile | 7m 10s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | javac | 7m 10s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 1m 52s | | the patch passed |
| +1 :green_heart: | mvnsite | 2m 46s | | the patch passed |
| +1 :green_heart: | javadoc | 2m 43s | | the patch passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 2m 39s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 5m 39s | | the patch passed |
| +1 :green_heart: | shadedclient | 34m 37s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 1m 18s | | hadoop-yarn-api in the patch
passed. |
| +1 :green_heart: | unit | 5m 52s | | hadoop-yarn-common in the patch
passed. |
| +1 :green_heart: | unit | 0m 48s | | hadoop-yarn-server-router in
the patch passed. |
| +1 :green_heart: | asflicense | 1m 2s | | The patch does not
generate ASF License warnings. |
| | | 197m 3s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5946/3/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5946 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint |
| uname | Linux e6512db2603f 4.15.0-212-generic #223-Ubuntu SMP Tue May 23
13:09:22 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 0d6adff94a09f16e94dec346f344df0f9f8f16cf |
| Default Java | Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5946/3/testReport/ |
| Max. process+thread count | 705 (vs. ulimit of 5500) |
[jira] [Commented] (YARN-11154) Make router support proxy server.
[
https://issues.apache.org/jira/browse/YARN-11154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17754849#comment-17754849
]
ASF GitHub Bot commented on YARN-11154:
---
zhengchenyu commented on code in PR #5946:
URL: https://github.com/apache/hadoop/pull/5946#discussion_r1295313354
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/main/java/org/apache/hadoop/yarn/server/router/clientrm/RouterClientRMService.java:
##
@@ -623,4 +648,22 @@ public void initUserPipelineMap(Configuration conf) {
YarnConfiguration.DEFAULT_ROUTER_PIPELINE_CACHE_MAX_SIZE);
this.userPipelineMap = Collections.synchronizedMap(new
LRUCacheHashMap<>(maxCacheSize, true));
}
+
+ private URL getRedirectURL() throws Exception {
+Configuration conf = getConfig();
+String webAppAddress = WebAppUtils.getWebAppBindURL(conf,
YarnConfiguration.ROUTER_BIND_HOST,
+WebAppUtils.getRouterWebAppURLWithoutScheme(conf));
+String[] hostPort = StringUtils.split(webAppAddress, ':');
+if (hostPort.length != 2) {
+ throw new YarnRuntimeException("Router can't get valid redirect proxy
url");
+}
+String host;
+if (StringUtils.isBlank(hostPort[0]) || hostPort[0].equals("0.0.0.0")) {
+ host = InetAddress.getLocalHost().getCanonicalHostName();
+} else {
+ host = hostPort[0];
+}
+return new URL(YarnConfiguration.useHttps(this.getConfig()) ? "https" :
"http", host,
+Integer.parseInt(hostPort[1]), "");
Review Comment:
@goiri Thanks for your review, I have fix it.
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/main/java/org/apache/hadoop/yarn/server/router/clientrm/RouterClientRMService.java:
##
@@ -623,4 +648,22 @@ public void initUserPipelineMap(Configuration conf) {
YarnConfiguration.DEFAULT_ROUTER_PIPELINE_CACHE_MAX_SIZE);
this.userPipelineMap = Collections.synchronizedMap(new
LRUCacheHashMap<>(maxCacheSize, true));
}
+
+ private URL getRedirectURL() throws Exception {
+Configuration conf = getConfig();
+String webAppAddress = WebAppUtils.getWebAppBindURL(conf,
YarnConfiguration.ROUTER_BIND_HOST,
+WebAppUtils.getRouterWebAppURLWithoutScheme(conf));
+String[] hostPort = StringUtils.split(webAppAddress, ':');
+if (hostPort.length != 2) {
+ throw new YarnRuntimeException("Router can't get valid redirect proxy
url");
+}
+String host;
+if (StringUtils.isBlank(hostPort[0]) || hostPort[0].equals("0.0.0.0")) {
Review Comment:
@goiri Thanks for your review, I have fix it.
> Make router support proxy server.
> -
>
> Key: YARN-11154
> URL: https://issues.apache.org/jira/browse/YARN-11154
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Affects Versions: 3.2.1
>Reporter: zhengchenyu
>Assignee: zhengchenyu
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Attachments: YARN-11154.draft.patch
>
>
> Detail message see: https://issues.apache.org/jira/browse/YARN-10775 and
> YARN-10775-design-doc.001.pdf
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11154) Make router support proxy server.
[
https://issues.apache.org/jira/browse/YARN-11154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17754834#comment-17754834
]
ASF GitHub Bot commented on YARN-11154:
---
zhengchenyu commented on code in PR #5946:
URL: https://github.com/apache/hadoop/pull/5946#discussion_r1295313293
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/main/java/org/apache/hadoop/yarn/server/router/clientrm/RouterClientRMService.java:
##
@@ -623,4 +648,22 @@ public void initUserPipelineMap(Configuration conf) {
YarnConfiguration.DEFAULT_ROUTER_PIPELINE_CACHE_MAX_SIZE);
this.userPipelineMap = Collections.synchronizedMap(new
LRUCacheHashMap<>(maxCacheSize, true));
}
+
+ private URL getRedirectURL() throws Exception {
+Configuration conf = getConfig();
+String webAppAddress = WebAppUtils.getWebAppBindURL(conf,
YarnConfiguration.ROUTER_BIND_HOST,
+WebAppUtils.getRouterWebAppURLWithoutScheme(conf));
+String[] hostPort = StringUtils.split(webAppAddress, ':');
+if (hostPort.length != 2) {
+ throw new YarnRuntimeException("Router can't get valid redirect proxy
url");
+}
+String host;
+if (StringUtils.isBlank(hostPort[0]) || hostPort[0].equals("0.0.0.0")) {
Review Comment:
@goiri Thanks for you review, I have fix it.
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/main/java/org/apache/hadoop/yarn/server/router/clientrm/RouterClientRMService.java:
##
@@ -623,4 +648,22 @@ public void initUserPipelineMap(Configuration conf) {
YarnConfiguration.DEFAULT_ROUTER_PIPELINE_CACHE_MAX_SIZE);
this.userPipelineMap = Collections.synchronizedMap(new
LRUCacheHashMap<>(maxCacheSize, true));
}
+
+ private URL getRedirectURL() throws Exception {
+Configuration conf = getConfig();
+String webAppAddress = WebAppUtils.getWebAppBindURL(conf,
YarnConfiguration.ROUTER_BIND_HOST,
+WebAppUtils.getRouterWebAppURLWithoutScheme(conf));
+String[] hostPort = StringUtils.split(webAppAddress, ':');
+if (hostPort.length != 2) {
+ throw new YarnRuntimeException("Router can't get valid redirect proxy
url");
+}
+String host;
+if (StringUtils.isBlank(hostPort[0]) || hostPort[0].equals("0.0.0.0")) {
+ host = InetAddress.getLocalHost().getCanonicalHostName();
+} else {
+ host = hostPort[0];
+}
+return new URL(YarnConfiguration.useHttps(this.getConfig()) ? "https" :
"http", host,
+Integer.parseInt(hostPort[1]), "");
Review Comment:
@goiri Thanks for you review, I have fix it.
> Make router support proxy server.
> -
>
> Key: YARN-11154
> URL: https://issues.apache.org/jira/browse/YARN-11154
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Affects Versions: 3.2.1
>Reporter: zhengchenyu
>Assignee: zhengchenyu
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Attachments: YARN-11154.draft.patch
>
>
> Detail message see: https://issues.apache.org/jira/browse/YARN-10775 and
> YARN-10775-design-doc.001.pdf
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11154) Make router support proxy server.
[
https://issues.apache.org/jira/browse/YARN-11154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17754708#comment-17754708
]
ASF GitHub Bot commented on YARN-11154:
---
goiri commented on code in PR #5946:
URL: https://github.com/apache/hadoop/pull/5946#discussion_r1294841343
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/main/java/org/apache/hadoop/yarn/server/router/clientrm/RouterClientRMService.java:
##
@@ -623,4 +648,22 @@ public void initUserPipelineMap(Configuration conf) {
YarnConfiguration.DEFAULT_ROUTER_PIPELINE_CACHE_MAX_SIZE);
this.userPipelineMap = Collections.synchronizedMap(new
LRUCacheHashMap<>(maxCacheSize, true));
}
+
+ private URL getRedirectURL() throws Exception {
+Configuration conf = getConfig();
+String webAppAddress = WebAppUtils.getWebAppBindURL(conf,
YarnConfiguration.ROUTER_BIND_HOST,
+WebAppUtils.getRouterWebAppURLWithoutScheme(conf));
+String[] hostPort = StringUtils.split(webAppAddress, ':');
+if (hostPort.length != 2) {
+ throw new YarnRuntimeException("Router can't get valid redirect proxy
url");
+}
+String host;
+if (StringUtils.isBlank(hostPort[0]) || hostPort[0].equals("0.0.0.0")) {
Review Comment:
Probably we can go directly:
```
String host= hostPort[0];
if (StringUtils.isBlank(host) || host.equals("0.0.0.0")) {
host = InetAddress.getLocalHost().getCanonicalHostName();
}
```
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/main/java/org/apache/hadoop/yarn/server/router/clientrm/RouterClientRMService.java:
##
@@ -623,4 +648,22 @@ public void initUserPipelineMap(Configuration conf) {
YarnConfiguration.DEFAULT_ROUTER_PIPELINE_CACHE_MAX_SIZE);
this.userPipelineMap = Collections.synchronizedMap(new
LRUCacheHashMap<>(maxCacheSize, true));
}
+
+ private URL getRedirectURL() throws Exception {
+Configuration conf = getConfig();
+String webAppAddress = WebAppUtils.getWebAppBindURL(conf,
YarnConfiguration.ROUTER_BIND_HOST,
+WebAppUtils.getRouterWebAppURLWithoutScheme(conf));
+String[] hostPort = StringUtils.split(webAppAddress, ':');
+if (hostPort.length != 2) {
+ throw new YarnRuntimeException("Router can't get valid redirect proxy
url");
+}
+String host;
+if (StringUtils.isBlank(hostPort[0]) || hostPort[0].equals("0.0.0.0")) {
+ host = InetAddress.getLocalHost().getCanonicalHostName();
+} else {
+ host = hostPort[0];
+}
+return new URL(YarnConfiguration.useHttps(this.getConfig()) ? "https" :
"http", host,
+Integer.parseInt(hostPort[1]), "");
Review Comment:
Extract the port.
```
String scheme = YarnConfiguration.useHttps(this.getConfig()) ? "https" :
"http";
int port = Integer.parseInt(hostPort[1]), "")
```
> Make router support proxy server.
> -
>
> Key: YARN-11154
> URL: https://issues.apache.org/jira/browse/YARN-11154
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Affects Versions: 3.2.1
>Reporter: zhengchenyu
>Assignee: zhengchenyu
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Attachments: YARN-11154.draft.patch
>
>
> Detail message see: https://issues.apache.org/jira/browse/YARN-10775 and
> YARN-10775-design-doc.001.pdf
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11154) Make router support proxy server.
[
https://issues.apache.org/jira/browse/YARN-11154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17753672#comment-17753672
]
ASF GitHub Bot commented on YARN-11154:
---
hadoop-yetus commented on PR #5946:
URL: https://github.com/apache/hadoop/pull/5946#issuecomment-1676291541
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 28s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 13m 51s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 21m 36s | | trunk passed |
| +1 :green_heart: | compile | 4m 59s | | trunk passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | compile | 4m 24s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | checkstyle | 1m 12s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 8s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 11s | | trunk passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 2m 3s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 3m 34s | | trunk passed |
| +1 :green_heart: | shadedclient | 21m 8s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 23s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 10s | | the patch passed |
| +1 :green_heart: | compile | 4m 16s | | the patch passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javac | 4m 16s | | the patch passed |
| +1 :green_heart: | compile | 4m 18s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | javac | 4m 18s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 1m 6s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 55s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 54s | | the patch passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 51s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 3m 40s | | the patch passed |
| +1 :green_heart: | shadedclient | 21m 34s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 0m 55s | | hadoop-yarn-api in the patch
passed. |
| +1 :green_heart: | unit | 4m 49s | | hadoop-yarn-common in the patch
passed. |
| +1 :green_heart: | unit | 0m 34s | | hadoop-yarn-server-router in
the patch passed. |
| +1 :green_heart: | asflicense | 0m 45s | | The patch does not
generate ASF License warnings. |
| | | 131m 2s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5946/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5946 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint |
| uname | Linux 32959ad2fad1 4.15.0-213-generic #224-Ubuntu SMP Mon Jun 19
13:30:12 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 7172c5245463fbd09315cd6aa96816feaa97e4cd |
| Default Java | Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5946/2/testReport/ |
| Max. process+thread count | 553 (vs. ulimit of 5500) |
[jira] [Commented] (YARN-11154) Make router support proxy server.
[
https://issues.apache.org/jira/browse/YARN-11154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17753665#comment-17753665
]
ASF GitHub Bot commented on YARN-11154:
---
zhengchenyu commented on code in PR #5946:
URL: https://github.com/apache/hadoop/pull/5946#discussion_r1292685883
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/main/java/org/apache/hadoop/yarn/server/router/clientrm/RouterClientRMService.java:
##
@@ -623,4 +650,22 @@ public void initUserPipelineMap(Configuration conf) {
YarnConfiguration.DEFAULT_ROUTER_PIPELINE_CACHE_MAX_SIZE);
this.userPipelineMap = Collections.synchronizedMap(new
LRUCacheHashMap<>(maxCacheSize, true));
}
+
+ private URL getRedirectURL() throws Exception {
+Configuration conf = getConfig();
+String webAppAddress = WebAppUtils.getWebAppBindURL(conf,
YarnConfiguration.ROUTER_BIND_HOST,
+WebAppUtils.getRouterWebAppURLWithoutScheme(conf));
+String[] hostPort = StringUtils.split(webAppAddress, ':');
+if (hostPort.length != 2) {
+ throw new YarnRuntimeException("Router can't get valid redirect proxy
url");
+}
+String host;
+if (null == hostPort[0] || hostPort[0].equals("") ||
hostPort[0].equals("0.0.0.0")) {
Review Comment:
refactor `null == hostPort[0] || hostPort[0].equals("")` to
`StringUtils.isBlank(hostPort[0])`.
Is it ok?
> Make router support proxy server.
> -
>
> Key: YARN-11154
> URL: https://issues.apache.org/jira/browse/YARN-11154
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Affects Versions: 3.2.1
>Reporter: zhengchenyu
>Assignee: zhengchenyu
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Attachments: YARN-11154.draft.patch
>
>
> Detail message see: https://issues.apache.org/jira/browse/YARN-10775 and
> YARN-10775-design-doc.001.pdf
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11154) Make router support proxy server.
[
https://issues.apache.org/jira/browse/YARN-11154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17753664#comment-17753664
]
ASF GitHub Bot commented on YARN-11154:
---
zhengchenyu commented on code in PR #5946:
URL: https://github.com/apache/hadoop/pull/5946#discussion_r1292685883
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/main/java/org/apache/hadoop/yarn/server/router/clientrm/RouterClientRMService.java:
##
@@ -623,4 +650,22 @@ public void initUserPipelineMap(Configuration conf) {
YarnConfiguration.DEFAULT_ROUTER_PIPELINE_CACHE_MAX_SIZE);
this.userPipelineMap = Collections.synchronizedMap(new
LRUCacheHashMap<>(maxCacheSize, true));
}
+
+ private URL getRedirectURL() throws Exception {
+Configuration conf = getConfig();
+String webAppAddress = WebAppUtils.getWebAppBindURL(conf,
YarnConfiguration.ROUTER_BIND_HOST,
+WebAppUtils.getRouterWebAppURLWithoutScheme(conf));
+String[] hostPort = StringUtils.split(webAppAddress, ':');
+if (hostPort.length != 2) {
+ throw new YarnRuntimeException("Router can't get valid redirect proxy
url");
+}
+String host;
+if (null == hostPort[0] || hostPort[0].equals("") ||
hostPort[0].equals("0.0.0.0")) {
Review Comment:
refactor `null == hostPort[0] || hostPort[0].equals("")` to
`StringUtils.isNotBlank(hostPort[0])`.
Is it ok?
> Make router support proxy server.
> -
>
> Key: YARN-11154
> URL: https://issues.apache.org/jira/browse/YARN-11154
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Affects Versions: 3.2.1
>Reporter: zhengchenyu
>Assignee: zhengchenyu
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Attachments: YARN-11154.draft.patch
>
>
> Detail message see: https://issues.apache.org/jira/browse/YARN-10775 and
> YARN-10775-design-doc.001.pdf
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11154) Make router support proxy server.
[
https://issues.apache.org/jira/browse/YARN-11154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17753663#comment-17753663
]
ASF GitHub Bot commented on YARN-11154:
---
zhengchenyu commented on code in PR #5946:
URL: https://github.com/apache/hadoop/pull/5946#discussion_r1292685574
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/main/java/org/apache/hadoop/yarn/server/router/clientrm/RouterClientRMService.java:
##
@@ -318,7 +329,23 @@ public GetClusterNodeLabelsResponse getClusterNodeLabels(
public GetApplicationReportResponse getApplicationReport(
GetApplicationReportRequest request) throws YarnException, IOException {
RequestInterceptorChainWrapper pipeline = getInterceptorChain();
-return pipeline.getRootInterceptor().getApplicationReport(request);
+GetApplicationReportResponse response = pipeline.getRootInterceptor()
+.getApplicationReport(request);
+if (getConfig().getBoolean(YarnConfiguration.ROUTER_WEBAPP_PROXY_ENABLE,
Review Comment:
I will extract a new method.
> Make router support proxy server.
> -
>
> Key: YARN-11154
> URL: https://issues.apache.org/jira/browse/YARN-11154
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Affects Versions: 3.2.1
>Reporter: zhengchenyu
>Assignee: zhengchenyu
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Attachments: YARN-11154.draft.patch
>
>
> Detail message see: https://issues.apache.org/jira/browse/YARN-10775 and
> YARN-10775-design-doc.001.pdf
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11154) Make router support proxy server.
[
https://issues.apache.org/jira/browse/YARN-11154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17753662#comment-17753662
]
ASF GitHub Bot commented on YARN-11154:
---
zhengchenyu commented on code in PR #5946:
URL: https://github.com/apache/hadoop/pull/5946#discussion_r1292685527
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/main/java/org/apache/hadoop/yarn/server/router/clientrm/RouterClientRMService.java:
##
@@ -157,6 +163,11 @@ protected void serviceStart() throws Exception {
YarnConfiguration.DEFAULT_ROUTER_CLIENTRM_ADDRESS,
YarnConfiguration.DEFAULT_ROUTER_CLIENTRM_PORT);
+if (getConfig().getBoolean(YarnConfiguration.ROUTER_WEBAPP_PROXY_ENABLE,
Review Comment:
I will fix it.
> Make router support proxy server.
> -
>
> Key: YARN-11154
> URL: https://issues.apache.org/jira/browse/YARN-11154
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Affects Versions: 3.2.1
>Reporter: zhengchenyu
>Assignee: zhengchenyu
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Attachments: YARN-11154.draft.patch
>
>
> Detail message see: https://issues.apache.org/jira/browse/YARN-10775 and
> YARN-10775-design-doc.001.pdf
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11154) Make router support proxy server.
[
https://issues.apache.org/jira/browse/YARN-11154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17753661#comment-17753661
]
ASF GitHub Bot commented on YARN-11154:
---
zhengchenyu commented on code in PR #5946:
URL: https://github.com/apache/hadoop/pull/5946#discussion_r1292674937
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/main/java/org/apache/hadoop/yarn/server/router/Router.java:
##
@@ -209,9 +215,30 @@ public void startWepApp() {
Builder builder =
WebApps.$for("cluster", null, null, "ws").with(conf).at(webAppAddress);
+if(conf.getBoolean(YarnConfiguration.ROUTER_WEBAPP_PROXY_ENABLE,
+YarnConfiguration.DEFAULT_ROUTER_WEBAPP_PROXY_ENABLE)) {
+ fetcher = new FedAppReportFetcher(conf);
+ builder.withServlet(ProxyUriUtils.PROXY_SERVLET_NAME,
ProxyUriUtils.PROXY_PATH_SPEC,
+ WebAppProxyServlet.class);
+ builder.withAttribute(WebAppProxy.FETCHER_ATTRIBUTE, fetcher);
+ String proxyHostAndPort = getProxyHostAndPort(conf);
+ String[] proxyParts = proxyHostAndPort.split(":");
+ builder.withAttribute(WebAppProxy.PROXY_HOST_ATTRIBUTE, proxyParts[0]);
+}
webApp = builder.start(new RouterWebApp(this));
}
+ public static String getProxyHostAndPort(Configuration conf) {
+String addr = conf.get(YarnConfiguration.PROXY_ADDRESS);
Review Comment:
We do not need a default value. In this function, router web address is just
the default value.
If PROXY_ADDRESS is configured, means that a standalone proxy server is
deployed.
If PROXY_ADDRESS is not configured, means that we will regard router web as
proxy server.
> Make router support proxy server.
> -
>
> Key: YARN-11154
> URL: https://issues.apache.org/jira/browse/YARN-11154
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Affects Versions: 3.2.1
>Reporter: zhengchenyu
>Assignee: zhengchenyu
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Attachments: YARN-11154.draft.patch
>
>
> Detail message see: https://issues.apache.org/jira/browse/YARN-10775 and
> YARN-10775-design-doc.001.pdf
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11154) Make router support proxy server.
[
https://issues.apache.org/jira/browse/YARN-11154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17753645#comment-17753645
]
ASF GitHub Bot commented on YARN-11154:
---
slfan1989 commented on code in PR #5946:
URL: https://github.com/apache/hadoop/pull/5946#discussion_r1292595901
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/main/java/org/apache/hadoop/yarn/server/router/clientrm/RouterClientRMService.java:
##
@@ -318,7 +329,23 @@ public GetClusterNodeLabelsResponse getClusterNodeLabels(
public GetApplicationReportResponse getApplicationReport(
GetApplicationReportRequest request) throws YarnException, IOException {
RequestInterceptorChainWrapper pipeline = getInterceptorChain();
-return pipeline.getRootInterceptor().getApplicationReport(request);
+GetApplicationReportResponse response = pipeline.getRootInterceptor()
+.getApplicationReport(request);
+if (getConfig().getBoolean(YarnConfiguration.ROUTER_WEBAPP_PROXY_ENABLE,
Review Comment:
extend
```
Configuration conf = getConfig();
boolean isWebProxyEnabled =
conf.getBoolean(YarnConfiguration.ROUTER_WEBAPP_PROXY_ENABLE,
YarnConfiguration.DEFAULT_ROUTER_WEBAPP_PROXY_ENABLE);
```
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/main/java/org/apache/hadoop/yarn/server/router/clientrm/RouterClientRMService.java:
##
@@ -318,7 +329,23 @@ public GetClusterNodeLabelsResponse getClusterNodeLabels(
public GetApplicationReportResponse getApplicationReport(
GetApplicationReportRequest request) throws YarnException, IOException {
RequestInterceptorChainWrapper pipeline = getInterceptorChain();
-return pipeline.getRootInterceptor().getApplicationReport(request);
+GetApplicationReportResponse response = pipeline.getRootInterceptor()
+.getApplicationReport(request);
+if (getConfig().getBoolean(YarnConfiguration.ROUTER_WEBAPP_PROXY_ENABLE,
Review Comment:
extend
```
Configuration conf = getConfig();
boolean isWebProxyEnabled =
conf.getBoolean(YarnConfiguration.ROUTER_WEBAPP_PROXY_ENABLE,
YarnConfiguration.DEFAULT_ROUTER_WEBAPP_PROXY_ENABLE);
```
> Make router support proxy server.
> -
>
> Key: YARN-11154
> URL: https://issues.apache.org/jira/browse/YARN-11154
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Affects Versions: 3.2.1
>Reporter: zhengchenyu
>Assignee: zhengchenyu
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Attachments: YARN-11154.draft.patch
>
>
> Detail message see: https://issues.apache.org/jira/browse/YARN-10775 and
> YARN-10775-design-doc.001.pdf
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11154) Make router support proxy server.
[
https://issues.apache.org/jira/browse/YARN-11154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17753644#comment-17753644
]
ASF GitHub Bot commented on YARN-11154:
---
slfan1989 commented on code in PR #5946:
URL: https://github.com/apache/hadoop/pull/5946#discussion_r1292597330
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/main/java/org/apache/hadoop/yarn/server/router/clientrm/RouterClientRMService.java:
##
@@ -623,4 +650,22 @@ public void initUserPipelineMap(Configuration conf) {
YarnConfiguration.DEFAULT_ROUTER_PIPELINE_CACHE_MAX_SIZE);
this.userPipelineMap = Collections.synchronizedMap(new
LRUCacheHashMap<>(maxCacheSize, true));
}
+
+ private URL getRedirectURL() throws Exception {
+Configuration conf = getConfig();
+String webAppAddress = WebAppUtils.getWebAppBindURL(conf,
YarnConfiguration.ROUTER_BIND_HOST,
+WebAppUtils.getRouterWebAppURLWithoutScheme(conf));
+String[] hostPort = StringUtils.split(webAppAddress, ':');
+if (hostPort.length != 2) {
+ throw new YarnRuntimeException("Router can't get valid redirect proxy
url");
+}
+String host;
+if (null == hostPort[0] || hostPort[0].equals("") ||
hostPort[0].equals("0.0.0.0")) {
Review Comment:
Can we refactor this part of the code? This part of the code doesn't look
easy to read.
> Make router support proxy server.
> -
>
> Key: YARN-11154
> URL: https://issues.apache.org/jira/browse/YARN-11154
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Affects Versions: 3.2.1
>Reporter: zhengchenyu
>Assignee: zhengchenyu
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Attachments: YARN-11154.draft.patch
>
>
> Detail message see: https://issues.apache.org/jira/browse/YARN-10775 and
> YARN-10775-design-doc.001.pdf
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11154) Make router support proxy server.
[
https://issues.apache.org/jira/browse/YARN-11154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17753642#comment-17753642
]
ASF GitHub Bot commented on YARN-11154:
---
slfan1989 commented on code in PR #5946:
URL: https://github.com/apache/hadoop/pull/5946#discussion_r1292595901
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/main/java/org/apache/hadoop/yarn/server/router/clientrm/RouterClientRMService.java:
##
@@ -318,7 +329,23 @@ public GetClusterNodeLabelsResponse getClusterNodeLabels(
public GetApplicationReportResponse getApplicationReport(
GetApplicationReportRequest request) throws YarnException, IOException {
RequestInterceptorChainWrapper pipeline = getInterceptorChain();
-return pipeline.getRootInterceptor().getApplicationReport(request);
+GetApplicationReportResponse response = pipeline.getRootInterceptor()
+.getApplicationReport(request);
+if (getConfig().getBoolean(YarnConfiguration.ROUTER_WEBAPP_PROXY_ENABLE,
Review Comment:
extend
```
Configuration conf = getConfig();
boolean isWebProxyEnabled =
conf.getBoolean(YarnConfiguration.ROUTER_WEBAPP_PROXY_ENABLE,
YarnConfiguration.DEFAULT_ROUTER_WEBAPP_PROXY_ENABLE);
```
> Make router support proxy server.
> -
>
> Key: YARN-11154
> URL: https://issues.apache.org/jira/browse/YARN-11154
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Affects Versions: 3.2.1
>Reporter: zhengchenyu
>Assignee: zhengchenyu
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Attachments: YARN-11154.draft.patch
>
>
> Detail message see: https://issues.apache.org/jira/browse/YARN-10775 and
> YARN-10775-design-doc.001.pdf
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11154) Make router support proxy server.
[
https://issues.apache.org/jira/browse/YARN-11154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17753641#comment-17753641
]
ASF GitHub Bot commented on YARN-11154:
---
slfan1989 commented on code in PR #5946:
URL: https://github.com/apache/hadoop/pull/5946#discussion_r1292594570
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/main/java/org/apache/hadoop/yarn/server/router/Router.java:
##
@@ -209,9 +215,30 @@ public void startWepApp() {
Builder builder =
WebApps.$for("cluster", null, null, "ws").with(conf).at(webAppAddress);
+if(conf.getBoolean(YarnConfiguration.ROUTER_WEBAPP_PROXY_ENABLE,
+YarnConfiguration.DEFAULT_ROUTER_WEBAPP_PROXY_ENABLE)) {
+ fetcher = new FedAppReportFetcher(conf);
+ builder.withServlet(ProxyUriUtils.PROXY_SERVLET_NAME,
ProxyUriUtils.PROXY_PATH_SPEC,
+ WebAppProxyServlet.class);
+ builder.withAttribute(WebAppProxy.FETCHER_ATTRIBUTE, fetcher);
+ String proxyHostAndPort = getProxyHostAndPort(conf);
+ String[] proxyParts = proxyHostAndPort.split(":");
+ builder.withAttribute(WebAppProxy.PROXY_HOST_ATTRIBUTE, proxyParts[0]);
+}
webApp = builder.start(new RouterWebApp(this));
}
+ public static String getProxyHostAndPort(Configuration conf) {
+String addr = conf.get(YarnConfiguration.PROXY_ADDRESS);
Review Comment:
Do we need a default value?
> Make router support proxy server.
> -
>
> Key: YARN-11154
> URL: https://issues.apache.org/jira/browse/YARN-11154
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Affects Versions: 3.2.1
>Reporter: zhengchenyu
>Assignee: zhengchenyu
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Attachments: YARN-11154.draft.patch
>
>
> Detail message see: https://issues.apache.org/jira/browse/YARN-10775 and
> YARN-10775-design-doc.001.pdf
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11154) Make router support proxy server.
[ https://issues.apache.org/jira/browse/YARN-11154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17753639#comment-17753639 ] ASF GitHub Bot commented on YARN-11154: --- slfan1989 commented on PR #5946: URL: https://github.com/apache/hadoop/pull/5946#issuecomment-1676167490 @zhengchenyu Thanks for your contribution! we need to add new properties in yarn-default.xml and fix blanks. > Make router support proxy server. > - > > Key: YARN-11154 > URL: https://issues.apache.org/jira/browse/YARN-11154 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Affects Versions: 3.2.1 >Reporter: zhengchenyu >Assignee: zhengchenyu >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Attachments: YARN-11154.draft.patch > > > Detail message see: https://issues.apache.org/jira/browse/YARN-10775 and > YARN-10775-design-doc.001.pdf -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11154) Make router support proxy server.
[
https://issues.apache.org/jira/browse/YARN-11154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17753638#comment-17753638
]
ASF GitHub Bot commented on YARN-11154:
---
slfan1989 commented on code in PR #5946:
URL: https://github.com/apache/hadoop/pull/5946#discussion_r1292578669
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/main/java/org/apache/hadoop/yarn/server/router/clientrm/RouterClientRMService.java:
##
@@ -157,6 +163,11 @@ protected void serviceStart() throws Exception {
YarnConfiguration.DEFAULT_ROUTER_CLIENTRM_ADDRESS,
YarnConfiguration.DEFAULT_ROUTER_CLIENTRM_PORT);
+if (getConfig().getBoolean(YarnConfiguration.ROUTER_WEBAPP_PROXY_ENABLE,
Review Comment:
Let's use `conf`, We have defined this variable.
```
conf.getBoolean(YarnConfiguration.ROUTER_WEBAPP_PROXY_ENABLE
```
> Make router support proxy server.
> -
>
> Key: YARN-11154
> URL: https://issues.apache.org/jira/browse/YARN-11154
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Affects Versions: 3.2.1
>Reporter: zhengchenyu
>Assignee: zhengchenyu
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Attachments: YARN-11154.draft.patch
>
>
> Detail message see: https://issues.apache.org/jira/browse/YARN-10775 and
> YARN-10775-design-doc.001.pdf
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11154) Make router support proxy server.
[
https://issues.apache.org/jira/browse/YARN-11154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17753556#comment-17753556
]
ASF GitHub Bot commented on YARN-11154:
---
hadoop-yetus commented on PR #5946:
URL: https://github.com/apache/hadoop/pull/5946#issuecomment-1675956495
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 29s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 13m 49s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 20m 26s | | trunk passed |
| +1 :green_heart: | compile | 5m 6s | | trunk passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | compile | 4m 20s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | checkstyle | 1m 12s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 24s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 24s | | trunk passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 16s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 2m 22s | | trunk passed |
| +1 :green_heart: | shadedclient | 21m 29s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 25s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 0m 44s | | the patch passed |
| +1 :green_heart: | compile | 4m 16s | | the patch passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javac | 4m 16s | | the patch passed |
| +1 :green_heart: | compile | 4m 18s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | javac | 4m 18s | | the patch passed |
| -1 :x: | blanks | 0m 0s |
[/blanks-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5946/1/artifact/out/blanks-eol.txt)
| The patch has 1 line(s) that end in blanks. Use git apply --whitespace=fix
<>. Refer https://git-scm.com/docs/git-apply |
| +1 :green_heart: | checkstyle | 1m 5s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 15s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 11s | | the patch passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 7s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 2m 27s | | the patch passed |
| +1 :green_heart: | shadedclient | 21m 34s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 0m 56s |
[/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5946/1/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api.txt)
| hadoop-yarn-api in the patch passed. |
| +1 :green_heart: | unit | 0m 33s | | hadoop-yarn-server-router in
the patch passed. |
| +1 :green_heart: | asflicense | 0m 44s | | The patch does not
generate ASF License warnings. |
| | | 117m 57s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.yarn.conf.TestYarnConfigurationFields |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5946/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5946 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 6898274e32b2 4.15.0-213-generic #224-Ubuntu SMP Mon Jun 19
13:30:12 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / e9b947dd5fe2b7ba00a17f5d96146b37e2a01359 |
| Default Java | Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Multi-
[jira] [Commented] (YARN-11154) Make router support proxy server.
[ https://issues.apache.org/jira/browse/YARN-11154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17753549#comment-17753549 ] ASF GitHub Bot commented on YARN-11154: --- zhengchenyu opened a new pull request, #5946: URL: https://github.com/apache/hadoop/pull/5946 ### Description of PR Federation: YARN running app web can't be unable to connect, because AppMaster can't redirect to the right address. Two steps: * YARN-11153: make proxy server support federation. * YARN-11154: make router support proxy server. This PR is the second step. ### How was this patch tested? unit test and running on production cluster more than one year. ### For code changes: * add embed proxy server for router * redirect tracking url to proxy server. > Make router support proxy server. > - > > Key: YARN-11154 > URL: https://issues.apache.org/jira/browse/YARN-11154 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Affects Versions: 3.2.1 >Reporter: zhengchenyu >Assignee: zhengchenyu >Priority: Major > Fix For: 3.4.0 > > Attachments: YARN-11154.draft.patch > > > Detail message see: https://issues.apache.org/jira/browse/YARN-10775 and > YARN-10775-design-doc.001.pdf -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11154) Make router support proxy server.
[ https://issues.apache.org/jira/browse/YARN-11154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17554034#comment-17554034 ] zhengchenyu commented on YARN-11154: [~slfan1989] Hi, I submit a draft patch firstly. After apply YARN-11153, I will submit PR. > Make router support proxy server. > - > > Key: YARN-11154 > URL: https://issues.apache.org/jira/browse/YARN-11154 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Affects Versions: 3.2.1 >Reporter: zhengchenyu >Assignee: zhengchenyu >Priority: Major > Fix For: 3.4.0 > > Attachments: YARN-11154.draft.patch > > > Detail message see: https://issues.apache.org/jira/browse/YARN-10775 and > YARN-10775-design-doc.001.pdf -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
