[jira] [Commented] (YARN-11307) Fix Yarn Router Broken Link
[ https://issues.apache.org/jira/browse/YARN-11307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17606923#comment-17606923 ] ASF GitHub Bot commented on YARN-11307: --- slfan1989 commented on PR #4905: URL: https://github.com/apache/hadoop/pull/4905#issuecomment-1251875424 @dineshchitlangia Please help to review the code again, thank you very much! > Fix Yarn Router Broken Link > --- > > Key: YARN-11307 > URL: https://issues.apache.org/jira/browse/YARN-11307 > Project: Hadoop YARN > Issue Type: Improvement > Components: federation, router >Affects Versions: 3.4.0 >Reporter: fanshilun >Assignee: fanshilun >Priority: Minor > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11308) Router Page display the db username and password in mask mode
[
https://issues.apache.org/jira/browse/YARN-11308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17606921#comment-17606921
]
ASF GitHub Bot commented on YARN-11308:
---
hadoop-yetus commented on PR #4908:
URL: https://github.com/apache/hadoop/pull/4908#issuecomment-1251873319
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 9s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 46m 32s | | trunk passed |
| +1 :green_heart: | compile | 26m 49s | | trunk passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 24m 7s | | trunk passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 1m 26s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 58s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 29s | | trunk passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 56s | | trunk passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 3m 31s | | trunk passed |
| +1 :green_heart: | shadedclient | 29m 0s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 18s | | the patch passed |
| +1 :green_heart: | compile | 29m 37s | | the patch passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 29m 37s | | the patch passed |
| +1 :green_heart: | compile | 23m 50s | | the patch passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 23m 50s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 1m 17s | | the patch passed |
| +1 :green_heart: | mvnsite | 2m 4s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 19s | | the patch passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 4s | | the patch passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 3m 34s | | the patch passed |
| +1 :green_heart: | shadedclient | 28m 49s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 20m 53s |
[/patch-unit-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4908/2/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt)
| hadoop-common in the patch passed. |
| +1 :green_heart: | asflicense | 1m 17s | | The patch does not
generate ASF License warnings. |
| | | 251m 22s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.metrics2.sink.TestPrometheusMetricsSink |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4908/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4908 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux e056d4e6f490 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4
01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / bfc51143a6b91a23be860d5145677fadb00f87a2 |
| Default Java | Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4908/2/testReport/ |
| Max. process+thread count | 2463 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output
[jira] [Commented] (YARN-11307) Fix Yarn Router Broken Link
[
https://issues.apache.org/jira/browse/YARN-11307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17606891#comment-17606891
]
ASF GitHub Bot commented on YARN-11307:
---
hadoop-yetus commented on PR #4905:
URL: https://github.com/apache/hadoop/pull/4905#issuecomment-1251846737
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 57s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 1s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 47m 3s | | trunk passed |
| +1 :green_heart: | compile | 0m 39s | | trunk passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 0m 35s | | trunk passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 0m 36s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 40s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 43s | | trunk passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 31s | | trunk passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 1m 8s | | trunk passed |
| +1 :green_heart: | shadedclient | 23m 19s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 26s | | the patch passed |
| +1 :green_heart: | compile | 0m 27s | | the patch passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 0m 27s | | the patch passed |
| +1 :green_heart: | compile | 0m 25s | | the patch passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 0m 25s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 19s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 27s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 23s | | the patch passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 21s | | the patch passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 0m 56s | | the patch passed |
| +1 :green_heart: | shadedclient | 23m 28s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 3m 40s | | hadoop-yarn-server-router in
the patch passed. |
| +1 :green_heart: | asflicense | 0m 39s | | The patch does not
generate ASF License warnings. |
| | | 109m 30s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4905/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4905 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux b65cc02fdc6b 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4
01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / a6479d74415d0b15c233821bcba129478bc8224a |
| Default Java | Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4905/2/testReport/ |
| Max. process+thread count | 681 (vs. ulimit of 5500) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4905/2/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
[jira] [Commented] (YARN-11305) Fix TestLogAggregationService#testLocalFileDeletionAfterUpload Failed After YARN-11241(#4703)
[
https://issues.apache.org/jira/browse/YARN-11305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17606861#comment-17606861
]
ASF GitHub Bot commented on YARN-11305:
---
slfan1989 commented on PR #4893:
URL: https://github.com/apache/hadoop/pull/4893#issuecomment-1251794396
@ayushtkn Please help to review the code again, thank you very much!
> Fix TestLogAggregationService#testLocalFileDeletionAfterUpload Failed After
> YARN-11241(#4703)
> -
>
> Key: YARN-11305
> URL: https://issues.apache.org/jira/browse/YARN-11305
> Project: Hadoop YARN
> Issue Type: Bug
> Components: log-aggregation
>Affects Versions: 3.4.0
>Reporter: fanshilun
>Assignee: fanshilun
>Priority: Major
> Labels: pull-request-available
>
> Yarn's unit tests, with parallel mode enabled
> {code:java}
> /usr/bin/mvn --batch-mode
> -Dmaven.repo.local=/home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-4846/yetus-m2/hadoop-trunk-patch-0
> -Ptest-patch -Dsurefire.rerunFailingTestsCount=2 -Pparallel-tests
> -P!shelltest -Pnative -Drequire.fuse -Drequire.openssl -Drequire.snappy
> -Drequire.valgrind -Drequire.zstd -Drequire.test.libhadoop -Pyarn-ui clean
> test -fae{code}
> Because the automatically generated application_id is the same,
> *testLocalFileDeletionAfterUpload* is affected by
> *testLocalFileRemainsAfterUploadOnCleanupDisable*
> testLocalFileDeletionAfterUpload needs to check that the log of an
> application is deleted, and testLocalFileRemainsAfterUploadOnCleanupDisable
> regenerates the log of the same application.
> Error Message
> {code:java}
> Directory
> [/home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-4846/ubuntu-focal/src/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/target/TestLogAggregationService-localLogDir/application_1234_0001]
> was not deleted {code}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11306) Refactor NM#FederationInterceptor#recover Code
[
https://issues.apache.org/jira/browse/YARN-11306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17606855#comment-17606855
]
ASF GitHub Bot commented on YARN-11306:
---
slfan1989 commented on code in PR #4897:
URL: https://github.com/apache/hadoop/pull/4897#discussion_r974848893
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/TestFederationInterceptor.java:
##
@@ -1242,4 +1255,130 @@ public void testRemoveAppFromRegistryApplicationFailed()
return null;
});
}
+
+ public void testRecoverWithBadSubCluster(final RegistryOperations
registryObj) throws IOException, InterruptedException {
+UserGroupInformation ugi =
+interceptor.getUGIWithToken(interceptor.getAttemptId());
+
+ugi.doAs((PrivilegedExceptionAction) () -> {
Review Comment:
Thanks for your help reviewing the code, I will refactor this part of the
code.
> Refactor NM#FederationInterceptor#recover Code
> --
>
> Key: YARN-11306
> URL: https://issues.apache.org/jira/browse/YARN-11306
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: federation, nodemanager
>Affects Versions: 3.4.0
>Reporter: fanshilun
>Assignee: fanshilun
>Priority: Major
> Labels: pull-request-available
>
> Refactor NM#FederationInterceptor#recover Code
> 1.Enhance code readability
> 2.Add empty check
> 3.When an exception is encountered, completely destroy the data of SubCluster
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11255) Support loading alternative docker client config from system environment
[
https://issues.apache.org/jira/browse/YARN-11255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17606854#comment-17606854
]
ASF GitHub Bot commented on YARN-11255:
---
ashutoshcipher commented on PR #4884:
URL: https://github.com/apache/hadoop/pull/4884#issuecomment-1251784604
Thanks @PrabhuJoseph for your reviewing and approving.
> Support loading alternative docker client config from system environment
>
>
> Key: YARN-11255
> URL: https://issues.apache.org/jira/browse/YARN-11255
> Project: Hadoop YARN
> Issue Type: New Feature
>Reporter: Ashutosh Gupta
>Assignee: Ashutosh Gupta
>Priority: Major
> Labels: pull-request-available
>
> When using YARN docker support, although the hadoop shell supported
> {code:java}
> -docker_client_config{code}
> to pass the client config file that contains security token to generate the
> docker config for each job as a temporary file.
> For other applications that submit jobs to YARN, e.g. Spark, which loads the
> docker setting via system environment e.g.
> {code:java}
> spark.executorEnv.* {code}
> will not be able to add those authorization token because this system
> environment isn't considered in YARN.
> Add genetic solution to handle these kind of cases without making changes in
> spark code or others
> Eg
> When using remote container registry, the
> {{YARN_CONTAINER_RUNTIME_DOCKER_CLIENT_CONFIG}} must reference the config.json
> file containing the credentials used to authenticate.
> {code:java}
> DOCKER_IMAGE_NAME=hadoop-docker
> DOCKER_CLIENT_CONFIG=hdfs:///user/hadoop/config.json
> spark-submit --master yarn \
> --deploy-mode cluster \
> --conf spark.executorEnv.YARN_CONTAINER_RUNTIME_TYPE=docker \
> --conf
> spark.executorEnv.YARN_CONTAINER_RUNTIME_DOCKER_IMAGE=$DOCKER_IMAGE_NAME \
> --conf
> spark.executorEnv.YARN_CONTAINER_RUNTIME_DOCKER_CLIENT_CONFIG=$DOCKER_CLIENT_CONFIG
> \
> --conf spark.yarn.appMasterEnv.YARN_CONTAINER_RUNTIME_TYPE=docker \
> --conf
> spark.yarn.appMasterEnv.YARN_CONTAINER_RUNTIME_DOCKER_IMAGE=$DOCKER_IMAGE_NAME
> \
> --conf
> spark.yarn.appMasterEnv.YARN_CONTAINER_RUNTIME_DOCKER_CLIENT_CONFIG=$DOCKER_CLIENT_CONFIG
> \
> sparkR.R{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11307) Fix Yarn Router Broken Link
[ https://issues.apache.org/jira/browse/YARN-11307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17606846#comment-17606846 ] ASF GitHub Bot commented on YARN-11307: --- slfan1989 commented on PR #4905: URL: https://github.com/apache/hadoop/pull/4905#issuecomment-1251782120 > Overall this looks good. @slfan1989 could you pls address the 6 new checkstyle warnings generated by your patch? > > Thank you for your contribution. @dineshchitlangia Thank you very much for helping to review the code, I will modify the code. > Fix Yarn Router Broken Link > --- > > Key: YARN-11307 > URL: https://issues.apache.org/jira/browse/YARN-11307 > Project: Hadoop YARN > Issue Type: Improvement > Components: federation, router >Affects Versions: 3.4.0 >Reporter: fanshilun >Assignee: fanshilun >Priority: Minor > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11303) Upgrade jquery ui to 1.13.2
[ https://issues.apache.org/jira/browse/YARN-11303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17606844#comment-17606844 ] ASF GitHub Bot commented on YARN-11303: --- ashutoshcipher commented on PR #4895: URL: https://github.com/apache/hadoop/pull/4895#issuecomment-1251779567 Thanks @goiri for reviewing and approving. We do that have a clean build except the following and I hope it should be fine for this PR. ``` -1 ❌ | test4tests | 0m 0s | | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. > Upgrade jquery ui to 1.13.2 > --- > > Key: YARN-11303 > URL: https://issues.apache.org/jira/browse/YARN-11303 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: D M Murali Krishna Reddy >Assignee: Ashutosh Gupta >Priority: Major > Labels: pull-request-available > > The current jquery-ui version used(1.13.1) in the trunk has the following > vulnerability > [CVE-2022-31160|https://nvd.nist.gov/vuln/detail/CVE-2022-31160] so we need > to upgrade to at least 1.13.2. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11308) Router Page display the db username and password in mask mode
[
https://issues.apache.org/jira/browse/YARN-11308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17606842#comment-17606842
]
ASF GitHub Bot commented on YARN-11308:
---
slfan1989 commented on code in PR #4908:
URL: https://github.com/apache/hadoop/pull/4908#discussion_r974842049
##
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/ConfServlet.java:
##
@@ -43,13 +48,18 @@ public class ConfServlet extends HttpServlet {
protected static final String FORMAT_JSON = "json";
protected static final String FORMAT_XML = "xml";
+
/**
* Return the Configuration of the daemon hosting this servlet.
* This is populated when the HttpServer starts.
*/
private Configuration getConfFromContext() {
Configuration conf = (Configuration)getServletContext().getAttribute(
HttpServer2.CONF_CONTEXT_ATTRIBUTE);
+List props = new ArrayList<>();
+props.add(FEDERATION_STATESTORE_SQL_USERNAME);
Review Comment:
I agree with you, I will refactor part of the code to make this part of the
ability more general.
> Router Page display the db username and password in mask mode
> -
>
> Key: YARN-11308
> URL: https://issues.apache.org/jira/browse/YARN-11308
> Project: Hadoop YARN
> Issue Type: Bug
> Components: federation
>Affects Versions: 3.4.0
>Reporter: fanshilun
>Assignee: fanshilun
>Priority: Major
> Labels: pull-request-available
> Attachments: image-2022-09-19-17-33-44-585.png,
> image-2022-09-19-17-35-02-471.png
>
>
> When using YRAN-Federation's SQLFederationStateStore, we need to configure
> yarn.federation.state-store.sql.username,
> yarn.federation.state-store.sql.password in the configuration file, When
> viewing Conf on the Router page, the user name and password are displayed in
> plaintext, which will bring security risks. We should display it in the form
> of a mask.
>
> before fixing
> {code:java}
>
>yarn.federation.state-store.sql.username
>federation
>false
>yarn-site.xml
>
>
>yarn.federation.state-store.sql.password
>federation123
>false
>yarn-site.xml
> {code}
> after fixing
> {code:java}
>
> yarn.federation.state-store.sql.username
> **
> false
> yarn-site.xml
>
>
> yarn.federation.state-store.sql.password
> **
> false
> yarn-site.xml
> {code}
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11308) Router Page display the db username and password in mask mode
[
https://issues.apache.org/jira/browse/YARN-11308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17606840#comment-17606840
]
ASF GitHub Bot commented on YARN-11308:
---
slfan1989 commented on code in PR #4908:
URL: https://github.com/apache/hadoop/pull/4908#discussion_r974841557
##
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/http/HttpServer2.java:
##
@@ -214,6 +215,12 @@ public final class HttpServer2 implements FilterContainer {
private StatisticsHandler statsHandler;
private HttpServer2Metrics metrics;
+ private static final String MASK = "**";
+ public static final String FEDERATION_STATESTORE_SQL_USERNAME =
+ "yarn.federation.state-store.sql.username";
+ public static final String FEDERATION_STATESTORE_SQL_PASSWROD =
Review Comment:
Your suggestion is very good, I read the code and I found that
`ConfigReactor` provides the ability to provide masks for key configurations.
When `ConfigReactor` is initialized, it will read some configuration
sensitive keywords. When encountering these keywords, it will return the value
in mask mode. But `ConfigReactor` currently only supports json configuration,
I will add a new method to support xml configuration
```
public ConfigRedactor(Configuration conf) {
String sensitiveRegexList = conf.get(
HADOOP_SECURITY_SENSITIVE_CONFIG_KEYS,
HADOOP_SECURITY_SENSITIVE_CONFIG_KEYS_DEFAULT);
List sensitiveRegexes =
Arrays.asList(StringUtils.getTrimmedStrings(sensitiveRegexList));
compiledPatterns = new ArrayList();
for (String regex : sensitiveRegexes) {
Pattern p = Pattern.compile(regex);
compiledPatterns.add(p);
}
}
public static final String HADOOP_SECURITY_SENSITIVE_CONFIG_KEYS_DEFAULT =
String.join(",",
"secret$",
"password$",
"username$",
"ssl.keystore.pass$",
"fs.s3.*[Ss]ecret.?[Kk]ey",
"fs.s3a.*.server-side-encryption.key",
"fs.s3a.encryption.algorithm",
"fs.s3a.encryption.key",
"fs.azure\\.account.key.*",
"credential$",
"oauth.*secret",
"oauth.*password",
"oauth.*token",
HADOOP_SECURITY_SENSITIVE_CONFIG_KEYS);
```
> Router Page display the db username and password in mask mode
> -
>
> Key: YARN-11308
> URL: https://issues.apache.org/jira/browse/YARN-11308
> Project: Hadoop YARN
> Issue Type: Bug
> Components: federation
>Affects Versions: 3.4.0
>Reporter: fanshilun
>Assignee: fanshilun
>Priority: Major
> Labels: pull-request-available
> Attachments: image-2022-09-19-17-33-44-585.png,
> image-2022-09-19-17-35-02-471.png
>
>
> When using YRAN-Federation's SQLFederationStateStore, we need to configure
> yarn.federation.state-store.sql.username,
> yarn.federation.state-store.sql.password in the configuration file, When
> viewing Conf on the Router page, the user name and password are displayed in
> plaintext, which will bring security risks. We should display it in the form
> of a mask.
>
> before fixing
> {code:java}
>
>yarn.federation.state-store.sql.username
>federation
>false
>yarn-site.xml
>
>
>yarn.federation.state-store.sql.password
>federation123
>false
>yarn-site.xml
> {code}
> after fixing
> {code:java}
>
> yarn.federation.state-store.sql.username
> **
> false
> yarn-site.xml
>
>
> yarn.federation.state-store.sql.password
> **
> false
> yarn-site.xml
> {code}
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11308) Router Page display the db username and password in mask mode
[
https://issues.apache.org/jira/browse/YARN-11308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17606837#comment-17606837
]
ASF GitHub Bot commented on YARN-11308:
---
slfan1989 commented on code in PR #4908:
URL: https://github.com/apache/hadoop/pull/4908#discussion_r974839419
##
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/http/HttpServer2.java:
##
@@ -214,6 +215,12 @@ public final class HttpServer2 implements FilterContainer {
private StatisticsHandler statsHandler;
private HttpServer2Metrics metrics;
+ private static final String MASK = "**";
+ public static final String FEDERATION_STATESTORE_SQL_USERNAME =
Review Comment:
Thanks a lot for your suggestion, I agree with you, I think we should use
separate config file configuration to store passwords and use KMS to encrypt
passwords. This part needs to submit a separate pr.
In this pr, when we get the configuration from the page (such as the router
webpage), the configuration containing some keywords can be displayed in the
form of a mask.
> Router Page display the db username and password in mask mode
> -
>
> Key: YARN-11308
> URL: https://issues.apache.org/jira/browse/YARN-11308
> Project: Hadoop YARN
> Issue Type: Bug
> Components: federation
>Affects Versions: 3.4.0
>Reporter: fanshilun
>Assignee: fanshilun
>Priority: Major
> Labels: pull-request-available
> Attachments: image-2022-09-19-17-33-44-585.png,
> image-2022-09-19-17-35-02-471.png
>
>
> When using YRAN-Federation's SQLFederationStateStore, we need to configure
> yarn.federation.state-store.sql.username,
> yarn.federation.state-store.sql.password in the configuration file, When
> viewing Conf on the Router page, the user name and password are displayed in
> plaintext, which will bring security risks. We should display it in the form
> of a mask.
>
> before fixing
> {code:java}
>
>yarn.federation.state-store.sql.username
>federation
>false
>yarn-site.xml
>
>
>yarn.federation.state-store.sql.password
>federation123
>false
>yarn-site.xml
> {code}
> after fixing
> {code:java}
>
> yarn.federation.state-store.sql.username
> **
> false
> yarn-site.xml
>
>
> yarn.federation.state-store.sql.password
> **
> false
> yarn-site.xml
> {code}
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11308) Router Page display the db username and password in mask mode
[
https://issues.apache.org/jira/browse/YARN-11308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17606823#comment-17606823
]
ASF GitHub Bot commented on YARN-11308:
---
slfan1989 commented on code in PR #4908:
URL: https://github.com/apache/hadoop/pull/4908#discussion_r974812563
##
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestConfServlet.java:
##
@@ -247,4 +254,67 @@ public void testBadFormat() throws Exception {
}
assertEquals("", sw.toString());
}
+
+ private void verifyReplaceProperty(Configuration conf, String format,
+ String propertyName) throws Exception {
+StringWriter sw = null;
+PrintWriter pw = null;
+ConfServlet service = null;
+try {
+ service = new ConfServlet();
+ ServletConfig servletConf = mock(ServletConfig.class);
+ ServletContext context = mock(ServletContext.class);
+ service.init(servletConf);
+ when(context.getAttribute(HttpServer2.CONF_CONTEXT_ATTRIBUTE))
+ .thenReturn(conf);
+ when(service.getServletContext())
+ .thenReturn(context);
+
+ HttpServletRequest request = mock(HttpServletRequest.class);
+ when(request.getHeader(HttpHeaders.ACCEPT))
Review Comment:
I will fix it.
##
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestConfServlet.java:
##
@@ -247,4 +254,67 @@ public void testBadFormat() throws Exception {
}
assertEquals("", sw.toString());
}
+
+ private void verifyReplaceProperty(Configuration conf, String format,
+ String propertyName) throws Exception {
+StringWriter sw = null;
+PrintWriter pw = null;
+ConfServlet service = null;
+try {
+ service = new ConfServlet();
+ ServletConfig servletConf = mock(ServletConfig.class);
+ ServletContext context = mock(ServletContext.class);
+ service.init(servletConf);
+ when(context.getAttribute(HttpServer2.CONF_CONTEXT_ATTRIBUTE))
+ .thenReturn(conf);
+ when(service.getServletContext())
+ .thenReturn(context);
+
+ HttpServletRequest request = mock(HttpServletRequest.class);
+ when(request.getHeader(HttpHeaders.ACCEPT))
+ .thenReturn(TEST_FORMATS.get(format));
+ when(request.getParameter("name"))
+ .thenReturn(propertyName);
+
+ HttpServletResponse response = mock(HttpServletResponse.class);
+ sw = new StringWriter();
+ pw = new PrintWriter(sw);
+ when(response.getWriter()).thenReturn(pw);
+
+ // response request
+ service.doGet(request, response);
+ String result = sw.toString().trim();
+
+ // For example, for the property
yarn.federation.state-store.sql.username,
+ // we set the value to test-user,
+ // which should be replaced by a mask, which should be **
+ // MASK_PROPERTIES.get("property
yarn.federation.state-store.sql.username")
+ // is the value before replacement, test-user
+ // result contains the replaced value, which should be **
+ assertTrue(result.contains(propertyName) &&
Review Comment:
I will fix it.
> Router Page display the db username and password in mask mode
> -
>
> Key: YARN-11308
> URL: https://issues.apache.org/jira/browse/YARN-11308
> Project: Hadoop YARN
> Issue Type: Bug
> Components: federation
>Affects Versions: 3.4.0
>Reporter: fanshilun
>Assignee: fanshilun
>Priority: Major
> Labels: pull-request-available
> Attachments: image-2022-09-19-17-33-44-585.png,
> image-2022-09-19-17-35-02-471.png
>
>
> When using YRAN-Federation's SQLFederationStateStore, we need to configure
> yarn.federation.state-store.sql.username,
> yarn.federation.state-store.sql.password in the configuration file, When
> viewing Conf on the Router page, the user name and password are displayed in
> plaintext, which will bring security risks. We should display it in the form
> of a mask.
>
> before fixing
> {code:java}
>
>yarn.federation.state-store.sql.username
>federation
>false
>yarn-site.xml
>
>
>yarn.federation.state-store.sql.password
>federation123
>false
>yarn-site.xml
> {code}
> after fixing
> {code:java}
>
> yarn.federation.state-store.sql.username
> **
> false
> yarn-site.xml
>
>
> yarn.federation.state-store.sql.password
> **
> false
> yarn-site.xml
> {code}
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11308) Router Page display the db username and password in mask mode
[
https://issues.apache.org/jira/browse/YARN-11308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17606821#comment-17606821
]
ASF GitHub Bot commented on YARN-11308:
---
slfan1989 commented on code in PR #4908:
URL: https://github.com/apache/hadoop/pull/4908#discussion_r974809560
##
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestConfServlet.java:
##
@@ -59,6 +59,8 @@ public class TestConfServlet {
new HashMap();
private static final Map TEST_FORMATS =
new HashMap();
+ private static final Map MASK_PROPERTIES =
Review Comment:
Thanks for your help reviewing the code, I will fix it.
> Router Page display the db username and password in mask mode
> -
>
> Key: YARN-11308
> URL: https://issues.apache.org/jira/browse/YARN-11308
> Project: Hadoop YARN
> Issue Type: Bug
> Components: federation
>Affects Versions: 3.4.0
>Reporter: fanshilun
>Assignee: fanshilun
>Priority: Major
> Labels: pull-request-available
> Attachments: image-2022-09-19-17-33-44-585.png,
> image-2022-09-19-17-35-02-471.png
>
>
> When using YRAN-Federation's SQLFederationStateStore, we need to configure
> yarn.federation.state-store.sql.username,
> yarn.federation.state-store.sql.password in the configuration file, When
> viewing Conf on the Router page, the user name and password are displayed in
> plaintext, which will bring security risks. We should display it in the form
> of a mask.
>
> before fixing
> {code:java}
>
>yarn.federation.state-store.sql.username
>federation
>false
>yarn-site.xml
>
>
>yarn.federation.state-store.sql.password
>federation123
>false
>yarn-site.xml
> {code}
> after fixing
> {code:java}
>
> yarn.federation.state-store.sql.username
> **
> false
> yarn-site.xml
>
>
> yarn.federation.state-store.sql.password
> **
> false
> yarn-site.xml
> {code}
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11293) [Federation] Router Support DelegationToken storeNewMasterKey/removeStoredMasterKey With MemoryStateStore
[ https://issues.apache.org/jira/browse/YARN-11293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17606820#comment-17606820 ] ASF GitHub Bot commented on YARN-11293: --- slfan1989 commented on PR #4852: URL: https://github.com/apache/hadoop/pull/4852#issuecomment-1251733241 @goiri Thank you very much for your help reviewing the code! > [Federation] Router Support DelegationToken > storeNewMasterKey/removeStoredMasterKey With MemoryStateStore > - > > Key: YARN-11293 > URL: https://issues.apache.org/jira/browse/YARN-11293 > Project: Hadoop YARN > Issue Type: Sub-task > Components: federation >Affects Versions: 3.4.0 >Reporter: fanshilun >Assignee: fanshilun >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > > Support storing storeNewMasterKey/removeStoredMasterKey in MemoryStore mode. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11283) [Federation] Fix Typo of NodeManager AMRMProxy.
[ https://issues.apache.org/jira/browse/YARN-11283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17606819#comment-17606819 ] ASF GitHub Bot commented on YARN-11283: --- slfan1989 commented on PR #4899: URL: https://github.com/apache/hadoop/pull/4899#issuecomment-1251733108 @goiri Thank you very much for your help reviewing the code! > [Federation] Fix Typo of NodeManager AMRMProxy. > --- > > Key: YARN-11283 > URL: https://issues.apache.org/jira/browse/YARN-11283 > Project: Hadoop YARN > Issue Type: Improvement > Components: federation, nodemanager >Affects Versions: 3.4.0 >Reporter: fanshilun >Assignee: fanshilun >Priority: Minor > Labels: pull-request-available > Fix For: 3.4.0 > > > Fix Typo of NodeManager amrmproxy -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11303) Upgrade jquery ui to 1.13.2
[ https://issues.apache.org/jira/browse/YARN-11303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17606804#comment-17606804 ] ASF GitHub Bot commented on YARN-11303: --- goiri commented on PR #4895: URL: https://github.com/apache/hadoop/pull/4895#issuecomment-1251707985 Can we get a clean build? > Upgrade jquery ui to 1.13.2 > --- > > Key: YARN-11303 > URL: https://issues.apache.org/jira/browse/YARN-11303 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: D M Murali Krishna Reddy >Assignee: Ashutosh Gupta >Priority: Major > Labels: pull-request-available > > The current jquery-ui version used(1.13.1) in the trunk has the following > vulnerability > [CVE-2022-31160|https://nvd.nist.gov/vuln/detail/CVE-2022-31160] so we need > to upgrade to at least 1.13.2. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11262) Upgrade JUnit from 4 to 5 in hadoop-yarn-server-resourcemanager
[
https://issues.apache.org/jira/browse/YARN-11262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17606787#comment-17606787
]
ASF GitHub Bot commented on YARN-11262:
---
hadoop-yetus commented on PR #4910:
URL: https://github.com/apache/hadoop/pull/4910#issuecomment-1251639019
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 3s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 5s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 300 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 42m 11s | | trunk passed |
| +1 :green_heart: | compile | 1m 14s | | trunk passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 1m 4s | | trunk passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 1m 15s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 9s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 3s | | trunk passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 50s | | trunk passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 2m 21s | | trunk passed |
| +1 :green_heart: | shadedclient | 25m 7s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 1s | | the patch passed |
| +1 :green_heart: | compile | 1m 9s | | the patch passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| -1 :x: | javac | 1m 9s |
[/results-compile-javac-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkUbuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4910/1/artifact/out/results-compile-javac-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkUbuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04.txt)
|
hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkUbuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 generated 9 new + 208
unchanged - 19 fixed = 217 total (was 227) |
| +1 :green_heart: | compile | 0m 59s | | the patch passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| -1 :x: | javac | 0m 59s |
[/results-compile-javac-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkPrivateBuild-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4910/1/artifact/out/results-compile-javac-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkPrivateBuild-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07.txt)
|
hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkPrivateBuild-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 generated 9 new
+ 165 unchanged - 19 fixed = 174 total (was 184) |
| -1 :x: | blanks | 0m 0s |
[/blanks-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4910/1/artifact/out/blanks-eol.txt)
| The patch has 53 line(s) that end in blanks. Use git apply --whitespace=fix
<>. Refer https://git-scm.com/docs/git-apply |
| -1 :x: | blanks | 0m 0s |
[/blanks-tabs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4910/1/artifact/out/blanks-tabs.txt)
| The patch 1 line(s) with tabs. |
| -0 :warning: | checkstyle | 1m 5s |
[/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4910/1/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt)
|
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 585 new + 2570 unchanged - 71 fixed = 3155 total (was
2641) |
| +1 :green_heart: | mvnsite | 1m 1s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 43s | | the
[jira] [Commented] (YARN-11306) Refactor NM#FederationInterceptor#recover Code
[
https://issues.apache.org/jira/browse/YARN-11306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17606752#comment-17606752
]
ASF GitHub Bot commented on YARN-11306:
---
goiri commented on code in PR #4897:
URL: https://github.com/apache/hadoop/pull/4897#discussion_r974657324
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/FederationInterceptor.java:
##
@@ -463,42 +461,67 @@ public void recover(Map recoveredDataMap)
{
// map as well.
UserGroupInformation appSubmitter;
if (UserGroupInformation.isSecurityEnabled()) {
-appSubmitter =
UserGroupInformation.createProxyUser(getApplicationContext().getUser(),
+appSubmitter = UserGroupInformation.createProxyUser(user,
UserGroupInformation.getLoginUser());
} else {
-appSubmitter =
UserGroupInformation.createRemoteUser(getApplicationContext().getUser());
+appSubmitter = UserGroupInformation.createRemoteUser(user);
}
- ApplicationClientProtocol rmClient =
- createHomeRMProxy(getApplicationContext(),
- ApplicationClientProtocol.class, appSubmitter);
- GetContainersResponse response = rmClient
- .getContainers(GetContainersRequest.newInstance(this.attemptId));
+ ApplicationClientProtocol rmClient =
createHomeRMProxy(applicationContext,
+ ApplicationClientProtocol.class, appSubmitter);
+
+ GetContainersRequest request =
GetContainersRequest.newInstance(this.attemptId);
+ GetContainersResponse response = rmClient.getContainers(request);
+
for (ContainerReport container : response.getContainerList()) {
-containerIdToSubClusterIdMap.put(container.getContainerId(),
-this.homeSubClusterId);
+ContainerId containerId = container.getContainerId();
+containerIdToSubClusterIdMap.put(containerId, this.homeSubClusterId);
containers++;
-LOG.debug(" From home RM {} running container {}",
-this.homeSubClusterId, container.getContainerId());
+LOG.debug("From home RM {} running container {}.",
this.homeSubClusterId, containerId);
}
- LOG.info("{} running containers including AM recovered from home RM {}",
+ LOG.info("{} running containers including AM recovered from home RM {}.",
response.getContainerList().size(), this.homeSubClusterId);
- LOG.info(
- "In all {} UAMs {} running containers including AM recovered for {}",
+ LOG.info("In all {} UAMs {} running containers including AM recovered
for {}.",
uamMap.size(), containers, this.attemptId);
- if (this.amRegistrationResponse != null) {
+ if (queue != null) {
// Initialize the AMRMProxyPolicy
-String queue = this.amRegistrationResponse.getQueue();
-this.policyInterpreter =
-FederationPolicyUtils.loadAMRMPolicy(queue, this.policyInterpreter,
-getConf(), this.federationFacade, this.homeSubClusterId);
+queue = this.amRegistrationResponse.getQueue();
+this.policyInterpreter = FederationPolicyUtils.loadAMRMPolicy(queue,
this.policyInterpreter,
+getConf(), this.federationFacade, this.homeSubClusterId);
}
} catch (IOException | YarnException e) {
throw new YarnRuntimeException(e);
}
+ }
+ private Map> getSCAMRMTokenIdentifierMap(
+ Map recoveredDataMap) throws IOException {
+Map> uamMap = new HashMap<>();
+ApplicationId applicationId = this.attemptId.getApplicationId();
+if (this.registryClient != null) {
+ uamMap = this.registryClient.loadStateFromRegistry(applicationId);
+ LOG.info("Found {} existing UAMs for application {} in Yarn Registry.",
+ uamMap.size(), applicationId);
+} else {
+ for (Entry entry : recoveredDataMap.entrySet()) {
Review Comment:
It might be good to have this large chunk in a separate method.
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/FederationInterceptor.java:
##
@@ -463,42 +461,67 @@ public void recover(Map recoveredDataMap)
{
// map as well.
UserGroupInformation appSubmitter;
if (UserGroupInformation.isSecurityEnabled()) {
-appSubmitter =
UserGroupInformation.createProxyUser(getApplicationContext().getUser(),
+appSubmitter = UserGroupInformation.createProxyUser(user,
UserGroupInformation.getLoginUser());
} else {
-appSubmitter =
UserGroupInformation.createRemoteUser(getApplicationContext().getUser());
+appSubmitter = UserGroupInformation.createRemoteUser(user);
}
- ApplicationClientProtocol rmClient =
- createHomeRMProxy(getApplicationContext(),
-
[jira] [Resolved] (YARN-11283) [Federation] Fix Typo of NodeManager AMRMProxy.
[ https://issues.apache.org/jira/browse/YARN-11283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri resolved YARN-11283. Hadoop Flags: Reviewed Resolution: Fixed > [Federation] Fix Typo of NodeManager AMRMProxy. > --- > > Key: YARN-11283 > URL: https://issues.apache.org/jira/browse/YARN-11283 > Project: Hadoop YARN > Issue Type: Improvement > Components: federation, nodemanager >Affects Versions: 3.4.0 >Reporter: fanshilun >Assignee: fanshilun >Priority: Minor > Labels: pull-request-available > Fix For: 3.4.0 > > > Fix Typo of NodeManager amrmproxy -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11283) Fix Typo of NodeManager amrmproxy
[ https://issues.apache.org/jira/browse/YARN-11283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17606749#comment-17606749 ] ASF GitHub Bot commented on YARN-11283: --- goiri merged PR #4899: URL: https://github.com/apache/hadoop/pull/4899 > Fix Typo of NodeManager amrmproxy > - > > Key: YARN-11283 > URL: https://issues.apache.org/jira/browse/YARN-11283 > Project: Hadoop YARN > Issue Type: Improvement > Components: federation, nodemanager >Affects Versions: 3.4.0 >Reporter: fanshilun >Assignee: fanshilun >Priority: Minor > Labels: pull-request-available > Fix For: 3.4.0 > > > Fix Typo of NodeManager amrmproxy -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-11283) [Federation] Fix Typo of NodeManager AMRMProxy.
[ https://issues.apache.org/jira/browse/YARN-11283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri updated YARN-11283: --- Summary: [Federation] Fix Typo of NodeManager AMRMProxy. (was: Fix Typo of NodeManager AMRMProxy) > [Federation] Fix Typo of NodeManager AMRMProxy. > --- > > Key: YARN-11283 > URL: https://issues.apache.org/jira/browse/YARN-11283 > Project: Hadoop YARN > Issue Type: Improvement > Components: federation, nodemanager >Affects Versions: 3.4.0 >Reporter: fanshilun >Assignee: fanshilun >Priority: Minor > Labels: pull-request-available > Fix For: 3.4.0 > > > Fix Typo of NodeManager amrmproxy -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-11283) Fix Typo of NodeManager AMRMProxy
[ https://issues.apache.org/jira/browse/YARN-11283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri updated YARN-11283: --- Summary: Fix Typo of NodeManager AMRMProxy (was: Fix Typo of NodeManager amrmproxy) > Fix Typo of NodeManager AMRMProxy > - > > Key: YARN-11283 > URL: https://issues.apache.org/jira/browse/YARN-11283 > Project: Hadoop YARN > Issue Type: Improvement > Components: federation, nodemanager >Affects Versions: 3.4.0 >Reporter: fanshilun >Assignee: fanshilun >Priority: Minor > Labels: pull-request-available > Fix For: 3.4.0 > > > Fix Typo of NodeManager amrmproxy -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Resolved] (YARN-11293) [Federation] Router Support DelegationToken storeNewMasterKey/removeStoredMasterKey With MemoryStateStore
[ https://issues.apache.org/jira/browse/YARN-11293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri resolved YARN-11293. Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > [Federation] Router Support DelegationToken > storeNewMasterKey/removeStoredMasterKey With MemoryStateStore > - > > Key: YARN-11293 > URL: https://issues.apache.org/jira/browse/YARN-11293 > Project: Hadoop YARN > Issue Type: Sub-task > Components: federation >Affects Versions: 3.4.0 >Reporter: fanshilun >Assignee: fanshilun >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > > Support storing storeNewMasterKey/removeStoredMasterKey in MemoryStore mode. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11308) Router Page display the db username and password in mask mode
[
https://issues.apache.org/jira/browse/YARN-11308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17606746#comment-17606746
]
ASF GitHub Bot commented on YARN-11308:
---
goiri commented on code in PR #4908:
URL: https://github.com/apache/hadoop/pull/4908#discussion_r974648275
##
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestConfServlet.java:
##
@@ -59,6 +59,8 @@ public class TestConfServlet {
new HashMap();
private static final Map TEST_FORMATS =
new HashMap();
+ private static final Map MASK_PROPERTIES =
Review Comment:
Single line
##
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/http/HttpServer2.java:
##
@@ -214,6 +215,12 @@ public final class HttpServer2 implements FilterContainer {
private StatisticsHandler statsHandler;
private HttpServer2Metrics metrics;
+ private static final String MASK = "**";
+ public static final String FEDERATION_STATESTORE_SQL_USERNAME =
+ "yarn.federation.state-store.sql.username";
+ public static final String FEDERATION_STATESTORE_SQL_PASSWROD =
Review Comment:
Should we make this more generic and mas any conf key with password on it?
##
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestConfServlet.java:
##
@@ -247,4 +254,67 @@ public void testBadFormat() throws Exception {
}
assertEquals("", sw.toString());
}
+
+ private void verifyReplaceProperty(Configuration conf, String format,
+ String propertyName) throws Exception {
+StringWriter sw = null;
+PrintWriter pw = null;
+ConfServlet service = null;
+try {
+ service = new ConfServlet();
+ ServletConfig servletConf = mock(ServletConfig.class);
+ ServletContext context = mock(ServletContext.class);
+ service.init(servletConf);
+ when(context.getAttribute(HttpServer2.CONF_CONTEXT_ATTRIBUTE))
+ .thenReturn(conf);
+ when(service.getServletContext())
+ .thenReturn(context);
+
+ HttpServletRequest request = mock(HttpServletRequest.class);
+ when(request.getHeader(HttpHeaders.ACCEPT))
+ .thenReturn(TEST_FORMATS.get(format));
+ when(request.getParameter("name"))
+ .thenReturn(propertyName);
+
+ HttpServletResponse response = mock(HttpServletResponse.class);
+ sw = new StringWriter();
+ pw = new PrintWriter(sw);
+ when(response.getWriter()).thenReturn(pw);
+
+ // response request
+ service.doGet(request, response);
+ String result = sw.toString().trim();
+
+ // For example, for the property
yarn.federation.state-store.sql.username,
+ // we set the value to test-user,
+ // which should be replaced by a mask, which should be **
+ // MASK_PROPERTIES.get("property
yarn.federation.state-store.sql.username")
+ // is the value before replacement, test-user
+ // result contains the replaced value, which should be **
+ assertTrue(result.contains(propertyName) &&
Review Comment:
Make it two separate assertTrue()
##
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/http/HttpServer2.java:
##
@@ -214,6 +215,12 @@ public final class HttpServer2 implements FilterContainer {
private StatisticsHandler statsHandler;
private HttpServer2Metrics metrics;
+ private static final String MASK = "**";
+ public static final String FEDERATION_STATESTORE_SQL_USERNAME =
Review Comment:
In a more general sense, does it make sense for passwords to be in the
configuration file itself?
Do we have other mechanisms to specify these?
##
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/ConfServlet.java:
##
@@ -43,13 +48,18 @@ public class ConfServlet extends HttpServlet {
protected static final String FORMAT_JSON = "json";
protected static final String FORMAT_XML = "xml";
+
/**
* Return the Configuration of the daemon hosting this servlet.
* This is populated when the HttpServer starts.
*/
private Configuration getConfFromContext() {
Configuration conf = (Configuration)getServletContext().getAttribute(
HttpServer2.CONF_CONTEXT_ATTRIBUTE);
+List props = new ArrayList<>();
+props.add(FEDERATION_STATESTORE_SQL_USERNAME);
Review Comment:
I think this should be more general.
We should have a generic part to set particular keys as passwords to not be
shown and then in the YARN router part to actually specify the key.
##
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestConfServlet.java:
##
@@ -247,4 +254,67 @@ public void testBadFormat() throws Exception {
}
assertEquals("", sw.toString());
}
+
+ private void verifyReplaceProperty(Configuration conf, String format,
+ String propertyName) throws Exception {
+
[jira] [Commented] (YARN-11293) [Federation] Router Support DelegationToken storeNewMasterKey/removeStoredMasterKey With MemoryStateStore
[ https://issues.apache.org/jira/browse/YARN-11293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17606747#comment-17606747 ] ASF GitHub Bot commented on YARN-11293: --- goiri merged PR #4852: URL: https://github.com/apache/hadoop/pull/4852 > [Federation] Router Support DelegationToken > storeNewMasterKey/removeStoredMasterKey With MemoryStateStore > - > > Key: YARN-11293 > URL: https://issues.apache.org/jira/browse/YARN-11293 > Project: Hadoop YARN > Issue Type: Sub-task > Components: federation >Affects Versions: 3.4.0 >Reporter: fanshilun >Assignee: fanshilun >Priority: Major > Labels: pull-request-available > > Support storing storeNewMasterKey/removeStoredMasterKey in MemoryStore mode. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11262) Upgrade JUnit from 4 to 5 in hadoop-yarn-server-resourcemanager
[ https://issues.apache.org/jira/browse/YARN-11262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17606702#comment-17606702 ] ASF GitHub Bot commented on YARN-11262: --- andreoss commented on PR #4910: URL: https://github.com/apache/hadoop/pull/4910#issuecomment-1251360519 @ashutoshcipher Understood. Any chance this can be merged? In this PR, only `Parameterized` tests are left intact, the rest are converted to Junit 5. The `Parameterized` tests require some refactoring which I would like to be in a separate PR to ease review > Upgrade JUnit from 4 to 5 in hadoop-yarn-server-resourcemanager > --- > > Key: YARN-11262 > URL: https://issues.apache.org/jira/browse/YARN-11262 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Affects Versions: 3.3.4 >Reporter: Ashutosh Gupta >Assignee: Ashutosh Gupta >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-11309) datatables@1.10.17 sonatype-2020-0988 vulnerability
[ https://issues.apache.org/jira/browse/YARN-11309?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Gupta reassigned YARN-11309: - Assignee: Ashutosh Gupta > datatables@1.10.17 sonatype-2020-0988 vulnerability > > > Key: YARN-11309 > URL: https://issues.apache.org/jira/browse/YARN-11309 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn-ui-v2 >Affects Versions: 3.3.4 >Reporter: Eugene Shinn (Truveta) >Assignee: Ashutosh Gupta >Priority: Major > > Our static analysis security tool detected that YARN's UI currently includes > a vulnerable version of datatables detected by Sonatype (sonatype-2020-0988). > From the vulnerability description: > _"The `datatables.net` package is vulnerable to Prototype Pollution. The > `setData` function in `jquery.dataTables.js` fails to protect prototype > attributes when objects are created during the application's execution. A > remote attacker can exploit this to modify the behavior of object prototypes > which, depending on their use in the application, may result in a Denial of > Service (DoS), Remote Code Execution (RCE), or other unexpected execution > flow."_ > This issue was addressed in v 1.11.5 (ref: [Fix: Protect developers from > inadvertantely introducing prototype pol… · > DataTables/Dist-DataTables@e2e19ea > (github.com)).|https://github.com/DataTables/Dist-DataTables/commit/e2e19eac7e5a6f140d7eefca5c7deba165b357eb#diff-e7d8309f017dd2ef6385fa8cdc1539a2R2765] > [HDFS-16777] datatables@1.10.17 sonatype-2020-0988 vulnerability - ASF JIRA > (apache.org) was filed to address the identical issue in HDFS' UI. > h4. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11262) Upgrade JUnit from 4 to 5 in hadoop-yarn-server-resourcemanager
[ https://issues.apache.org/jira/browse/YARN-11262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17606694#comment-17606694 ] ASF GitHub Bot commented on YARN-11262: --- ashutoshcipher commented on PR #4910: URL: https://github.com/apache/hadoop/pull/4910#issuecomment-1251343476 Hi @andreoss, Thanks for you PR. I have already been working on it in my local and make some good progress as well. It would be great if you can ask on JIRA before starting the work if assigned JIRA person is working on it or not. Thanks. > Upgrade JUnit from 4 to 5 in hadoop-yarn-server-resourcemanager > --- > > Key: YARN-11262 > URL: https://issues.apache.org/jira/browse/YARN-11262 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Affects Versions: 3.3.4 >Reporter: Ashutosh Gupta >Assignee: Ashutosh Gupta >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-11262) Upgrade JUnit from 4 to 5 in hadoop-yarn-server-resourcemanager
[ https://issues.apache.org/jira/browse/YARN-11262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated YARN-11262: -- Labels: pull-request-available (was: ) > Upgrade JUnit from 4 to 5 in hadoop-yarn-server-resourcemanager > --- > > Key: YARN-11262 > URL: https://issues.apache.org/jira/browse/YARN-11262 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Affects Versions: 3.3.4 >Reporter: Ashutosh Gupta >Assignee: Ashutosh Gupta >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11262) Upgrade JUnit from 4 to 5 in hadoop-yarn-server-resourcemanager
[ https://issues.apache.org/jira/browse/YARN-11262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17606689#comment-17606689 ] ASF GitHub Bot commented on YARN-11262: --- andreoss opened a new pull request, #4910: URL: https://github.com/apache/hadoop/pull/4910 ### Description of PR Upgrade to Junit 5 ### How was this patch tested? mvn test ### For code changes: - [x] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')? - [ ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation? - [x] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files? > Upgrade JUnit from 4 to 5 in hadoop-yarn-server-resourcemanager > --- > > Key: YARN-11262 > URL: https://issues.apache.org/jira/browse/YARN-11262 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Affects Versions: 3.3.4 >Reporter: Ashutosh Gupta >Assignee: Ashutosh Gupta >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-11309) datatables@1.10.17 sonatype-2020-0988 vulnerability
Eugene Shinn (Truveta) created YARN-11309: - Summary: datatables@1.10.17 sonatype-2020-0988 vulnerability Key: YARN-11309 URL: https://issues.apache.org/jira/browse/YARN-11309 Project: Hadoop YARN Issue Type: Bug Components: yarn-ui-v2 Affects Versions: 3.3.4 Reporter: Eugene Shinn (Truveta) Our static analysis security tool detected that YARN's UI currently includes a vulnerable version of datatables detected by Sonatype (sonatype-2020-0988). >From the vulnerability description: _"The `datatables.net` package is vulnerable to Prototype Pollution. The `setData` function in `jquery.dataTables.js` fails to protect prototype attributes when objects are created during the application's execution. A remote attacker can exploit this to modify the behavior of object prototypes which, depending on their use in the application, may result in a Denial of Service (DoS), Remote Code Execution (RCE), or other unexpected execution flow."_ This issue was addressed in v 1.11.5 (ref: [Fix: Protect developers from inadvertantely introducing prototype pol… · DataTables/Dist-DataTables@e2e19ea (github.com)).|https://github.com/DataTables/Dist-DataTables/commit/e2e19eac7e5a6f140d7eefca5c7deba165b357eb#diff-e7d8309f017dd2ef6385fa8cdc1539a2R2765] [HDFS-16777] datatables@1.10.17 sonatype-2020-0988 vulnerability - ASF JIRA (apache.org) was filed to address the identical issue in HDFS' UI. h4. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11308) Router Page display the db username and password in mask mode
[
https://issues.apache.org/jira/browse/YARN-11308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17606649#comment-17606649
]
ASF GitHub Bot commented on YARN-11308:
---
hadoop-yetus commented on PR #4908:
URL: https://github.com/apache/hadoop/pull/4908#issuecomment-1251199268
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 15s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 41m 46s | | trunk passed |
| +1 :green_heart: | compile | 25m 48s | | trunk passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 22m 24s | | trunk passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 1m 25s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 58s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 26s | | trunk passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 59s | | trunk passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 3m 5s | | trunk passed |
| +1 :green_heart: | shadedclient | 26m 25s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 6s | | the patch passed |
| +1 :green_heart: | compile | 25m 1s | | the patch passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 25m 1s | | the patch passed |
| +1 :green_heart: | compile | 26m 20s | | the patch passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 26m 20s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 1m 35s | | the patch passed |
| +1 :green_heart: | mvnsite | 2m 10s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 25s | | the patch passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 58s | | the patch passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 3m 10s | | the patch passed |
| +1 :green_heart: | shadedclient | 26m 27s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 18m 28s |
[/patch-unit-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4908/1/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt)
| hadoop-common in the patch passed. |
| +1 :green_heart: | asflicense | 1m 11s | | The patch does not
generate ASF License warnings. |
| | | 234m 54s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.http.TestHttpServer |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4908/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4908 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 7563b7f43232 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4
01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 055258d7e683b794e23fbbfd4fe786a8b6766b68 |
| Default Java | Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4908/1/testReport/ |
| Max. process+thread count | 1253 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
[jira] [Updated] (YARN-11308) Router Page display the db username and password in mask mode
[
https://issues.apache.org/jira/browse/YARN-11308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated YARN-11308:
--
Labels: pull-request-available (was: )
> Router Page display the db username and password in mask mode
> -
>
> Key: YARN-11308
> URL: https://issues.apache.org/jira/browse/YARN-11308
> Project: Hadoop YARN
> Issue Type: Bug
> Components: federation
>Affects Versions: 3.4.0
>Reporter: fanshilun
>Assignee: fanshilun
>Priority: Major
> Labels: pull-request-available
> Attachments: image-2022-09-19-17-33-44-585.png,
> image-2022-09-19-17-35-02-471.png
>
>
> When using YRAN-Federation's SQLFederationStateStore, we need to configure
> yarn.federation.state-store.sql.username,
> yarn.federation.state-store.sql.password in the configuration file, When
> viewing Conf on the Router page, the user name and password are displayed in
> plaintext, which will bring security risks. We should display it in the form
> of a mask.
>
> before fixing
> {code:java}
>
>yarn.federation.state-store.sql.username
>federation
>false
>yarn-site.xml
>
>
>yarn.federation.state-store.sql.password
>federation123
>false
>yarn-site.xml
> {code}
> after fixing
> {code:java}
>
> yarn.federation.state-store.sql.username
> **
> false
> yarn-site.xml
>
>
> yarn.federation.state-store.sql.password
> **
> false
> yarn-site.xml
> {code}
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11308) Router Page display the db username and password in mask mode
[
https://issues.apache.org/jira/browse/YARN-11308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17606563#comment-17606563
]
ASF GitHub Bot commented on YARN-11308:
---
slfan1989 opened a new pull request, #4908:
URL: https://github.com/apache/hadoop/pull/4908
JIRA: YARN-11308. Router Page display the db username and password in mask
mode.
When using YRAN-Federation's SQLFederationStateStore, we need to configure
yarn.federation.state-store.sql.username,
yarn.federation.state-store.sql.password in the configuration file, When
viewing Conf on the Router page, the user name and password are displayed in
plaintext, which will bring security risks. We should display it in the form of
a mask.
before fixing
```
yarn.federation.state-store.sql.username
federation
false
yarn-site.xml
yarn.federation.state-store.sql.password
federation123
false
yarn-site.xml
```
after fixing
```
yarn.federation.state-store.sql.username
**
false
yarn-site.xml
yarn.federation.state-store.sql.password
**
false
yarn-site.xml
```
> Router Page display the db username and password in mask mode
> -
>
> Key: YARN-11308
> URL: https://issues.apache.org/jira/browse/YARN-11308
> Project: Hadoop YARN
> Issue Type: Bug
> Components: federation
>Affects Versions: 3.4.0
>Reporter: fanshilun
>Assignee: fanshilun
>Priority: Major
> Attachments: image-2022-09-19-17-33-44-585.png,
> image-2022-09-19-17-35-02-471.png
>
>
> When using YRAN-Federation's SQLFederationStateStore, we need to configure
> yarn.federation.state-store.sql.username,
> yarn.federation.state-store.sql.password in the configuration file, When
> viewing Conf on the Router page, the user name and password are displayed in
> plaintext, which will bring security risks. We should display it in the form
> of a mask.
>
> before fixing
> {code:java}
>
>yarn.federation.state-store.sql.username
>federation
>false
>yarn-site.xml
>
>
>yarn.federation.state-store.sql.password
>federation123
>false
>yarn-site.xml
> {code}
> after fixing
> {code:java}
>
> yarn.federation.state-store.sql.username
> **
> false
> yarn-site.xml
>
>
> yarn.federation.state-store.sql.password
> **
> false
> yarn-site.xml
> {code}
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-11308) Router Page display the db username and password in mask mode
[
https://issues.apache.org/jira/browse/YARN-11308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
fanshilun updated YARN-11308:
-
Description:
When using YRAN-Federation's SQLFederationStateStore, we need to configure
yarn.federation.state-store.sql.username,
yarn.federation.state-store.sql.password in the configuration file, When
viewing Conf on the Router page, the user name and password are displayed in
plaintext, which will bring security risks. We should display it in the form of
a mask.
before fixing
{code:java}
yarn.federation.state-store.sql.username
federation
false
yarn-site.xml
yarn.federation.state-store.sql.password
federation123
false
yarn-site.xml
{code}
after fixing
{code:java}
yarn.federation.state-store.sql.username
**
false
yarn-site.xml
yarn.federation.state-store.sql.password
**
false
yarn-site.xml
{code}
was:
When using YRAN-Federation's SQLFederationStateStore, we need to configure
yarn.federation.state-store.sql.username,
yarn.federation.state-store.sql.password in the configuration file, When
viewing Conf on the Router page, the user name and password are displayed in
plaintext, which will bring security risks. We should display it in the form of
a mask.
!image-2022-09-19-17-33-44-585.png|width=499,height=87!
> Router Page display the db username and password in mask mode
> -
>
> Key: YARN-11308
> URL: https://issues.apache.org/jira/browse/YARN-11308
> Project: Hadoop YARN
> Issue Type: Bug
> Components: federation
>Affects Versions: 3.4.0
>Reporter: fanshilun
>Assignee: fanshilun
>Priority: Major
> Attachments: image-2022-09-19-17-33-44-585.png,
> image-2022-09-19-17-35-02-471.png
>
>
> When using YRAN-Federation's SQLFederationStateStore, we need to configure
> yarn.federation.state-store.sql.username,
> yarn.federation.state-store.sql.password in the configuration file, When
> viewing Conf on the Router page, the user name and password are displayed in
> plaintext, which will bring security risks. We should display it in the form
> of a mask.
>
> before fixing
> {code:java}
>
>yarn.federation.state-store.sql.username
>federation
>false
>yarn-site.xml
>
>
>yarn.federation.state-store.sql.password
>federation123
>false
>yarn-site.xml
> {code}
> after fixing
> {code:java}
>
> yarn.federation.state-store.sql.username
> **
> false
> yarn-site.xml
>
>
> yarn.federation.state-store.sql.password
> **
> false
> yarn-site.xml
> {code}
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-11308) Router Page display the db username and password in mask mode
[ https://issues.apache.org/jira/browse/YARN-11308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] fanshilun updated YARN-11308: - Attachment: image-2022-09-19-17-35-02-471.png > Router Page display the db username and password in mask mode > - > > Key: YARN-11308 > URL: https://issues.apache.org/jira/browse/YARN-11308 > Project: Hadoop YARN > Issue Type: Bug > Components: federation >Affects Versions: 3.4.0 >Reporter: fanshilun >Assignee: fanshilun >Priority: Major > Attachments: image-2022-09-19-17-33-44-585.png, > image-2022-09-19-17-35-02-471.png > > > When using YRAN-Federation's SQLFederationStateStore, we need to configure > yarn.federation.state-store.sql.username, > yarn.federation.state-store.sql.password in the configuration file, When > viewing Conf on the Router page, the user name and password are displayed in > plaintext, which will bring security risks. We should display it in the form > of a mask. > > !image-2022-09-19-17-33-44-585.png|width=499,height=87! > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-11308) Router Page display the db username and password in mask mode
[ https://issues.apache.org/jira/browse/YARN-11308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] fanshilun updated YARN-11308: - Description: When using YRAN-Federation's SQLFederationStateStore, we need to configure yarn.federation.state-store.sql.username, yarn.federation.state-store.sql.password in the configuration file, When viewing Conf on the Router page, the user name and password are displayed in plaintext, which will bring security risks. We should display it in the form of a mask. !image-2022-09-19-17-33-44-585.png|width=499,height=87! was:When using YRAN-Federation's SQLFederationStateStore, we need to configure yarn.federation.state-store.sql.username, yarn.federation.state-store.sql.password in the configuration file, When viewing Conf on the Router page, the user name and password are displayed in plaintext, which will bring security risks. We should display it in the form of a mask. > Router Page display the db username and password in mask mode > - > > Key: YARN-11308 > URL: https://issues.apache.org/jira/browse/YARN-11308 > Project: Hadoop YARN > Issue Type: Bug > Components: federation >Affects Versions: 3.4.0 >Reporter: fanshilun >Assignee: fanshilun >Priority: Major > Attachments: image-2022-09-19-17-33-44-585.png > > > When using YRAN-Federation's SQLFederationStateStore, we need to configure > yarn.federation.state-store.sql.username, > yarn.federation.state-store.sql.password in the configuration file, When > viewing Conf on the Router page, the user name and password are displayed in > plaintext, which will bring security risks. We should display it in the form > of a mask. > > !image-2022-09-19-17-33-44-585.png|width=499,height=87! > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-11308) Router Page display the db username and password in mask mode
[ https://issues.apache.org/jira/browse/YARN-11308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] fanshilun updated YARN-11308: - Attachment: image-2022-09-19-17-33-44-585.png > Router Page display the db username and password in mask mode > - > > Key: YARN-11308 > URL: https://issues.apache.org/jira/browse/YARN-11308 > Project: Hadoop YARN > Issue Type: Bug > Components: federation >Affects Versions: 3.4.0 >Reporter: fanshilun >Assignee: fanshilun >Priority: Major > Attachments: image-2022-09-19-17-33-44-585.png > > > When using YRAN-Federation's SQLFederationStateStore, we need to configure > yarn.federation.state-store.sql.username, > yarn.federation.state-store.sql.password in the configuration file, When > viewing Conf on the Router page, the user name and password are displayed in > plaintext, which will bring security risks. We should display it in the form > of a mask. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-11308) Router Page display the db username and password in mask mode
fanshilun created YARN-11308: Summary: Router Page display the db username and password in mask mode Key: YARN-11308 URL: https://issues.apache.org/jira/browse/YARN-11308 Project: Hadoop YARN Issue Type: Bug Components: federation Affects Versions: 3.4.0 Reporter: fanshilun Assignee: fanshilun When using YRAN-Federation's SQLFederationStateStore, we need to configure yarn.federation.state-store.sql.username, yarn.federation.state-store.sql.password in the configuration file, When viewing Conf on the Router page, the user name and password are displayed in plaintext, which will bring security risks. We should display it in the form of a mask. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11255) Support loading alternative docker client config from system environment
[
https://issues.apache.org/jira/browse/YARN-11255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17606471#comment-17606471
]
ASF GitHub Bot commented on YARN-11255:
---
PrabhuJoseph commented on PR #4884:
URL: https://github.com/apache/hadoop/pull/4884#issuecomment-1250691046
Thanks @ashutoshcipher for the patch. LGTM, +1. Will commit it if no other
comments.
> Support loading alternative docker client config from system environment
>
>
> Key: YARN-11255
> URL: https://issues.apache.org/jira/browse/YARN-11255
> Project: Hadoop YARN
> Issue Type: New Feature
>Reporter: Ashutosh Gupta
>Assignee: Ashutosh Gupta
>Priority: Major
> Labels: pull-request-available
>
> When using YARN docker support, although the hadoop shell supported
> {code:java}
> -docker_client_config{code}
> to pass the client config file that contains security token to generate the
> docker config for each job as a temporary file.
> For other applications that submit jobs to YARN, e.g. Spark, which loads the
> docker setting via system environment e.g.
> {code:java}
> spark.executorEnv.* {code}
> will not be able to add those authorization token because this system
> environment isn't considered in YARN.
> Add genetic solution to handle these kind of cases without making changes in
> spark code or others
> Eg
> When using remote container registry, the
> {{YARN_CONTAINER_RUNTIME_DOCKER_CLIENT_CONFIG}} must reference the config.json
> file containing the credentials used to authenticate.
> {code:java}
> DOCKER_IMAGE_NAME=hadoop-docker
> DOCKER_CLIENT_CONFIG=hdfs:///user/hadoop/config.json
> spark-submit --master yarn \
> --deploy-mode cluster \
> --conf spark.executorEnv.YARN_CONTAINER_RUNTIME_TYPE=docker \
> --conf
> spark.executorEnv.YARN_CONTAINER_RUNTIME_DOCKER_IMAGE=$DOCKER_IMAGE_NAME \
> --conf
> spark.executorEnv.YARN_CONTAINER_RUNTIME_DOCKER_CLIENT_CONFIG=$DOCKER_CLIENT_CONFIG
> \
> --conf spark.yarn.appMasterEnv.YARN_CONTAINER_RUNTIME_TYPE=docker \
> --conf
> spark.yarn.appMasterEnv.YARN_CONTAINER_RUNTIME_DOCKER_IMAGE=$DOCKER_IMAGE_NAME
> \
> --conf
> spark.yarn.appMasterEnv.YARN_CONTAINER_RUNTIME_DOCKER_CLIENT_CONFIG=$DOCKER_CLIENT_CONFIG
> \
> sparkR.R{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
